Evaluating Multimodal Conversational Agents in Second Language Learning: Lexical Proficiency Enhancement with bespoke LLMs
Disciplines
Applied Linguistics | Bilingual, Multilingual, and Multicultural Education | Computational Linguistics | German Linguistics | Language Interpretation and Translation | Modern Languages | Morphology | Other Computer Engineering | Semantics and Pragmatics | Syntax | Technology and Innovation
Abstract (300 words maximum)
This research evaluates the German Language Coach (GLC), a bespoke intelligent agent powered by OpenAI’s fourth-generation Generative Pre-Trained Transformer (GPT-4), focusing on its efficacy in second language learning enhancement, through an analysis of student communicative proficiency and perceptions of the tool. Central to our methodology is a survey designed to collect, on a voluntary basis, text and voice conversations between participants and the GLC, providing a rich dataset for analysis. This survey, conducted at several intervals throughout the study, measures key outcomes such as lexical proficiency, user engagement, and the integration of this AI tool into study practices, especially in scenarios where direct instructor involvement is minimal. By examining the interactions with the GLC and assessing lexical proficiency as a primary metric, the research sheds light on the potential and limitations of employing advanced large language models and bundled technologies in educational settings, contributing crucial insights for the development of effective, technology-assisted second language education strategies.
Academic department under which the project should be listed
RCHSS - Foreign Languages
Primary Investigator (PI) Name
Dylan Goldblatt
Evaluating Multimodal Conversational Agents in Second Language Learning: Lexical Proficiency Enhancement with bespoke LLMs
This research evaluates the German Language Coach (GLC), a bespoke intelligent agent powered by OpenAI’s fourth-generation Generative Pre-Trained Transformer (GPT-4), focusing on its efficacy in second language learning enhancement, through an analysis of student communicative proficiency and perceptions of the tool. Central to our methodology is a survey designed to collect, on a voluntary basis, text and voice conversations between participants and the GLC, providing a rich dataset for analysis. This survey, conducted at several intervals throughout the study, measures key outcomes such as lexical proficiency, user engagement, and the integration of this AI tool into study practices, especially in scenarios where direct instructor involvement is minimal. By examining the interactions with the GLC and assessing lexical proficiency as a primary metric, the research sheds light on the potential and limitations of employing advanced large language models and bundled technologies in educational settings, contributing crucial insights for the development of effective, technology-assisted second language education strategies.