Findings from Studies on English-Based Conversational AI Agents (including ChatGPT) Are Not Universal

Abstract

A common, but largely untested, assumption in artificial intelligence (AI) and speech systems is that results from experiments using only the English language as the communication medium will hold true across any other language or cultural context. We argue here, based on emerging recent scientific evidence, that such an assumption appears to be invalid. In fact, there appear to be stark differences across languages and cultures when experiments are conducted using the same artificial speech system setup to be able to communicate in more than one language. Moreover, using those AI systems with bilingual human speakers shows that their behavior, social cues, and communication patterns change when language "code-switching" occurs within the same experiment session. To illustrate our point further, in the second half of the paper we give the specific example of ChatGPT (as the backbone speech content for artificial speech systems) being used for older adults with dementia and Alzheimer’s, who often have altered speech patterns (e.g. slurred pronunciation). There are emerging reports from such research of severe limitations of ChatGPT in such contexts, which highlights the dangers of assuming findings from a narrow range of linguistic and/or cultural contexts can fully capture some universal truths about human communication with artificial agents. Finally, we point out that the reluctance of scientific journals and conferences to publish negative results means many of those emerging reports are only being reported anecdotally, which is problematic for the field of conversational user interfaces (CUI).

Ähnliche Arbeiten