The impact of text topic and assumed human vs. AI authorship on competence and quality assessment

Abstract

While Large Language Models (LLMs) are considered positively with respect to technological progress and abilities, people are rather opposed to machines making moral decisions. But the circumstances under which algorithm aversion or algorithm appreciation are more likely to occur with respect to LLMs have not yet been sufficiently investigated. Therefore, the aim of this study was to investigate how texts with moral or technological topics, allegedly written either by a human author or by ChatGPT, are perceived.In a randomized controlled experiment, n=164 participants read six texts, three of which had a moral and three a technological topic (predictor text topic). The alleged author of each text was randomly either labeled “ChatGPT” or “human author” (predictor authorship). We captured three dependent variables: assessment of author competence, assessment of content quality, and participants’ intention to submit the text in a hypothetical university course (sharing intention). We hypothesized interaction effects, that is, we expected ChatGPT to score lower than human authors for moral topics and higher than human authors for technological topics and vice versa.In contrast to the hypotheses, we did not find any significant interaction effects. However, ChatGPT was consistently devalued compared to human authors across all dependent variables: There were main effects of authorship for assessment of the author competence, β=.78, 𝑡(326)=7.36, p<.001, 𝑑=0.81; for assessment of content quality, β=.30, 𝑡(326)=3.31, p<.001, 𝑑=0.37; as well as for sharing intention, β=.85, 𝑡(326)=5.07, p<.001, 𝑑=0.56. There was also a main effect of text topic on the assessment of text quality, β=.21, 𝑡(326)=2.35, p=.019, d=0.26. These results are rather in line with previous findings on algorithm aversion than with algorithm appreciation. We discuss the implications of these findings for the acceptance of the use of LLMs for text composition.

Ähnliche Arbeiten