Investigating the Effect of Sample Size and Respondent Characteristics on Usability Measurement: The Case of ChatGPT

Abstract

Standardized usability questionnaires are a fast and relatively effortless way of assessing usability of software products. Despite their long use, so far, little attention has been paid to the effect of sample size and the level of respondents’ acquaintance with the evaluated software on the measurement. This paper addresses this gap and uses SUS and mTAM measurements of ChatGPT to illustrate how the deviation from the mean usability score decreases with increasing sample size, and to confirm the significant effect of usage frequency and knowledge of the evaluated software and its alternatives on the measurement results. It also exposes no demographic bias due to participation of respondents of different gender, country of stay, and academic major.

Ähnliche Arbeiten