How to measure AI literacy (Fast): Development and norming of the AI competence objective scale short version (AICOS-S)

Abstract

As artificial intelligence (AI) becomes increasingly embedded in occupational, educational, and everyday contexts, the need for reliable and efficient assessment of AI literacy grows substantially. Despite the availability of existing instruments, significant gaps remain in test length, scalability, and, particularly, the possibility of standardized, norm-based interpretation. This study aims to overcome these limitations by developing and validating a new, psychometrically robust, and economical short version (AICOS-S) based on the existing AI Competency Objective Scale. The AICOS-S was derived through content-related and item-based reduction procedures and evaluated in a large, heterogeneous German-speaking sample recruited via an online panel. The findings demonstrate solid reliability and clear evidence of convergent and discriminant validity. Comparative analyses further reveal systematic differences in AI literacy by gender, age, occupational background, and educational qualification, underscoring the need for differentiated norm values, which are provided here for the first time. By integrating these norm values with a rigorously constructed short scale, the AICOS-S enables a scientifically sound, practically scalable, and contextually sensitive assessment of AI literacy. The resulting norming framework provides a reference structure that enhances the interpretability of AI literacy scores in research and applied settings, thereby supporting diagnostic precision and comparability. Future research should examine its applicability across different populations and cultural contexts. Overall, the AICOS-S advances the methodological foundation for AI literacy assessment and represents a substantial step toward establishing standardized, population-level benchmarks in a rapidly evolving technological landscape.

Ähnliche Arbeiten