Effect of Artificial Intelligence on Learning: A Meta-Meta-Analysis

Abstract

Artificial intelligence (AI) and large language models (LLMs) are believed to revolutionize education. This claim is supported by empirical evidence from dozens of meta-analyses that report large and statistically significant average effects of AI/LLMs on learning outcomes. We challenge these findings by a study-level meta-meta-analysis of 1,840 effect sizes from 67 meta-analyses. The results show strong evidence of severe publication bias and extreme between-study heterogeneity. Publication bias-adjusted analyses reveal model-averaged effects approximately one-third the magnitude of unadjusted estimates (standardized mean difference, SMD = 0.196, 95% credible interval from 0.000 to 0.323), along with wide prediction intervals spanning both large negative and positive effects (−1.521 to 1.908). Subgroup analyses fail to identify specific moderators that yield consistent benefits. No substantial difference exists between studies published before or after 2023. Overall, broad claims of generalized learning gains resulting from AI/LLMs appear premature; the current evidence is insufficient to support robust policy or practice recommendations.

Ähnliche Arbeiten