When are item nuances useful for prediction in organizations? Comparing the validity of item-level, scale-level, and ensemble machine learning models.

Abstract

Recent evidence suggests that prediction models using item scores, instead of the traditionally used scale scores, can more accurately predict outcomes of interest. However, little is known about the conditions under which item- or scale-level models are more suitable for prediction in organizational practice. To address this gap, we examined several real-world data sets for the presence of valid item nuances (i.e., criterion-valid item-specific variance that is lost when aggregating to scale scores). We then designed and conducted a Monte Carlo simulation based on empirical estimates to investigate the criterion-related validity of item- and scale-level models. In the simulation, we varied (a) the distribution of nuances among items, (b) the effect size of nuances, (c) the effect size of constructs, (d) scale internal consistency, and (e) training sample size. Results suggested that item-level models are recommended when relatively few items in a scale carry nuances, the nuance effect sizes are similar or larger in magnitude to the scale effect sizes, internal consistency is high, and training sample size is large. Our review of prior studies and analyses of real-world data suggest that the conditions favoring item-level models are not uncommon in organizational data. For conditions that do not substantiate the use of either item- or scale-level models, we also examined ensemble models and found that they can be an attractive alternative when the choice is unclear. We provide R code and recommendations for examining valid nuances in one's data and developing item-level, scale-level, and ensemble predictive models. (PsycInfo Database Record (c) 2026 APA, all rights reserved).

Ähnliche Arbeiten