Implicit and explicit research quality score probabilities from ChatGPT

Abstract

Abstract The large language model (LLM) ChatGPT’s quality scores for journal articles correlate more strongly with human judgments than some citation-based indicators in most fields. Averaging multiple ChatGPT scores improves the results, apparently exploiting its internal probability model. To leverage these probabilities, we test two novel strategies: requesting percentage likelihoods for scores and extracting the probabilities of alternative tokens in the responses. These probability estimates were used to calculate weighted average scores. Both strategies were evaluated with five iterations of ChatGPT 4o-mini on 96,800 articles submitted to the U.K. Research Excellence Framework (REF) 2021, using departmental average REF2021 quality scores as a proxy for article quality. The data were analyzed separately for each of the 34 field-based REF Units of Assessment. For the first strategy, explicit requests for tables of score percentage likelihoods substantially decreased the value of the scores (lower correlation with the proxy quality indicator). In contrast, weighed averages of score token probabilities slightly increased the correlation with the quality proxy indicator and these probabilities reasonably accurately reflected ChatGPT’s outputs. The token probability leveraging approach is therefore the most accurate method for ranking articles by research quality as well as being cheaper than comparable ChatGPT strategies.

Ähnliche Arbeiten