Classifying research activity and impact indicators: Leveraging ChatGPT and the ReAct taxonomy for in-context learning

Abstract

This study evaluates the potential of LLMs for identifying and classifying self-formulated indicators using the ReAct taxonomy, a structured framework focused on micro-level research activities and impacts. Drawing on 6,752 indicators from 317 academic international cooperation projects funded by VLIR-UOS, the study explores zero- and few-shot prompting strategies to automate classification across ReAct’s three dimensions: participation in person, participation by product, and research inflow and 98 related categories. We assess and compare a simple classification prompt including limited detail with contextually rich prompts that contain category definitions, classification examples with reasoning statements, and hierarchical information from the taxonomy. Results show that ChatGPT, when guided by the rich contextual prompts, achieves high accuracy (up to 0.86) and F1 scores (up to 0.77), especially for well-represented classes. A manual validation of a stratified sample confirms strong model performance at the top taxonomy levels, though classification at deeper levels showed greater ambiguity. Limitations are discussed and include: the emphasis of the taxonomy on academic activities, underrepresented categories, and label ambiguity. The study demonstrates that large language models can support scalable, taxonomy-driven classification of unstructured textual impact data, providing a promising step toward more systematic societal impact identification and monitoring in diverse research reporting or assessment contexts.

Ähnliche Arbeiten