Comparing AI and Human Coding of NIH Grant Abstracts to Identify Innovations in Opioid Addiction Treatment (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Artificial intelligence (AI) methods such as Large Language Models (LLMs) are gaining attention as tools for qualitative data analysis. </sec> <sec> <title>OBJECTIVE</title> This study compares the effectiveness of ChatGPT4.0, compared to human coders, in coding the basic innovation of NIH grants. </sec> <sec> <title>METHODS</title> 118 NIH grant abstracts were coded using ChatGPT and human coders. In a first step, a description of the main “innovation” of the grant (e.g., the idea or practice being tested) was produced by both ChatGPT and humans. In a second step, the quality of those outputs was rated by both ChatGPT and human coders for depth/detail (DD) and relevance/completeness (RC), on 5-point Likert scales. Identical instructions were given to ChatGPT and human coders at both steps. </sec> <sec> <title>RESULTS</title> GPT-generated innovation outputs were consistently rated higher on both DD and RC, by both humans and ChatGPT (F (1, 176) = 133.9, P &lt; 0.001)). On average, human evaluators gave ChatGPT outputs a rating of 4.47 on both DD and RC, compared to 3.33 (DD) and 3.24 (RC) for human-generated outputs, suggesting that ChatGPT-produced outputs were more detailed and relevant than human outputs, as judged by human evaluators. </sec> <sec> <title>CONCLUSIONS</title> ChatGPT 4.0 shows potential in qualitative analysis of grant abstracts. ChatGPT output was rated higher in depth/detail and relevance/completeness by both humans and ChatGPT itself, compared to human outputs. When used with carefully designed prompts and instructions, LLMs can augment and enhance traditional qualitative coding, leading to more efficient, detailed, and consistent results. These findings highlight the potential to incorporate LLMs in research evaluation processes. </sec>

Ähnliche Arbeiten