Assessing the performance of large language model Artificial Intelligence in thyroid cancer multidisciplinary team (MDT) decision-making

Abstract

Abstract Background Large language models (LLMs) have potential in augmenting clinical decision-making. The performance of three leading LLMs (ChatGPT, MetaAI, and DeepSeek) was evaluated in thyroid cancer MDT decision-making. Method Forty clinical cases (20 simulated/20 genuine), median age 45 (25–74), 26F, were discussed in a regional thyroid cancer MDT to provide the reference decisions. Cases were then blind-assessed by three LLMs that had been trained on contemporary guidelines, using uniform prompts. LLM outcomes were compared with reference decisions via a 4-point concordance score. Thematic analysis was performed. Results ChatGPT demonstrated the highest overall concordance score, 97/120 (36 cases concordant/near-concordant), DeepSeek scored 93/120 (33 concordant/near-concordant) and MetaAI performed least well 87/120. Despite identical guideline references, LLMs offered differing interpretation and application. ChatGPT offered the greatest nuance, with outputs that closely mirrored MDT phrasing and reasoning, especially around complex/borderline cases. ChatGPT and DeepSeek explicitly acknowledged patient choice, but ChatGPT more consistently contextualised it within guideline-compliant treatment options. MetaAI demonstrated more variability, with several examples of overtreatment or misalignment with current clinical practice. None of the LLMs recommended more recent treatment options for poorly-differentiated/anaplastic disease such as neoadjuvant or combination chemotherapy. Conclusion LLMs showed good concordance with clinical MDT decisions, even in complex real-world cases. However, despite receiving the same prompts there were differences in reasoning. ChatGPT’s nuanced, context-aware outputs appear most promising as a decision-support tool. Last, because LLMs lack intrinsic capability for continual learning at present, they have clear limitations, for example in keeping pace with some areas of advancing clinical practice.

Ähnliche Arbeiten