A Pilot Study on the Accuracy of Large Language Models (ChatGPT, Google Bard, and Microsoft Copilot) for Selecting the Correct Modality for Musculoskeletal Clinical Cases According to the ACR's Appropriateness Criteria

Abstract

ABSTRACT Background Large language models (LLMs) are becoming more commonly used in many aspects of radiology. Some authors have previously tested the capacity of various LLMs to suggest the correct imaging modality according to the guidelines of various associations, such as the American College of Radiology. This study aims to test whether free LLMs can suggest the most appropriate imaging modality in various musculoskeletal radiological cases. Methods We tested ChatGPT 3.5, Google Bard, and Copilot (Precise) to see if they could correctly suggest the appropriate imaging modality per the American College of Radiology's Appropriateness Criteria, using clinical vignettes from the musculoskeletal section. Seventy‐six vignettes were submitted to each chatbot, with the answer only being considered correct if it was the most appropriate according to the guidelines. Results ChatGPT 3.5 was correct in 82% of cases, Bard in 66%, and Copilot in 89% of cases. Bard was unable to answer in four cases, claiming that as a LLM, it was not capable of answering the question. Conclusions We found that all three LLMs were able to suggest the correct modality in a majority of cases. However, there was variability in the performance of the LLMs, with Copilot performing the best overall, with an accuracy of 89%.

Ähnliche Arbeiten