527 Identifying the Validity of ChatGPT in the Diagnosis of Orthopaedic Emergencies

Abstract

Abstract Aim Chat Generated Pre-Trained Transformer (ChatGPT) is a large language model that is pre-trained to predict the next token in a document. Our aim is to evaluate the clinical accuracy of ChatGPT in the diagnosis and management of orthopaedic emergencies. ChatGPT has been tested to pass high level medical licencing and board exams. However, there remains concerns regarding the use of artificial intelligence (AI) models, specifically surrounding data governance and accuracy. Method Ten clinical scenarios were drafted and then vetted by fellowship level orthopaedic surgeons. The scenarios cover the breadth of orthopaedic emergencies such as septic arthritis, open fractures, and cauda equina syndrome. The scenarios were run through ChatGPT which generated differentials and management options. The answers were then scored by orthopaedic registrars. Results The results demonstrated that ChatGPT is largely accurate and safe in the diagnosis of orthopaedic emergencies. In particular, it demonstrated excellent scoring regarding conversation structure with clearly written diagnosis and management plans. Conclusions This novel validity study demonstrates that ChatGPT has potential in the diagnosis of orthopaedic emergencies. Given the nature of the AI model, the answers are continually changing as more information is fed into the transformer, thus reproducibility remains a major challenge in testing the model. Furthermore, scoring the quality of the answers provided by the model is subjective. Nevertheless, this study suggests that ChatGPT could provide clinical assistance and more research in the area is necessary to test its limits and safety.

Ähnliche Arbeiten