Abstract 66: Diagnostic Accuracy of ChatGPT4.o for TIA or Stroke Using Patient Symptoms and Demographics

Abstract

Introduction: Many patients may not recognize the initial symptoms of TIA or stroke and delay seeking urgent medical care, leading to missed treatments and worse outcomes. Diagnostic decision support (DDSS) systems may help these patients recognize and act on early symptoms of stroke or TIA. Large language models (LLMs) are available and utilized by the public. We evaluated the efficacy of GPT 4.o, a recent LLM, for the prediction of stroke or TIA from data collected at presentation among patients admitted to an emergency department TIA/Stroke observational unit (ED-OU). Methods: 1466 patients admitted to the ED-OU for suspected TIA in a large, urban, academic ED, from 3/2013 - 2/2020 were included. A thorough history and physical were obtained, including presenting symptoms, symptom time course, vital signs, and past medical history. The outcome was a discharge diagnosis of stroke, TIA, or an alternate diagnosis, from a consulting neurologist and confirmed with neuroimaging. For a random sample of 500 records demographics and symptom data were entered into GPT 4.o via OpenAI’s Application Programming Interface (API). We used a validated prompt that requests the 5 most likely diagnoses. The sensitivity of the GPT 4.o diagnoses for the neurologist’s diagnosis was calculated as a match in the top 1 (M1), top 3 (M3) or top 5 (M5). Sensitivity to a combined diagnosis of TIA or stroke was also calculated. Results were compared to (1) a random sample of 100 patients from the same dataset manually entered into the ChatGPT4.o interface by a research assistant, and (2) results from evaluation of ChatGPT 4.0 with data from a study of patients requesting urgent primary care, who entered their clinical data into a DDSS app. Results: The 500 cases included 257 with TIA, 73 with stroke, and 170 with other diagnoses. Table 1 shows results of diagnostic matches. Diagnostic lists of 18.4% of cases had no match between GPT4.o and the neurologist’s diagnosis. 5.4% of cases had no clear neurologist diagnosis. GPT4 sensitivity just for combined diagnosis of TIA or Stroke was 98.8%. Conclusions: DDSS like ChatGPT/GPT4.o have potential to aid patients’ prompt recognition of TIA or stroke symptoms which could shorten time to care. To better define usability, accuracy and safety of DDSS, we are studying direct data collection from patients in the ED or urgent primary care, including stroke patients or their companions, and evaluating other DDSS tools including symptom checkers.

Ähnliche Arbeiten