Performance of AI Chatbots in Preliminary Diagnosis of Maxillofacial Pathologies

Abstract

BACKGROUND Artificial intelligence (AI) has shown significant potential in transforming healthcare by enabling accurate, data-driven decision-making. This study compared the performance of the AI chatbots ChatGPT, Grok, Blackbox, and Claude AI in preliminary diagnosis of maxillofacial pathologies. MATERIAL AND METHODS This study included 23 patients (9 cysts, 14 neoplasms) who underwent operations at Dicle University Faculty of Dentistry between 2017 and 2024 and had their diagnoses histopathologically confirmed. For each case, 4 differential diagnosis options were prepared in question format and directed to the AI platforms. The accuracy of the answers given by the chatbots was analyzed by comparing them with the definitive histopathological diagnoses of the cases. Statistical analysis used the chi-square ad Fisher-Freeman-Halton tests to compare performance among the chatbots. Statistical significance was set at p<0.05. RESULTS ChatGPT answered 15 out of 23 questions correctly, achieving a success rate of 65.2%. Grok and Blackbox AI each achieved a success rate of 52.17%, while Claude AI achieved the lowest success rate, at 30.43%. When cases were categorized into cysts and neoplasms, Blackbox AI showed the highest accuracy for cyst cases (66.6%), while ChatGPT had the highest accuracy for neoplasm cases (71.4%). No statistically significant difference was observed in the distribution of correct and incorrect answers among the chatbots (p=0.125). No statistically significant difference was observed in the distribution of cysts and neoplasms answers among the chatbots (p=0.654). CONCLUSIONS Although all 4 AI chatbots achieved certain levels of accuracy, ChatGPT showed superior performance compared to other chatbots. The development of these chatbots could be beneficial for diagnostic accuracy and treatment recommendations in dentistry.

Ähnliche Arbeiten