Chatgpt Vs. Google Gemini: Assessment of Performance Regarding the Accuracy and Repeatability of Responses to Questions in Implant-Supported Prostheses

Abstract

Purpose: This study aimed to determine the accuracy and repeatability of the responses of different large language models to questions regarding implant-supported prostheses and assess the impact of pre-prompt utilization and the time of day. Materials & Methods: A total of 12 open-ended questions related to implant-supported prostheses were generated and the content validity of the questions was verified by a specialist. Following that, questions were posed to 2 different LLMs: ChatGPT-4.0 and Google Gemini (morning, afternoon, evening; with and without pre-prompt). The responses were evaluated by two expert prosthodontists with a holistic rubric; the concordance between the graders' responses and repeated responses by C and G software programs was calculated with the Brennan and Prediger coefficient, Cohen kappa coefficient, Fleiss kappa, and Krippendorff alpha coefficients. Kruskal-Wallis, Mann-Whitney U, independent t-test, and ANOVA analyses were used to compare the responses obtained in the implementations. Results: The results showed that the accuracy of ChatGPT and Google Gemini was 34.7% and 17.4%, respectively. The implementation of pre-prompt significantly increased accuracy in Gemini (p = 0.026). No significant difference was found according to the time of day (morning, afternoon, evening) or inter-week implementations. In addition, inter-rater reliability and repeatability showed high levels of consistency. Conclusion: The use of pre-prompt positively affected accuracy and repeatability in both ChatGPT and Google Gemini. However, LLMs can still produce hallucinations. Therefore, LLMs may assist clinicians but they should be aware of these limitations. Keywords: Chatbot, ChatGPT, Prostheses and Implant.

Ähnliche Arbeiten