Assessing the Ethical and Professional Capabilities of AI: A Study of ChatGPT and Google Gemini versus PREview (Situational Judgement Test) for Medical Student Applicant

Abstract

Introduction: Artificial intelligence (AI) is increasingly integrated into healthcare, supporting tasks ranging from administrative functions to clinical decision-making. This study evaluates the ethical reasoning and professionalism of two AI models, ChatGPT and Google Gemini, by examining their responses to the Association of American Medical Colleges’ (AAMC) PREview exam—a situational judgment test assessing ethical and professional competencies in healthcare. Methods: ChatGPT 4.0 and Google Gemini 1.5 were evaluated using two sets of AAMC PREview scenarios. Each response was rated on a three-point effectiveness scale: “not correct,” “partially correct,” and “correct.” Full points were awarded for responses closely matching the AAMC’s ideal answers, while partial points were given for responses that were similar but varied. Statistical significance in performance differences was analyzed using a one-way ANOVA test. Results: ChatGPT and Google Gemini were conducted separately on both tests. ChatGPT achieved an accuracy rate of 79.3% on the first test and 77% on the second, while Google Gemini scored 68% and 70%, respectively. A statistically significant difference was observed in Test 1 (p = 0.002) but not in Test 2 (p = 0.101). Overall, ChatGPT demonstrated a stronger alignment with AAMC’s ethical standards than Google Gemini. Conclusion: Both AI models exhibited competence in ethical reasoning, with ChatGPT achieving a higher degree of alignment with medical ethics standards. While these models show promise as tools for ethical decision-making, they should complement, not replace, human judgment in complex healthcare contexts.

Ähnliche Arbeiten