Algorithmic Bias in AI-Driven Medical Education: Uncovering Implicit Racial and Gender Disparities

Abstract

INTRODUCTION: As AI technology becomes increasingly prevalent, its role in medical education is expanding. Many educational startups now incorporate large language models (LLMs), such as ChatGPT APIs, into their platforms. However, since these models are trained on existing data, they may inadvertently perpetuate gender- and race-based inequities, reflecting broader historic societal biases. Prior research has demonstrated a decline in the percentage of underrepresented individuals in medicine (URiM) along the surgical career pipeline. However, it remains unclear whether AI-driven career guidance reinforces these disparities. This study aims to identify potential biases in AI-generated career advice by assessing the influence of sex and race on ChatGPT’s 4o specialty recommendations. METHODS: A total of 200 unique applicant profiles were generated, systematically varying sex, race, and academic achievement. Each profile was queried for specialty recommendations using a standardized prompt that demonstrated an interest in a surgical residency, and ChatGPT 4o top three suggestions were recorded. Specialties were ranked on a 4-point scale by competitiveness based on NRMP data. For each profile, a competitiveness score was calculated as the sum of assigned points for recommended specialties, and the number of surgical specialties suggested was recorded. Multivariate linear regression assessed the effects of academic achievement, race, and sex on (1) the number of surgical specialties recommended and (2) the competitiveness score. Bonferroni correction was applied for multiple comparisons (p<0.05). Statistical analyses were conducted in RStudio 2023. RESULTS: Racial disparities were observed, with Black, Hispanic, and South Asian students receiving fewer surgical specialties and lower competitiveness scores compared to White students. Specifically, Black students were recommended significantly fewer surgical specialties (Coef = -0.48, p = 0.003) and had the largest decrease in competitiveness scores (Coef = -2.58, p < 0.0001). Hispanic students exhibited a similar pattern, with fewer surgical specialties recommended (Coef = -0.43, p = 0.008) and lower competitiveness scores (Coef = -2.23, p < 0.0001). Asian students showed lower competitiveness scores (Coef = -0.90, p = 0.03) but did not experience a significant difference in the number of surgical specialties recommended (Coef = -0.25, p = 0.115). Higher Step 2 scores and publication counts were significantly correlated with higher competitiveness scores and number of surgical specialties recommended. Female students were significantly less likely to receive surgical specialty recommendations, though their competitiveness scores did not differ significantly from male students (Coef = -0.69, p < 0.0001; Coef = -0.23, p = 0.3773). CONCLUSION: ChatGPT specialty recommendations exhibit algorithmic implicit biases related to sex and race, with female and URIM students receiving fewer surgical recommendations and lower competitiveness scores compared to white students with similar academic standing. These findings are particularly concerning given the rapid expansion of AI-driven startups utilizing these LLM in medical education. This underscores the urgent need for transparency and bias mitigation strategies to prevent the reinforcement of existing inequities.

Ähnliche Arbeiten