Classification of Living Kidney Donation Experiences on Reddit: Understanding the Sensitivity of ChatGPT to Prompt Engineering (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Living kidney donation (LKD), where individuals donate one kidney while alive, plays a critical role in increasing the number of kidneys available for those suffering from kidney failure. Prior studies show that many generous people are interested in becoming living donors, but a huge gap exists between the number of patients on the waiting list and the number of living donors yearly. </sec> <sec> <title>OBJECTIVE</title> To bridge this gap, we investigated how to identify potential living donors from discussion on public social media forums so that educational interventions could later be directed to them. </sec> <sec> <title>METHODS</title> Utilizing Reddit forums as an example, this study describes the classification of Reddit content shared about LKD into three classes: (1) Present (presently dealing with LKD personally), (2) Past (dealt with LKD personally in the past), and Other (LKD general comments). An evaluation comparing a fine-tuned DistilBERT model (hereafter written BERT for simplicity) and inference with GPT-3.5 (ChatGPT) occurred. To systematically evaluate ChatGPT’s sensitivity to distinguishing between the three prompt categories, we employed a comprehensive prompt engineering strategy encompassing a full factorial analysis in 48 runs. A novel prompt engineering approach, Dialogue Until Classification Consensus (DUCC), was introduced to simulate a deliberation between two domain experts until a consensus on classification was achieved. </sec> <sec> <title>RESULTS</title> BERT and GPT-3.5 exhibited classification accuracies of approximately 75% and 78%, respectively. Recognizing the inherent ambiguity between classes, a post-hoc analysis of incorrect predictions was performed, revealing sensible reasoning and acceptable errors from the predictive models. Considering these acceptable mis-matched predictions, the accuracy improved to 89.33% for BERT, 90.67% for GPT-3.5, and 92.67% for GPT-4. </sec> <sec> <title>CONCLUSIONS</title> Large Language Models (LLMs) like GPT-3.5 are highly capable of detecting and categorizing LKD targeted content on social media forums. They are sensitive to instructions, and the introduced DUCC method exhibited superior performance over standalone reasoning, highlighting the merit in advancing prompt engineering methodologies. The models can produce appropriate contextual reasoning, even when final conclusions differ from human counterparts. </sec>

Ähnliche Arbeiten