6ER-011 Evaluating the potential of ChatGPT to simulate clinical trial data: implications for the integrity of scientific research

Abstract

<h3>Background and Importance</h3> Large Language Models (LLMs) like ChatGPT represent a significant opportunity for healthcare. However, they may also facilitate fraudulent scientific practices. The ability of LLMs to generate plausible clinical trial (CT) data poses a potential threat to research integrity. <h3>Aim and Objectives</h3> This study aimed to assess the capability of GPT-4 ADA (OpenAI) to generate a dataset resembling that of a real CT, specifically in the context of comparing two drugs without previous direct comparisons. <h3>Material and Methods</h3> Instructions for the LLM were adapted from those described by Taloni et al. (2023) for a CT comparing two pharmacological treatments. Through the ChatGPT-4 interface, a request was made to generate a database of 500 patients diagnosed with advanced clear cell renal carcinoma. The dataset was designed to simulate a CT comparing pembrolizumab + axitinib with nivolumab + cabozantinib. The data were required to show a statistically significant difference in progression-free survival (PFS) between the two treatment groups. <h3>The following variables were specified</h3> Patient code. Sex: 39% male, 61% female. Date of birth. Treatment: Nivolumab + cabozantinib (50%), pembrolizumab + axitinib (50%). Recruitment region: North America (24%), Western Europe (26%), Rest of the World (50%). Combined Positive Score for PD-L1: ≥1 (59%), &lt;1 (41%). Number of organs with metastasis: 1 (26%), ≥2 (74%). Prior radiotherapy: 10%. Prior nephrectomy: 82%. Time to death and PFS: measured in months. A Cox proportional hazards model was used to estimate the hazard ratio (HR) and 95% confidence interval (CI), confirming the statistical significance of the differences observed. <h3>Results</h3> Within two minutes, ChatGPT generated a downloadable Excel database containing information on 500 pseudo-anonymised patients. The data met all predefined criteria. The analysis showed a median PFS of 18.78 months for nivolumab + cabozantinib and 25.25 months for pembrolizumab + axitinib, with a HR of 0.4 (95% CI: 0.32–0.5, p&lt;0.001). <h3>Conclusion and Relevance</h3> ChatGPT’s ability to generate CT-like datasets demonstrates a significant risk to the integrity of scientific research, as it could facilitate fraudulent publications. Awareness of this threat and increased transparency in study registration and data handling are essential to safeguard research integrity. Hospital pharmacists should recognise the potential for LLMs to enable scientific fraud, highlighting the need for study registration and transparency in data handling. <h3>References and/or Acknowledgements</h3> https://jamanetwork.com/journals/jamaophthalmology/fullarticle/2811505 <h3>Conflict of Interest</h3> No conflict of interest

Ähnliche Arbeiten