Artificial intelligence approach to estimating trial feasibility with OncoLLM.

Abstract

822 Background: Assessing trial feasibility and predicting accrual is difficult and often inaccurate, resulting in the costly activation of studies that do not achieve their intended accrual. Artificial intelligence (AI) tools, including large language models (LLM), may complement expert judgment by systematically identifying trials with limited enrollment potential. The objective of this study was to compare investigator and LLM-based accrual estimates against actual accrual outcomes for gastrointestinal (GI) oncology trials. A secondary objective was to evaluate whether LLM-based accrual estimates could flag trials at risk of non-accrual. Methods: This study retrospectively examined GI oncology trials that closed before July 2024, prior to the adoption of the Triomics PRISM platform. For each trial, three measures were compared: investigator-derived accrual estimates generated before activation, actual patient accrual, and projections from OncoLLM (an LLM fine-tuned on institutional data and oncology guidelines). Accuracy was evaluated using two complementary approaches. First, the mean absolute error (MAE) was calculated between projected and actual accrual across trials. Second, a binary classification framework was applied using a pragmatic cutoff of ≤5 projected patients to flag trials at risk of non-accrual. Sensitivity, specificity, and overall accuracy were measured. Results: A total of 8 gastrointestinal oncology trials were included, with an average total accrual of 0.9 patients (SD 0.8). Investigator estimates were more accurate for continuous prediction (MAE 3.9 vs. 8.8 for OncoLLM). In the binary analysis, 4 of 8 trials (50%) closed with zero accrual. OncoLLM correctly classified 3 of the 4 trials as at risk of non-accrual (sensitivity 75%, specificity 100%). Across all 8 trials, OncoLLM correctly classified 7 (88%) as either at risk of non-accrual or not. Conclusions: While investigator estimates were closer to observed accrual when measured continuously, OncoLLM demonstrated value as a screening tool for identifying trials at risk of non-accrual. The model correctly classified 7 of 8 trials and flagged 3 of 4 non-accruing studies as at risk of non-accrual. LLM-based feasibility screening can be a complementary safeguard prior to trial activation, supporting more efficient resource allocation. Comparison of OncoLLM and physician estimates for trial accrual. Trial Physician Estimate OncoLLM Estimate Actual Accrual OncoLLM (At Risk of Non-Accrual) Actual No Accrual OncoLLM Correct 1 5 4 0 Yes Yes Yes 2 5 6 1 No No Yes 3 5 19 2 No No Yes 4 ≤5 0 0 Yes Yes Yes 5 5 26 2 No No Yes 6 4 3 0 Yes Yes Yes 7 5 9 0 No Yes No 8 5 10 2 No No Yes

Ähnliche Arbeiten