Treatment Effect Estimates in Randomised Trials With Ai-augmented Control Arms

Abstract

<title>Abstract</title> This study empirically illustrates the risk of errors and misleading conclusions associated with trial augmentation with controls generated by AI using solely trial data. We generated 8,000,000 trials with AI-augmented control arms using the data from two large trials (IST and IST3), removed a fraction of the original control arm and replaced it with generated data with algorithms trained on the patients already included.We varied the nature of the removed patients, the sizes of the remaining original patient control group (from 10% to 50%), the deep learning architecture (CTGAN and TVAE), and the sampling methods to obtain the treatment effect estimate. To assess the augmented trials, we use the differences in treatment effect estimates between augmented and original trials, the number of significance disagreements, i.e., augmented and original trials reaching different conclusions, as lead by their confidence intervals (CIs) and the number of incompatible results, i.e., augmented and original trials with disjoint CIs.In IST, the absolute risk difference (ARD) of aspirin versus no aspirin on death or dependency at 6 months between the two arms, was − 0.012 (95% CI, -0.026 to 0.002). When augmenting a trial where the first 1000 participants in the control arm (10%) were kept and others were replaced with control patients using CTGAN and using the average procedure, we observed an ARD of 0.004 (95% CI, -0.010 to 0.018), representing a relative difference of 133%. While the original IST did not find a statistically significant treatment effect, 54% of these augmented trials defined above found a statistically significant treatment effect. Finally, 139/1000 (14%) of augmented trials had incompatible results with the original trial. We obtained similar results with IST3 and in all other scenarios.

Ähnliche Arbeiten