P088: Leveraging OpenAI GPTs for precision oncology: A custom guideline-constrained model in hereditary cancer

Abstract

Introduction: Effective management of hereditary cancer requires integrating evolving genetic evidence, risk models, and surveillance recommendations across various syndromes.The National Comprehensive Cancer Network (NCCN) guidelines for hereditary cancer are comprehensive but are updated frequently and can be complex to navigate in clinical settings.Healthcare providers often face challenges in accessing and interpreting these guidelines at the point of care, leading to inconsistent management practices.While Large language models (LLMs) have demonstrated potential to summarize complex medical information, they can also hallucinate, omit critical details, or generate non-guideline-concordant responses.This study outlines the design of a custom-trained LLM grounded in guidelines and evaluates its effectiveness in improving the accuracy and transparency of hereditary cancer recommendations compared with general-purpose models.Methods: This cross-sectional comparative study evaluates the concordance of an AI model with national hereditary cancer guidelines.The NCCN guidelines (Version 1.2025) were integrated into OpenAI's Custom GPT platform.Prompt engineering followed the RICCE framework (Role, Input, Context, Constraints, Examples) to ensure standardized, guideline-based outputs, including structured sections for summary, recommendations, rationale, citations, disclaimers, and version stamps.The model was configured to automatically decline out-of-scope queries.Ten representative provider questions covering cancer risk, surveillance, surgical management, and cascade testing were independently posed to the custom GPT, ChatGPT, and Gemini.Responses were anonymized and scored by two independent reviewers blinded to model identity using a 3-point rubric (0 = off-guideline, 1 = partially concordant, 2 = fully concordant).Interrater agreement and descriptive statistics will be calculated.Results: (anticipated) Data collection and scoring are ongoing.Preliminary review suggests that the NCCN-custom GPT consistently produces structured responses that explicitly cite relevant guideline algorithms, footnotes, and version identifiers.The model reliably refuses to answer questions beyond the NCCN's scope and maintains internal consistency across related queries.In contrast, general-purpose models occasionally provide recommendations that, while plausible, do not consistently adhere to current guidelines, or they may omit essential citation and version details.Metrics for quantitative concordance scoring, interrater reliability, and an analysis of the output structure will be presented. Conclusion:This ongoing evaluation examines the potential for a guideline-embedded LLM to deliver accurate, transparent, and up-to-date decision support for hereditary cancer care.Preliminary qualitative findings suggest that applying a structured prompt architecture (RICCE) and embedding authoritative NCCN content help mitigate LLMs' limitations, including hallucinations and deviations from established guidelines.If validated through comprehensive analysis, this approach could support scalable, auditable AI tools that complement, not replace, clinical judgment in genetics and oncology practice.

Ähnliche Arbeiten