GPT-4 in a Cancer Center: Institute-Wide Deployment Challenges and Lessons Learned

Abstract

Generative AI has both significant untapped potential and unique risks for healthcare. Here we report on the implementation challenges, technology solutions, and lessons learned in broad deployment of large language models to the workforce of Dana-Farber Cancer Institute. In order that other organizations might benefit from our experience, we detail the aspects of sponsorship, governance, technical implementation and configurations, program launch, socialization, and ongoing support that went into making generative AI available to our entire workforce in a compliant, auditable, and secure manner. ----- The enormous potential for generative pre-trained transformers and other artificial intelligence large language models to improve healthcare has become increasingly clear since the release of ChatGPT in November 2022. Software tools based on large language models have been shown to perform as well as or better than humans on many healthcare-related tasks, including generation of clinical documentation, extraction of structured data from medical records, performance on a growing number of medical board exams, and writing accurate and empathetic responses to patients’ medical questions. However, healthcare settings pose unique ethical, legal, regulatory, and technical challenges for large-scale deployment and adoption of large language models, including the essentiality of patient data privacy and security, the direct negative consequences of errors and biases, the need for model interpretability and supporting evidence, the need to safeguard intellectual property and proprietary data, and the difficulty of modifying clinical and operational workflows. Because of these risks and challenges, few large language models are in use in hospitals outside of controlled research studies or small pilots, and none to our knowledge are yet broadly deployed in a dedicated cancer center. In this case study, we report on the implementation challenges and lessons learned in the evaluation and deployment of large language models at Dana-Farber Cancer Institute for use by all business areas including research and clinical operations. In early discussions about whether and how to proceed, it became clear that while some inherent risks could be mitigated by clear policy guardrails and a secure technical environment, significant risks would remain, including compliance with rapidly evolving regulations (e.g., for academic use of LLMs). We also recognized the substantial, ongoing work that would be required for appropriate ethical consideration of each use case, and to ensure patient- and human-centric decision-making. After robust discussion over many months, we believed it would be better for our institution to tackle these questions and challenges together as a community, with broad participation and discussion, in an audited, continuous-learning environment, than to prohibit all use of LLMs. Here we detail aspects of sponsorship, governance, technical implementation, program launch, socialization and ongoing support that went into making generative AI broadly available to our 12,500-member workforce in a compliant, auditable, and secure manner. We also share our source code and infrastructure-as-code to benefit others who may follow the same path. We hope other institutions can benefit from our experience as they consider deployment of generative AI that furthers their medical and research missions.

Ähnliche Arbeiten