Assessing the performance of Watson for oncology, a decision support system, using actual contemporary clinical cases.

Abstract

8023 Background: IBM Watson for Oncology (WFO), trained by Memorial Sloan Kettering (MSK), is a cognitive computing system designed to assist medical oncologists making treatment decisions for individual patients. Recommendations are consistent with established guidelines and published evidence, as reflected in MSK’s practice and historical cases. Treatment options are classified as Recommended (WFO-REC), For Consideration (WFO-FC), or Not Recommended (WFO-Not REC). Published evidence, medical logic, and drug information are presented for each treatment option. We sought to assess the current performance of WFO to benchmark accuracy and identify areas for development. Methods: 20 de-identified cases were selected from the practices of two MSK thoracic medical oncologists. Patients presented for initial consultation regarding first-line systemic therapy during 2014 and all necessary information to make a treatment decision was available at the time of initial consultation or within two weeks, including molecular pathology. Cases were entered into WFO using structured attributes. WFO recommendations were compared to those of the MSK thoracic medical oncologist (MSKMD-REC). Results: WFO-REC and MSKMD-REC matched 50% of the time. 25% of the MSKMD-RECs appeared in the WFO-FC category and 25% in WFO-Not REC. In the 16 cases with metastatic lung cancers, 88% of administered regimens were returned as WFO-REC or WFO-FC. All choices were within established guidelines. Cases where the MSKMD-REC appeared as WFO-Not REC involved elderly patients with co-morbidities not yet included in WFO. Conclusions: While WFO‘s choices today fall within evidence-based standards, WFO has the capacity to provide greater precision through iterative training and development. Elderly patients for whom care choices are heterogeneous based on co-morbid illnesses represent a challenge. Benchmarking against actual cases has helped us to prioritize development work to increase the number of attributes to include more co-morbid conditions and to incorporate patient preferences to improve precision.

Ähnliche Arbeiten