Leveraging Large Language Models to Optimise Automated PET/CT Tumour Segmentation Performance in Retrospective Data

Abstract

Deep-learning, particularly nnUNet, has become the baseline for medical image segmentation, yet accuracy remains limited for undifferentiated tumours. While clinical implementation research uses only imaging data, retrospective research benefits from additional information sources. Our work developed a pipeline using radiology reports to enhance tumour segmentation in a retrospective sarcoma dataset, with an overarching aim to enable larger discovery research in retrospective standard of care data. The pipeline developed had three key component parts; radiology report interpretation using a large language model (LLM), mapping text-based output to image data and model predictions. LLM responses were systematically scored for consistency and only responses with a consistency score greater than 50% used. Text-based output were then mapped to imaging data using TotalSegmentator generated organ labels. This was used to inform the bounding box on which nnUNet was deployed. A single-site dataset comprised 132<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">18</sup> FDGPET/CT studies from 60 sequential sarcoma patients was used for development and evaluation under ethical approval (24/HRA/1339). Model results were compared to a baseline nnUNet with set-up based on successful approaches from the autoPET challenge. Incorporating radiology report data markedly improved the mean dice coefficient. The baseline nnUNet achieved a mean Dice coefficient of 0.41, whilst the LLM-assisted pipeline attained 0.62. This enhancement primarily stemmed from accurate identification of true negative cases (disease-free images). The developed methodology maintained high interpretability, potentially enabling case stratification for targeted operator input. Future work will focus on pipeline optimisation and development of an integrated multimodal approach.

Ähnliche Arbeiten