LLM-HeMO: A Large Language Model Driven Multimodal Fusion Framework for Intraoperative Hemodynamic Intervention Prediction

Abstract

Accurate prediction of intraoperative hemodynamic interventions is essential for timely decisionmaking and improved perioperative outcomes, yet most existing models rely on single modalities or target a specific intervention, limiting their generalizability. We propose LLM-HeMO, a multimodal fusion framework to deeply integrate structured electronic medical records (EMR) and high-frequency vital-sign time series for predicting intraoperative hemodynamic intervention. Prompt engineering was first employed to convert raw time series into clinically meaningful narratives, and further align multi-modal data with task-oriented semantics via LLM-generated semantic summaries. Two architectures were then compared: JointContext which is based on simple concatenation, and DualEncoder, which incorporates a timeseries encoder and bidirectional cross-attention for deep crossmodal integration. On 626 surgical cases from the VitalDB dataset, LLMHeMO significantly outperformed unimodal and concatenationbased baselines, achieving AUROC of 0.976 for binary task and F1-score of 0.708 for multi-class task. Visualization and ablation analysis showed the differentiated contributions of semantic summaries and time-series representations for different tasks. While semantic summaries were most effective for coarse-level binary prediction on whether intervention was required, both semantic summaries and temporal dynamics proved essential for fine-grained multi-class prediction of specific intervention types. In conclusion, LLM-HeMO demonstrates that LLMdriven multimodal fusion framework enables more accurate and interpretable intraoperative intervention prediction, offering a promising step toward intelligent perioperative decision-support systems.

Ähnliche Arbeiten