Real-Time Voice-Controlled Interface for AR/VR Surgical Simulations

Abstract

This study introduces a Context-Adaptive Real-Time Voice-AI Hybrid Control Interface for AR-assisted surgical environments, combining multimodal context sensing, predictive intent modeling, and adaptive voice control. The proposed system integrates visual, audio, and biometric cues through a ResNet-50 backbone with transformer-based temporal modeling to predict surgeon intent and assist with task sequencing. The custom dataset, including 24 simulated surgical procedures (≈14.5 hours), supports robust multimodal training. Results demonstrate high accuracy (Phase Recognition: 88.5 ± 1.2%; Intent Prediction: 90.2 ±1.0%) with sub-200 ms inference latency on RTX 3080 and ~250 ms on edge hardware. The system's low latency and noise-robust voice fusion enable seamless surgeon interaction, establishing readiness for pre-clinical validation and controlled simulation trials.

Ähnliche Arbeiten