Generative AI in Simulation Debriefings: An Exploratory Study Using the Team-FIRST Framework

Abstract

<title>Abstract</title> <bold>Background</bold> Effective debriefings in simulation-based education require accurate observation of team interactions, yet facilitators face challenges due to cognitive load, observer bias, and the complexity of team dynamics. Generative artificial intelligence (AI) offers a potential means to support this process by analyzing verbal communication and providing structured feedback. This study explored how AI can contribute to teamwork observation and debriefing in high-fidelity medical simulations. <bold>Methods</bold> We conducted a qualitative, exploratory study using thematic analysis of simulation participants’ and debriefers’ experiences with AI-generated teamwork reports. Forty-one participants (anesthesia nurses, residents, and attendings) participated in high-fidelity scenarios at the University Hospital Zurich simulation center. Verbal interactions were transcribed with AI-assisted speech recognition and analyzed using two large language model–based systems (Isaac and ChatGPT-4o) guided by a prompt based on the Team-FIRST framework. Structured reports were generated for each scenario and reviewed by four experienced debriefers. Semi-structured interviews captured learners’ perspectives on being observed by AI. <bold>Results</bold> A total of 26 AI-generated reports and 27 learner interviews were analyzed. Debriefers valued the detailed transcripts and illustrative quotes, which supported structured feedback and captured observations that might otherwise be missed. Limitations included inaccuracies in categorization, misattribution of speakers, overly generalized interpretations, and the absence of contextual or nonverbal information. Learners expressed openness and optimism about AI’s potential benefits: efficiency, objectivity, and enhanced perception, while also raising concerns about transparency, data protection, interpretation errors, and risks of overreliance. Both groups emphasized the necessity of human oversight. <bold>Conclusion</bold> Generative AI can complement simulation debriefings by structuring communication data and highlighting teamwork patterns, thereby supporting reflective practice. Current limitations highlight the need for multimodal approaches, refined prompting strategies, and integration with expert facilitation to ensure AI functions as a support tool rather than a replacement in simulation-based education. <bold>Trial Registration</bold> BASEC ID: Req-2024-01642.

Ähnliche Arbeiten