12. Institutional Experience Improving Feedback Through Implementation of AI Driven Operative Entrustability Assessments

Abstract

Background: Traditional surgical education relies on immediate in-person feedback or broader end-of-rotation evaluations. These extremes yield vague feedback that is not specific to operative skill development, rendering residents unsure of their operative competency and faculty uncertain about assigning autonomy. This ambiguity compromises the efficiency of surgical training. Additionally, resident satisfaction with the quality of feedback received is measured through the ACGME annual resident survey and is a common source of training program citations. One solution to this problem is the longitudinal capture of case specific evaluations. With timely, specific, and frequent feedback, trainees can more effectively focus on ongoing improvement in their surgical skills. While this type of feedback was originally developed with competency-based residency models in mind, we feel it is beneficial for all trainees. We studied the implementation of an EMR integrated, HIPAA compliant surgical education platform, Firefly Lab (Firefly) in our program and the effect on resident satisfaction with faculty feedback. Firefly lab synchronizes with the OR schedule and this data triggers push notifications for faculty assessment and resident self-assessment after each case. Translation of assessment data into learning curves provides insights into a resident’s projected journey towards surgical autonomy. Firefly integration with the ACGME case logging system streamlines case logging for the residents with algorithm driven CPT suggestions and assists in transferring case log data to the ACGME interface, increasing trainee compliance with case logging. Methods: In this single institution study within an integrated training program, we analyzed data from Sept 2022 to Sept 2023 from the Firefly platform. 926 assessments were received during this time. We compared resident ACGME survey results and program evaluation scores in 2021 (pre-Firefly implementation) to results in 2022 (post-Firefly implementation). We also conducted a resident survey regarding the experience using Firefly. Results: In 2021, our score on the “Satisfied with faculty’s feedback” question of the program resident survey revealed a score of 3.6, lagging behind the national average of 4.3. This led to an ACGME citation of our residency program. Post-Firefly in 2022, our score climbed to 4.3, aligning with the national average of 4.4. A resident survey post-Firefly implementation revealed 77% of participating residents credit their improved feedback to the platform, while 70% acknowledged a surge in their case logging frequency. Discussion: Since introducing Firefly for our program in 2022, it has had a profound effect on our program’s feedback culture. Firefly’s integration of immediate feedback and predictive analytics addresses core challenges in surgical education. Observations from this institution highlight Firefly’s potential in transforming surgical feedback dynamics.

Ähnliche Arbeiten