Evaluation of basic life support practice skills based on artificial intelligence technology: system construction and evaluation equivalence validation

Abstract

BACKGROUND: Basic life support (BLS) skills are the core competencies of healthcare professionals in responding to emergency situations. Traditional manual assessments suffer from subjective bias, low efficiency, and are affected by examiner fatigue. Artificial intelligence (AI) technology brings innovative opportunities for skill assessment, but currently there is a lack of mature and validated BLS AI assessment systems. AIM: To construct a basic life support practice skills evaluation system based on artificial intelligence technology, and verify its equivalence to manual evaluation. The purpose is to construct an evaluation system of basic life support practice skills based on artificial intelligence technology, verify its equivalence with manual evaluation, and analyse the advantages and development direction of artificial intelligence technology in the evaluation of basic life support practice skills. METHODS: The BLS practical skills evaluation system based on AI technology was developed based on RTMPose, ST-GCN, SVM, YOLOX and Whisper AI base models, and the BLS practical skills assessment environment was constructed, which contains audio and video capture systems, CPR simulators with distance sensors, and displays for human-computer interaction with candidates. Using a paired design, the BLS practical skills assessment was conducted in August 2024 among 85 new nurses of the class of 2024 at Ruijin Hospital of Shanghai Jiaotong University School of Medicine, where each candidate's BLS skills performance was assessed by both the examiner and the AI system. The differences between the examiner scores and the AI system scores were compared, and at the same time, the self-assessment results of the Visual Analogue Scale of Fatigue Severity (VAS-F) of the four examiners before and after the scores were collected, as well as the candidates' satisfaction and acceptance of the application of the system for the BLS practical skills assessment. RESULTS: There was no significant difference between AI system scores and examiner scores (t = -0.294, p = 0.769), with good absolute agreement further confirmed by an intraclass correlation coefficient of 0.868 (95% CI: 0.803-0.912). Examiner VAS-F self-ratings were 73.50 ± 19.33 before and 82.25 ± 25.10 after the examination, respectively, and 51 (64.56%) of the candidates indicated that the AI system assessment helped to reduce their nervousness. CONCLUSION: AI system marking is consistent with the results of examiner marking and can be used to evaluate BLS practical skills, and, the application of AI system can improve the effectiveness of the assessment organisation. Meanwhile, the candidates' acceptance of the AI system was good, and the application can be expanded after further optimisation of the system. Candidates showed good acceptance of the AI system; however, expansion of application should proceed only after further optimization of system stability to reduce failure rates below 5%.

Ähnliche Arbeiten