Assessment of Artificial Intelligence Credibility in Evidence-Based Healthcare Management with “AERUS” Innovative Tool

Abstract

Background: Artificial Intelligence (AI) technologies have found applications across various arenas, and Evidence-Based Management (EBMgt) is no exception.However, in this context, the careful assessment of AI outcomes becomes essential to confirm their credibility and ensure adherence to ethical standards.Mirroring recent similar explorations in the literature, this study introduced the "AERUS" tool, designed to evaluate AI trustworthiness in healthcare administration, focusing on five key areas: Accuracy, Efficiency, Reliability, Usability, and Security. Methods:The AERUS instrument evaluated AI's reliability in healthcare administration.It underwent minor revisions after an initial test with thirty healthcare administrators and internal consistency confirmation via Cronbach's alpha.The final version was tested on four AI models (ChatGPT 3.5, ChatGPT 4, Microsoft Bing, Google Bard) over six managerial topics, with evaluation by two raters using Cohen's kappa. Results:The refined AERUS tool assessed five areas: AI accuracy in management data, operational efficiency impact, decisionmaking reliability, user-friendliness for managers, and security protocol adherence.Initial testing with ten healthcare management statements showed high internal consistency (Cronbach's alpha of .911).Among six assessments, Microsoft Bing scored highest (mean 22.93, SD 1.11), followed by ChatGPT-4 (mean 22.00, SD 1.21), ChatGPT-3.5 (mean 20.00, SD 1.21), and Google Bard (mean 19.60, SD 1.22).Inter-rater agreement resulted in Cohen's kappa values ranging from 0.358 to 0.885 for the AI models.Conclusions and Recommendations: AERUS presents a supporting instrument for addressing AI credibility concerns in EBMgt, with recommendations for further research and widespread implementation to ensure the trustworthiness and reliability of AI in professional managerial decision-making.

Ähnliche Arbeiten