Individual and Organisational Capacities for Assessing “Trustworthiness” of AI Systems in Healthcare Settings

Abstract

The integration of AI systems in healthcare settings fundamentally transforms professional work practices and introduces new requirements for professionals' competence in assessing AI system trustworthiness.While regulatory frameworks such as the EU AI Act mandate sufficient AI knowledge from employees, current studies reveal a significant skills gap in managing AI systems.This article examines what governance structures organizations must establish to enable employees to assess AI system trustworthiness and ensure continuous capability development.Drawing on Structural Empowerment Theory (SET), it analyses how human-AI interaction evolves from a tool-based to a collaborative-like relationship and identifies the organizational conditions required to empower professionals.Through scenario-based case analyses from the medical field, the study demonstrates that assessing AI trustworthiness is not solely a matter of individual competence but depends significantly on organizational structures that ensure access to information, resources, and participatory governance processes.The analysis reveals that with -at least seemingly -ever-advancing AI systems, the complexity of trustworthiness assessment grows, making individual evaluations alone insufficient.The findings indicate that successful AI integration goes beyond technical training and requires a comprehensive approach that embeds structural empowerment across multiple organizational levels.AI governance should therefore not only focus on technology regulation but shape human-AI interaction in ways that strengthen professional autonomy, competence, and trust.This necessitates establishing new learning and design processes, communication channels, and reflection spaces.The article develops human-centric design principles and presents a capabilityoriented framework for effective organizational AI governance.While structural empowerment represents a promising approach to fostering AI competencies, its limitations in complex and dynamic environments are also highlighted.Embedding structural empowerment within an adaptive governance framework proves crucial for equipping healthcare professionals to navigate the challenges of (potentially increasing) autonomous AI systems while maintaining human-centered decision-making and ethical standards.

Ähnliche Arbeiten