Towards the Reliability of Agentic Artificial Intelligence Models in Healthcare

Abstract

SUMMARY & CONCLUSIONSRecent advancements in Artificial Intelligence (AI) have revealed promising opportunities across industries. Continuous learning systems such as agentic AI models are designed to work collaboratively, adapt decision-making strategies, and solve complex problems. In healthcare, current practice has shown that it is challenging to certify the reliability of systems that are powered by self-learning AI models. This paper presents a methodology for evaluating and optimizing the reliability of agentic models, based on a multi-step approach.The case study presented in this work involves agentic models comprising a Generative Pre-trained Transformer (GPT) model, agents, and tools. The model configurations took event data and users’ prompt as input. The prompts were decomposed and orchestrated to appropriate tools to generate general and specific recommendations as output through tool learning. The model configurations were evaluated against performance metrics such as task adherence, tool call accuracy and intent resolution. A reliability index was introduced, based on the metrics obtained.A key strength of the proposed approach is its extended reach to knowledge resources and tools, beyond the specialty of foundation models. The proposed reliability evaluation approach can be applied to situations where scalability is required to monitor and optimize AI systems.

Ähnliche Arbeiten