An AI Coach for Patient Centered Communication: PCOF-Based Evaluation and Feedback Using Large Language Models

Abstract

<h3>Context</h3> Primary care providers’ behaviours and interactions with patients are critical for delivering effective, patient-centered care. The Patient-Centered Observation Form (PCOF) offers formalized guidelines to identify skill gaps, coach providers, and recommend improvements in competencies. <h3>Objective</h3> To develop and evaluate the feasibility of an AI-driven coaching system, trained on PCOF guidelines, capable of automatically assessing physician–patient interactions and providing constructive, guideline-based feedback. <h3>Study Design and Analysis</h3> A multi-agent architecture was implemented using the LangGraph framework. Three instructed LLM agents—Evaluator, Reviewer, and Judge—were integrated with an OpenAI embedding-based document vectorizer and database system. This architecture analyzed physician–patient conversation transcripts, assessing semantic content, manner, and sentiment against PCOF skill sets, generating explainable reasoning and targeted feedback. <h3>Setting or Dataset</h3> The dataset comprised 1,200 physician–patient transcripts, covering diverse clinical contexts. These were used to engineer effective prompts and train the AI coach for PCOF-based evaluation. <h3>Population Studied</h3> Conversations spanned a range of patient case types, including frailty, diabetes, chest pain, and insomnia. <h3>Intervention/Instrument</h3> The intervention was “Larry-AI,” a conversational AI coach designed to monitor physician behaviors and assess adherence to PCOF competencies, delivering structured, actionable feedback. <h3>Outcome Measures</h3> Performance was assessed through field studies and qualitative physician satisfaction surveys, focusing on the AI’s ability to identify PCOF elements and provide relevant coaching feedback. <h3>Results</h3> In independent evaluations of five visit transcripts covering 50+ PCOF skill elements, Larry-AI matched human expert annotations in 88% of cases, with 100% of its feedback accompanied by identifiable, interpretable reasoning. <h3>Conclusion</h3> Larry-AI demonstrated the capacity to process and understand physician–patient interactions, delivering PCOF-aligned, constructive feedback that supports skill enhancement, professional development, and improved patient engagement. The study highlights the feasibility of using orchestrated, instructed LLM agents as AI coaches in clinical training and quality improvement.

Ähnliche Arbeiten