Assessment of AI/ML Approaches for Qualitative Analysis in Obstructive Sleep Apnea

Abstract

Abstract Rationale: Positive airway pressure (PAP) therapy is the first-line treatment for obstructive sleep apnea (OSA). Understanding the user's therapy experience is essential in promoting PAP usage and achieving treatment goals. Artificial intelligence/machine learning (AI/ML) techniques like traditional natural language processing (NLP) techniques and large language models (LLMs) have become popular approaches for understanding real-world patient experience using large qualitative datasets. However, selecting an appropriate AI/ML model to examine themes and sentiment from real-world patient feedback datasets can be challenging. Methods: We compare the strengths and weaknesses of three AI/ML approaches for examining a large dataset of qualitative survey response data: NLP (BERTopic), LLM-based RAG (Retrieval-Augmented Generation using Mistral, LLama3) and hybrid NLP+LLM-based RAG. Models were assessed for model input and output, prompt engineering, result interpretability, hallucination and reproducibility of results. Qualitative survey response data regarding PAP therapy experience was previously collected from adult PAP therapy users (ResMed, San Diego CA) with OSA. Response data was examined only from users responding in the English language and residing in the United States. The three AI/ML models were used to identify key topic themes and assess sentiment related to PAP therapy experience. All analyses were run locally. Results: Qualitative responses from 5,633 PAP users were assessed. Compared to traditional NLP models, which give keyword-based themes, and standard LLMs, which give broad themes and sentiments, the hybrid RAG framework appeared to provide more accurate sub-themes and sentiments for each user. Furthermore, hallucination was limited as the hybrid model was grounded in real-world evidence and worked well with existing computational resources (Table). Conclusion: As PAP therapy technology continues to evolve, understanding the user experience will remain a central part of effective care management. A hybrid RAG framework for theme identification and sentiment analysis presents a promising solution for gaining valuable real-world user experience insights while efficiently handling large-scale qualitative datasets. However, human oversight is still required for model refinement and future work is needed to explore how a hybrid framework may perform across diverse datasets.

Ähnliche Arbeiten