The Impact of Acoustic and Informational Noise on AI-Generated Clinical Summaries

Abstract

Abstract Objectives To investigate the effect of the introduction of environmental noise, microphone identity & position, and informational noise on the accuracy of a commercial Clinical AI Scribe (CAIS). Design Consultations on five medical conditions (memory loss, diarrhoea, headaches, skin rash, or prostate symptoms) were recorded. A ground-truth audio file was produced for a noise free consultation. This in turn produced a ground-truth clinical summary via the AI scribe, typically containing 30 facts. Different types of ‘noise’ at different levels were then investigated by (i) increasing the distance of the consultation from the microphone, and changing microphone type (ii) introducing background noise: heavy rain, baby crying, construction noise and a toddler at a range of levels between -10 to +5dB relative to the consultation audio and (iii) introducing informational noise: the medical consultations were modified to include additional conversations between the clinician and the patient to introduce irrelevant discussions (medical and non-medical, shorter and longer). The resulting 160 summaries were then compared with the corresponding ground-truth summaries and the errors introduced were classified and enumerated. Setting Simulated primary care setting. Participants The roles of the doctor and patient were played by actors, rotating between roles. Main outcome measures Outcomes measured were the quantification of the degradation of accuracy of the clinical summary, i.e. omissions, hallucinations and inclusions, relative to the ground-truth summary, following the introduction of noise of all types at different levels. Results Error rates and absolute error counts increased for all noise levels and noise types; the principal error being omissions. At 4.5 m from the consultation, for all microphones tested, all facts were omitted. At up to 2 m all microphones, except the laptop, only introduced a small number (<5) of omissions. For background noise, the type of noise was found to correlate with omissions – toddler and heavy rain were particularly deleterious, 7 and 15 omissions at 0 and +5dB respectively, i.e. when the noise loudness was comparable to or greater than that of the consultation. The CAIS was remarkably apt at rejecting additional informational noise. This was true for the addition of either short or long, medical or non-medical, information. Conclusions The CAIS was very good at capturing a doctor-patient consultation and producing accurate clinical summaries, however error rate (especially omissions) increased notably when acoustic noise was introduced. The system was better able to handle informational noise. Summary Box What is already known on this topic The use of AI is rapidly increasing, with many commercial products available, both specific to clinical use and not. Commercial Clinical AI Scribes (generating clinical summaries from consultation audio) are being used in healthcare settings, sometimes with little guidance or knowledge of employers. These tools can operate well in perfect conditions, but there is limited data on nosier environments. What this study adds The commercial Clinical AI Scribe only introduced a small number of errors when introduced to informational noise, however large numbers of errors (especially omissions) were caused by acoustic noise (dependant on the particular sound type and volume), especially with low-quality poorly-located microphones. Users of Clinical AI Scribes must be particularly vigilant in checking the summary when the environment contains certain deleterious background noises, even at lower volumes, to ensure errors are not introduced to the patient records.

Ähnliche Arbeiten