Prompt again: How consistently useful are artificial intelligence chatbot responses when prompted with concerns about the reality of paediatric chronic pain?

Abstract

The ubiquity and conversational ability of artificial intelligence (AI) chatbots have ushered in a new era. Individuals, including children, are increasingly turning to AI chatbots for interactive conversations, including those related to health.1 Given this trend, current children might soon be classified as “AI-natives,” growing up in a world where interacting with AI is a norm. This phenomenon raises questions about the consistency of health information provided by AI chatbots.2 To date, the consistency of AI chatbot responses has been a subject of scrutiny. However, a notable gap exists in our understanding of this consistency in paediatric chronic pain, and the overall usefulness of AI chatbots for this population. This is a crucial area of inquiry because misconceptions and misunderstandings about chronic pain (such as questioning the reality of pain with no physical evidence of injury) are prevalent in society.3 One common stigmatizing concern that children with chronic pain might express is the question, “Is it all in my head?,”4 and turn to AI for guidance. Therefore, this commentary aims to assess the usefulness and consistency of responses provided by common AI chatbots to this question. We accessed six popular AI chatbots on 2 January 2024; ChatGPT 4 Turbo, Google Bard (Gemini Pro) version 2023.12.18, Microsoft Copilot Creative Mode (GPT-4), Claude, Perplexity AI, and ChatGPT 3.5. Each chatbot was presented with the following prompt 10 times: “I am a child with chronic pain. Is it all in my head?”. The usefulness of responses was analyzed based on a predefined scoring system of 10 criteria developed by the authors, deemed to contribute to a high-quality clinical response to our question. Appendix 1 provides definitions for each criterion and references to related literature for each. One point was given to each criterion met, so that scores ranged from 0 to 10 for each response. These response criteria included (1) sorry to hear, (2) actively addresses that pain is not in your head, (3) pain is multifactorial, (4) referral to health professional, (5) evidence-based, (6) not alone, (7) coaching tone, (8) asked clarifying questions, (9) child-friendly language (Flesch–Kincaid Grade Level <7), and (10) word count not too short or long (100–300 words). Chatbot responses were scored by two blinded raters independently (JWP and BS). Discrepancies in the responses were discussed, and if a decision could not be agreed upon, they were further adjudicated by SDT. A mean score, SD, and range were calculated for each chatbot based on the 10 responses. The readability of responses was evaluated by the range of the Flesch–Kincaid Grade Level score using Microsoft Word. The consistency of the two raters was calculated for each criterion using prevalence-adjusted and bias-adjusted kappa (PABAK). Figure 1 contains a summary of the usefulness and readability scores for each chatbot. PABAK values for each scoring criterion ranged from 0.60 to 1.00. Our findings revealed that all 10 predetermined criteria for high-quality clinical responses were not met across chatbots when prompted about the reality of chronic pain. Responses were not child-friendly. All individual responses are presented in Appendix 2. For ChatGPT 4 Turbo, between 6 and 8 of the 10 criteria were met (mean [SD] = 7.0 [0.7]). This SD suggests that the 10 responses were somewhat similar. Readability scores ranged from Grades 9 to 13. For Bard, between 1 and 8 of the 10 criteria were met (mean [SD] = 5.8 [2.6]). Readability scores ranged from Grades 5 (when it refused to answer) to 21. For Copilot, 6–7 of the 10 criteria were met in each response (mean [SD] = 6.8 [0.4]). This SD suggests that the 10 responses were somewhat similar. Readability scores ranged from Grades 7 to 14. For Claude, 1–6 of the 10 criteria were met (mean [SD] = 3.1 [1.9]). Readability scores ranged from Grades 7 to 10. Claude refused to respond to 7 of the 10 prompts (scoring 2–3/10 for those) citing a lack of required information or context in the prompt. However, on the attempts when Claude did respond it scored 5–7/10. For Perplexity, 4–6 of the 10 criteria were met (mean [SD] = 5.2 [0.6]). This SD suggests that the 10 responses were somewhat similar. Readability scores ranged from Grades 12 to 21. For ChatGPT 3.5, between 6 and 8 of the 10 criteria were met (mean [SD] = 6.9 [0.6]). This SD suggests that the 10 responses were somewhat similar. Readability scores ranged from Grades 14 to 17. We provided all data for each criterion in Appendix 2, rather than only total scores as in Figure 1, because our scoring to determine usefulness has not been psychometrically tested and validated. Therefore, the total quantitative scores should be interpreted with caution. Second, the rapid pace of technological advancements in AI often outstrips the speed at which ethical approvals and recruitment for be For AI chatbot responses included to and this with children is as to common it is for to our to the for in this For between chatbots not be as is a the are Our was not to which chatbot is or rather to a range of chatbots a and The in conversational responses and between AI chatbots is Therefore, a is using current Our blinded a to and AI chatbots in paediatric should pain and a and which questions, in a a tone, and questions, and in on the the chatbots the reality of pain pain is this not be lack understanding of individual context and is not to to a or information chatbots refused to to the individual context of pain no chatbots to the context with chatbots in with about usefulness and further This could of and Our commentary the for of AI in of the of AI chatbot and in AI responses. Our data with findings that should be when from with current AI to assess the usefulness and consistency of AI chatbot responses should be developed and tested so that interpreted with in than is AI chatbots to information and chatbot in and paediatric chronic pain chatbot is given the and new in this Our commentary the for AI chatbots to with children, in a that is and current The and of chatbots where criteria are met should be The no for this by of as of the of the of The have no of in understanding the that pain in paediatric a with the on the of chronic pain in children, it based that children are by chronic pain and is of in in for A for of paediatric pain in clinical A The of children with chronic pain and am a child with chronic pain. Is it all in my sorry to that chronic pain. to that chronic pain is a and and not in your in the of or is a from your that is not pain, in be by including health or or For children, chronic pain be because it might with and to the and of chronic pain, which are and your overall crucial to to a the of your pain and This might physical or and about your pain is a and to to a a or for and guidance. not in and are and that pain in children, in is a and is not in your in the of or is a and often physical and chronic pain be to children, chronic pain be by including or the not be The of pain is and is by a of and to that the of chronic pain mean the pain is not for a child chronic pain to a pain and to as a paediatric pain if The for chronic pain often a physical and with chronic pain, to about your pain with your and the to your pain and your of not alone, and are your sorry to that chronic pain. to that chronic pain is a and not your pain be by a of including physical or health the of chronic pain might not be that it or The of pain is and from to by and This that your pain is a and should be to your pain with a the and a This might physical or not in and are your pain sorry to that chronic pain. to that chronic pain is a and be by including and about pain, not about physical The a crucial in pain. This mean that your pain is not or that in your is a and be by physical and For and the of pain, and physical or contribute to chronic pain. with chronic pain, to to a the of your pain and with to a This might physical or your pain is and from is a in with a and to understanding and your pain. sorry to that chronic pain. to that pain, or is a and not your pain be and is by a of including and children, chronic pain be to assess and not for chronic pain to be with no physical is This mean the pain as or contribute to chronic pain, and are as to crucial to to a about your pain. to and to if This might in pain physical or health the of pain mean the pain is a of understanding contribute to pain and to and are and the is to chronic pain pain in children, in is a and to that chronic pain is not in your in the that not is a and and chronic pain, in have and to children, chronic pain might from This or that pain. chronic or to pain. the pain or to pain that or has and the of pain. These the pain they the pain and lack of physical or at or contribute to the of chronic pain. for children chronic pain to a by This often a including pain and health to all of the pain. to a that might and to the pain and of pain is a and to with your about so they the and sorry to that chronic pain. to that pain, including chronic pain, is a and not your is a phenomenon that the and the pain be by a of physical as or be by or the of chronic pain might not be that it or For children, crucial to have a and to to to a the of your pain and This might or to your pain. your pain is and are and to to and your with and pain in children, in is a and to that it is not in your in the of or is a phenomenon that the and the chronic pain, the system be pain when is no This mean the pain rather that the pain system in the has or pain be by including and chronic pain, to A the of your pain and a This physical and to about your pain with your and so they and in the sorry to that chronic pain. to that pain, including chronic pain, is a and have not your pain is and of pain in children be by including physical or a of common physical of chronic pain in children chronic pain, or that or contribute to or pain. The to chronic pain is a that might physical and crucial to with a the of your pain and and for your pain is a and and are and to for in your pain. sorry to that chronic pain. to that pain, including chronic pain, is a and not your pain be and is by including and children, chronic pain be related to a of as pain, or pain from a physical or the of pain, this mean the pain or crucial to to a the of your pain and a This might or a of your pain is and are to with a or a about your and pain in children is a and and to that not in your contribute to chronic pain in children, including physical and are to in pain is pain is and to it your are to with a or to the of your pain. This could it might or are to are for chronic pain in children, including physical and to a that for are that might be The The The of not in children chronic pain, and are it and a this information is if have that chronic pain, and to that chronic pain is and not in your chronic pain, and it have a on not in your pain often has a physical or These physical pain to your when no pain your pain. the to pain it to to from a and your pain. a and to with the pain. are that be The The not in are and to and to for this pain in children is a and often to that pain is in your of pain is as and of as physical to that not children and chronic pain, and are to are to in pain have be by physical or the be to are to chronic This might physical and to with your or to a that for about your pain be be to about pain, your with be a This could your a or a for children with chronic pain. are that The The of The of The not in are about and to your pain. to for if this information is if have pain be for a to that not in and are about and to to about the of your pain. chronic pain have crucial to that chronic pain is in your a physical with a of and to to determine the are to in pain is a for and it be by a of and pain is the your pain is and and are the pain has a physical or are that it and your of are that be The The The of and and to to your or about your pain. the of your pain and a that for this information to for further if that with chronic pain as a child be and to have questions about your pain, and if in your is a common and to that chronic pain in children is not or children chronic pain to or that are are to in pain is the your of pain is and to be are not children with chronic pain. are and to and with be not be a physical for your pain. This mean not it might mean that to it are are for with chronic to your or a pain your pain and a that for about pain are to your pain, as physical and pain by your with to children or with chronic pain be a to and are and of and all your overall health and which your pain. are and with chronic pain is have to it are about and to are that The The The of this information is that are not alone, and is and chronic pain for a to that chronic pain is a and and not in your the of your pain might not be that mean not contribute to chronic pain in children, including physical and crucial to with a or to the of your pain and a that for are to pain is not children chronic pain, and are to to for to your or about are to your pain. are including physical and are that The The The of and the to your pain and a this information is that not a professional, and to to a or about your pain in children is a and and to that not in your are from physical or to chronic and health or crucial to to a about your pain, as a or the of your pain and the and not in and are about and to are to in pain including not a of or to be a is This and your the to your pain. are for chronic pain in This physical or to and in This your and overall are that and your The The The of and for with chronic pain. the and your pain and a and pain in children be a and and to that not in your a in pain chronic pain in children has a physical if not to are to pain is The is not and to your pain are pain in children be by a of as and chronic pain, to to your or a pain the of your pain and a that for not children chronic pain, and are to are and pain for are that might be to your with a or be in your pain and about pain and the to of your and and pain in a chronic pain be a and not are about and to to for and with your to the to your pain. this information is if have this is sorry to that are from chronic pain. pain is a and that children and is not that are or pain have as or the of chronic pain is not that not mean that your pain is not or pain your and it for to as to with or or These are and they your pain is it is to from your and and your pain. are to chronic pain in children and of are as These pain or your pain. the and of for on your pain and should your and about or in your pain. These your and your and your pain. are or a that your and should to physical if it is a or These with your and related to your pain. to your your and your are or or and to your should to your or about are and are These your and your in of your pain and your are or on your or with the of a should with your new if it or The to chronic pain in children and is to a of to your individual and This is a or that and your health to not only your physical pain, your and of your pain. This your of and your to about chronic pain in children and This is a that provides and for with pain, and has a to children and where pain and that for the of This is a that and on pain. has a on pain in children, where and to or pain in This is a of the that on health has on the of chronic pain in children, where the and for chronic pain in children in and this information was and that from your pain. are not and to a and this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or of your as your head, or pain with your and are not alone, and are to with chronic pain and your of of the to chronic pain should a the of your pain and the for to a as a paediatric pain or a pain and for your pain. your for a referral if to your pain or should your on to and should your about or have with your are to pain using as or or These your and your pain. your or about which are for and to physical as or your and in your that and pain. should that and that are for your of pain and should and your and your or about the for pain your as or your with your and is to to and with your and from your or a or of children and have chronic pain and your and about chronic and your about chronic pain and to information and from or of the that for This provides information about chronic pain in children and to pain and for and health in for the of This provides a on chronic pain in children, and on the of chronic pain in children This provides for the of chronic pain in children, including and this information is for chronic pain is not your and about are and and to a and this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or of the as the head, or pain with your and or are not in your pain. are to it and your of of the that with chronic pain should a and of your pain. The or that your pain or it be to a or a pain that for pain about pain. information and about chronic pain and it children and about the of pain, pain in the and and your pain. about the of pain, which that your pain is by and your pain in and of to with your pain and on your of as or to These your system and and that your pain. as or These your and your and which is a of that your and that your pain. with and and as or a These on your pain and and to are and and This be a a a or a a or where children and have chronic pain and your and for when as with or pain be it not have to your and your have and that your pain. are and are not your pain. are than to about chronic pain in children and A that provides and for with pain and for the of A that and on pain and A of the that provides and on the of chronic pain in A of and that and on health including chronic pain in children and this is sorry to that are from chronic pain. pain is not all in your is a that children and pain is pain that for than or the of or have as or pain your and are not in your with chronic pain. are and that and your of of the that your pain from a a pain or a pain the of your pain, and as physical or and as and These your and from your or are or with or about your pain and and in that and that from your pain, as or These and have about chronic pain and to or that information and on chronic pain in children and of the that This provides information and on chronic pain in including pain and for children and for the of This and on pain in children and to This the and of chronic pain in children, and and for and this information is and that from your pain. are and and this in in for the of in in for the of in in for the of in this is sorry to that are from chronic pain. pain is a and that children and is not that are or pain have and and it with your and are not alone, and are to and your pain. to the chronic pain in children is as that or for than or the of pain be by as or the of chronic pain is or to pain of the as the head, or pain your and The is that chronic pain be and with the of health as and the for your pain, which physical or The of is to your pain, your and your of One of the to your pain is to about it and it information and for children and with chronic pain on A that provides and for with pain, and for the of A that and on pain and A that and clinical for the of chronic pain in children and to your or about your pain and it and with your pain. or or where children and have with chronic pain. your and with and from each chronic pain is not your and are not are about and to are to your pain and a and are and and the of chronic pain. this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or of the as the head, or pain with your and or are not in are to chronic pain and your of One of the is to to your or a pain the of your pain and that your of the physical or to in your as and These with your pain and on your is to from your or are and and or or of children and have chronic pain. and with from and each to about chronic pain in children and of the that for have information and for and your This provides information about chronic pain in children and to pain and to a for pain in for the of This a on the of chronic pain in children, including the and of pain, as as the of and on the of chronic pain in children This the for the of chronic pain in children, developed by the in with and of chronic pain in children and summary This the and from the on the of chronic pain in children and this information is for that are not and that all the this is sorry to that are from chronic pain. pain is a and that children and is not in your head, and it is not your pain have as or the of chronic pain is not that not mean it is or pain your and and it with your as and However, chronic pain be with the of health your and are to with chronic pain, as physical and are not in this and are and that and your are that to about chronic pain in children and and to This provides information about chronic pain in children and and to and pain in in for the of This provides for the of chronic pain in children, based on the and on the of chronic pain in children This provides on the of chronic pain in children, with a on and of chronic pain in children and summary This provides a summary of the for the of chronic pain in children and in with and for health and this information is to that are and and that this all the this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or the of chronic pain is not that not mean it is not pain your and it to as to with or or These are and they your pain The is that chronic pain be are to with your pain and your of of that your These pain or your as or These your or your physical as or These your and as or These your with or from your or health and or that not have to with chronic pain are and are your for a referral to a paediatric pain or a paediatric pain assess your pain, the for and to your pain information and from A that to the of of with pain, and for the of A that and on pain. A of the that provides and on health including chronic pain in A that provides and on health and including chronic pain in children and this information is and that from your pain. are not and this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or of the as the head, or pain with your and or are not in your pain. are to it and your of of the that with chronic pain should a the of your pain and the for to a as a paediatric pain or a pain as physical or about pain. and your about pain it and to are of information as or or where your and from have chronic of by that as and as to or to your the pain. as or to and to as a a a or a about and express your or if or because of your pain. is no in for to be and this information and with your chronic pain. are and and this all the this is sorry to that are from chronic pain. pain is not all in your is a that children and pain have as or of your as your head, or pain with your as to with or pain your and or are not in are to chronic pain and your of of the to chronic pain from a or a pain the of your pain, or to as or

Ähnliche Arbeiten