Enhancing the accuracy and consistency of ChatGPT in hematology–oncology manuscript preparation: A critical perspective

Abstract

We read with great interest the article, “An objective cross-sectional assessment of ChatGPT in hematology–oncology manuscript composition: Balancing promise with factual inaccuracies” published in Cancer Research, Statistics and Treatment.[1] The study sheds light on the potential of large language models (LLMs), specifically Chat Generative Pre-Training Transformer (ChatGPT 3.5), in aiding manuscript preparation and literature search. This is particularly valuable in accelerating the writing process and allowing researchers to focus on more critical scientific tasks. We commend the authors for their methodological rigor and the objective framework used in the study. Using predefined scoring criteria and independent reviews by experts in hematology/oncology ensures a robust and credible assessment. The high degree of inter-observer agreement (Cronbach’s alpha and intraclass correlation coefficient both at 0.995) underscores the reliability and consistency of the evaluation process.[1] However, we would like to highlight several concerns that need to be addressed to enhance the robustness and practical applicability of the findings. While the study is insightful, we believe it could be further strengthened by a more extensive evaluation encompassing a broader set of queries and multiple versions of the model such as ChatGPT 4.0. In addition, the article notes the presence of factual inaccuracies in ChatGPT’s responses with an overall score of 34.2 out of 90. However, it does not provide a detailed breakdown of the types of inaccuracies encountered, which is crucial for a comprehensive understanding of the limitations of ChatGPT. A more comprehensive analysis, such as categorizing inaccuracies by type such as incorrect references, wrong data points, and misinterpretations of scientific concepts would provide valuable insights. Cloesmeijer et al.[2] observed that while using ChatGPT in the field of pharmacometrics, a simple query yielded different outcomes, indicating the variability of the chatbot with respect to the input query. Similarly, in this article, the authors could have considered the variability of responses generated by ChatGPT by using the same prompts repeatedly.[1] This variability can be a significant limitation, affecting the reproducibility and reliability of artificial intelligence (AI)–generated content. An analysis of this aspect would provide valuable insights into the consistency of ChatGPT’s performance and its implications for research and is therefore an important consideration for future studies. In conclusion, although the study by Singh et al. offers vital insights into the potential and limitations of ChatGPT, further exploration is essential in various areas.[1] These include extensive evaluation using a broader set of queries, the types and frequency of factual inaccuracies, and the variability of the chatbot’s responses concerning the input query. Although AI has the potential to revolutionize many aspects of scientific work, we must remain vigilant about its limitations and ethical implications. Before widespread acceptance of an article, an extensive evaluation of authorship, accountability, bias, and potential misinformation must be done.[3,4] Researchers should be encouraged to use AI as an innovative tool rather than a primary source of information, ensuring that human expertise remains at the forefront of scientific inquiry. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.

Ähnliche Arbeiten