ChatGPT surges ahead: GPT-4 has arrived in the arena of medical research

Abstract

At the end of November 2022, the OpenAI company (San Francisco, CA, USA) launched Chat Generative Pre-trained Transformer (ChatGPT), which quickly became famous around the world and sparked a wide range of practical and research applications.1,2 In the medical field, ChatGPT has been extensively tested for triage, translation, clinical workflow, consultation, medical research, medical education, and so on.3 ChatGPT itself is a product of natural language processing and artificial intelligence (AI) with a large language model. As the chat in its name suggests, ChatGPT was originally designed to be able to converse with people in natural language. Despite the presence of artificial hallucinations,4 researchers have found that ChatGPT can accurately answer the questions in examinations with a surprisingly high correct rate. In the medical field, the earliest study was published on January 11, 2023, on the parasitology examination of the first-year medical students at Hallym University in South Korea.5 ChatGPT was able to answer 60.8% of the test questions correctly, which was lower than the overall student performance. The second study was published on February 8, 2023. The examination questions were taken from two question banks of the United States Medical Licensing Examination (USMLE) step 1 and step 2 examinations.6 ChatGPT only passed the step 1 examination of one question bank. The study that received the most media attention worldwide was published on February 9, 2023, using the actual USMLE 2022 June step 1, step 2 clinical knowledge, and step 3 examinations.7 ChatGPT performed at or near the passing threshold of examinations without any prior training. Dozens of articles followed in a short time, focusing on the national medical licensing examinations of various countries and the board examinations of various specialties.8,9 Recently, the Journal of the Chinese Medical Association (JCMA) has also published two original articles on the performance of ChatGPT in examinations for medical professionals. These works may be the first of their kind in Taiwan. In the article published online ahead of print on May 25, 2023, ChatGPT was used to test the Taiwanese Pharmacist Licensing Examination in 2023.10 To overcome the language discrepancies of ChatGPT’s performance, the original Chinese version and its translated English version were tested for comparison. As expected, ChatGPT performed better in the English version than in the Chinese version. More importantly, ChatGPT passed the stage 2 examination and approached the threshold of the stage 1 examination. According to the official statistics for this examination, only 76.72% of the candidates passed the stage 2 examination and 13.82% passed the stage 1 examination.11 In another JCMA’s article published online ahead of print on June 9, 2023, ChatGPT was used to test Taiwan’s annual family medicine board examination in 2022.12 Although ChatGPT failed this examination with a correct rate of 41.6% (threshold: 60%), it was able to correctly answer in 58.3% of the questions with mutually exclusive options, that is, questions in which the selection of one option automatically excludes the possibilities of the remaining options and the candidates do not have to judge each option individually. For questions with multiple correct answers, ChatGPT had the worst performance (accuracy 33.3%). Understandably, when ChatGPT has to judge each option individually, the error rate increases cumulatively. According to the report of the Taiwan Association of Family Medicine, 94.07% of the candidates passed the examination.13 Obviously, the specialty board examination seemed to be more difficult for ChatGPT. Similar results have been also reported for other countries and specialties. The possible reason for this was that there was less information about the specialty board examinations available on the Internet, so ChatGPT’s training set had less relevant information. At the end of May 2023, PubMed of the United States National Library of Medicine had recorded 563 bibliographic entries related to ChatGPT, published by 261 journals. Five journals had 10 or more articles: Cureus (75), Annals of Biomedical Engineering (31), Nature (19), Aesthetic Plastic Surgery (18), and Radiology (10). In the first 6 months after the birth of ChatGPT, most publications in medical journals were indeed editorials, reviews, news, comments and letters to the editor. The overall speed of publication was remarkable, from three articles in December 2022 to 212 in May 2023 (Fig. 1). Before the medical researchers became familiar with ChatGPT, Generative Pre-trained Transformer 4 (GPT-4) was launched by OpenAI on March 14, 2023. The newer version was said to be more creative, powerful and reliable. A new wave of research immediately followed: three articles in March, 10 in April, and 33 in May 2023. Original and experimental studies have also caught up.14,15Fig. 1: Monthly PubMed records related to ChatGPT. The records related to ChatGPT also include those related to GPT-4. Status: May 31, 2023. ChatGPT = Chat Generative Pre-trained Transformer; GPT-4 = Generative Pre-trained Transformer 4.Although ChatGPT’s capabilities currently have many limitations, such as fabricating facts, weak reasoning, and no long-term memory, it has the potential to drastically change the healthcare ecology in the near future. Medical publishing is clearly lagging behind the current development of AI. The JCMA welcomes more articles on new technologies and strives to publish them efficiently with rigorous peer review.

Ähnliche Arbeiten