Diagnostic Performance of ChatGPT from Patient History and Imaging Findings on the Diagnosis Please Quizzes

Introduction

Artificial intelligence advancements, particularly in large language models like ChatGPT which is based on the GPT-4 architecture (1), have started showing remarkable capabilities in diverse areas (2,3). However, the potential of ChatGPT in radiology remains underexplored. This study will investigate the ability of ChatGPT to solve diagnostic quizzes from the journal Radiology to assess its potential as a diagnostic aid and decision support system.

Materials and Methods

This study evaluated the diagnostic ability of GPT-4–based ChatGPT (4) using patient history and image findings from the educational Diagnosis Please quizzes in Radiology. First, the clinical history and images are given in the case presentation. Then, the image findings and diagnosis answers are presented in the case commentary. As ChatGPT cannot process images, it was given image findings. Since this study relied on published articles, no ethical approval was needed. This study was designed in accordance with Standards for Reporting of Diagnostic Accuracy Studies guidelines (5). Study overview is presented in the Figure.

Download as PowerPoint

We consecutively collected 313 Diagnosis Please cases from April 1998 to April 2023 and extracted patient history and imaging findings. ChatGPT was asked to list the differential diagnoses, first based only on the patient’s history, then based only on the imaging findings, then based on both. ChatGPT was then asked to generate a final diagnosis. Two radiologists (D.H., Y. Mitsuyama; 7 and 4 years of experience, respectively) compared and confirmed that the generated diagnoses coincided with the published diagnoses. The legitimacy of differential diagnoses generated by ChatGPT was also evaluated by the same radiologists on a five-point Likert scale ranging from 1 (highly illegitimate) to 5 (highly legitimate). In addition, the specialty of each case was determined. If there was a discrepancy, a third senior radiologist (H. Takita, 8 years of experience) made the decision.

Given that Radiology articles may have been incorporated into the GPT-4 training, accuracies for cases before and after the September 2021 cutoff (1) were compared using the Fisher exact test with R software (version 4.0.0; https://www.r-project.org/).

Results

The diagnostic performance of ChatGPT in differential diagnoses (a) from submitter-provided patient history was 22% (68 of 313), (b) from submitter-identified image findings was 57% (177 of 313), (c) from both history and image findings was 61% (191 of 313), and (d) for the final diagnosis was 54% (170 of 313). All P values between the periods showed no significant differences (ranging from P = .124 to P >.99). In the final diagnosis, the highest accuracy was achieved in cardiovascular radiology, at 79% (23 of 29), whereas the lowest accuracy was in musculoskeletal radiology, at 42% (28 of 66). These are shown in the Table. The evaluation of ChatGPT-generated diagnoses yielded a median legitimacy score of 5.0. Scores were at least 3.0 for the lower quartile and reached 5.0 for the upper quartile.

Overall Results by Time Period with P Values

Discussion

The diagnostic performance of ChatGPT for Diagnosis Please cases was 54% using both the history and the imaging findings, with a median legitimacy score of 5.0, indicating modestly reliable results. Since the performance varies by specialty, users should be aware when they use it.

Identification of differential diagnoses and settling on a final diagnosis can be challenging for radiologists. ChatGPT may lighten their workload by delivering immediate and dependable diagnostic outcomes (6). This tool could prove particularly beneficial in circumstances where there is a shortage of radiologists (7).

However, this study had limitations. These include that ChatGPT cannot evaluate images directly, Diagnosis Please represents a controlled setting that may not reflect real-world complexities, and the use of post hoc author-written findings might have contributed to the relatively high performance. In addition, cases published through September 2021 might have been used for ChatGPT training.

In conclusion, this study indicates the potential of ChatGPT as a decision support system in radiology.