"skipMainNavigation"
Search
Quick Search in Journals
Quick Search anywhere
Advanced Search
Radiology
Menu
Previous
Next
Original Research

Accuracy of Large Language Model–based Automatic Calculation of Ovarian-Adnexal Reporting and Data System MRI Scores from Pelvic MRI Reports

  • Deputy Editor: Linda Moy
  • Scientific Editor: Sarah Atzen
Published Online:Apr 1 2025https://doi.org/10.1148/radiol.241554
Figure 1:

Figure 1: Study flowchart. (A) Test set 1 included lesions with at least one assigned Ovarian-Adnexal Reporting and Data System (O-RADS) MRI score in the original radiology report (173 lesions; 158 lesions with O-RADS scores). (B) Because reports with O-RADS scores are more likely to use standard O-RADS terminology and lesions typical of a specific category, additional reports that preceded O-RADS MRI were collected as test set 2. O-RADS MRI categorizes risk into one of five scores: 1 (normal), 2 (almost certainly benign; positive predictive value [PPV] of malignancy < 0.5%), 3 (low risk; PPV, approximately 5%), 4 (intermediate risk; PPV, approximately 50%), and 5 (high risk; PPV, approximately 90%).

Figure 2:

Figure 2: Large language model (LLM)–based strategies. To automatically calculate Ovarian-Adnexal Reporting and Data System (O-RADS) MRI scores from report descriptions, two different LLM-based strategies were evaluated. The first involved using GPT-4 (OpenAI) prompted with O-RADS MRI system rules and two examples (few-shot learning) to assign a score to each lesion (LLM only). The second involved leveraging GPT-4 with few-shot learning to extract and classify key descriptions (ie, septations and solid enhancing tissue), and output a structured JSON object for each lesion (hybrid). These features were then automatically passed into a deterministic formula to apply complex system rules and calculate the final O-RADS MRI score. O-RADS MRI categorizes risk into one of five scores: 1 (normal), 2 (almost certainly benign; positive predictive value [PPV] of malignancy, <0.5%), 3 (low risk; PPV, approximately 5%), 4 (intermediate risk; PPV, approximately 50%), and 5 (high risk; PPV, approximately 90%).

Figure 3:

Figure 3: An 82-year-old woman presented with a right adnexal cystic mass that required further evaluation at MRI. This right adnexal mass was described in the original MRI report as containing enhancing septations but no papillary projections or nodularity. The reference standard category for this lesion was Ovarian-Adnexal Reporting and Data System (O-RADS) MRI 3. The original MRI report assigned an O-RADS MRI score of 4, which was incorrect. Both the large language model (LLM)–only (GPT-4 with in-context learning; OpenAI) and the hybrid application (GPT-4 for feature classification followed by deterministic formula to calculate the O-RADS score) correctly classified the lesion as O-RADS MRI 3. O-RADS MRI categorizes risk into one of five scores: 1 (normal), 2 (almost certainly benign; positive predictive value [PPV] of malignancy, <0.5%), 3 (low risk; PPV, approximately 5%), 4 (intermediate risk; PPV, approximately 50%), and 5 (high risk; PPV, approximately 90%).

Figure 4:

Figure 4: Accuracy of Ovarian-Adnexal Reporting and Data System (O-RADS) MRI scores compared with the reference standard review for lesions with an O-RADS score in the original report (n = 158). The hybrid model (GPT-4 for feature classification [OpenAI] followed by deterministic formula to calculate O-RADS score) accuracy (97%; 153 of 158) was higher than original report scores (88%; 139 of 158; P = .004) and large language model [LLM] only (GPT-4 with in-context-learning [OpenAI]; 89%; 141 of 158; P = .01). O-RADS MRI categorizes risk into one of five scores: 1 (normal), 2 (almost certainly benign; positive predictive value [PPV] of malignancy, <0.5%), 3 (low risk; PPV, approximately 5%), 4 (intermediate risk; PPV, approximately 50%), and 5 (high risk; PPV, approximately 90%).