External Evaluation of a Mammography-based Deep Learning Model for Predicting Breast Cancer in an Ethnically Diverse Population
Abstract
External evaluation of a mammography-based deep learning tool for breast cancer prediction in a dataset from a high-risk population demonstrated that detection of precancerous changes is likely a major driver of model performance.
Purpose
To externally evaluate a mammography-based deep learning (DL) model (Mirai) in a high-risk racially diverse population and compare its performance with other mammographic measures.
Materials and Methods
A total of 6435 screening mammograms in 2096 female patients (median age, 56.4 years ± 11.2 [SD]) enrolled in a hospital-based case-control study from 2006 to 2020 were retrospectively evaluated. Pathologically confirmed breast cancer was the primary outcome. Mirai scores were the primary predictors. Breast density and Breast Imaging Reporting and Data System (BI-RADS) assessment categories were comparative predictors. Performance was evaluated using area under the receiver operating characteristic curve (AUC) and concordance index analyses.
Results
Mirai achieved 1- and 5-year AUCs of 0.71 (95% CI: 0.68, 0.74) and 0.65 (95% CI: 0.64, 0.67), respectively. One-year AUCs for nondense versus dense breasts were 0.72 versus 0.58 (P = .10). There was no evidence of a difference in near-term discrimination performance between BI-RADS and Mirai (1-year AUC, 0.73 vs 0.68; P = .34). For longer-term prediction (2–5 years), Mirai outperformed BI-RADS assessment (5-year AUC, 0.63 vs 0.54; P < .001). Using only images of the unaffected breast reduced the discriminatory performance of the DL model (P < .001 at all time points), suggesting that its predictions are likely dependent on the detection of ipsilateral premalignant patterns.
Conclusion
A mammography DL model showed good performance in a high-risk external dataset enriched for African American patients, benign breast disease, and BRCA mutation carriers, and study findings suggest that the model performance is likely driven by the detection of precancerous changes.
Keywords: Breast, Cancer, Computer Applications, Convolutional Neural Network, Deep Learning Algorithms, Informatics, Epidemiology, Machine Learning, Mammography, Oncology, Radiomics
Supplemental material is available for this article.
© RSNA, 2023
See also commentary by Kontos and Kalpathy-Cramer in this issue.
References
- 1. . The current status of risk-stratified breast screening. Br J Cancer 2022;126(4):533–550.
- 2. . Breast cancer screening in women at higher-than-average risk: recommendations from the ACR. J Am Coll Radiol 2018;15(3 Pt A):408–414.
- 3. . Breast imaging reporting and data system. In: StatPearls. Treasure Island, FL: StatPearls Publishing, 2022. http://www.ncbi.nlm.nih.gov/books/NBK459169/. Accessed October 24, 2022.
- 4. . Breast density and parenchymal patterns as markers of breast cancer risk: a meta-analysis. Cancer Epidemiol Biomarkers Prev 2006;15(6):1159–1169.
- 5. . Mammographic density, breast cancer risk and risk prediction. Breast Cancer Res 2007;9(6):217.
- 6. . Toward robust mammography-based models for breast cancer risk. Sci Transl Med 2021;13(578):eaba4373.
- 7. . A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 2019;292(1):60–66.
- 8. . Deep learning to improve breast cancer detection on screening mammography. Sci Rep 2019;9(1):12495.
- 9. . Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. J Natl Cancer Inst 2019;111(9):916–922.
- 10. . A deep learning model to triage screening mammograms: a simulation study. Radiology 2019;293(1):38–46.
- 11. . Detecting and classifying lesions in mammograms with deep learning. Sci Rep 2018;8(1):4165.
- 12. . Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans Med Imaging 2020;39(4):1184–1194.
- 13. . Deep-LIBRA: an artificial-intelligence method for robust quantification of breast density with independent validation in breast cancer risk assessment. Med Image Anal 2021;73:102138.
- 14. . Automated mammographic breast density estimation using a fully convolutional network. Med Phys 2018;45(3):1178–1190.
- 15. . Multi-institutional validation of a mammography-based breast cancer risk model. J Clin Oncol 2022;40(16):1732–1740.
- 16. . Cancer yield of mammography, MR, and US in high-risk women: prospective multi-institution breast cancer screening study. Radiology 2007;244(2):381–388.
- 17. . Deep learning vs traditional breast cancer risk models to support risk-based mammography screening. J Natl Cancer Inst 2022;114(10):1355–1363.
- 18. . Comparison of mammography artificial intelligence algorithms for 5-year breast cancer risk prediction. medRxiv 2022.01.05.22268746 [preprint]. Posted January 7, 2022. Accessed October 1, 2022.
- 19. . Peering into the black box of artificial intelligence: evaluation metrics of machine learning methods. AJR Am J Roentgenol 2019;212(1):38–43.
- 20. . Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on a deep learning model for ECG analysis. Circ Arrhythm Electrophysiol 2020;13(3):e007988.
- 21. . Preparing medical imaging data for machine learning. Radiology 2020;295(1):4–15.
- 22. . Racial disparities in survival outcomes among breast cancer patients by molecular subtypes. Breast Cancer Res Treat 2021;185(3):841–849.
- 23. . The impact of coronavirus disease 2019 on the quality of life and treatment disruption of patients with breast cancer in a multiethnic cohort. Cancer 2021;127(21):4072–4080.
- 24. . Evaluation of abnormal mammography results and palpable breast abnormalities. Ann Intern Med 2003;139(4):274–284.
- 25. . Breast density classification with deep convolutional neural networks. In:
2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018,6682–6686. - 26. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction — umap 0.5 documentation. https://umap-learn.readthedocs.io/en/latest/. Accessed September 27, 2022.
- 27. . Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44(3):837–845.
- 28. . Evaluating the yield of medical tests. JAMA 1982;247(18):2543–2546.
- 29. . Mammographic density and the risk and detection of breast cancer. N Engl J Med 2007;356(3):227–236.
- 30. . Breast cancer detection using automated whole breast ultrasound and mammography in radiographically dense breasts. Eur Radiol 2010;20(3):734–742.
- 31. . Breast cancer screening using tomosynthesis and digital mammography in dense and nondense breasts. JAMA 2016;315(16):1784–1786.
- 32. . Intensive surveillance with biannual dynamic contrast-enhanced magnetic resonance imaging downstages breast cancer in BRCA1 mutation carriers. Clin Cancer Res 2019;25(6):1786–1794.
- 33. . Field effect in cancer—-an update. Ann Clin Lab Sci 2009;39(4):331–337.
- 34. . Mapping geographic zones of cancer risk with epigenetic biomarkers in normal breast tissue. Clin Cancer Res 2006;12(22):6626–6636.
- 35. . Breast tumor characteristics as predictors of mammographic detection: comparison of interval- and screen-detected cancers. J Natl Cancer Inst 1999;91(23):2020–2028.
Article History
Received: Jan 2 2023Revision requested: Feb 15 2023
Revision received: May 25 2023
Accepted: July 3 2023
Published online: July 26 2023