Original Research

External Evaluation of a Mammography-based Deep Learning Model for Predicting Breast Cancer in an Ethnically Diverse Population

Published Online:Jul 26 2023https://doi.org/10.1148/ryai.220299

See related article

Abstract

External evaluation of a mammography-based deep learning tool for breast cancer prediction in a dataset from a high-risk population demonstrated that detection of precancerous changes is likely a major driver of model performance.

Purpose

To externally evaluate a mammography-based deep learning (DL) model (Mirai) in a high-risk racially diverse population and compare its performance with other mammographic measures.

Materials and Methods

A total of 6435 screening mammograms in 2096 female patients (median age, 56.4 years ± 11.2 [SD]) enrolled in a hospital-based case-control study from 2006 to 2020 were retrospectively evaluated. Pathologically confirmed breast cancer was the primary outcome. Mirai scores were the primary predictors. Breast density and Breast Imaging Reporting and Data System (BI-RADS) assessment categories were comparative predictors. Performance was evaluated using area under the receiver operating characteristic curve (AUC) and concordance index analyses.

Results

Mirai achieved 1- and 5-year AUCs of 0.71 (95% CI: 0.68, 0.74) and 0.65 (95% CI: 0.64, 0.67), respectively. One-year AUCs for nondense versus dense breasts were 0.72 versus 0.58 (P = .10). There was no evidence of a difference in near-term discrimination performance between BI-RADS and Mirai (1-year AUC, 0.73 vs 0.68; P = .34). For longer-term prediction (2–5 years), Mirai outperformed BI-RADS assessment (5-year AUC, 0.63 vs 0.54; P < .001). Using only images of the unaffected breast reduced the discriminatory performance of the DL model (P < .001 at all time points), suggesting that its predictions are likely dependent on the detection of ipsilateral premalignant patterns.

Conclusion

A mammography DL model showed good performance in a high-risk external dataset enriched for African American patients, benign breast disease, and BRCA mutation carriers, and study findings suggest that the model performance is likely driven by the detection of precancerous changes.

Keywords: Breast, Cancer, Computer Applications, Convolutional Neural Network, Deep Learning Algorithms, Informatics, Epidemiology, Machine Learning, Mammography, Oncology, Radiomics

Supplemental material is available for this article.

See also commentary by Kontos and Kalpathy-Cramer in this issue.

References

1. Clift AK, Dodwell D, Lord S, et al. The current status of risk-stratified breast screening. Br J Cancer 2022;126(4):533–550.
MedlineGoogle Scholar
2. Monticciolo DL, Newell MS, Moy L, et al. Breast cancer screening in women at higher-than-average risk: recommendations from the ACR. J Am Coll Radiol 2018;15(3 Pt A):408–414.
MedlineGoogle Scholar
3. Magny SJ, Shikhman R, Keppke AL. Breast imaging reporting and data system. In: StatPearls. Treasure Island, FL: StatPearls Publishing, 2022. http://www.ncbi.nlm.nih.gov/books/NBK459169/. Accessed October 24, 2022.
Google Scholar
4. McCormack VA, dos Santos Silva I. Breast density and parenchymal patterns as markers of breast cancer risk: a meta-analysis. Cancer Epidemiol Biomarkers Prev 2006;15(6):1159–1169.
MedlineGoogle Scholar
5. Vachon CM, van Gils CH, Sellers TA, et al. Mammographic density, breast cancer risk and risk prediction. Breast Cancer Res 2007;9(6):217.
MedlineGoogle Scholar
6. Yala A, Mikhael PG, Strand F, et al. Toward robust mammography-based models for breast cancer risk. Sci Transl Med 2021;13(578):eaba4373.
MedlineGoogle Scholar
7. Yala A, Lehman C, Schuster T, et al. A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 2019;292(1):60–66.
Google Scholar
8. Shen L, Margolies LR, Rothstein JH, et al. Deep learning to improve breast cancer detection on screening mammography. Sci Rep 2019;9(1):12495.
MedlineGoogle Scholar
9. Rodriguez-Ruiz A, Lång K, Gubern-Merida A, et al. Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. J Natl Cancer Inst 2019;111(9):916–922.
MedlineGoogle Scholar
10. Yala A, Schuster T, Miles R, Barzilay R, Lehman C. A deep learning model to triage screening mammograms: a simulation study. Radiology 2019;293(1):38–46.
Google Scholar
11. Ribli D, Horváth A, Unger Z, Pollner P, Csabai I. Detecting and classifying lesions in mammograms with deep learning. Sci Rep 2018;8(1):4165.
MedlineGoogle Scholar
12. Wu N, Phang J, Park J, et al. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans Med Imaging 2020;39(4):1184–1194.
MedlineGoogle Scholar
13. Haji Maghsoudi O, Gastounioti A, Scott C, et al. Deep-LIBRA: an artificial-intelligence method for robust quantification of breast density with independent validation in breast cancer risk assessment. Med Image Anal 2021;73:102138.
MedlineGoogle Scholar
14. Lee J, Nishikawa RM. Automated mammographic breast density estimation using a fully convolutional network. Med Phys 2018;45(3):1178–1190.
MedlineGoogle Scholar
15. Yala A, Mikhael PG, Strand F, et al. Multi-institutional validation of a mammography-based breast cancer risk model. J Clin Oncol 2022;40(16):1732–1740.
MedlineGoogle Scholar
16. Lehman CD, Isaacs C, Schnall MD, et al. Cancer yield of mammography, MR, and US in high-risk women: prospective multi-institution breast cancer screening study. Radiology 2007;244(2):381–388.
Google Scholar
17. Lehman CD, Mercaldo S, Lamb LR, et al. Deep learning vs traditional breast cancer risk models to support risk-based mammography screening. J Natl Cancer Inst 2022;114(10):1355–1363.
MedlineGoogle Scholar
18. Arasu VA, Habel LA, Achacoso NS, et al. Comparison of mammography artificial intelligence algorithms for 5-year breast cancer risk prediction. medRxiv 2022.01.05.22268746 [preprint]. Posted January 7, 2022. Accessed October 1, 2022.
Google Scholar
19. Handelman GS, Kok HK, Chandra RV, et al. Peering into the black box of artificial intelligence: evaluation metrics of machine learning methods. AJR Am J Roentgenol 2019;212(1):38–43.
MedlineGoogle Scholar
20. Noseworthy PA, Attia ZI, Brewer LC, et al. Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on a deep learning model for ECG analysis. Circ Arrhythm Electrophysiol 2020;13(3):e007988.
MedlineGoogle Scholar
21. Willemink MJ, Koszek WA, Hardell C, et al. Preparing medical imaging data for machine learning. Radiology 2020;295(1):4–15.
Google Scholar
22. Zhao F, Copley B, Niu Q, et al. Racial disparities in survival outcomes among breast cancer patients by molecular subtypes. Breast Cancer Res Treat 2021;185(3):841–849.
MedlineGoogle Scholar
23. Zhao F, Henderson TO, Cipriano TM, et al. The impact of coronavirus disease 2019 on the quality of life and treatment disruption of patients with breast cancer in a multiethnic cohort. Cancer 2021;127(21):4072–4080.
MedlineGoogle Scholar
24. Kerlikowske K, Smith-Bindman R, Ljung BM, Grady D. Evaluation of abnormal mammography results and palpable breast abnormalities. Ann Intern Med 2003;139(4):274–284.
MedlineGoogle Scholar
25. Wu N, Geras KJ, Shen Y, et al. Breast density classification with deep convolutional neural networks. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018,6682–6686.
Google Scholar
26. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction — umap 0.5 documentation. https://umap-learn.readthedocs.io/en/latest/. Accessed September 27, 2022.
Google Scholar
27. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44(3):837–845.
MedlineGoogle Scholar
28. Harrell FE Jr, Califf RM, Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982;247(18):2543–2546.
MedlineGoogle Scholar
29. Boyd NF, Guo H, Martin LJ, et al. Mammographic density and the risk and detection of breast cancer. N Engl J Med 2007;356(3):227–236.
MedlineGoogle Scholar
30. Kelly KM, Dean J, Comulada WS, Lee SJ. Breast cancer detection using automated whole breast ultrasound and mammography in radiographically dense breasts. Eur Radiol 2010;20(3):734–742.
MedlineGoogle Scholar
31. Rafferty EA, Durand MA, Conant EF, et al. Breast cancer screening using tomosynthesis and digital mammography in dense and nondense breasts. JAMA 2016;315(16):1784–1786.
MedlineGoogle Scholar
32. Guindalini RSC, Zheng Y, Abe H, et al. Intensive surveillance with biannual dynamic contrast-enhanced magnetic resonance imaging downstages breast cancer in BRCA1 mutation carriers. Clin Cancer Res 2019;25(6):1786–1794.
MedlineGoogle Scholar
33. Chai H, Brown RE. Field effect in cancer—-an update. Ann Clin Lab Sci 2009;39(4):331–337.
MedlineGoogle Scholar
34. Yan PS, Venkataramu C, Ibrahim A, et al. Mapping geographic zones of cancer risk with epigenetic biomarkers in normal breast tissue. Clin Cancer Res 2006;12(22):6626–6636.
MedlineGoogle Scholar
35. Porter PL, El-Bastawissi AY, Mandelson MT, et al. Breast tumor characteristics as predictors of mammographic detection: comparison of interval- and screen-detected cancers. J Natl Cancer Inst 1999;91(23):2020–2028.
MedlineGoogle Scholar

Article History

Received: Jan 2 2023
Revision requested: Feb 15 2023
Revision received: May 25 2023
Accepted: July 3 2023
Published online: July 26 2023

Vol. 5, No. 6

Supplemental Material

Abbreviations

Abbreviations:
AUC	area under the receiver operating characteristic curve
BDC	breast density classifier
BI-RADS	Breast Imaging Reporting and Data System
DL	deep learning
HER2	human epidermal growth factor receptor 2
HR	hormone receptor
WD	weighted density

Metrics

Downloaded 629 times

Altmetric Score

PDF download