Evaluation of an AI Model to Assess Future Breast Cancer Risk

Patients

Female patients included in this retrospective observational matched case-control study attended the United Kingdom NHS Breast Screening Programme from February 2010 to September 2019 at OPTIMAM Mammography Image Database sites (Appendix S1) (15). Previous studies using this data are listed at https://medphys.royalsurrey.nhs.uk/omidb/publications/. Data were fully anonymized and received ethical approval for research (REC reference: 19/SC/0284, IRAS reference: 265403). The current study reports new work using this database (16).

Patients aged 46–74 years were eligible for inclusion if they had undergone a standard four-view mammographic screening examination with “for presentation” (ie, processed) images from a Hologic mammography system (Lorad Selenia or Selenia Dimensions) at two screening sites and did not have breast implants. Patients were excluded if they had benign disease.

Prognostic Model

Mirai (version 0.3.1, https://github.com/yala/Mirai) was developed by Yala et al (12) using a U.S. cohort and the computer code is freely available (17). The model estimates risk annually for 5 years. It was run following the developer instructions (17) using mammograms up to 6 years (two screening visits) prior to diagnosis. The percentage of mammographic breast density as determined using Volpara (Volpara Health) software was provided from the OPTIMAM registry for all screening examinations with unprocessed Digital Imaging and Communications in Medicine, or DICOM, images.

Study Design

The primary end point of the study was diagnosis of biopsy-confirmed invasive carcinoma or DCIS as recorded in the NHS Breast Screening Programme database from February 2010 to September 2019. The main prognostic factor was 3-year absolute risk. Most female individuals have triennial screening from 50 to 70 years of age in England, with some starting at 47 years or ending at 73 years of age during the study period due to a trial (ISRCTN registry no. ISRCTN33292440). All included patients had at least 3 years of follow-up after the index mammogram. Patients with breast cancer (cases) were included if they attended a screening visit and (a) the cancer was screen-detected on mammograms at a screening visit 3 years prior or (b) they were diagnosed with interval cancer between two screening visits and a mammogram was available from the screening visit less than 3 years earlier. Patients without breast cancer (controls) were matched 1:1 with cases according to age at the time the mammogram was obtained (within 1 year) and the mammography system used, with at least 3 years of follow-up to a subsequent normal screening assessment. Exploratory outcomes were breast cancer subtypes recorded in the NHS Breast Screening Programme database as follows: advanced cancer stage (18,19) (≥stage II, indicated by positive nodal status or tumor size ≥20 mm), cancer grade, estrogen receptor (ER) status, hormone receptor status (positive if ER positive and/or progesterone receptor positive), human epidermal growth factor receptor 2 (HER2; also called ERBB2) status, and triple-negative breast cancer (positive if ER negative, progesterone receptor negative, and HER2 negative).

The study sample size was a priori judged sufficient because the area under the receiver operating characteristic curve (AUC) of breast density is approximately 0.60 (20,21). We expected that the model would be stronger than mammographic density and strongest for interval cancer (12), and we determined the model had approximately 90% power to test for heterogeneity if the AUC is 0.67 for interval cancer versus 0.60 for screen-detected cancer.

Statistical Analysis

A statistical analysis plan was finalized before analysis by two authors (C.D., A.R.B.), with analysis performed by three authors (C.D., A.R.B., E.L.). The primary analysis objective was to assess the strength of the AI model in individuals attending the NHS Breast Screening Programme. The secondary objective was to assess potential heterogeneity for interval versus screen-detected cancers, age, and cancer type (invasive vs DCIS), with calibration of relative risk. Primary analysis was performed on a complete matched-pair basis. The odds ratio per SD in controls of the natural logarithm of absolute 3-year risk was estimated. To reflect the study design, matching was considered in the analysis through use of conditional logistic regression (odds ratios with 95% profile likelihood CIs) and a concordance index adjusted for matching factors (AUCs with 95% CIs from the Wilson method [21]). This AUC indicates the chance that a case has a greater risk score than the matched control. Heterogeneity of performance by subgroup was assessed using likelihood ratio tests for interaction. To assess whether the relative risks of the AI model were accurate, a calibration slope was estimated, which used the logistic regression coefficient associated with log absolute risk such that a perfect calibration would yield a coefficient of 1.0. Calibration of the predicted relative risk was plotted using a generalized additive model fit to continuous values of the AI model relative risk (22). Positive predictive value (or prevalence) was estimated according to risk decile, assuming a prevalence of 8.5 patients per 1000 at screening (14) and interval cancer rates of approximately three per 1000 (23). Other predefined subgroups were cancer type (invasive vs DCIS) and age (<55 years, 55−64 years, ≥65 years). Exploratory analysis assessed performance according to subtypes of invasive cancer using the same methodology. Performance at diagnosis was evaluated in screen-detected cancers and 6 years prior. The AI model was compared with breast density by using conditional logistic regression χ² statistics with matched pairs that had Volpara-determined breast density available for both the case and control. Interval cancer subgroups were examined based on national screening program review from at least two reviewers as (a) radiologically normal, (b) seen with hindsight but not obvious, or (c) obviously malignant (24). This review is conducted in the NHS Breast Screening Programme as part of routine quality assurance, and the identity of the reviewers is anonymized in the database. P < .05 was considered indicative of a statistically significant difference. Statistical analysis was performed using R (version 4.1.2; The R Foundation) (25).

Patient Characteristics

Patient inclusion into this study from the OPTIMAM Mammography Image Database is shown in Figure 1. After exclusions, there were 2044 patient cases with screen-detected cancer and 696 patient cases with interval cancer. They were matched 1:1 to controls in our primary analysis data using the factors shown in Table 1. The median ages were 59 years (IQR, 53–65 years) for patients with interval cancer and 60 years (IQR, 55–65 years) for those with screen-detected cancer. Proportionally more interval cancers than screen-detected cancers were invasive (91% vs 75%) (Table 1).

Figure 1:

Download as PowerPoint

Table 1: Baseline Patient Characteristics according to Case or Control Status

Analysis and Presentation

Correlation with age and breast density.—The AI model was weakly positively correlated with age (Spearman correlation in controls, 0.18) and breast density (Spearman correlation, 0.15; both P < .001).

Association with breast cancer risk.—Overall, 3-year risk was strongly associated with development of both future interval cancer and screen-detected cancer (Table 2). There was no evidence of a significant difference in model performance between interval and screen-detected cancers (AUC, 0.69 vs 0.67; P = .085) or invasive cancer versus DCIS (AUC, 0.68 vs 0.66; P = .057).

Table 2: Predictive Ability of the AI Model for Future Breast Cancer over the 3 Years after a Negative Screening Result with Predefined Subgroup Analysis

Calibration slope.—To evaluate whether the implied relative risks from the AI model were accurate, we estimated the calibration slope of observed versus expected relative risks. This analysis showed that the 3-year risk slightly underestimated the spread of implied relative risks (overall calibration slope [observed vs expected odds ratio], 1.13; 95% CI: 1.01, 1.26) (Fig 2), indicating that the observed performance had slightly more discrimination than was expected by the AI model.

Figure 2:

Download as PowerPoint

Positive predictive value.—Patients with an AI model risk greater than the top 10% of our sample were estimated to be at more than 10 times the risk of breast cancer over the next 3 years compared with patients with an AI model risk in the bottom 10% (Table S1). At the next screening round, the highest decile had a positive predictive value of approximately 25.2 patients per 1000 screened versus 3.2 per 1000 for the lowest decile (Table S1). The corresponding projected risks for interval cancer were 15.9 patients per 1000 versus 0.9 per 1000, respectively.

Invasive cancer subtypes.—The AI model showed discrimination for all subtypes, although there was some evidence of heterogeneity in strength of effect as follows (Table 3). First, the model was more predictive of stage II or higher cancers than cancers that were less advanced (<stage II) (AUC, 0.72 vs 0.66; P = .037). Second, there was evidence that the model was more predictive for ER-positive cancers (AUC, 0.69 vs 0.65; P = .015), being higher for that of screen-detected cancers (AUC, 0.70 vs 0.61; P = .001) (Tables S2, S3). Third, the model was less predictive for screen-detected grade 3 cancers (AUC, 0.62) than grade 2 cancers (AUC, 0.71; P = .005).

Table 3: Exploratory Subgroup Analysis of the AI Model Outputs Considering Subtypes of Invasive Cancer with Screen-detected and Interval Cancers Combined

Timing of mammograms.—Mirai was evaluated at the time of cancer diagnosis and showed strong discrimination (AUC, 0.89; 95% CI: 0.88, 0.91) (Table S4), with an approximate 70-fold difference in risk between the top and bottom deciles, suggesting that the AI algorithm is detecting visible signs of breast cancer in the mammogram. Change in the projected 3-year risk over the 6-year period to diagnosis in cases with complete data (n = 1260) is shown graphically in Figure 3. In the subset of cases that had matched controls, the AUC increased from 0.69 (95% CI: 0.63, 0.74) at 6 years and 0.71 (95% CI: 0.65, 0.77) at 3 years prior to diagnosis to 0.88 (95% CI: 0.83, 0.92) at the time of diagnosis (Tables S5, S6).

Figure 3:

Download as PowerPoint

Comparison with breast density.—A total of 1477 matched case and control pairs (482 with interval cancer and 995 with screen-detected cancer) had Volpara-determined breast density data available (Table S7). Overall, the model had a higher likelihood of predicting a future breast cancer than breast density alone (AUC, 0.67 vs 0.56; P < .001). Breast density improved performance of the AI model for interval cancer prediction (AUC, [model alone] 0.69, [combined] 0.71; P < .001), but did not significantly increase predictive power for screen-detected cancers (P = .38) (Table 4).

Table 4: Direct Comparison of the Performances of the AI Algorithm and Breast Density in Patients with Both Measures

Interval cancer reader review subgroups.—Upon review, 81% of interval cancers were “not radiologically apparent” using the previous screening mammograms and only 1% were classified as “radiologically obvious” (Table S8) (24).