Range of Radiologist Performance in a Population-based Screening Cohort of 1 Million Digital Mammography Examinations

Introduction

Breast cancer is the most common cancer type among women in the Western world, and its incidence is increasing (1). The mortality reduction resulting from mammography screening was estimated at 20% in a meta-analysis and up to 40% in a single study (2,3). For women who undergo biennial screening, the interval cancer rate is around 28% (4). These interval cancers are more aggressive and have a higher mortality rate (5).

Even if breast cancer screening programs deliver important health benefits, the overall costs are substantial owing to the large number of participants. The estimated cost of mammography screening in the United States in 2010 was $7.8 billion (6). To enhance screening performance, computer-aided detection (CAD) has been developed to facilitate detection of tumors in screening mammograms. However, traditional CAD, based on human-specified formulas for identification of tumors, results in many false-positive findings per image (7). There is increasing optimism for use of image analysis methods based on artificial intelligence (AI)—specifically deep neural networks—to reach human-level performance in the detection of suspicious findings on mammograms (8,9).

AI CAD systems might increase diagnostic performance and lower costs through use as stand-alone readers. However, before AI CAD systems for screening mammography can be clinically implemented, they must be validated and their performance compared with that of radiologists. Since the introduction of full-field digital mammography, there have been a few studies of radiologist performance in large screening cohorts, but none of them have divided performance into quartiles, and none have been performed in a European setting (10–13).

Our aim with this study was to establish performance benchmarks of human-level performance for comparison with stand-alone AI CAD systems as a replacement for the first reader in a geographically defined population-based screening cohort of around 1 million screening mammograms.

Materials and Methods

The ethics review board approved the research for this retrospective study and waived the need for written informed consent.

Study Participants

Our retrospective multicenter cohort study was performed with anonymized full-field digital mammograms obtained during screening mammography. The study population was the Swedish Cohort of Screen-Aged Women, which consisted of all women aged 40–74 in Stockholm County who were invited for screening mammograms from 2008 to 2015. Screening was performed every 18–24 months, with a 70% participation rate. All participants in the current study had been included in a prior publication that described the collection method of the Swedish Cohort of Screen-Aged Women and gave an overview of the population characteristics (14). The prior article dealt with an overall description of the data set, whereas here we report relevant benchmarks after having examined the performance of high-volume readers. Data on cancer diagnoses including tumor characteristics and radiologic assessments were obtained through linkage with the Regional Cancer Centre Stockholm Gotland breast cancer quality and screening registry; the Swedish personal identification numbers were used.

We included all consecutive screening mammographic examinations performed between January 1, 2008, and September 31, 2015. Women were considered to have had breast cancer at the time of screening if the breast cancer quality registry indicated the diagnosis within 12 months after the mammogram had been obtained.

High- versus Low-Volume Readers

There were 110 known interpreting radiologists. We distinguished between low- and high-volume readers based on the number of annual screening mammograms: those who read less than 5000 screening mammograms for at least 1 year within the time range of our data set were considered low-volume readers, and those who read 5000 or more were considered high-volume readers.

Mammography Screening System

The general mammography screening system in Sweden includes two-view mammography of each breast assessed with double reading. If a suspicious finding is observed by one or both radiologists, the mammogram gets flagged for consensus discussion. If the consensus discussion concludes that there is radiologic suspicion for an abnormal lesion, the woman is recalled. Another reason for recall is if the woman reports breast symptoms at the time of screening. The screening assessments were performed at five different institutions—two private and three public. The total number of screening mammograms ranged from 151 360 to 344 306. Mammography devices from Hologic (Marlborough, Mass), Philips (Amsterdam, the Netherlands), Sectra (Linkoping, Sweden), Fujifilm (Minato, Japan), GE Healthcare (Boston, Mass), and Siemens (Munich, Germany) were used.

Study Definitions

We determined performance levels for each radiologist for the following measures: sensitivity, specificity, abnormal interpretation rate, cancer detection rate, false-negative rate, accuracy, and positive predictive value. Measures based on first-reader assessments form the basis for the main tables, whereas measures based on second-reader and consensus assessments can be found in Table E1 (online). For the false-negative screenings, the cancer was detected either by the second reader or clinically (ie, interval cancer). In Sweden, clinical detection is mainly based on self-detection because clinical breast examination of healthy women is rarely performed. The mean numbers reported in all tables for first and second readers were calculated as the average of radiologist-level measures, whereas the mean number reported for the consensus discussion was calculated as the overall average (because the consensus discussion involved various combinations of radiologists). To make a more robust estimate of the individual measures, we grouped into a single-radiologist category all examinations by radiologists who identified fewer than 10 cancers among the mammograms they had assessed. We reported performance benchmarks overall and for each quartile of high-volume readers, wherein quartile one (Q1) included the worst-performing ones and quartile four (Q4) included the best-performing ones for each metric. Minimum and maximum values per quartile were determined. Based on a screening population of 10 000 women, we calculated screening outcomes separately for each quartile of high-volume readers grouped according to sensitivity levels to enable future comparisons with AI CAD systems operating at different sensitivity levels. Finally, to facilitate nuanced future evaluations of AI CAD systems, we determined specific sensitivity levels for various tumor characteristics: molecular subtype and histologic type including tumor size and invasiveness. The tumor size was based on pathologic measurements and was classified as minimal (≤10 mm), small (11–19 mm), or large (≥20 mm). Invasiveness was classified based on pathologic analysis as either in situ only or invasive (with possible in situ components). The histologic type was divided into ductal, lobular, or other. Ductal type included ductal-only tumors and mixed tumors with a ductal component. The molecular subtypes were defined according to the St Gallen classification (15).

Statistical Analysis

Computer software (Stata Statistical Software, release 15.1; Stata, College Station, Tex) was used for all statistical analyses. All statistical tests were two sided. The level for statistical significance was set at P < .05. To test for an association across quartiles of radiologist performance, we fitted linear regression models with the quartile as the predictor. The 95% confidence interval (CI) for sensitivity and specificity was estimated through bootstrapping.

Results

Participant and Mammogram Characteristics

From 2008 to 2015, 504 566 women were invited to undergo screening; 83 225 did not participate and thus were not part of our study population. We excluded 23 033 screening mammograms in 3300 women owing to unknown radiologist identification. The final study population included 1 186 045 screening mammograms in 418 041 women; 972 899 mammograms were read by high-volume readers, and the remaining 213 146 were read by low-volume readers. The study population flowchart is presented in Figure 1. Participant and cancer characteristics are shown in Table 1. Among women who underwent screening, 4723 were diagnosed with breast cancer at screening or within 12 months thereafter. There were 3514 true-positive screenings, 1209 false-negative screenings, 1 138 619 true-negative screenings, and 41 969 false-positive screenings. Example images are shown in Figure 2. The mean age at screening was 54 years ± 9.5 (standard deviation), and the mean age at diagnosis was 59 years ± 10.1.

Figure 1:

Download as PowerPoint

Figure 2a:

Download as PowerPoint

Figure 2b:

Download as PowerPoint

Figure 2c:

Download as PowerPoint

Figure 2d:

Download as PowerPoint

Reader Performance Measures

Table 2 reports first-reader performance measures. There were 24 high-volume readers and 86 low-volume readers (Table E3 [online]). For high-volume readers, the maximum, mean, and minimum annualized volumes were 19 850 mammograms, 8243 mammograms, and 3168 mammograms, respectively. Sensitivities were 73% (95% CI: 69%, 77%), 75% (95% CI: 67%, 81%), and 73% (95% CI: 68%, 77%) for all, low-volume, and high-volume readers, respectively. The corresponding values for specificity were 96% (95% CI: 95%, 97%), 97% (95% CI: 94%, 97%), and 96% (95% CI: 95%, 97%), respectively. With regard to the performance of high-volume readers, the mean values per quartile ranged from 63% to 84% for sensitivity and from 95% to 98% for specificity. The percentage of cancers that were missed by first readers but were detected by second readers was 44%, and the percentage of cancers that were detected through clinical detection within 12 months was 56%. Performance measures for second readers and for the consensus discussion can be found in Table E1 (online); sensitivities were 80% and 85% for second readers and consensus discussion, respectively. Performance is further stratified according to mammographic density and age in Table E2 (online).

In Table 3, we examine the screening outcomes according to sensitivity quartile of the first reader based on 10 000 screening mammograms per quartile. For the most sensitive radiologists (Q4), 45 cancers were diagnosed at screening or within 12 months thereafter, of which 37 (82%) were detected and seven were missed by the first reader (per 10 000 screened). For the least sensitive radiologists (Q1), there were 38 cancers, of which 24 (63%) were detected and 14 were missed (per 10 000 screened). There were 497 and 281 abnormal interpretations for Q4 and Q1, respectively (per 10 000 screenings).

Reader Sensitivity for Each Tumor Subgroup

First-reader sensitivity for each tumor subgroup is shown in Table 4. The results are presented overall and by quartile of sensitivity of high-volume readers (each quartile contains the same radiologists as those in Table 3). The overall sensitivity for high-volume readers was 77% for ductal cancers and 73% for lobular cancers, 76% for invasive cancers and 83% for in situ only (any grade) cancers, 77% for luminal A, and 69% for basal cancers. Analysis by quartile of high-volume readers showed that the sensitivities for the most (Q4) and the least (Q1) sensitive high-volume readers were 85% and 67%, respectively, for ductal cancers; 84% and 63%, respectively, for lobular cancers; 85% and 67%, respectively, for all invasive cancers; 93% and 75%, respectively, for in situ cancers only (any grade); 99% and 75%, respectively, for high-grade in situ cancers only; 85% and 67%, respectively, for luminal A cancers; and 89% and 53%, respectively, for basal cancers (P < .001 for the comparison of sensitivity levels between quartiles for each of the aforementioned subgroups).

Discussion

Our multicenter study provided performance benchmarks for first readers of screening mammograms in Sweden based on more than 1 million full-field digital mammograms acquired between 2008 and 2015. For first readers, second readers, and consensus discussion, the sensitivities were 73%, 80%, and 85%, respectively, and the specificities were 96%, 96%, and 98%, respectively. For high-volume (≥5000 mammograms per year) first readers, sensitivity ranged from 63% to 84% for the least sensitive and most sensitive quartile of radiologists, respectively. In a standardized screening population of 10 000 women, the most sensitive quartile of radiologists and the least sensitive quartile would miss seven and 14 cancers, respectively, that were later detected by the second reader or through clinical detection during 12 months after screening. Top sensitive radiologists were consistently better across tumor subgroups.

When our high-volume first-reader results are compared with the most recent update from the Breast Cancer Surveillance Consortium, the sensitivity is lower (73% vs 87%) and the specificity is higher (96% vs 89%) (13). Even though the first-reader sensitivity is markedly lower, the entire process with double reading shows a sensitivity of 85% after consensus discussion. Higher sensitivity and lower specificity in the United States compared with European countries is well known and has been stated in previous studies (16). The main analysis of this study was performed without considering that often there were repeated observations over time for each woman. As a secondary analysis, we fitted a generalized estimating equation, with repeated measures taken into account. This showed only a minimal difference in estimated sensitivity (lower by 0.3 percentage points) and no change in specificity.

Both sensitivity and specificity were similar between low- and high-volume readers. There have been conflicting data on the association between the experience level of radiologists and their accuracy in interpreting mammograms (17–19). However, we observed large differences in sensitivity between individual high-volume readers. Major differences in mammography assessment accuracy were also reported in a recent study that showed a range of sensitivities between 76% and 84% and a range of specificities between 49% and 79% (9).

Our analysis revealed a lower number of first-reader false-negative screenings (ie, potential interval cancers) in readers operating at a higher sensitivity level at the expense of a higher rate of abnormal interpretations (ie, potential recalls). These findings are in line with the association between increased recall rate and decreased interval cancer rate determined in a prior study (20).

The aforementioned studies determined performance benchmarks, but none examined sensitivity differences based on tumor characteristics. The top quartile of radiologists achieved a sensitivity level of 99% for high-grade in situ cancers compared with 85% for invasive cancers. The higher sensitivity for in situ cancers might be explained by their mammographic manifestation of calcifications, which are relatively easy to see, as well as a frequent absence of clinical manifestations that are less likely to be detected (21–23). Lobular cancers are detected at a larger size than ductal cancers, both at screening and clinically, based on symptoms. The sensitivity figures reported in our study concern the ability of screening mammography to facilitate detection of cancers otherwise detected clinically within 12 months. Cancers that are hard to detect clinically, such as lobular cancers or small cancers, will have an inflated sensitivity measure (because the cancers will not cause clinical symptoms until after 12 months have elapsed). Our observation that first-reader sensitivity for diagnosing larger tumors is lower than for smaller tumors might in part be explained by the fact that larger tumor size may be associated with an increased probability that women will self-detect the tumor during the 12-month interval after a negative screening result. In our study, the most sensitive radiologists were better across most tumor characteristics. The largest performance difference was observed for the basal molecular subtype, in which the least sensitive quartile of radiologists detected only 53% of cancers. Basal cancers often have benign or indeterminate mammographic findings, potentially making them easier to miss (24).

There is much interest in developing and implementing AI-based computer-aided CAD systems (9). Our results showed that the performance requirements for an AI CAD system are considerably higher for replacing both double reading and consensus discussion than merely replacing the first reader.

Our study had some limitations. The generalizability was limited by the ethnic composition of Stockholm, screening age range of 40–74 years, biennial screening schedule, double-reading setting, breast density, and some missing data regarding tumor characteristics, such as tumor histologic and molecular subtypes. Also, tumor size was based on pathologic measurements without consideration of neoadjuvant therapy, so the size at the time of screening could have been larger than at the time of pathologic postoperative assessment. This should not affect the comparison across reader quartiles but could affect the absolute performance measures. Our study was conducted based on two-dimensional screening mammography, and the corresponding metrics in a three-dimensional mammographic setting are likely to be different.

In conclusion, our study determined a range of screening mammography benchmarks, which can be useful in evaluating the performance and choosing the operating point of standalone artificial intelligence–based computer-aided detection systems.

Disclosures of Conflicts of Interest: M.S. disclosed no relevant relationships. K.D. disclosed no relevant relationships. M.E. disclosed no relevant relationships. P.L. disclosed no relevant relationships. F.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: is a consultant for Collective Minds Radiology. Other relationships: disclosed no relevant relationships.

Author Contributions

Author contributions: Guarantor of integrity of entire study, F.S.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, M.S., F.S.; clinical studies, M.S., P.L., F.S.; statistical analysis, F.S., M.S., M.E.; and manuscript editing, M.S., F.S.