Computer-aided Detection of Masses at Mammography: Interactive Decision Support versus Prompts

Introduction

In breast cancer screening, computer-aided detection (CAD) systems are used to prevent perceptual oversight of abnormalities on mammograms. The positive effects of CAD were shown in certain studies (1–4), while other studies demonstrated no performance increases related to CAD (5–8). Most radiologists agree that CAD is helpful for detection of microcalcifications, for which the sensitivity of CAD is high. However, there is less agreement regarding CAD’s ability to find masses and architectural distortions. Many radiologists argue that CAD indicates too many false-positive results to have a beneficial effect on mass detection (9,10).

CAD may lead to disappointing results because masses are often missed because of incorrect interpretation (11,12), which is not the focus of existing CAD technology. Interestingly, previous research indicated that reader performance could be improved by combining reader scores with the presence and probability of CAD marks (13,14). These results motivated us to develop a CAD system that helped radiologists interpret suspicious regions, rather than to assist with the initial detection of these regions. In our interactive system, CAD marks are only displayed “on demand” for queried regions with a suspiciousness score.

Other interactive CAD systems provide additional information to justify the CAD marks so that they are not ignored. Additional interactive CAD systems aid characterization of lesions in a clinical setting by using reference libraries of similar cases (15–17). Our system is aimed at screening. Instead of providing justification for the marks that are used in the existing CAD systems, our system relies on the reader for the initial detection process and aims to improve only interpretation and recall decisions. A recent study found that this way of using CAD for the detection of masses was effective (18), although they did not compare it with the regular use of CAD prompts. That study used digitized film mammograms. The purpose of our study was to assess the effectiveness of an interactive CAD system in which CAD marks and their associated suspiciousness scores remain hidden unless their location is queried by the reader and to compare it with the effect of traditional CAD prompts used in current clinical practice for the detection of malignant masses in full-field digital mammograms.

Materials and Methods

Study Population

This retrospective study was carried out in accordance with the rules in the Netherlands that are relevant to the review of research ethics committees and informed consent. Study material was made anonymous and institutional review board approval was waived. Mammograms used in this study were acquired in a 2003–2008 digital screening pilot project that was conducted in Utrecht, the Netherlands (19). In the screening program, women who were aged 50–74 years were invited to participate every 2 years. Digital mammograms were acquired with a Selenia digital mammography system (Hologic, Danbury, Conn). All mammograms were independently read by two radiologists, with recalls based on consensus. For subsequent screenings, prior digitized film mammograms were available preceding the first digital screening examination. The material from which we selected cases for the study included over 1200 recalls in which 202 cancers were detected by using a mass or architectural distortion as the dominant sign of abnormality. Abnormalities were annotated with the supervision of a radiologist.

Case Selection

We collected material from a prospective study on the effect of digital screening (19) and used it in our observer study on the effect of interactive CAD. We selected 80 biopsy-proved cancer cases and 120 negative cases because previous observer studies showed this was a reasonable balance. All selected cases were full-field digital mammograms. The cases were selected as follows: First, cases in which the lesion was rated as obvious, cases with only microcalcifications, and cases in which not all four views (ie, both cranial-caudal and mediolateral oblique views of both breasts) were available were excluded. Then we checked for digital screening mammograms, acquired prior to detection, in which a malignant lesion was already visible. This yielded a total of 17 mammograms. From the remaining cases, we randomly selected 63 screen-detected cancer cases from incident screening rounds. For the negative cases, we included 20 false-positive cases that were verified by normal follow-up to make the study series more challenging. In these 20 cases, a suspicious mass or architectural distortion had been reported in screening, while a biopsy was not deemed necessary during diagnostic assessment, and recall was not due to a known benign abnormality such as cyst. In this way, we sampled suspicious findings due to projection shadows that appeared normal during assessment. The remaining 100 negative mammograms were randomly selected from the nonreferred digital mammograms with at least one normal follow-up screening examination. We made sure that the proportion of initial screenings was the same for positive (four of 80) and negative (six of 120) cases.

CAD and Reading Environment

This CAD system (20,21) used in our study was designed to detect malignant masses and architectural distortions, and was trained with a large set of digitized film mammograms (11 793 images that contained 1853 malignant mass regions). By using a preprocessing module, this system can be used for full-field digital mammograms (22). A special feature of the CAD system is that it automatically links regions in the mediolateral and mediolateral oblique views of the same breast if they correspond to the same lesion (20,23–25). This linking information is used when computing the suspiciousness score for a region.

The reader study was performed by using an experimental reading environment for screening mammography (18), developed in house, which included hanging protocols for navigation between views and comparison of current and prior mammograms. Images were displayed by using a 30-inch (76-cm), calibrated, color Digital Imaging and Communications in Medicine display panel (FlexScan SX3031W; Eizo, Ishikawa, Japan) with a resolution of 2560 × 1600. Readers’ actions were logged by the system to facilitate detailed analyses of the sessions.

CAD results were viewed in two modes: the prompting mode and the interactive mode. When activated and in the prompting mode, all CAD regions were shown by a display of contours without providing suspiciousness scores. Prompts were shown for CAD regions with suspiciousness scores above a particular threshold. This threshold was adjusted so that a reference set of normal mammograms was given an average of two prompts per case. This prompting mode, with the threshold we used, was similar to the way in which CAD is used in current clinical practice.

In the interactive mode, regions that were defined by CAD remained hidden until they were activated by the reader, who could query for CAD results with a mouse-click on a mammographic region. If a CAD result was available at the queried location, the contour of this region was presented to the reader with its suspiciousness score. A CAD result was available if the queried location was inside of the contour of the CAD region or if the distance between the queried location and the center point of the CAD region was less than 0.5 cm. View correspondence was also used; if an activated CAD region was linked to a region in the other view, this other region’s contour and score were also shown.

CAD results were displayed in the interactive mode by obtaining suspiciousness scores that were above the threshold. This threshold was chosen so that an average of eight CAD regions were available on a normal four-view mammogram. The increased accessibility of the interactive mode was an inherent advantage because more CAD results were accessible in this mode than they were in the prompting mode. To prevent too many false-positive determinations, we did not provide all marks that were accessible with interactive CAD as prompts in the traditional reading mode, which may have undermined our intention to compare the interactive approach with the use of CAD in current clinical practice. In the interactive mode, contours of queried regions were displayed in color by using a continuous scale from yellow (less suspicious) to red (highly suspicious). A numeric value that represented suspiciousness was also shown next to the contour, ranging from 0 (not suspicious) to 100 (highly suspicious).

Observer Study Design

Twelve readers (nine radiologists with 1–24 years of experience in mammography and three residents trained in breast imaging) participated in the study. Before the study sessions, readers were offered a short training session so that they could become familiar with the experimental setup. Most readers had previously used conventional CAD in screening practice, and five of the radiologists participated in earlier studies with a commercial CAD system where they used traditional prompts (18, 26). Three radiologists (readers 5, 8, and 11) participated in a prior study on interactive CAD, where they gained experience with the interactive display (18). None of the readers had experience with our CAD system. One reader interpreted some of the original mammograms that were used for the screening that occurred more than 2 years before we conducted the observer study. Two readers helped supervise the annotation of the abnormalities, which was completed more than a year before the observer study took place.

In the observer study, each reader interpreted all images in both modes during two sessions. In the first session, the first 100 mammograms were interpreted by using either CAD with prompts or interactive CAD, while the second series of 100 mammograms were subsequently interpreted in the alternate mode. In the second session, conducted at least 4 weeks after the first session was completed, the same mammograms were interpreted again with the reading modes switched.

To obtain sufficient data for analysis, radiologists were asked to report more of their findings than they would usually do in screening practice. Readers marked the location of their findings (not the contour) on both views, and assigned a suspiciousness score from 0 to 100 to each of their findings. In the prompted mode, readers first assigned a suspiciousness score to each image without using CAD. CAD was then made available, and readers could adjust their scores and add new findings. Results for the unaided, regular CAD, and interactive CAD modes were obtained.

Data Analysis

The standalone performance of CAD was computed by using free-response receiver operating characteristic analysis, which provided the case-based true-positive fraction as a function of the false-positive rate on images that did not depict cancer. The location of a finding was considered correct if its distance to the center of the reference standard was less than 2 cm. On one image, two malignant masses were present, and the correctly localized finding with the highest score was used. For comparison, we determined the sensitivity and specificity of the digital CAD system (R2 ImageChecker, V1.4; Hologic) on the study set. This system was adjusted to a specific setting.

Reader performance was computed by using location receiver operating characteristic (LROC) analysis, which determined the true-positive fraction for each false-positive fraction and was computed from the finding with the highest score on each of the images that did not depict cancer. For each reader and reading mode, the partial area under the LROC curve (pAUC) for false-positive fractions that were less than 0.2 was computed by using linear interpolation between the operating points. The low false-positive range was chosen to match the operating points used by radiologists in screening. In the dataset we used, 16.7% (20 of 120) of the normal cases were recalled by the original screening radiologists. The raw LROC curves for each mode were averaged by computing true-positive fraction values for a standard set of false-positive fraction values with linear interpolation. This was also calculated for false-positive fraction values that were obtained by LROC curves from all readers. Statistical analysis on the performance differences for the three modes was performed by using a method that treats readers and data as random samples and uses the jackknife method of analysis (27). This method did not take the location of the findings into account. P values less than .05 were considered to indicate statistical significance only if the global test on the null hypothesis of equal treatments also indicated significance (P < .05).

In the interactive mode, the average number of queries with and without a CAD response was computed in normal cases. Because most images were normal at screening, this number reflects the number of clicks that would be expected during screening practice. We also computed the median reading time for the normal cases for each reader. Instead of average reading times, median reading times were computed because the median is less affected by excessively long reading times caused by interruptions during the sessions. To investigate a potential effect of the experience of the readers on the benefit of CAD, the Pearson correlation coefficient was computed between the number of years readers practiced as qualified breast imagers and the increase in pAUC with interactive CAD (R version 2.11.1; Institute of Science and Medicine, Vienna, Austria).

Results

The performance of CAD on the 200 study cases is shown in Figure 1. In the prompting mode, the standalone sensitivity of CAD was 84% (67 of 80 cases), with an average of 3.2 false-positive results per normal image. In the interactive mode, the sensitivity was 91% (73 of 80 cases), with an average of 8.2 false-positive results per negative case. More false-positive marks were available than expected (2.0 for prompting and 8.0 for interactive CAD). The sensitivity of the commercial system was 0.75 at 0.21 false-positive mark per image. The sensitivity of our CAD system was 0.76 at the same false-positive rate.

Figure 1:

Download as PowerPointOpen in Image Viewer

In total, 10 031 findings were reported by 12 readers in the three modes. By considering a distance of more than 2 cm unique, an average of 958 unique findings was reported for each mode. All abnormalities were correctly localized by at least one reader in the unaided mode. Figure 2 shows the LROC curves for the three modes averaged over the readers. For each reader, the endpoint of the LROC curve was at a false-positive fraction value that was higher than the threshold used for computing the pAUC (0.2), and no extrapolation was needed to compute the pAUC values. For nine out of 12 readers, the use of interactive CAD yielded a higher performance than did unaided reading. The average pAUC (± standard deviation) increased from 0.57 ± 0.088 to 0.62 ± 0.051. The difference in reader performance for these two modes was statistically significant (P = .009; P = .015 for the global test on the three modes). The average pAUC, obtained with prompting, was 0.57 ± 0.087, which was not better than that of unaided reading (Table 1). Reader performance with prompting was significantly worse than reader performance with interactive CAD (P = .016). The performance of three readers who achieved a relatively high performance in the unaided mode was lower with interactive CAD.

Figure 2:

Download as PowerPointOpen in Image Viewer

Table 1 pAUC for False-Positive Fractions Less than 0.2

Note.— pAUC is given for the unaided mode. Increase in pAUC is given for the other modes and compared with the unaided mode. Trained residents have 0 years of experience.

The correlation between the radiologists’ number of years of mammography experience and the performance increase with interactive CAD is shown in Figure 3. The correlation coefficient was −0.53, which suggested that more experienced readers had less of a performance increase. The average number of clicks per normal case is listed in Table 2, and the median reading time per radiologist is listed in Table 3. It appears that two of the most experienced readers (each with 19 years of experience) probed regions often, which led them to display most of the available CAD marks. Their performance declined compared with that of unaided reading.

Figure 3:

Download as PowerPointOpen in Image Viewer

Table 2 Number of Clicks per Normal Case with and without CAD Response

Table 3 Median Reading Time in Seconds per Case

Note.— The increase or decrease from the unaided mode is given in seconds in parentheses.

Discussion

We found that reader performance increased when CAD results were interactively displayed, and that prompts had no major effect on reader performance. The reading process of the proposed interactive system was not disrupted by appearance of false-positive prompts at unexpected locations, as was the case in the conventional system, which may have led to these results. In the interactive mode, marks remained hidden unless regions that corresponded were probed. Because most radiologists probed only a limited number of regions, and only those that they were interested in, fewer false-positives were displayed. CAD suspiciousness scores generally corresponded well with the observers’ interpretations. When this was not the case, readers were alerted and paid more attention, which may have led to better decisions.

There was a large variance in the effect of interactive CAD for the 12 readers. Results indicated that readers with more experience had a higher unaided performance and had less or no benefit with CAD. The decrease in performance of the two readers who probed regions extremely often may have been related to this deviation. That two readers were involved in supervising the abnormality annotation may have biased the results. However, annotation was performed more than 1 year before the observer study was conducted, and a potential for recall existed with both prompted and interactive modes.

To make a fair and relevant comparison, the threshold for displaying prompts in the conventional system was set to a level that corresponded to the level used in current clinical practice. In the study series, we measured 3.2 marks per normal case that were false-positive for mass. This rate was in the range of settings used in commercial systems but was slightly higher than we expected, because the thresholds on the classifier output were based on a reference set of digitized film mammograms. It appeared that average CAD scores for digital images were slightly higher than those for digitized film images. To validate a comparison with clinical practice, the sensitivity of CAD should be high. Compared with the commercial R2 ImageChecker system, the sensitivity of our system was slightly higher at the same false-positive rate.

The threshold used to determine which CAD results were available in the interactive system was different than the threshold used to display prompts in the conventional system for reasons described in the Materials and Methods section. We do not know if other results would have been obtained by using another threshold. A further study would be necessary to determine the optimal threshold.

The effect of prompts may be underestimated in this study. Our results were different from the results that were obtained in prospective studies, where prompts had a positive effect on performance (1–4). A retrospective observer study like ours may have had fewer search errors, and prompts would have had less of an effect. The lack of effect from regular CAD might be due to a sequential-reading-order design that was used to compare it with unaided performance. This sequential approach to reading has been criticized because it could bias reader performance for one of the two modes (unaided reading or reading with prompts). However, a study by Beiden et al (28) shows little difference between mean effects of CAD measured in independent and sequential approaches. Furthermore, reading with interactive CAD yielded an average reader performance that was significantly higher than both unaided reading and reading with prompts.

We found that the use of CAD (interactive or with prompts) lengthened the time that radiologists took to read the images by approximately 10 seconds per image. Owing to the sequential scoring of each case, prompts could only increase the time it would take to read the images compared with the unaided mode. Some readers reported that, out of curiosity, they took more time to explore the CAD results in the interactive system, which would have increased the time it took to read the images. We expect that, with increased experience, the reading time for interactive CAD will be reduced. An earlier study showed no increase in reading times with interactive CAD (18).

To ensure that we obtained enough data for LROC analysis, radiologists were asked to report more findings than they ordinarily would in screening practice. This might have changed the behavior of the readers compared with their routine practice. However, for our analysis, we used pAUC for low false-positive fractions. Therefore, findings with very low suspiciousness scores did not influence the results.

A limitation of this study was that the effect size of interactive CAD did not easily translate to screening practice. We selected a challenging set of cases for this study, in which the proportions of normal and abnormal cases were different from those in screening practice. Microcalcification cases, in which no mass or architectural distortion was visible, were excluded. Reader performance depends on the subtlety of the cases in the study set. However, we used the same study set for each mode; therefore, we believe that the relative differences between unaided reading, with prompts and with interactive CAD, are valid.

In conclusion, the interactive use of the results from CAD as decision support for detection of malignant masses on mammograms may be more effective than the current use of CAD, which is aimed at prevention of perceptual oversights.

Disclosures of Conflicts of Interest: R.H. No relevant conflicts of interest to disclose. M.S. No relevant conflicts of interest to disclose. M.B.L. No relevant conflicts of interest to disclose. R.M. Mann Financial activities related to the present article: none to disclose. Financial activities not related to the present article: payment for lectures from Siemens Healthcare. Other relationships: none to disclose. R. Mus No relevant conflicts of interest to disclose. G.J.d.H. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: consultancy for Philips ICT Research Eindhoven as an expert in breast radiology; payment for lectures from Philips Healthcare Netherlands; stock/stock options in Sigmascreening. Other relationships: none to disclose. D.B. No relevant conflicts of interest to disclose. R.M.P. No relevant conflicts of interest to disclose. C.B. No relevant conflicts of interest to disclose. N.K. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: grants from Matakina Technology, Qview Medical, Riverain Medical; patent from MeVis Medical Solutions; stock in Matakina Technology, Qview Medical. Other relationships: none to disclose.

Author Contributions

Author contributions: Guarantors of integrity of entire study, R.H., N.K.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, R.H., M.S., G.J.d.H., C.B., N.K.; clinical studies, M.B.L., R.M. Mann, R. Mus, G.J.d.H., R.M.P.; experimental studies, R.H., M.S., R.M. Mann, R. Mus, D.B., C.B., N.K.; statistical analysis, R.H., N.K.; and manuscript editing, R.H., M.B.L., R.M. Mann, R. Mus, G.J.d.H., D.B., R.M.P., N.K.