Deep Learning for Detecting Cerebral Aneurysms with CT Angiography

Data Acquisition

Inclusion criteria included availability of head or head and neck CT angiography images with at least one radiologically reported untreated intracranial aneurysm, regardless of age, indication, clinical manifestation, and morphologic features. Images with severe motion artifacts were excluded. The images were consecutively retrieved from the imaging databases of two hospitals of Huazhong University of Science and Technology (Union Hospital and Union West Hospital) and were acquired with four different scanners between January 2015 and June 2019. Images from each scanner were divided equally and randomly between training and validation sets. An additional 400 CT angiograms acquired between July and December 2019 were used for external validation. The imaging protocol included standard CT angiography of the head or head and neck with variable imaging parameters (Table E1 [online]).

Development of the Algorithm

CT angiograms with reported intracranial aneurysms were used to develop and assess the algorithm. A three-dimensional detection algorithm based on a convolutional neural network was developed. All images were rescaled to a spacing of 0.39 × 0.39 × 0.39 mm³. During training, the rescaled images were cropped to 128 × 128 × 128 pixels and then fed into a one-stage detector as described previously (17). Convolutional block attention module (18) was added into Res-block of ResNet-18 (19); moreover, dense atrous convolution and residual multikernel pooling blocks (20) were used between encoding and decoding stages (Fig E1 [online]).

For testing, the trained algorithm was used to predict the probability of aneurysm existence and the size at each voxel and to draw a bounding box on the feature suspected to be an aneurysm. Those locations with high probabilities were selected as candidates. The overlapped candidates were then filtered out by using nonmaximum suppression.

Hyperparameters were selected by using fivefold cross-validation on the training set. During training, a stochastic gradient descent optimizer with a learning rate of 0.01 and weight decay of 10⁻⁴ was used. The algorithm converged after 100 epochs. The threshold of overlap in nonmaximum suppression was set to 0.02. All images were automatically preprocessed with SimpleITK version 1.2.0 (http://www.simpleitk.org), and the detector was implemented by using Pytorch framework 0.4.1 (https://pytorch.org). Code is available at https://github.com/CTA-detection/DLCA.

Evaluation of the Algorithm

Two general radiologists (X. Long and M.X.) with 7 and 5 years of experience in CT angiography of the head, respectively, independently annotated the aneurysms by using software (3D Slicer version 4.10.1; https://www.slicer.org). Reference standard for diagnosis included, in addition to the radiologic report (by at least two of 49 radiologists with 3–23 years of experience in CT angiography of the head, three of whom are authors X. Long, M.X., and J.L.), a careful independent read by each of the two radiologists who performed the annotation. Any disagreement was solved by discussion and consensus between the two annotating radiologists.

Two radiologists (X. Long and M.X.) reviewed the algorithm predictions on the validation set and evaluated the result of the aneurysm recall.

Both internal and external validation were performed. In internal validation, 100 CT angiograms positive for aneurysms and extracted from the validation set were interpreted by four general radiologists (T.J., J.L., M.X., and X. Long) of different experience levels (3, 3, 5, and 7 years for radiologists 1 through 4, respectively) without and with algorithm assistance. External validation was carried out by four general radiologists (with 1, 3, 5, and 7 years of experience in CT angiography of the head) who were blinded to the clinical indication, radiologic reports, and other imaging examinations. Random samples of 200 of the 400 CT angiograms were assigned to each of the four radiologists for random interpretation either with or without aid of the algorithm. Following a 2-week washout period, the same selected sample was randomly interpreted again by the same corresponding radiologist without or with aid of the algorithm (if the first read was with algorithm aid, then the second read was without algorithm aid, and vice versa). The aneurysm type, number, location, and size and time to diagnosis were recorded for each interpreted image.

Statistical Analysis

The proposed algorithm was evaluated on the validation set by calculating its sensitivity and free-response receiver operating characteristic curve. Difference between recalled and nonrecalled groups in patients' sex, number of aneurysms, aneurysm type, and aneurysm location was tested by using χ² test, and the difference in age and mean aneurysm size was tested by using t test.

On the external validation set, the jackknife alternative free-response receiver operating characteristic analysis was performed with freely available software (JAFROC, version 4; http://www.devchakraborty.com) with a random-readers and random-cases model (21). By considering the clinical importance of each lesion, the area under weighted alternative free-response receiver operating characteristic (wAFROC) curve on the basis of equal weight for each lesion in multiple-lesion CT angiograms was used to assess the overall performance of readers. Multilevel generalized linear models were used for comparing the performance among repeated measures. Multilevel linear models were used to evaluate the time to diagnosis of readers. Descriptive statistical analysis was performed by using GLIMMIX, MIXED, and LOGISTIC procedures in SAS software (version 9.4; SAS Institute, Cary NC), and confidence intervals of proportions were calculated by using the Clopper-Pearson method. P values less than .05 were considered to indicate a statistically significant result.

Patient and Cerebral Aneurysm Characteristics

After exclusion of CT angiograms with no aneurysms (n = 8187) and those with severe motion artifact (n = 7), 1068 CT angiograms encompassing 1337 cerebral aneurysms from 1068 patients (mean age, 57 years ± 11 [standard deviation]; 660 women) were used for algorithm development. CT angiograms were acquired with four different scanners, including Discovery CT750 HD (GE Healthcare, Chicago, Ill; n = 440), Somatom Definition AS+ (Siemens Healthineers, Erlangen, Germany; n = 283), and Aquilion ONE (Toshiba, Tokyo, Japan; n = 251) at Union Hospital and Ingenuity CT scanner (Philips Healthcare, Best, the Netherlands; n = 94) at Union West Hospital. The detailed characteristics of patients and aneurysms are listed in Table 1. The flowchart of data acquisition, selection, and division are presented in Figure 1.

Table 1: Characteristics of 1068 Patients and 1337 Aneurysms Included in Algorithm Development

Figure 1:

Download as PowerPoint

The training set involved 688 cerebral aneurysms from 534 patients. Of these, 197 were men (age range, 7–90 years; mean age, 55 years ± 12), and 337 were women (age range, 21–89 years; mean age, 58 years ± 11). The size of aneurysms ranged from 1.2 mm to 45.6 mm, with a mean size of 5.3 mm ± 3.9. The training set contained 649 saccular aneurysms and 39 fusiform aneurysms.

In the validation set, there were 649 cerebral aneurysms from 534 patients, including 211 men (age range, 10–83 years; mean age, 56 years ± 10) and 323 women (age range, 25–93 years; mean age, 57 years ± 10). The size of aneurysms ranged from 1.2 mm to 30.8 mm (mean size, 5.2 mm ± 3.6). There were 626 saccular aneurysms and 23 fusiform aneurysms.

The external validation set involved an extra 400 CT angiograms that were acquired from the same two hospitals (Union Hospital, n = 302; Union West Hospital, n = 98) with the same four scanners (GE Healthcare, n = 101; Siemens, n = 99; Toshiba, n = 102; and Philips, n = 98). Of the 400 CT angiograms, there were 188 CT angiograms that were positive for aneurysms, encompassing 206 aneurysms in 66 male patients (age range, 29–85 years; mean age, 59 years ± 11) and 122 female patients (age range, 31–86 years; mean age, 59 years ± 10). The size of aneurysms ranged from 1 mm to 22 mm (mean size, 4.1 mm ± 2.6). There were 193 saccular aneurysms and 13 fusiform aneurysms. The characteristics of the training, validation, and external validation sets are provided in Table 2.

Table 2: Characteristics of Training, Validation Including Results of Aneurysms Recall, and External Validation Sets

Algorithm Performance

The training and validation sets and results of aneurysm recall on the validation set are given in Table 2. The maximum sensitivity of aneurysm detection by using the proposed algorithm was 97.5% (633 of 649; 95% CI: 96.0, 98.6), with a rate of false-positive findings per case of 13.8. The free-response receiver operating characteristic curve is shown in Figure 2. Representative images with false-positive findings are illustrated in Figure 3.

Figure 2:

Download as PowerPoint

Figure 3a:

Download as PowerPoint

Figure 3b:

Download as PowerPoint

Figure 3c:

Download as PowerPoint

Figure 3d:

Download as PowerPoint

Nonrecalled aneurysms were mostly smaller than 3 mm in longest diameter and found near the bony structures. Typical examples are shown in Figure 4. The performance of the developed algorithm stratified according to the type, size, and location of aneurysm and the imaging scanner is presented in Table 2. The sensitivities of detection of saccular aneurysms and fusiform aneurysms were 97.6% (611 of 626; 95% CI: 96.1, 98.7) and 95.7% (22 of 23; 95% CI: 78.1, 99.9), respectively. The algorithm performance showed a gradual improvement in sensitivity regarding aneurysm size, reaching 100% (49 of 49) in aneurysms 10 mm or larger. In terms of location, the lowest sensitivity was observed in aneurysms located on the anterior and posterior cerebral arteries; however, most of these aneurysms were smaller than 3 mm.

Figure 4a:

Download as PowerPoint

Figure 4b:

Download as PowerPoint

There was no difference between recalled and nonrecalled aneurysm groups in patients’ sex (P = .74), age (P = .30), aneurysm type (P = .44), aneurysm location (P = .16), or scanner type (P = .76). However, there was a difference in aneurysm size between the two groups (P < .01; Table 2).

The annotating radiologists found eight newly discovered aneurysms that had been overlooked during the initial reporting. Among these, six aneurysms were smaller than 3 mm and two were in the size range of 3–5 mm. Three of these aneurysms were located on the basilar artery, two were located on the internal carotid artery, and the other three aneurysms were distributed on the anterior cerebral artery, posterior cerebral artery, and posterior communicating artery. Typical examples of these aneurysms are shown in Figure 5.

Figure 5a:

Download as PowerPoint

Figure 5b:

Download as PowerPoint

Algorithm Validation

Results of radiologists’ performance in image interpretation without and with algorithm assistance in internal validation are presented in Table E2 (online). In external validation, 400 new CT angiograms (188 positive findings and 212 negative findings) were individually interpreted by four radiologists without and with the aid of the algorithm. Table 3 lists the results of external validation at each individual level. The overall areas under the wAFROC curve without and with algorithm aid were 0.60 and 0.61, respectively. The difference was 0.01 (95% CI: 0.00, 0.03). Sensitivity per lesion without and with algorithm aid was 79.09% and 88.94%, respectively. The difference was 9.85% (95% CI: 4.92, 14.79). Sensitivity per case without and with algorithm aid was 81.63% and 91.86%, respectively. The difference was 10.23% (95% CI: 5.48, 15.00). Specificity per case without and with algorithm aid was 95.94% and 90.93%, respectively. The difference was −5.01% (95% CI: −8.35, −1.68). Time to diagnosis was slightly lower (difference, −3.62 seconds; 95% CI: −9.85, 2.59); however, this difference was not statistically significant. In particular, the area under the wAFROC curve for radiologists 1 and 2, with less experience, improved by 0.03 (95% CI: 0.00, 0.06) and 0.02 (95% CI: 0.00, 0.04), respectively, with aid of the algorithm. For the wAFROC plot of each reader, see Figure E2 (online). For detailed results of the performance and comparisons, as well as the estimates of multilevel generalized linear models of radiologists’ performance and the estimates of multilevel linear models for reading time, see Tables E3–E7 (online) and Figure E3 (online).

Table 3: Four Radiologists’ Performance in Detection of Cerebral Aneurysms without and with Aid of Algorithm