Pancreatic Cancer Detection on CT Scans with Deep Learning: A Nationwide Population-based Study

Local Data Set

Contrast-enhanced portal venous phase CT studies in 546 patients with PC diagnosed between January 2006 and July 2018 were extracted from the imaging archive of a tertiary referral center for further manual labeling and analysis using convenience sampling. Inclusion criteria were as follows: (a) age of at least 18 years; (b) findings recorded in cancer registry; and (c) pancreatic adenocarcinoma confirmed at histology or cytology. If multiple studies were identified in a patient, then only the study immediately preceding the PC diagnosis was used.

CT studies in 1465 individuals with a normal pancreas obtained between January 2004 and December 2019 were extracted from the imaging archive; these individuals comprised the control group. The control studies were selected from liver or renal donors and randomly selected individuals with a normal or unremarkable pancreas (details in Appendix E1 [online]). The patients with PC were randomly divided (8:2 ratio) into the training and validation set and the local test set, respectively. The control group was randomly divided into two portions, one for constructing the training and validation set and the local test set and the other as part of the nationwide test set. Section thickness and image size were 0.7–1.5 mm and 512 × 512 pixels, respectively. CT scans were obtained in the portal venous phase, 70–80 seconds after intravenous administration of contrast medium.

National Health Insurance Data Set

The National Health Insurance (NHI) test set was provided by the Applications of Artificial Intelligence in Medical Images Database of NHI of Taiwan. The NHI is a single-payer comprehensive compulsory health insurance system covering in- and outpatient care for 99.8% of the Taiwanese population. Among the 23 164 health care facilities throughout Taiwan, 21 463 (93%) have a contract with the NHI (7). All patients with newly confirmed PC between January 2018 and July 2019 throughout Taiwan were identified by searching the registry of the NHI Major Illness/Injury Certificate (details in Appendix E1 [online]). For selection of control subjects with a normal pancreas, we identified all kidney donors and liver donors during the same period and extracted the CT studies performed for predonation evaluation from the NHI database. Demographic information and the imaging protocol used were unavailable. The CT studies of control subjects were reviewed, and the absence of radiologic abnormalities in the pancreas was confirmed by a radiologist (P.T.C.). In total, CT studies of 669 patients with PC and 72 control subjects were extracted from the NHI image database and were further combined with CT studies of 732 control subjects from the tertiary referral center imaging archive to construct the nationwide test data set.

Training of Segmentation and Classification Models

We had previously trained a segmentation CNN (8–10) using local data with radiologist-labeled pancreas and tumor and three US data sets with pancreatic parenchyma and masses labeled (11–13). The tumor and pancreas segmented by the segmentation CNN were combined into a volume of interest to be analyzed by the classification CNNs; therefore, the performance of the classification CNNs would not be negatively affected if the tumor was segmented as pancreas. To provide more robust performance and quantitative information on the confidence of the classification, five independently operating classification CNNs were trained using the same model structure but different subsets of the training and validation set from the tertiary referral center (437 patients with PC, 586 control subjects). CT was considered to show PC if the number of positive-predicting CNNs was equal to or greater than the smallest number yielding a positive likelihood ratio (LR) greater than one in the validation set (details in Appendix E1 [online]).

Visual Assessment of Tumor Location Identified by the Segmentation Model

Consensus review by two experienced abdominal radiologists (P.T.C., K.L.L.) was conducted to assess whether the segmentation model correctly predicted the location of the tumor in the CT study classified as PC by the ensemble classifier. To explore whether the secondary signs of PC might have contributed to classification, the dilated pancreatic duct was masked by randomly selected neighboring pixels of the pancreas parenchyma.

Statistical Analyses

The performance of the segmentation CNN was evaluated with Dice score per patient. The performance of each classification CNN and the ensemble classifier was assessed in various test sets to ascertain the respective sensitivity, specificity, and accuracy and the exact 95% CI for each. The sensitivity of the CAD tool was further assessed with stratification by tumor size and stage and was compared with sensitivity of radiologist interpretation as assessed in the original radiology report (details in Appendix E1 [online]). The receiver operating characteristic curves of each classification CNN and the ensemble classifier were constructed by plotting sensitivity against 1-specificity, as the threshold varied from 0 to 1. The area under the receiver operating characteristic curve (AUC) and its asymptotic 95% CI were calculated to evaluate the performance of the classifiers. LR was calculated as the ratio between the probability of the specific test result in individuals with the disease and the probability of the specific test result in those without the disease (14). The Fisher exact test and the Mann-Whitney U test were used to compare categorical and continuous variables between groups, respectively. The exact McNemar test was used to compare sensitivity for PC between the CAD tool and radiologist interpretation for CT studies in which radiologist reports were available (15). Trend analysis for proportions (ie, P trend) were implemented in STATA 17 (Stata, College Station, Tex) (16). Statistical analysis was conducted (T.W., D.C.), and P < .05 was considered to indicate a significant difference.

Data Set Characteristics

In the local data set, the mean age of patients with PC (54% male, 46% female) was 65 years, and the mean age of control subjects (51% male, 49% female) was 54 years (Table 1). The median age of patients with PC in Taiwan was 67 years in male patients and 69 years in female patients, with a slight male predominance (54%) (17). There were 106 healthy donors in the local training and validation set and 22 in the local test set. The nationwide test set included 210 healthy donors, 72 from the NHI and 138 from the tertiary referral center. In the local test set, the segmentation model yielded median Dice scores of 0.70 (IQR, 0.63–0.75) and 0.54 (IQR, 0.28–0.73) in segmenting the pancreas and the tumor, respectively, and 0.76 (IQR, 0.71–0.80) for the pancreas and tumor combined. The performance of the segmentation model could not be assessed in the NHI data set because segmentation of the pancreas and tumor by radiologists as the ground truth was not feasible due to NHI regulations.

Table 1: Characteristics of Pancreatic Cancer and Control Groups in Local Data Sets

Differentiation between PC and Control Groups and Associated Positive Likelihood Ratios

Figure 3A–3C shows ROC curves of classification CNNs in the training and validation set, local test set, and nationwide test set, respectively. The AUCs of the five classification CNNs ranged from 0.95 to 0.97 in the local test set and from 0.94 to 0.95 in the nationwide test set. The final classification (with PC vs without PC) was determined as 4 in the training phase because the corresponding positive LR began to exceed 1 at this threshold in the validation set (Table 2). In the local test set of 256 individuals, the ensemble classifier achieved 89.9% sensitivity (98 of 109; 95% CI: 82.7, 94.9), 95.9% specificity (141 of 147; 95% CI: 91.3, 98.5), and 93.4% accuracy (239 of 256; 95% CI: 89.6, 96.1) (AUC, 0.96; 95% CI: 0.94, 0.99) in distinguishing CT images in the PC group from those in the control group (Table 3, Fig 3B). In the nationwide test set of 1473 individuals, the ensemble classifier achieved 89.7% sensitivity (600 of 669, 95% CI: 87.1, 91.9), 92.8% specificity (746 of 804, 95% CI: 90.8, 94.5), and 91.4% accuracy (1346 of 1473; 95% CI: 89.8, 92.8) (AUC, 0.95; 95% CI: 0.94, 0.96] (Table 3, Fig 3C). There is no statistical difference (P = .20) between specificity in the 72 control patients in the NHI database (97.2%; 95% CI: 90.3, 99.7) and specificity in the 732 control patients in the local data set (92.3%; 95% CI: 90.2, 94.2).

Figure 3:

Download as PowerPoint

Table 2: Positive Likelihood Ratio According to Number of Classification Convolutional Neural Networks Predicting as Having Pancreatic Cancer

Table 3: Performance of Computer-aided Detection Tool and Radiologists Based on the Original Report in Differentiating Between CT Studies with and Those without Pancreatic Cancer

In both the local test set and the nationwide test set, the number of classification CNNs predicting PC was positively associated with the positive LR (P_trend = .04 and P_trend = .03, respectively) (Table 2). The positive LRs for prediction as PC by four and five CNNs were 2.22 (95% CI: 1.46, 3.37) and 25.01 (95% CI: 17.10, 36.56), respectively, in the nationwide test set and 8.09 (95% CI: 1.85, 35.42) and 29.0 (95% CI: 10.98, 76.60), respectively, in the local test set.

Comparison between Model Segmentation and Radiologist Interpretation with Respect to Tumor Location

In the local test set, 98 PC cases were correctly classified by the ensemble classifier, and the segmentation model-predicted tumor location was correct in 86 (87.8%) (Fig 3D–3F) and immediately adjacent to the radiologist-determined tumor in three (3.1%). In the remaining nine (9.2%) PC cases with a median tumor size of 2.3 cm, the location of the segmentation model-predicted tumor was inaccurate. In the nine patients with PC in which segmentation CNN-predicted tumor location was inaccurate, the segmentation model-predicted pancreas encompassed the tumor in six patients and included approximately half of the tumor in one patient. In the remaining two cases, the segmentation CNN classified the tumor as neither pancreas nor tumor but correctly segmented the nontumorous portion of the pancreas, which displayed secondary signs of PC (18), including upstream dilation of the pancreatic duct with abrupt duct cutoff in one case and upstream dilation of the pancreatic duct with parenchymal atrophy in the other (Fig 4A, 4B). To understand whether the secondary signs of PC in those two patients might have contributed to their classification as PC, we repeated the analysis after masking the dilated duct in the former PC case (Fig 4A), and the case was classified as noncancerous by the ensemble classifier (Fig 5). In the other case (Fig 4B), masking of secondary signs was not feasible due to lack of normal-appearing pancreas parenchyma. Among the six control patients in whom PC was incorrectly predicted by the ensemble classifier in the local test set, no apparent tumor was identified by the segmentation CNN in four cases. In the remaining two control patients, collateral veins secondary to idiopathic portal vein thrombosis were segmented as tumor in one patient by the segmentation CNN (Fig 4C); in the other, the pancreas parenchyma beside a biliary stent for palliation of obstructive jaundice from hepatocellular carcinoma was segmented as tumor (Fig 4D).

Figure 4:

Download as PowerPoint

Figure 5:

Download as PowerPoint

Sensitivity according to Tumor Size and Stage and Comparison with Radiologist Interpretation

In the local test set excluding seven patients without formal radiologist reports, there was no significant difference between the sensitivity of the CAD tool and that of the original radiologist report (CAD tool, 90.2%; radiologist report, 96.1%; P = .11) (Table 3). Four PCs were missed by radiologists, of which two (1.3 and 3.3 cm in size) were correctly classified by the ensemble classifier. Stratified analysis according to primary tumor size and stage also showed similar sensitivity between the CAD tool and attending radiologists (Table 4). For tumors smaller than 2 cm, the sensitivities for detecting PC with the CAD were 87.5% (21 of 24, 95% CI: 67.6, 97.3) in the local test set and 74.7% (68 of 91, 95% CI: 64.5, 83.3) in the nationwide test set. Radiologist reports and information on cancer stage were not available in the NHI data set. Comparison between the CAD tool and radiologists with respect to specificity was not feasible, given that control patients were selected based on the statement of a normal or unremarkable pancreas in the radiologist report, because otherwise it was inappropriate to exclude the possibility of PC.

Table 4: Sensitivity of Computer-sided Detection Tool and Radiologists Stratified by Tumor Stage and Size