Development and Validation of a Convolutional Neural Network for Automated Detection of Scaphoid Fractures on Conventional Radiographs

Datasets

This retrospective study was approved by the medical ethical review boards of the hospitals Jeroen Bosch Ziekenhuis (JBZ) and Radboud University Medical Center (Radboudumc) in the Netherlands. Informed written consent was waived, and data collection and storage were performed in accordance with local guidelines. Three different datasets consisting of hand, wrist, and scaphoid radiographs were prepared for training and testing a scaphoid segmentation CNN (dataset 1: 1039 radiographs [from 775 patients] acquired during 2017–2019 at JBZ and Radboudumc), for training a fracture detection CNN (dataset 2: 3000 radiographs [in 1846 patients] acquired during 2003–2019 at Radboudumc), and for testing the entire fracture detection system (dataset 3: 190 radiographs [in 190 patients] acquired during 2011–2020 at JBZ). Datasets 2 and 3 were gathered at different hospitals to assess the generalization performance of the fracture detection CNN. All radiographs were extracted from the picture archiving and communication system of JBZ and Radboudumc and were de-identified by removing metadata. Details of the datasets are provided in Table 1.

Table 1: Details of the Experimental Datasets

The datasets were manually selected and annotated by N.H. in consultation with a musculoskeletal radiologist with 30 years of experience (M.R.). For datasets 1 and 2, the software Visual Geometry Group (VGG; https://www.robots.ox.ac.uk/~vgg/software/via/) Image Annotator (14) was used to create scaphoid segmentation masks and bounding boxes. For datasets 2 and 3, binary labels for scaphoid fracture detection were derived from the original radiology reports. Dubious scaphoid fractures were reevaluated by radiologist M.R. (only for dataset 2). The diagnoses of all fractures in dataset 3 were confirmed with a follow-up CT scan within 4 weeks (results of which were considered the ground truth).

Only anteroposterior and posteroanterior radiographs were selected for the datasets, because these showed the least amount of overlap between the scaphoid and other bones. Radiographs were excluded when accurate annotation or fracture diagnosis was impossible as a result of screws or other implants, resection, excessive damage or malformation, or a cast. Moreover, radiographs depicting old scaphoid fractures were excluded from dataset 3, because the focus of the current study was on early fracture diagnosis. Additional patient information and an overview of all criteria are provided in Appendixes E1 and E2 (supplement).

Model Pipeline

Figure 1 shows an overview of the model pipeline, which consisted of two main components: a scaphoid segmentation and fracture detection CNN (from now on referred to as segmentation and detection CNN). A radiograph was first preprocessed by normalizing the scaphoid size and fixing the image size by padding or cropping. Then, the segmentation CNN localized the scaphoid on the radiograph to remove irrelevant regions by cropping. A lightweight custom architecture was designed to preserve the original image resolution as much as possible, because the segmentation task is complicated by the overlap of the scaphoid with surrounding carpal bones. After the cropping operation, the image was rescaled to a fixed size, and its contrast was normalized. A detailed description of the architecture, preprocessing steps, and training procedure is provided in Appendixes E3–E5 (supplement).

Figure 1:

Download as PowerPoint

Next, the detection CNN processed the cropped image and returned a probability of whether the scaphoid contained a fracture. This CNN was based on a DenseNet-121 architecture (15). A class activation map was then calculated using the Smooth Grad-CAM++ method (16), which showed regions in the image that were most relevant for predicting whether a fracture was present (see Appendix E6 [supplement] for implementation details). These regions were expected to correlate with any existing fracture line. The class activation map was projected on the input image as a heatmap for easy localization of potential fractures (Fig 1).

Implementation Details

The CNNs were implemented on a graphics processing unit (Nvidia RTX Titan) using the PyTorch machine learning framework (17). Through supervised learning, the segmentation CNN was trained on a random subset of dataset 1 (80%), and the detection CNN was trained on all samples from dataset 2. The Adam optimizer (18) was used to update the weights by a linearly decaying learning rate schedule. A detailed account of the training procedure is provided in Appendix E5 (supplement).

Observer Study

For comparing the performance of the fracture detection system with that of radiologists, an observer study was conducted among 11 radiologists (three of whom were radiology residents; B.V., E.S., M.d.J., B.M., S.D., S. Bollen, A.S., S. Bruijnen, S.S., M.R., M.d.R. [random order]). The radiologists independently assessed 190 radiographs from dataset 3 for the presence of a scaphoid fracture and indicated their confidence for each radiograph on a continuous scale from 0 to 1.0, where 1.0 indicated absolute certainty of a fracture. They used 0.5 as the cutoff point for the decision whether a fracture was present. We defined an average radiologist by averaging the scores of the 11 readers.

Statistical Analysis

The segmentation CNN was evaluated on an unseen subset of dataset 1 (20%, no patient overlap) by calculating the Dice similarity coefficient (DSC) and symmetric Hausdorff distance (HD). The detection CNN was internally evaluated in dataset 2 by means of fivefold cross-validation (manual scaphoid crops, no patient overlap) by calculating the sensitivity, specificity, positive predictive value (PPV), and area under the receiver operating characteristic curve (AUC). The whole fracture detection system was evaluated on hold-out set dataset 3 by calculating the sensitivity, specificity, PPV, and AUC (automated scaphoid crops, no patient overlap with datasets 1 and 2). The classification threshold was set at 0.5 for both CNNs.

The evaluation metrics were calculated using the scikit-learn machine learning library (version 0.22.1, 2020) (19) and the MedPy medical image processing library (version 0.4.0, 2019) (20) for Python. Stratified bootstrapping with 5000 iterations was applied for estimating 95% CIs. To compare both the AUC of the CNN with that of the radiologists and the AUC of the attending radiologists with that of the radiology residents, multiple-reader, multiple-case (MRMC) receiver operating characteristic (ROC) analyses were conducted using the iMRMC software (version 1.2.0, 2020) (21). These analyses were based on a t test, where the degrees of freedom were estimated as proposed by Obuchowski et al (22). A difference with a P value smaller than .05 was considered significant.

Model Availability

This system is freely available at https://grand-challenge.org/algorithms/scaphoid-fracture-detection/, where it can be run in a web browser in real time.

Segmentation CNN Results

The segmentation CNN achieved an average DSC of 97.4% ± 1.4 (standard deviation) with an HD of 1.31 mm ± 1.03 on the test set (subset of dataset 1). One test sample led to a segmentation failure in which no pixels were segmented, and for this sample no HD could be calculated. The average height and width of the ground truth scaphoid segmentation masks were 23.12 mm ± 3.84 and 14.30 mm ± 2.03, respectively, which were estimated by calculating the major and minor axis length of ellipses fitted to the masks. The total training time of this network was approximately 15 hours.

Detection CNN Results

On dataset 2, the detection CNN achieved a mean sensitivity of 66% ± 3, specificity of 90% ± 1, PPV of 81% ± 2, and AUC of 0.86 ± 0.01 over the validation folds. The mean ROC curve on this set from the fivefold cross-validation procedure is provided in Appendix E7 (supplement). The ROC curve indicates that the CNN had a sensitivity of 73% at the same specificity as measured on the test set (dataset 3), 84%.

The training time of the detection CNN was approximately 2 hours. The average processing time per test sample (dataset 3) was 1.7 seconds (including all preprocessing steps, as listed in Appendix E4 [supplement]). The average reading time per test sample of the radiologists was 13.9 seconds.

Comparison of the CNN to Radiologists

Table 2 presents the sensitivity, specificity, PPV, and AUC with their 95% CIs for the CNN and the radiologists on the test set (dataset 3). The ROC curves for the CNN (with the 95% CI band), the average of radiologists, and the radiologists with the highest and lowest AUCs are shown in Figure 2A. The same ROC curve for the CNN with individual operating points of the CNN and radiologists are plotted in Figure 2B. These operating points indicate the sensitivity and false-positive rate at a threshold of 50%. The CNN and radiologists achieved similar performance for fracture detection (AUC, 0.87 [95% CI: 0.81, 0.91] vs 0.83 [range, 0.79–0.85]; P = .09 with MRMC ROC analysis). At a fixed false-positive rate of 5.0%, the CNN achieved a sensitivity of 65% (61 of 95 patients with a scaphoid fracture would be recommended for follow-up procedures). In comparison, at the same false-positive rate, the average of radiologists (n = 11) achieved a sensitivity of 55% (equating to 51 of 95 patients). Among the radiologists, the attending radiologists (n = 8) and radiology residents (n = 3) achieved similar performance (AUC, 0.83 [range, 0.79–0.85] vs 0.82 [range, 0.81–0.84], P = .86 with MRMC analysis).

Table 2: Evaluation Results of the Detection CNN and the Radiologists

Figure 2:

Download as PowerPoint

To inspect test cases that were misdiagnosed by either radiologists or the CNN, the fracture confidence score of the CNN versus the average fracture confidence score of the radiologists per case are plotted in Figure 3. The CNN produced four false-negative results and 14 false-positive results not made by the average radiologist. Among these cases of misdiagnosis, with high certainty, the CNN missed one fracture (confidence score = 0.01) and misclassified six nonfractures as fractures (confidence score, ≥ 0.8) that were all easily identified by the radiologists (confidence scores ≥ 0.8 and ≤ 0.2; respectively). The average radiologist made 18 false-negatives and one false-positive not made by the CNN. Among the false-negative cases, with high certainty the average radiologist missed five fractures (confidence score ≤ 0.2) that were confidently identified by the CNN (confidence score ≥ 0.8). Additional examples of scaphoid fractures that were occult to the average radiologist but not for the CNN are provided in Appendix E8 (supplement).

Figure 3:

Download as PowerPoint

Fracture Localization with Class Activation Maps

Examples of class activation maps that were generated for localizing (potential) fractures in the radiographs from dataset 3 are shown in Figure 4. The image crops that were fed into the detection CNN and the corresponding class activation maps are presented in pairs. On manual inspection, highlighted regions in the class activation maps were found to overlap with the fracture lines in the scaphoid, as illustrated in Figure 4.

Figure 4:

Download as PowerPoint