Artificial Intelligence for Classification of Soft-Tissue Masses at US

Patient Population

This retrospective study was approved by the institutional review board, and a waiver of consent was obtained. A systematic search of the radiology report database was performed by a single musculoskeletal radiologist (L.P. [1 year of experience]) to identify musculoskeletal US examinations performed for evaluation of a soft-tissue mass at the institution between May 2010 and June 2019. Included examinations comprised patients with masses with a histologic diagnosis confirmed by the results of a biopsy or surgical excision (n = 227) or masses that demonstrated pathognomonic imaging characteristics of lipoma, benign peripheral nerve sheath tumor, or vascular malformation (n = 192), as determined by a consensus read of three experienced musculoskeletal radiologists (L.P., C.B. [7 years of experience], R.S.A. [30 years of experience]). Examinations of benign masses without histologic diagnosis were included, as a large proportion of masses demonstrating pathognomonic features of a benign lesion at the time of imaging do not undergo subsequent biopsy or surgical excision. A total of 419 patients who underwent US were identified (mean age, 52 years ± 18 [standard deviation]; 250 women). The anatomic distribution of the masses included 163 in the upper extremities (38.9%), 165 in the lower extremities (39.4%), 78 in the trunk (18.6%), and 13 in the head and neck region (3.1%). Each patient included had one soft-tissue mass. The distribution of the pathologic findings is listed in Table 1.

Table 1: Distribution of Pathologic Findings of Soft-Tissue Masses from 419 Patients

US Imaging

For each patient, up to two grayscale and two Doppler images of the same mass at differing locations were saved from the picture archiving and communication system for training and evaluating the CNN models. The large majority of examinations were acquired by using Acuson S2000 or Acuson S3000 US systems (Siemens Healthineers) with a 14-MHz or 9-MHz linear transducer. As the images were extracted from examinations retrospectively, some patients had fewer than two grayscale or Doppler images available, either because a second image was not acquired at the time of the examination or because the additional images were of poor quality. Additionally, grayscale and Doppler images were not obtained in a coregistered manner, and variations in scanning parameters such as scanning depth, gain, and Doppler flow sensitivity could not be controlled for.

Data Preparation

Images were de-identified and stored in the tagged image file format, or TIFF. Individual images were cropped manually by a single musculoskeletal radiologist (L.P.) in Adobe Photoshop (Adobe Systems) into a rectangle that best fit the mass and excluded surrounding background soft tissue. Zero padding was achieved by overlaying the cropped image onto a 512 × 512-pixel black canvas in Photoshop.

To train the CNNs with both grayscale and Doppler information simultaneously, pairs of cropped grayscale and Doppler images were concatenated side by side to generate up to two 1024 × 512-pixel images per patient. For patients with fewer than two grayscale or Doppler images, only one image was generated.

Two separate groups were created for the purpose of training and evaluating the two different CNN models. The first CNN model was trained to classify between benign and malignant masses. Since histologic diagnosis is considered the reference standard for tumor characterization, only masses with a histologic diagnosis were selected for training and evaluation of this model, which accounted for 227 masses (344 images). For the purpose of categorization, a small subset of intermediate-grade lesions (n = 5) was considered as malignant for training and evaluation of the model. In total, there were 147 benign masses (245 images) and 80 malignant masses (99 images). Twenty percent of the dataset (46 masses [68 images]) was withheld as a holdout test dataset. The remaining 181 masses (276 images) were used for training the CNN model, with a 75%-25% training-validation split and fourfold cross-validation. Patient clinical characteristics and relative distribution of patients across the training, validation, and test datasets are summarized in Table 2.

Table 2: Clinical Characteristics Across Training, Validation, and Test Datasets

A second CNN model was trained to classify the three most common benign masses depicted on US images at our institution (lipoma, benign nerve sheath tumor, and vascular malformation). These masses were chosen as they are the only masses seen with sufficient frequency (>50 patients out of 419 patients) to reasonably train a CNN classifier. For this group, both histologically proven cases and imaging pathognomonic cases were included, for a total of 275 patients with soft-tissue masses (480 images in total consisting of 219 lipomas [from 119 patients], 102 benign nerve sheath tumors [from 65 patients], and 159 vascular malformations [from 91 patients], Table 1). Twenty percent of the dataset (55 masses [94 images]) was withheld as a holdout test dataset. The remaining 220 masses (386 images) were used to train the CNN model with a 75%-25% training-validation split and fourfold cross-validation. Patient clinical characteristics and relative distribution of benign masses across the training, validation, and test datasets are summarized in Table 2.

Model Architecture

We selected the VGG16 network architecture pretrained on the ImageNet database to build the two CNN models to take advantage of the benefits of transfer learning, as well as the performance of a deep neural network architecture (24). The VGG16 architecture was implemented on the Keras machine learning platform (version 2.3.1) (https://keras.io/), running the TensorFlow (version 1.15.0) (https://www.tensorflow.org/) backend (25,26).

The VGG16 network was loaded with pretrained weights from ImageNet. The fully connected layers of the original network were replaced with a customized classifier block, consisting of two fully connected layers with 64 nodes each, and the rectified linear unit activation function, followed by a two-class or three-class output layer by using the softmax activation function. As the size of our dataset was small and prone to overfitting on a large network such as VGG16, the VGG16 architecture was simplified by removing the final block of three convolution layers. To further reduce overfitting, L2 regularization was applied to the network, and dropout layers were applied between the fully connected layers of the classifier block (Fig 1). The first convolution layer of the model was frozen, and the remaining layers were set as trainable. Finally, the training examples were loaded through Keras with real-time data augmentation implemented. Fourfold cross-validation was used to select the optimal hyperparameters for training. The models were trained on a high-performance compute cluster running an NVIDIA Tesla V100-SXM2 graphics processing unit (GPU) (NVIDIA) with 16 GB random-access memory (RAM).

Figure 1:

Download as PowerPoint

Statistical Analysis

The performance metrics monitored during training of the CNN model for classifying benign and malignant masses included validation loss, accuracy, and area under the receiver operating characteristic curve (AUC). The model with the highest validation AUC across the four folds of cross-validation was selected for evaluation on the holdout test set. Overall performance of the model was measured by the accuracy, recall, precision, and AUC of the selected model on the holdout test set. As AUC is the primary performance metric being optimized for hyperparameter and model selection, the reported model accuracy, recall, and precision are based on the default threshold of 0.5.

The performance metrics monitored during training of the CNN model for classifying lipoma, benign nerve sheath tumor, and vascular malformation included validation loss, accuracy, and the weighted average of F1 scores. The model with the highest weighted-average F1 score across the four folds of cross-validation was selected for evaluation on the holdout test set. Overall performance of the model was measured by the accuracy, recall, precision, and F1 scores of the selected model on the holdout test set.

In addition, we compared the performance of the CNN model for classifying benign and malignant masses against two experienced musculoskeletal radiologists with 30 and 7 years of experience (R.S.A. and C.B., respectively). For this reader study, each radiologist, blinded to clinical history and patient demographics, evaluated the grayscale and Doppler image pairs in the test set and labeled the mass either benign or suspicious for malignancy on the basis of imaging appearance alone. Accuracy, sensitivity, and specificity for identifying malignant masses were calculated for each reader and compared with the CNN model on a receiver operating characteristic (ROC) curve. The performance of the model and the radiologists was compared by using the McNemar test (27). The 95% CIs for performance metrics were estimated by using the Clopper-Pearson method for accuracy, sensitivity (recall), specificity, and precision, and the DeLong method for AUC (28–30). A P value of .05 was considered statistically significant. Statistical analysis was performed by using RStudio, version 1.2.5042 (https://rstudio.com/).

Benign versus Malignant Classifier Performance Metrics

The CNN trained to distinguish between benign and malignant masses demonstrated a mean validation AUC of 0.85 ± 0.11 and validation accuracy of 78% ± 9 across the four folds of cross-validation. The best-performing model was tested on the holdout test set of 46 masses (68 images) and achieved an accuracy of 79% (54 of 68); a recall (sensitivity) of 90% (19 of 21) and 74% (35 of 47) for classifying malignant and benign masses, respectively; and a precision (positive predictive value) of 61% (19 of 31) and 95% (35 of 37) for classifying malignant and benign masses, respectively. The confusion matrix of the model and its performance metrics with 95% CIs are summarized in Figure 2.

Figure 2:

Download as PowerPoint

The performance of the CNN on the test dataset was plotted on an ROC curve, which demonstrated an AUC of 0.91 (95% CI: 0.84, 0.98). The two experienced musculoskeletal radiologists classified the same test set with an accuracy of 82% (56 of 68) and 68% (46 of 68), sensitivity of 76% (16 of 21) and 90% (19 of 21), and specificity of 85% (40 of 47) and 57% (27 of 47), respectively. Comparison of the model’s performance with those of the two radiologists is shown on an ROC curve in Figure 3.

Figure 3:

Download as PowerPoint

The sensitivity, specificity, and accuracy of the two readers were compared with that of the model by using the McNemar test, which demonstrated no difference in performance between the model and the readers. The results are summarized in Table 3.

Table 3: Comparison of Accuracy, Sensitivity, and Specificity of the Convolutional Neural Network Model for Predicting Malignant Masses in the Holdout Test Dataset with Two Expert Radiologists

Four sample images of masses in the test dataset and the model’s corresponding predications are shown in Figure 4. The selected images depict one example each of true-negative, true-positive, false-positive, and false-negative predictions by the model. In all four examples, the model prediction matched the predictions of both musculoskeletal radiologists.

Figure 4:

Download as PowerPoint

Benign Mass Classifier Performance Metrics

The CNN trained to distinguish among lipomas, benign nerve sheath tumors, and vascular malformations demonstrated a mean weighted-average F1 score of 0.764 ± 0.069 and mean validation accuracy of 77% ± 7 across the four folds of cross-validation. The best-performing model across the four folds was tested on the holdout test set and achieved a weighted-average F1 score of 0.698 and accuracy of 71% (67 of 94). Precision and recall of the model were 78% (39 of 50) and 93% (39 of 42), respectively, for classification of lipomas; 71% (10 of 14) and 42% (10 of 24), respectively, for classification of benign nerve sheath tumors; and 60% (18 of 30) and 64% (18 of 28), respectively, for classification of vascular malformations. The confusion matrix of the model and its performance metrics are summarized in Figure 5.

Figure 5:

Download as PowerPoint

Examples of masses correctly classified by the CNN model are provided in Figure 6. In each example, the mass presented to the model demonstrates classic US features of lipomas (Fig 6, A), benign nerve sheath tumors (Fig 6, B), and vascular malformations (Fig 6, C). Examples of incorrectly classified masses are shown in Figure 7. Two images of benign nerve sheath tumors were misclassified as a lipoma (Fig 7, A) and vascular malformation (Fig 7, B). An image of a hemangioma was misclassified as a lipoma (Fig 7, C). All masses shown in Figures 6 and 7 were diagnosed histologically by the results of a biopsy or surgical excision.

Figure 6:

Download as PowerPoint

Figure 7:

Download as PowerPoint