Deep Learning for the Detection, Localization, and Characterization of Focal Liver Lesions on Abdominal US Images

Study Design

Institutional review board approval (no. IRB00011591) was obtained for this retrospective, multicenter study, and informed consent was waived. Data for this pilot study were obtained in collaboration with a clinical data warehouse registered under the number 1980120. US images were extracted from the picture archiving and communication system (PACS) of two university hospitals, center 1 (Necker Hospital, Paris) and center 2 (Saint Louis Hospital, Paris). Only adult patients aged 18 years and older were selected. Patients with liver parenchyma containing lesions were included if they met the following criteria: Lesions were visible on the US images as decided unanimously by an adjudication panel, patients had not previously received local therapy, and a definite pathologic diagnosis of lesions was obtained (cyst, angioma, focal nodular hyperplasia, adenoma, metastasis, or hepatocellular carcinoma). Patients without lesions in the liver parenchyma were selected in case of a definite absence of pathologic diagnosis. All images from abdominal US performed at centers 1 and 2 between 2014 and 2019 were selected. For the training and development set, images from examinations performed between 2014 and 2018 were selected. For the test set, images from examinations performed in 2019 were selected, provided that the corresponding patients had not undergone any examination between 2014 and 2018. Extremely magnified images and images obtained using degraded mode simultaneously with contrast-enhanced US images were excluded.

Data Acquisition

Data collected retrospectively in center 1 consisted exclusively of multivendor US images with FLLs (four vendors). Data collected retrospectively in center 2 consisted of multivendor images of healthy liver parenchyma and FLLs (two vendors).

Data Preprocessing

The Digital Imaging and Communications in Medicine (DICOM) red-green-blue images extracted from the PACS were converted to JPEG format. They were processed using a de-identification tool to remove identifying data and metadata. This tool uses the DICOM US attributes to remove the upper band of the image before converting it to JPEG format. A parametric method was used to detect the US fan area and remove annotations and other machine-dependent characteristics outside this region (11). In addition, biometric measurements present inside the region were removed using existing inpainting methods (Fig 1) (11). This enables networks to be trained on raw data with no manually added annotations by examiners and therefore may be more suitable for real-time use by nonexpert examiners.

Figure 1:

Download as PowerPoint

Determination of the Ground Truth

For each center, the diagnosis associated with each US image was assigned by two radiologists (J.M.C., Pierre Bourrier, MD), each with more than 15 years of experience in US image interpretation, by cross-referencing with other imaging modalities (contrast-enhanced US, CT, or MRI, or biopsy, when available) and clinical files. The final diagnosis was used for the characterization task (liver parenchyma and characterization of FLLs) and did not include the number of lesions in the US image or their localization. To determine the ground truth boxes for the localization task (ie, boxes around the liver and FLLs), an adjudication panel was used as an external standard of reference. The panel consisted of four physicians (including A.L.R., C.d.M.M., and Mariama Bah, MD) who either are radiologists or hold a national diploma in US imaging and four sonographers (including K.K.) who hold a national diploma in US imaging from six different health institutions and who each have more than 3 years of experience. The annotators, who worked on a custom annotation platform, were first asked to localize the liver and classify it as a “homogeneous liver” (ie, without lesions) or as a “liver with lesion(s)” with respect to the final diagnosis associated with the image. The selection of liver with lesion(s) led to the second task (ie, localization of lesions). Each lesion was required to be classified in accordance with the final diagnosis, with two possibilities: benign lesion(s) or malignant lesion(s). Benign lesions were further subclassified as cyst, angioma, focal nodular hyperplasia, or adenoma. Malignant lesions were further subclassified as metastasis or hepatocellular carcinoma. Examples of such annotations are shown in Figure 2. Each image was annotated by two experts and at least once by a physician. During this phase, in case of a disagreement between two annotators for the localization task, four additional annotators analyzed the questionable image. If annotation of the image was not unanimous between the additional annotators, the image was excluded from the study.

Figure 2:

Download as PowerPoint

Data Partitions

A total of 1026 patients (n = 2551 images) met the inclusion criteria for the training and development set. This set was randomly split into two subsets containing approximately the same proportions for each class, with 80% and 20%, respectively, for training and development. The objective of the development set is to choose the network settings that achieve the best performance on images the network has never seen. A total of 48 additional patients (n = 155 images) met the inclusion criteria for the test set. All images and clinical reports were de-identified within the centers, and no demographics information from the study population was retained.

Models.—Two object detection networks were used. Detection Transformer (DETR) is an end-to-end object detection vision transformer network (12) and is more suitable for real-time application (13). Faster region-based convolutional neural network (Faster R-CNN) is a two-stage object detection network that showed more robust performance on natural image datasets (14). Figure 3 compares the workflows of both networks. The networks were trained for 100 epochs, after which the validation loss stopped decreasing and the overfitting occurred. Stochastic gradient descent with Nesterov momentum (15) was used for optimization (Appendix E1 [supplement]), with a learning rate of 0.0048, a batch size of 4, a momentum of 0.9, and a weight decay of 0.0001. For the first epoch, a warm-up schedule was applied (16). For the rest of the training, the learning rate of each parameter group was decreased by 0.9 once the number of epochs reached 75 and 90.

Figure 3:

Download as PowerPoint

Data augmentation.—To improve the performance of the networks, a data augmentation strategy based on the Google Brain team bounding box augmentation policies was applied (17). The data augmentation scheme is based on a learned set of specific subpolicies that were proven to improve generalization performance (Appendix E1 [supplement]). These subpolicies consist of a list of intensity, geometric, and bounding box operations (rotation, cutout, sharpness, translation, contrast, brightness, solarization, shear) applied to the image or to a specific object inside the bounding box. During training, one of those policies is randomly selected and then applied to the current image as reported by Zoph et al (17).

Model evaluation.—Performances of all raters (expert and nonexpert caregivers, networks) were compared against the ground truth. The nonexpert, an emergency physician, had 5 years of experience with US imaging, both acquiring and interpreting US images. The experts, a radiologist (expert 1, S.B.) and an advanced practice sonographer (expert 2, F.J.), each had a national degree in US imaging with 9 and 8 years of experience, respectively. All three caregivers and the networks were blinded to clinical history and reports. Performances were evaluated in detail with regard to the capability to detect liver parenchyma with FLLs, localize FLLs using bounding boxes, regardless of the label assigned to them, characterize the FLLs correctly localized by all raters as either benign or malignant, and characterize FLLs into benign and malignant subcategories. For each of these tasks and for all raters, the positive predictive value (PPV) (Eq [1]) or the sensitivity (Eq [2]) and the specificity (Eq [3]) are reported. For the localization task, the F1 score (Eq [5]) is reported as well. For binary classification (ie, tasks 1 and 2), the accuracy (Eq [4]) and the Matthews correlation coefficient (MCC) (Eq [6]) metrics are reported as well. For multiclass classification, the macro specificity, sensitivity, and accuracy are used instead. The formulas of all metrics are defined as follows:

To report all of these metrics, a class-specific threshold was set to select the probabilities output by the networks; the threshold was defined as the value that maximizes the F1 score of each class during the validation step. In this setting, a prediction (box + label) is considered a TP if it has an intersection over union score with a ground truth box greater than 0.3 and the same label as the ground truth box and an FP otherwise. Ground truth boxes with no intersections with predicted boxes are FNs.

Statistical Analysis

Because images in the same patient are not independent, bootstrapping experiments were used, as described by Linnet (18). The test set was resampled to contain only one randomly selected image per patient, and the inference was repeated 1000 times. At each iteration, the resampled test set had the same size as the patient population size (48 patients, 48 images). For accuracy, specificity, sensitivity, PPV, F1 score, and MCC values, we report the mean value across the observations, as well as 95% CIs computed using the 2.5 and 97.5 percentiles of the ranked observations. The sign test was used to report the significant results for all metrics. Given multiple comparisons, the Bonferroni correction method (the six classes are cyst, angioma, focal nodular hyperplasia, adenoma, metastasis, and hepatocellular carcinoma) was used to adjust P values (19), and a P value less than .05 was considered statistically significant.

Data Availability

All code used for this study is publicly available at https://gitlab.inria.fr/hdadoun/focal-liver-lesions-us.

Detection of Lesions in the Liver Parenchyma

Table 1 shows performance in detecting liver parenchyma with or without FLLs in terms of accuracy, specificity, sensitivity, and MCC. The nonexpert caregiver achieved an accuracy of 78% (95% CI: 70, 86), expert 1 achieved a slightly higher accuracy of 80% (95% CI: 74, 86), and expert 2 had the best accuracy score overall, achieving 99% (95% CI: 98, 100). Faster R-CNN achieved an accuracy of 93% (95% CI: 88, 98), and DETR achieved an accuracy of 96% (95% CI: 92, 98). Overall, the networks were consistent in their predictions, with an MCC of 81% (95% CI: 66, 95) for Faster R-CNN and 88% (95% CI: 77, 95) for DETR. Overall, we found no difference between expert 2 and DETR in terms of accuracy, specificity, and sensitivity. In addition, both expert 2 and DETR showed higher accuracy and sensitivity compared with the nonexpert caregiver and expert 1 (P < .001).

Table 1: Detection of Liver Parenchyma with or without Focal Liver Lesions: Performance by Rater

Localization of Lesions

Table 2 shows performance in the localization of FLLs in terms of PPV, sensitivity, and F1 score. Because there are no TNs for this task, accuracy and specificity are not reported. Expert 2 and the networks achieved comparable results for this task, with a mean PPV of 76% (95% CI: 72, 79) and mean sensitivity of 78% (95% CI: 74, 83). Expert 1 achieved a PPV of 73% (95% CI: 63, 84) and sensitivity of 69% (95% CI: 62, 77). This result was consistent with the detection accuracy of liver parenchyma with FLLs, wherein the performance of expert 1 was lower than that of expert 2 and the networks.

Table 2: Localization of Focal Liver Lesions: Performance by Rater

Characterization of Lesions

To report the characterization performance, a subset of lesions that were detected and localized both by the networks and the experts was selected. Table 3 shows performance in the characterization of FLLs in terms of accuracy, specificity, sensitivity, and MCC. DETR achieved the highest accuracy, 81% (95% CI: 68, 94), compared with 61% (95% CI: 50, 71) for the best-performing expert (expert 2). The accuracy achieved by expert 2 was significantly different from that of the networks (P < .001) but not from that of expert 1 (P = .25), while the accuracy achieved by DETR was significantly different from both experts but not from Faster R-CNN (P = .18). Expert 2 achieved the highest sensitivity among all raters (87% [95% CI: 73, 100]) and the lowest specificity (33% [95% CI: 25, 44]). DETR achieved the second highest sensitivity (82% [95% CI: 62, 100]) and the highest specificity (81% [95% CI: 67, 91]) among all raters. Both networks also achieved a higher MCC with the true labels compared to experts: 63% (95% CI: 37, 88) for DETR compared to 25% (95% CI: 4, 50) for expert 2.

Table 3: Characterization and Subcharacterization of Focal Liver Lesions: Performance by Rater

Subcharacterization

The subcharacterization performance on the subset of lesions that were detected and localized both by the networks and the experts is also reported in Table 3. DETR achieved the highest accuracy, with 76% (95% CI: 62, 91) compared with 52% (95% CI: 36, 70) for the best-performing expert (expert 1). All raters (experts and networks) showed a comparable specificity for the subcharacterization task, with a mean value of 91% ± 3 (SD) and a mean sensitivity of 59% ± 10. Pairwise comparisons for this task did not show significant differences between all raters (experts and networks).

In summary, for the detection of lesions in the liver parenchyma, no significant difference was found between the best-performing network (DETR) and the best-performing expert (expert 2) in the small test set; for the localization of FLLs, all raters achieved comparable results; for the characterization of FLLs detected and localized by all raters, both networks achieved higher performance compared with both experts; and for the subcharacterization of benign and malignant FLLs, pairwise comparisons between raters were not significantly different. More details on the discrimination performance of the networks are reported in Appendix E1 (supplement).