Endotracheal Tube Position Assessment on Chest Radiographs Using Deep Learning

Study Design

This retrospective study was institutional review board–exempt and used de-identified Health Insurance Portability and Accountability Act–compliant radiographs and corresponding reports obtained from the picture archiving and communication system at Thomas Jefferson University Hospital, Philadelphia, Pa. A total of 22 960 de-identified frontal chest radiographs from 11 153 patients (average age, 60.2 years ± 19.9 [standard deviation], 55.6% men) were obtained consecutively by the use of natural language queries on a repository of chest radiology reports from Thomas Jefferson University Hospital between the dates of 2010 to 2018. The inclusion criteria for this study were reports and images in which a specific ETT-carina distance was mentioned in the report from the previously mentioned dates and patients 18 years or older. At our institution, radiologists commonly indicate a specific measurement (eg, “ETT is 4.6 cm above the carina”) in the report itself. Queries were developed to extract such reports and corresponding images in which mention of an ETT was made and with a specific distance measurement from the carina. There were 12 categories in total, including bronchial insertion, distance from the carina at 1.0-cm intervals up to 10 cm (0.0–0.9 cm, 1.0–1.9 cm, …, 9.0–9.9 cm), and 10 cm or greater. Images were randomly split into training (80%, 18 368 images), validation (10%, 2296 images), and test sets (10%, 2296 images) (Table 1, Fig 1). The images were evenly divided across training, validation, and test sets such that the relative percentage of images per interval categories were the same (Table 1). The previously mentioned test set of 2296 images was also referred to as the internal test set, derived from the same institution as the training data (Thomas Jefferson University Hospital, Philadelphia, Pa).

Table 1: Examinations and Patients per Category

Figure 1:

Download as PowerPoint

On the internal test dataset, ground truth ETT position was determined by ETT-carina distance as indicated in the original radiology report. A second board-certified radiologist practicing in the cardiothoracic division (P.L., 13 years of experience) also visually inspected all test images for quality assurance reasons to ensure presence of the ETT and whether the stated ETT-carina distance in centimeters was reasonable.

Radiograph Acquisition

Images were acquired from multiple different computed radiography (indicated as CR below) and digital radiography (indicated as DR below) manufacturers and models. This included the following systems: Agfa CR ADC Compact, Solo, 51xx series, Agfa CR 35, 75, and 85 series, and Agfa DR DX-G and DX-M series (Agfa Health Care, Carlstadt, NJ); Kodak CR 825, 850, 950, and 975 series (Carestream Health Care, Rochester, NY); Siemens DR Axiom-Multix (Siemens Healthcare, Erlangen, Germany), Canon DR CXDI (Canon Medical Systems USA, Tustin, Calif), Philips DR Digital Diagnost (Philips Healthcare, Andover, Mass), Carestream DR DRX-1, DRX-Evolution, DRX-Revolution, and DRX-G series (Carestream Health Care, Rochester, NY).

Interrater Agreement

On 100 random images from the internal test dataset (Fig 1), the ETT-carina distance was remeasured in centimeters by a board-certified radiologist (reader 1) practicing in the cardiothoracic division (P.L.) using a picture archiving and communication system workstation (Philips Intellispace 4.4; Koninklijke Philips, Amsterdam, the Netherlands) to assess interrater agreement. The measurement by reader 1 was compared with that stated in the original radiology report itself (original reader), which was interpreted by any one of the board-certified radiologists practicing in our department (six radiologists, 6–40 years of experience).

This same process was repeated for an external test dataset (Fig 1), which consisted of 100 consecutive chest radiographs obtained from a different hospital (Kennedy University Hospital, Cherry Hill, NJ) from March 30, 2020, to April 30, 2020, in which an ETT was present and the ETT-carina distance was specified in the report.

In both cases, reader 1 was blinded to the data in the original radiology report (original reader).

Image Preprocessing

Contrast-limited adaptive histogram equalization was performed on the original Digital Imaging and Communications in Medicine (DICOM) images. The images were then converted to 8-bit grayscale portable network graphics format. A top-center crop of the original DICOM image was performed (by dividing the original matrix size by 0.5 in both the x and y direction, and then adding 200 pixels in the x and 400 pixels in the y dimension). This reduced the dimensionality of the original image but preserved the carina and ETT for all images inspected on the test set. The cropped images were subsequently resized to 512 × 512 pixels.

Deep Learning Architecture and Training Details

The TensorFlow framework (TensorFlow 1.4, Google, Mountain View, Calif) and the Keras library (Keras version 2.12, https://keras.io) were used for network training. The Inception V3 deep convolutional neural network architecture was used for this study which consists of 48 layers (18). In the network, the last fully connected layers were set to random initialization of weights. The remainder of the layers were initialized using pretrained weights from ImageNet (9). The very last layer was set to a single node that predicted ETT-carina distance ranging from 0.5 cm (bronchial insertion) to 10.5 cm above the carina, with a linear activation function. The loss function was set to mean absolute error. Real-time augmentation during training consisted of rotation, sheer, horizontal flipping, and image translation. Training was performed for 200 epochs, using an Adam optimizer, and fixed learning rate of 0.0001, which achieved the lowest loss on the validation dataset after multiple experiments. The model with the lowest loss on the validation dataset was chosen for inference on the test datasets.

Model Inference on Test Data

The deep neural network was used to predict ETT distance from the carina in centimeters. Test-time augmentation was employed during inference on the test datasets (random rotation, sheer, horizontal flipping, and image translation) for 10 different times per image, and the average of these predictions was used for the final classification.

Statistical Analysis

For assessing the interrater agreement for both internal and external datasets, the mean absolute differences and standard deviations were calculated for the ETT-carina distances for reader 1 compared with the original reader and for the AI-generated result compared with each reader. Intraclass correlation coefficients (ICCs) and corresponding 95% CIs were calculated per the methods outlined by Koch (19). For the qualitative interpretation of ICC interrater agreement measures, we considered less than 0.40 as poor, 0.40–0.59 as fair, 0.60–0.74 as good, and more than 0.75 as excellent, per guidelines used by Cicchetti (20).

A Bland-Altman plot was constructed for AI versus reader 1 and the original reader, and between reader 1 and the original reader (Fig 2), for both internal and external test datasets (18).

Figure 2:

Download as PowerPoint

Two thresholds were defined as low positions of the ETT: less than 1 cm and less than 2 cm for ETT-carina distance. A threshold of 7 cm or greater for ETT-carina distance was defined as high ETT position. Model performance in assessing low and high ETT positions is described using confusion matrices (Figs 3, 4), sensitivity, and specificity on the entire internal test data of 2296 images (Table 2). The statistical analysis was performed using R (version 3.6.2, R Foundation, Vienna, Austria). Sensitivity and specificity analysis of the thresholds was not performed on the external test set because of low sample size of the malpositions (11 low positions and six high positions of the 100 total images).

Figure 3:

Download as PowerPoint

Figure 4a:

Download as PowerPoint

Figure 4b:

Download as PowerPoint

Figure 4c:

Download as PowerPoint

Table 2: Model Performance for Assessing Low and High ETT on Internal Test Set

Additional Data Analysis and Visualization

Class activation maps were created using the methods by Zhou et al (21) to inspect the areas of the image that were activated by the network (Fig 5).

Figure 5a:

Download as PowerPoint

Figure 5b:

Download as PowerPoint

Figure 5c:

Download as PowerPoint

ETT to Carina Distance Comparisons

The predicted ETT-carina distance for the AI system had mean absolute differences of 0.69 cm ± 0.70 on the internal test set and 0.63 cm ± 0.55 on the external test set compared with the average radiologist (Table 3). The radiologists themselves had mean absolute differences of 0.44 cm ± 0.44 and 0.63 cm ± 0.67 from each other on the internal and external datasets, respectively (Table 3).

Table 3: ETT-Carina Distance Mean Absolute Differences: Radiologists and AI

On the internal and external datasets, respectively, the ICCs of AI and radiologists were 0.84 (95% CI: 0.78, 0.92) and 0.89 (95% CI: 0.77, 0.94); the ICCs of the radiologists were 0.93 (95% CI: 0.90, 0.95) and 0.84 (95% CI: 0.71, 0.90) (Table 4).

Table 4: Interrater Assessment Between Radiologists and AI

The Bland-Altman plot (Fig 2) shows that the mean difference between the radiologists (average of reader 1 and original reader) and AI was −0.22 cm. The upper limit of agreement (+1.96 standard deviation) was +1.55 cm, and the lower limit of agreement (−1.96 standard deviation) was −1.98 cm.

Confusion Matrices and Class Activation Maps

The confusion matrix comparing the ground truth ETT-carina distance to that of the AI system on the full internal test data are provided in Figure 3. Additional confusion matrices at different cutoffs for low ETT (< 1 cm and < 2 cm) and high ETT (≥ 7 cm) are shown in Figure 4. Class activation maps for sample test images are shown in Figure 5.

Sensitivity and Specificity of AI Determination of ETT to Carina Distance

On the internal test data, the sensitivity and specificity for the two different low (ETT-carina distances of < 1 cm and < 2 cm) and high (≥ 7 cm) thresholds are shown in Table 2. For differentiating ETT-carina distance of less than 1 cm from all others, the sensitivity was 93.9% (95% CI: 90.0, 96.7) and specificity 97.7% (95% CI: 96.9, 98.3); for differentiating ETT-carina distance of less than 2 cm from all others, the sensitivity was 90.1% (95% CI: 86.6, 92.9) and specificity 92.4% (95% CI: 91.1, 93.5); for differentiating ETT-carina distance of greater than or equal to 7 cm from all others, the sensitivity was 66.5% (95% CI: 59.9, 72.2) and specificity 99.2% (95% CI: 98.6, 99.6).