AI Improves Nodule Detection on Chest Radiographs in a Health Screening Population: A Randomized Controlled Trial

Participants and Trial Design

We conducted a pragmatic, randomized controlled trial at a health screening center affiliated with a tertiary referral hospital. All individuals who visited the center and underwent chest radiography for health checkup purposes between June 2020 and December 2021 were enrolled and randomized into either the intervention arm (AI group) or control arm (non-AI group) at a 1:1 ratio. Individuals aged 18 years or younger were excluded. For those in the AI group, the designated radiologists interpreted chest radiographs aided by AI-based CAD, while for those in the non-AI group, the radiologists interpreted chest radiographs without AI-based CAD results (Fig 1). Most chest radiographs were interpreted by one of three designated board-certified radiologists (E.H.L., H.J.K., and M.N., with 36, 27, and 13 years of experience in chest radiography reading, respectively) at our health screening center. The radiologists, physicians in the health screening center, and outcome assessors were aware of the allocation (single blinded).

Figure 1:

Download as PowerPoint

Baseline characteristics including age; sex; smoking status; history of lung cancer, other malignancy, lung surgery, and pulmonary tuberculosis; family history of lung cancer; and comorbidities (hypertension, diabetes mellitus, dyslipidemia, and chronic hepatitis) were collected using a self-reported health questionnaire (Fig S1, Appendix S1). For those who underwent chest CT within 3 months after chest radiography, CT images were reviewed, and the presence of actionable lung nodules was determined (Appendix S2) (21–23). For participants who underwent pathologic evaluation, the pathology reports were investigated. All radiologic and clinical information was recorded using an electronic case report (Appendix S1).

Chest Radiographs

All chest radiographs were obtained in the posteroanterior projection without lateral views using INOVISION-EXII (Dong Kang Medical Systems) (tube voltage, 120 kVp; tube current, 320 mA; variable exposure time, typically 15–18 msec). For some participants who underwent two chest radiographic examinations during the trial, only the latter images were used in the analyses.

CT Protocol

Most CT scans were obtained with low-dose protocol CT without contrast material enhancement. The median volume CT dose index and dose-length product were 0.52 mGy (IQR, 0.43–2.00 mGy) and 20.6 mGy · cm (IQR, 16.5–66.5 mGy · cm), respectively (Appendix S3).

AI-PACS: AI-based CAD Implementation and Randomization

We used an AI-based CAD–implemented PACS (AI-PACS) devised for this trial. We embedded a commercial AI-based CAD software (Lunit INSIGHT CXR version 2.0.2.0; Lunit) approved by the Ministry of Food and Drug Safety of Korea (4–6) into a commercial PACS (M6; Infinitt Healthcare). When each chest radiograph was acquired, the image was immediately allocated a unique five-digit random number and randomized into the AI or non-AI group. In the AI group, the AI-PACS overlaid the results of AI-based CAD onto the chest radiograph, whereas the AI-based CAD results were blinded in the non-AI group (Fig 2). The AI-based CAD system analyzed frontal chest radiographs to assess the possibility of having major thoracic abnormalities (ie, pulmonary nodule, pneumonia, and pneumothorax) on a percentage scale (0%–100%) and localized them as overlaid heat maps (Fig 3). The radiologists used a structured reporting system implemented in AI-PACS (Fig S2) to interpret all chest radiographs from both groups.

Figure 2:

Download as PowerPoint

Figure 3:

Download as PowerPoint

Clinical Outcomes

The primary outcome of our trial was the detection rate of actionable lung nodules on chest radiographs. Actionable lung nodules were defined as solid nodules larger than 8 mm or subsolid nodules with a solid portion larger than 6 mm in average diameter measured on two axes (Lung Imaging Reporting and Data System, or Lung-RADS, category 4), regarding their clinical significance and visibility on chest radiographs (21). The detection rate was defined as the number of true-positive chest radiographs divided by the total number of chest radiographs.

The secondary outcomes included the false-referral rate of chest radiographs, positive-report rate of chest radiographs, rate of performing chest CT, positive rate for actionable lung nodules on chest CT scans, rate of performing pathologic evaluations, detection rate of malignant lung nodules on chest radiographs, detection rate of lung cancer on chest radiographs, positive rate of malignant lung nodules, and positive rate of lung cancer. Definitions are listed in Appendix S4.

Sample Size Estimation

We conducted a preliminary analysis using a retrospective cohort of 3073 individuals who underwent chest radiography and chest CT in our health screening center (16). In this sample, the estimated detection rates of actionable lung nodules were 1.73% and 1.49% with and without AI-based CAD, respectively. Assuming a two-tailed test with a type I error of .05 and power of 80%, the estimated sample size to determine the difference in detection rates was 83 552. We planned to stop the trial within 18 months (expected sample size, 8000–10 000), considering (a) continuous improvements in AI-based CAD performance, (b) policies of the research grant, and (c) expected enhanced effect of AI-based CAD in real clinical practice with low disease prevalence.

Statistical Analyses

Primary and secondary clinical outcomes were compared between the two groups using the χ² test, Fisher exact test, or Wilcoxon rank sum test, and the impact of the intervention (use of AI-based CAD) on primary and secondary clinical outcomes was evaluated using univariable logistic regression analyses. Subgroup analyses were conducted to evaluate the interactions of confounding factors on the clinical outcomes. Age; sex; smoking status; history of lung cancer, other malignancy, lung surgery, and pulmonary tuberculosis; presence of prior chest radiographs; and the reporting radiologist were included as potential confounders. The subgroup effects of the confounding factors were tested by the significance of the interaction term added to the regression model. The Firth correction was applied when the number of events was relatively too low (24). We also performed multivariable logistic analyses to assess the associations of baseline characteristics of the participants with the performance of chest CT and positive-report rates of chest radiographs, respectively, and the association of nodule characteristics with the sensitivity of chest radiography. All analyses except those for the primary outcome were considered exploratory. Statistical analyses were conducted using SAS 9.4 (SAS Institute) by statisticians with 10 and 15 years of experience (N.P. and J.K., respectively). P < .05 was considered to indicate a statistically significant difference.

Participant Characteristics

In total, 10 478 participants were enrolled in AI-PACS, and two participants aged 18 years or younger were excluded. In the final analysis, 10 476 participants (median age, 59 years [IQR, 50–66 years]; 5121 men) were allocated into the AI (n = 5238) and non-AI (n = 5238) groups (Fig 1). In the overall cohort, 11% (1138 of 10 476 participants) and 25% (2633 of 10 476 participants) were current and former smokers, respectively. Of the 10 476 participants, 0.6% (n = 59) had a history of lung cancer and 10% (n = 1038) had a history of other malignancy (Table 1). Other information is summarized in Table 1, and there was no evidence of differences in baseline characteristics between the two groups (P = .15–.91).

Table 1: Baseline Characteristics of the Participants

The chest radiographs were reported to contain nodules in 2% of the 10 476 participants (n = 222). Older age (P < .001), a history of lung cancer (P = .002) or pulmonary tuberculosis (P < .001), and the absence of a prior chest radiographic examination (P = .001) were associated with positive results, and the positive-report rates differed among the reporting radiologists (P < .001) (Table S1). Of the 10 476 participants, 47% (n = 4886) underwent chest CT within 3 months after chest radiography and 0.3% (n = 30) underwent pathologic evaluation of lung nodules. Participants who were aged 65–74 years (P < .001), male (P = .004), and former or current smokers (P < .001) and those who did not have a prior chest radiograph (P = .001) were more likely to undergo chest CT (Table S2).

Clinical Outcomes

The detection rate for actionable lung nodules on chest radiographs was higher in the AI group than in the non-AI group (0.59% [31 of 5238 participants] vs 0.25% [13 of 5238 participants], respectively; odds ratio [OR], 2.4; 95% CI: 1.3, 4.6; P = .008) (Table 2), accomplishing the primary outcome of the trial (Table 1). The positive-report rate of chest radiography showed no evidence of a difference between the AI group (2.3% [122 of 5238 participants]) and non-AI group (1.9% [100 of 5238 participants]; OR, 1.4; 95% CI: 0.94, 1.6; P = .14). Among the 222 participants with positive chest radiographs, the AI and non-AI groups showed a similar false-referral rate (45.9% [56 of 122 participants] vs 56.0% [56 of 100 participants], respectively; OR, 0.67; 95% CI: 0.39, 1.1; P = .14). Representative images are presented in Figures 3 and 4.

Table 2: Summary of Analyses for the Primary and Secondary Outcomes

Figure 4:

Download as PowerPoint

However, because chest CT was performed for 47% of the participants regardless of the chest radiography results (mostly for health checkup purposes), considerable actionable lung nodules were found initially at CT. There was no evidence of a difference in CT performance between the AI group (46.3% [2425 of 5238 participants]) and non-AI group (47.0% [2461 of 5238 participants]; OR, 0.97; 95% CI: 0.90, 1.1; P = .48). In the overall cohort, 1.1% (111 of 10 476 participants) showed actionable lung nodules at chest CT, and the positive rate of actionable lung nodules was similar between the AI group (1.1% [55 of 5238 participants]) and the non-AI group (1.1% [56 of 5238 participants]; OR, 0.98; 95% CI: 0.68, 1.4; P = .92). When the diagnostic performance of chest radiography was evaluated among the participants who underwent chest CT (n = 4886), the AI group showed higher sensitivity (56.4% [31 of 55 participants] vs 23.2% [13 of 56 participants], respectively; P < .001), positive predictive value (35.6% [31 of 87 participants] vs 18.8% [13 of 69 participants]; P = .02), and negative predictive value (99.0% [2314 of 2338 participants] vs 98.2% [2349 of 2392 participants]; P = .03) in detecting actionable lung nodules (Table 3).

Table 3: Diagnostic Performance of Chest Radiography for Detecting Actionable Lung Nodules in the 4886 Participants Who Underwent Chest CT

In the overall cohort, 0.28% of participants (30 of 10 476) underwent pathologic evaluation for lung nodules (22 by means of surgical resection and eight by means of bronchoscopic or percutaneous needle biopsy), where the proportions were similar between the AI group (0.34% [18 of 5238 participants]) and non-AI group (0.23% [12 of 5238 participants]; OR, 1.5; 95% CI: 0.72, 3.1; P = .28). Twenty-four participants were diagnosed with malignant lung cancer (20 adenocarcinomas, one squamous cell carcinoma, one adenosquamous carcinoma, and two minimally invasive adenocarcinoma) and four with other malignancies (two metastasis, one atypical carcinoid, and one lymphoma). The detection rates of malignant lung nodules on chest radiographs (0.15% [eight of 5238] in the AI group vs 0% [0 of 5238] in the non-AI group; P = .008) and lung cancer on chest radiographs (0.11% [six of 5238] vs 0% [0 of 5238]; P = .03) were higher in the AI group; however, there was no evidence of a difference at logistic regression analyses (OR, 17.0 [95% CI; 0.98, 295.1; P = .05] and 13.0 [95% CI: 0.73, 231.1; P = .08], respectively) (Table 2).

Subgroup Analyses

The impact of AI-based CAD on the detection of actionable lung nodules on chest radiographs was consistent across the subgroups (Table 4). There was no evidence of significant interactions for age; sex; smoking status; history of lung cancer, other malignancy, lung surgery, and pulmonary tuberculosis; the presence of prior chest radiographs; and the reporting radiologists in detecting actionable lung nodules on chest radiographs (P = .05–.98) (Table 4). Specifically, age and sex showed P values of .06 and .05, respectively. None of the aforementioned factors showed evidence of significant interactions with AI-based CAD in detecting malignant lung nodules on chest radiographs (P > .05 for all) (Table S3) or performing CT (P > .05 for all) (Table S4). The sensitivity of chest radiography was not affected by nodule characteristics including size, location, and overlapping structures (Table S5).

Table 4: Subgroup Analyses for the Primary Outcome