Determinants of Chest Radiography Sensitivity for COVID-19: A Multi-Institutional Study in the United States

Introduction

In December 2019, the first outbreak of COVID-19 was reported in China in Wuhan City, Hubei Province (1). It rapidly achieved pandemic status, and the United States currently has the highest number of cases in the world. In the United States, the medical community has been primarily focused on controlling viral spread through mitigation measures (2). Until favorable developments occur from drug discovery and vaccine development efforts, rapid and sensitive detection of COVID-19 infection is essential for triage of patients, both for timely isolation of patients confirmed positive for COVID-19 and for appropriate allocation of hospital resources and providers in hospital units with limited capacity.

The reference standard for diagnostic confirmation of COVID-19 infection in patients is a positive result from reverse-transcription polymerase chain reaction, but the time to result from laboratories (which may be as long as several days) has been a rate-limiting step in hospital care. As a result, imaging-based diagnostic tools are attractive as a companion option, with some reports suggesting high sensitivity of COVID-19 detection with CT studies (3,4). However, questions have been raised about the reported high sensitivity of CT in those studies, which likely suffer from selection bias toward more severe cases (5). Some early reports have also described high specificity in the diagnosis of COVID-19 compared with other entities, but these too may be biased by selection of more severe cases and, at least in one case, an artificial control group of other viral pneumonias (6,7). In addition, the ability to detect COVID-19 pneumonia at imaging likely varies over time but remains to be fully elucidated (8).

Here we evaluate the imaging characteristics, sensitivity, and specificity of both chest radiography and chest CT in detecting COVID-19 infection in patients with laboratory-confirmed SARS-CoV-2 viral infection over time. We also evaluate the contributions of patient demographics and comorbidities that influenced false-negative findings at imaging.

Materials and Methods

Patient Selection

This retrospective study was conducted at two large urban academic medical centers, including three tertiary care hospitals and one community hospital. The respective institutional review boards at the two academic medical centers approved the study, with waiver of informed consent. Inclusion criteria were patients who had undergone both at least one nucleic acid amplification–based COVID-19 detection test and at least one chest radiography or CT examination that was performed within a week of testing. Patients younger than 18 years of age were excluded. A patient was regarded as positive for COVID-19 if they had any positive test within the time interval. At the first institution, 182 patients positive for COVID-19 infection from March 1, 2020, through April 1, 2020, were randomly selected and then paired with age- and sex-matched control patients who tested negative for COVID-19. At the second institution, 72 patients who were positive for COVID-19 infection between March 1, 2020, through April 14, 2020, and age- and sex-matched control patients were then included in a similar process.

Image Review

At each site, two board-certified, fellowship-trained thoracic radiologists reviewed each study separately. At the first institution, images were reviewed by two of the thoracic radiologists (M.M.H., A.R.H., F.L.J., H.H., and S.C.B., who each had between 5 and 29 years of experience). At the second institution, R.A.H., with 4 years of experience, and C.W., with 29 years of experience, reviewed the studies. Reviewers were blinded to the diagnosis.

Chest Radiograph Evaluation

Reviewers evaluated for the predominant pattern on chest radiographs, selecting from interstitial opacities, interstitial and airspace (IA) opacities, atelectasis, diffuse airspace (DA) opacities, lobar consolidation, or peripheral opacities; if none of these patterns applied, the reviewers would select “normal”. Reviewers also assessed for a craniocaudal gradient and the presence of pleural effusions. Reviewers were asked to assign a severity score for the chest radiography findings: normal, mild, moderate, or severe. Reviewers also assessed the likelihood of COVID-19 on the basis of the radiography findings, with a score between 1 and 5 (1 being “very unlikely,” 3 being “intermediate likelihood,” and 5 being “highly likely”). Up to three serial chest radiographs were scored per patient.

Chest CT Evaluation

Reviewers were asked to look for the presence of nodules, ground glass opacities, consolidation, and septal thickening. The presence of pleural effusions and reversed halo sign were also assessed. Additionally, reviewers evaluated craniocaudal gradient, anterior/posterior gradient, and central/peripheral gradient; reviewers could score cases as having a gradient or diffuse involvement. The reviewers were asked to assign a chest CT severity score and a chest CT COVID-19 likelihood score.

Patient Demographics

Patient sex, age, ethnicity, height, and admission weight were collected. Patient history was reviewed for positive history for hypertension, diabetes mellitus, cardiac disease or anomaly, chronic lung disease, autoimmune disease, malignancy, stem cell transplant, solid organ transplant, or chronic kidney disease.

Data Entry and Statistics

Data were entered using the Castor EDC platform (https://www.castoredc.com/) (9) and the REDCap platform (https://projectredcap.org/) (10). Graphpad Prism 5.0 and Graphpad Prism 8.4.2 (Graphpad Software) as well as JMP (v15, SAS Institute) statistical software packages were used for formal analysis of data.

To combine data from the two reviewers, for craniocaudal gradient, anterior/posterior gradient, and central/peripheral gradient, diffuse distribution was assigned to a case if (a) at least one reviewer scored diffuse or (b) if both reviewers selected opposing categories (for example, if reviewer 1 selected anterior distribution and reviewer 2 selected posterior distribution). Chest radiographs and chest CT scans were characterized as having normal patterns only if both reviewers characterized them as normal. If the two reviewers selected different abnormal patterns, both patterns were scored for purposes of analysis; if one reviewer selected an abnormal pattern while the other selected normal, then only the abnormal pattern was scored. For severity and COVID-19 likelihood scores, the scores were averaged for the two reviewers.

For purposes of sensitivity and specificity, a chest radiograph or CT scan scored as 3 (intermediate COVID-19 likelihood) or above in the COVID-19 likelihood score was classified as a positive test.

The time intervals between symptom onset and chest radiography were evaluated, and the quartiles were used to generate time interval bins for analysis of chest radiograph and CT scan changes over time. Comparisons of categorical variables were done with one-tailed Fisher exact test for 2 × 2 tables or the χ² test for variables with multiple levels. Trend analysis of categorical variables across time interval bins was done using the Cochran-Armitage test. Comparison of continuous variables across time interval bins was done with analysis of variance. Multivariable logistic regression analysis was performed to evaluate factors associated with false-negative findings on chest radiographs. A P value threshold of .05 was used for statistical significance. To adjust for multiple comparisons, we applied the Benjamini-Hochberg correction to the P values for each table of univariable analyses, using a false-discovery rate of 0.05, and we named the P values “adjusted P value” in these instances.

CIs and standard errors of the areas under the receiver operating characteristic curves (AUCs) were obtained using bootstrapping with 2500 samples. The comparison of paired AUCs was performed using a previously described method (11).

Results

Patient Characteristics

A total of 508 patients were included, 254 of whom tested positive for COVID-19, with 254 matched COVID-19-controls. The demographics and clinical characteristics of the cohort are given in Table 1. Five hundred (98%) of the patients underwent chest radiography at least once, 195 (38%) underwent chest radiography at least twice, 112 (22%) underwent chest radiography at least three times, and 169 (33%) underwent at least one chest CT. Chest radiographs were obtained at the following time intervals after onset of symptoms: 254 (32%) at ≤ 2 days, 192 (24%) at 3–6 days, 183 (23%) at 7–11 days, and 177 (22%) at > 11 days. The median time from symptoms to CT was 4 days. The available clinical indications for chest radiography and CT between patients positive for COVID-19 and those negative for COVID-19 are listed in Tables E1–E4 (supplement).

Chest Radiograph Evolution over Time

The severity of chest radiograph scores increased over time for patients confirmed positive for COVID-19 (from mean of 1.11 at ≤ 2 days to mean of 1.97 at > 11 days, P < .001, Fig 1, A). However, the severity scores for patients without COVID-19 did not change significantly over time (from mean of 0.93 at ≤ 2 days to mean of 1.10 at > 11 days, P = .21).

Figure 1:

Download as PowerPointOpen in Image Viewer

The most common chest radiography finding in patients positive for COVID-19 was IA opacities (IA pattern), present in 62% of cases. IA pattern increased over time, present in 51% at ≤ 2 days versus 73% at > 11 days (P < .001 for trend, Fig 1, B). Chest radiograph disease distribution also changed over time, with increasing prevalence of diffuse opacities (3% at ≤ 2 days vs 25% at > 11 days, P < .001 for trend, Fig 1, C). Examples of evolving chest radiography in patients positive for COVID-19 are given in Figure 2.

Figure 2:

Download as PowerPointOpen in Image Viewer

Sensitivity of Imaging over Time

The rise of chest radiograph severity over the course of COVID-19 infection was reflected in the sensitivity of chest radiography COVID-19 detection over time (Fig 3, A). The sensitivity of chest radiography at ≤ 2 days is 55%, increasing to 79% at > 11 days (P < .001 for trend). Specificity declined slightly over time, 83% at ≤ 2 days to 70% at > 11 days (P = .02 for trend).

Figure 3:

Download as PowerPointOpen in Image Viewer

We evaluated sequential imaging tests per patient for diagnosis of COVID-19. The first set of chest radiographs (chest radiograph 1) was obtained a median of 3 days after symptoms, the second set of chest radiographs (chest radiograph 2) was obtained a median of 8 days after symptoms, and CT scans were obtained a median of 4 days after symptoms. In patients with both chest radiograph 1 and chest radiograph 2 (n = 195), AUCs were 0.80 for chest radiograph 1 and 0.871 for chest radiograph 2. In patients who had both chest radiograph 1 and a CT study performed (n = 160), chest radiograph 1 AUC was 0.80 and CT AUC was 0.90. In patients who had chest radiograph 1, chest radiograph 2, and CT (n = 85), AUC was 0.79 (95% CI: 0.67, 0.88) for chest radiograph 1, 0.88 (95% CI: 0.79, 0.93) for chest radiograph 2, and 0.92 (95% CI: 0.84, 0.97) for CT (Fig 3, B). Chest radiograph 2 and CT AUCs were both significantly higher than the chest radiograph 1 AUC (P = .02 and P = .003, respectively), whereas comparison of chest radiograph 2 and CT AUCs found no significant difference (P = .11). In this group, chest radiograph 1 had a sensitivity of 73% and specificity of 80%, chest radiograph 2 had a sensitivity of 83% and specificity of 73%, and CT had a sensitivity of 88% and specificity of 77%.

Specific Chest Radiograph 1 Findings for COVID-19

Frequencies of chest radiograph 1 findings in COVID-19 cases and controls are shown in Table 2 (with frequencies from total chest radiography shown in Table E5 [supplement]). As noted above, the most common chest radiograph 1 finding in COVID-19 was the IA pattern, present in 51% of COVID-19–positive cases, versus 26% of COVID-19–negative cases (P < .001). Peripheral airspace opacities and DA disease were also more common in COVID-19–positive cases than controls (21% vs 4%, P < .001; and 6% vs 2%, P = .03). Conversely, COVID-19–positive chest radiograph 1 demonstrated a reduced frequency of atelectasis, lobar consolidation, and pleural effusions compared with COVID-19–negative chest radiograph 1 (P ≤ .001, P = .03, P < .001, respectively).

Specific CT Findings for COVID-19

Frequencies of single CT findings in COVID-19–positive cases and controls are given in Table 3. Diffuse involvement across craniocaudal and anterior/posterior spatial distributions of chest CT findings predominated in patients positive for COVID-19 relative to patients negative for COVID-19 (58% vs 28%, P < .001, and 69% vs 47%, P = .01, respectively), as did ground glass opacities (88% vs 49%, P < .001). Additionally, the reversed halo sign was significantly more prevalent in patients positive for COVID-19 (33% in COVID-19–positive cases vs 5% in COVID-19–negative cases, P < .001). Among these COVID-19–positive characteristics in total CT scans, only the frequency of consolidation significantly changed over the course of COVID-19 infection (Fig 4), increasing from 32% at ≤ 2 days to 91% at > 11 days (P < .001 for trend).

Figure 4:

Download as PowerPointOpen in Image Viewer

Features Associated with False-Negative Imaging Findings

There were 113 false-negative (FN) initial chest radiography findings and 23 FN second chest radiography findings in patients positive for COVID-19. Normal chest radiographs comprised 51 (45%), three (14%), and 54 (40%) of FN initial chest radiography findings, second chest radiography findings, and the summed initial and second chest radiography findings. The combined normal and mild severity FN chest radiography findings for these three respective groups were 106 (94%), nine (39%), and 118 (88%). Of the 113 patients with FN chest radiograph 1 studies, 35 had a chest radiograph 2, of which 16/35 were positive. Analysis of patient demographics and comorbidities initially identified female sex, chronic lung disease, absence of hypertension, ethnicity, and young age to be associated with FN initial chest radiography finding (Table 4). A multivariable logistic regression analysis showed young age and African American ethnicity increased FN finding rates, whereas Hispanic American ethnicity decreased FN finding rates.

Of 10 FN CT findings, two were normal and seven (70%) were either normal or mild severity. Female sex (odds ratio [OR] 5.44, P = .03) and chronic lung disease (OR 5.56, P = .02) also contributed to FN chest CT studies.

Discussion

In summary, we evaluated the imaging characteristics and evolution of severity, sensitivity, and specificity of chest radiography and chest CT in patients with laboratory-verified COVID-19 and matched patients negative for COVID-19 from two academic medical centers in two major U.S. cities. Our results indicate that the sensitivity of chest radiography increases over the course of COVID-19 infection, from as low as 55% at ≤ 2 days to 79% at > 11 days after symptom onset, and serial chest radiography approaches the diagnostic accuracy of chest CT (AUC of 0.875 vs 0.916). Normal imaging was the most prevalent factor underlying FN chest radiography findings and chest CT scans. Patient characteristics including young age and certain ethnicities were associated with greater rates of FN examination findings. Finally, we found a number of specific imaging findings at chest radiography and CT that were helpful in the diagnosis of COVID-19, including IA pattern (OR, 2.84) and peripheral airspace opacities (OR, 7.94) on chest radiographs and reversed halo sign on chest CT scans (OR, 9.36).

We showed that the severity and sensitivity of COVID-19–positive chest radiography findings increases over time, with sensitivity markedly increasing after day 6 following symptom onset. This coincides with the known time course of pathophysiologic deterioration in patients, as the time from illness onset to dyspnea, sepsis, and acute respiratory distress syndrome in patients positive for COVID-19 has been reported as 7 days, 9 days, and 12 days, respectively (12). A previous study analyzed baseline chest radiographs from 64 patients positive for COVID-19 and found a sensitivity of 44/64 (69%) in COVID-19 detection, comparable to our results (13). In our study, we stratified chest radiographs on the basis of time since the onset of symptoms and were able to identify when imaging is expected to be most useful. Due to an inadequate number of CT scans in our patient population, we could not perform a similar analysis for CT sensitivity.

From our study population, we determined that the overall sensitivity of chest CT for COVID-19 infection was 88%, which is comparable to three other studies reporting 86%, 86%, and 88% each but lower than studies reporting sensitivities as high as 100% (1,3,4,14–16). Of note, our article specifically addressed diagnostic accuracy within a relevant clinical scenario (distinguishing patients positive for COVID-19 from those negative for COVID-19) rather than assessing sensitivity alone without a control group or with artificially generated control groups.

Our finding that serial chest radiography approaches the accuracy of CT in detecting COVID-19 has potential value to enhancing biocontainment and workflow in hospitals. The American College of Radiology has recommended decontamination of CT imaging rooms after each use for COVID-19 evaluation to prevent viral transmission from CT scanners (17). In addition to the significant cleaning duties and personal protective equipment donning and doffing required of radiology technicians and janitorial staff, the process of transporting inpatients to a commonly utilized CT unit accumulates risk for viral transmission to other patients and health care staff. These concerns potentially limit widespread adoption of CT for COVID-19 detection. Conversely, chest radiography uses less radiation, and portable chest radiography machines can be brought to patient bedsides to perform chest radiography and require far less time to decontaminate, making it an attractive option to employ.

The majority of FN findings were caused by chest radiographs or CT scans characterized as normal, which likely results from imaging taken during early viral infection when infectious changes are not yet visible in lung parenchyma. We observed that older patients with COVID-19 had a higher frequency of true-positive chest radiography findings, which corresponds to reports that COVID-19 has greater severity in older patients (12,18). We noted increased FN finding rates in African American patients and a decreased FN finding rate in Hispanic-American patients relative to White patients. This may be related to differential testing in these minority populations, with some patients having access to testing for more mild cases and others not; alternatively, there may be true biologic differences (or different rates of comorbidities) that contribute to different chest radiography finding severities. Regardless, these findings may have implications for equity of care across minority populations.

Our study had several limitations, including its retrospective nature, as there was not a consistent clinical or imaging algorithm for these patients. Other limitations lay in patient selection, specifically in that the reasons for patient imaging were unknown and that serial chest radiography imaging and CT imaging was likely biased toward patients with more severe disease. However, the receiver operating characteristic comparisons performed in this study were done among patients who had the same set and number of imaging tests, so those patients should be comparable to each other.

In conclusion, the results from our study imply that strategic serial chest radiography can detect COVID-19 infection in symptomatic patients and may be a helpful adjunct to reverse-transcription polymerase chain reaction testing. In an inpatient hospital setting, in which testing kit availability, laboratory resources, and/or laboratory staff are compromised and threaten delay to patient care and hospital workflow, serial chest radiographs could potentially be utilized in an adjunct diagnostic role in patients with suspected COVID-19.

Disclosures of Conflicts of Interest: S.S. disclosed no relevant relationships. T.S. disclosed no relevant relationships. H.C. disclosed no relevant relationships. S.R.C. disclosed no relevant relationships. A.H. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: author is a resident at University of Maryland. Other relationships: disclosed no relevant relationships. F.L.J. disclosed no relevant relationships. H.H. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: author received consultancy fees from Mitsubishi Chemical; author’s institution has grants/grants pending from Canon Medical Systems and Konica-Minolta; author received money (less than US$5000 per year) from Canon Medical Systems for medical advisory board duties. Other relationships: disclosed no relevant relationships. S.C.B. disclosed no relevant relationships. K.S. disclosed no relevant relationships. T.A. disclosed no relevant relationships. J.A.H. disclosed no relevant relationships. C.S.W. assoicate editor of Radiology: Cardiothoracic Imaging. R.H. disclosed no relevant relationships. A.R.H. disclosed no relevant relationships. M.M.H. disclosed no relevant relationships.

Author Contributions

Author contributions: Guarantors of integrity of entire study, S.S., T.S., F.L.J., C.S.W., A.R.H., M.M.H.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, S.S., T.S., H.C., S.R.C., F.L.J., K.S., T.A., J.A.H., M.M.H.; clinical studies, S.S., T.S., H.C., S.R.C., A.H., F.L.J., H.H., S.C.B., K.S., T.A., C.S.W., R.H., A.R.H., M.M.H.; statistical analysis, S.S., T.S., A.H., M.M.H.; and manuscript editing, S.S., T.S., H.C., S.R.C., A.H., F.L.J., H.H., S.C.B., T.A., J.A.H., C.S.W., R.H., A.R.H., M.M.H.