Performance of Radiologists in Differentiating COVID-19 from Non-COVID-19 Viral Pneumonia at Chest CT

Introduction

Since the initial outbreak of coronavirus disease 2019 (COVID-19) in Wuhan, China, in late December 2019 (1), 105 586 cases have been confirmed, and 3584 deaths have been reported across 60 countries as of March 9, 2020 (2,3). The majority of COVID-19 cases (77%) have been found in China (3,4). Patients infected with COVID-19 typically present with fever, cough, dyspnea, and muscle aches while imaging frequently reveals bilateral pneumonia (5).

The standard diagnostic method used is real-time reverse-transcription polymerase chain reaction (RT-PCR), which helps detect viral nucleotides from specimens obtained with oropharyngeal swab, nasopharyngeal swab, bronchoalveolar lavage, or tracheal aspirate (6). However, recent reports have revealed that RT-PCR has a sensitivity as low as 60%–71% for helping detect COVID-19 (5,7,8). This can possibly be attributed to the low viral load present in test specimens or laboratory error (7,9). These false-negative findings hinder quarantine efforts, necessitate repeat testing, and have the potential to overload the current supply of testing kits and related infrastructure (8). By contrast, chest CT has demonstrated about 56%–98% sensitivity in in the detection of COVID-19 at initial presentation and can be helpful in rectifying false-negative findings obtained with RT-PCR during the early stages of disease development (7,8). Chest CT images in patients with COVID-19 reveal areas of consolidation and ground-glass opacity with bilateral peripheral involvement in multiple lobes progressing to “crazy paving” patterns and consolidation. CT signs gradually improve beginning approximately 14 days after symptom onset (10–12).

Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT had low specificity (25%) in a recent report of 1014 patients with COVID-19 (5). Previous studies have not directly compared chest CT patterns of COVID-19 with those of viral pneumonia at chest CT. The purpose of this study was to assess the performance of radiologists from the United States and China in differentiating COVID-19 from viral pneumonia at chest CT.

Materials and Methods

Patient Cohort

The institutional review board of all seven hospitals in Hunan Province, China, and Rhode Island Hospital in Providence, RI, approved this retrospective study, and the requirement to obtain written informed consent was waived. A total of 256 patients with both positive COVID-19 results with RT-PCR and chest CT imaging within 2 weeks were retrospectively identified from seven hospitals in Hunan Province, China, from January 6 to February 20, 2020. The RT-PCR results were extracted from the patients’ electronic medical records in the hospital information system. The RT-PCR assays were performed by using TaqMan One-Step Real-Time RT-PCR kits (Thermo Fisher Scientific, Waltham, Mass) from Shanghai Huirui Biotechnology or Shanghai BioGerm Medical Biotechnology, both of which have approved use by the National Medical Products Administration in China. For patients with multiple RT-PCR assays, positive results on the last performed test were adopted as confirmation of diagnosis. The number of cases included from each hospital is shown in Table E1 (online). Among these patients, 37 with negative findings at chest CT were excluded, resulting in a final cohort of 219 patients. The chest CT protocols from the seven hospitals are shown in Table E2 (online).

The radiology search engine Montage (Montage Healthcare Solutions, Philadelphia, Pa) at Rhode Island Hospital was used to identify cases that contain the word “pneumonia” in the impression section of the radiology CT reports from 2017 to 2019. The identified CT scans were directly downloaded from the hospital picture archiving and communications system, and nonchest CT scans were excluded. Positive results from the respiratory pathogen panel (RPP) were used to identify patients with viral pneumonia from 2017 to 2019. The RPP tests (ePlex; GenMark Diagnostics, Carlsbad, Calif) were performed in the microbiology laboratory of the Rhode Island Hospital pathology department according to its manufacturing protocol (13). A diagram illustrating the initial breakdown of RPP search results is shown in Figure E1 (online). The two lists were cross-referenced to generate a final list that contained CT chest scans with the word “pneumonia” in the final impression and positive RPP test results within 7 days of each other. Then, the impression sections of these CT reports were reviewed by a research assistant (B.H.) and a radiologist (H.X.B.) board certified in general diagnostic radiology and interventional radiology with 1 year of practice experience to identify 205 cases with final CT impression being “consistent with” or “highly suspicious for” pneumonia. Our final cohort consisted of 424 patients. A flowchart illustrating patient inclusion and exclusion is shown in Figure 1. A diagram illustrating the final breakdown of RPP results is shown in Figure 2.

Figure 1:

Download as PowerPoint

Figure 2:

Download as PowerPoint

Radiologists’ Interpretation

Three radiologists from China, who were blinded to RT-PCR results, reviewed all chest CT images and scored each case as COVID-19, pneumonia of other cause, or neither. Fifty-eight age-matched patients from the COVID-19 and non-COVID-19 pneumonia groups were randomly selected from the entire cohort and evaluated in the same way by four radiologists from the United States. All identifying information was removed from the CT scans, which were shuffled and uploaded to 3D Slicer software (https://www.slicer.org/) for interpretation. Information on the radiologists regarding location of practice, years in practice, cardiothoracic imaging fellowship, and COVID-19-specific training experience is shown in Table 1.

Table 1: Backgrounds of Radiologists Who Reviewed the Cases

Results

Our final cohort consisted of 424 patients, including 205 patients with non-COVID-19 pneumonia from the United States and 219 patients with COVID-19 from China. The average number of days between CT examination and a positive RT-PCR test result was 4.1 days ± 4.4 (standard deviation) for the COVID-19 group; the average number of days between CT examination and a positive RPP test result was 1.0 day ± 1.7 for the non-COVID-19 pneumonia group. Patients with COVID-19 were younger than those with non-COVID-19 pneumonia (mean age, 45 vs 65 years, respectively; P < .001) and less likely to have an elevated white blood cell count (29% vs 55%, P < .001). However, they were more likely to have a reduced lymphocyte count (16% vs 44%, P < .001). Clinical characteristics, including comorbidities comparing the two groups, are shown in Table 2.

Table 2: Clinical Characteristics of COVID-19 Patient Cohort from China and Non-COVID-19 Patient Cohort from the United States

For the entire cohort, the accuracy of the three radiologists from China in differentiating COVID-19 from non-COVID-19 pneumonia was 83% (95 CI: 79%, 86%), 80% (95% CI: 76%, 83%), and 60% (95% CI: 55%, 65%), respectively (Table 3). Sensitivity ranged from 72% to 94%, and specificity demonstrated high variation (24%–94%) (Table 3).

Table 3: Performance Results of the Three Radiologists from China Who Reviewed All 424 Chest CT Cases of COVID-19 and Non-COVID-19 Viral Pneumonia

In a randomly selected age-matched cohort of 58 patients, the accuracy of four radiologists from the United States in differentiating COVID-19 from non-COVID-19 pneumonia was 97% (95% CI: 88%, 100%), 88% (95 CI: 77%, 95%), 83% (95% CI: 71%, 91%), and 85% (95% CI: 73%, 93%), respectively (Table 4). Sensitivity ranged from 70% to 93%, and specificity ranged from 93% to 100% (Table 4).

Table 4: Results of Three Radiologists from China and Four Radiologists from the United States Who Reviewed 58 Age-matched Cases

COVID-19 pneumonia was more likely than non-COVID-19 pneumonia to have a peripheral distribution (80% vs 57%, P < .001), ground-glass opacity (91% vs 68%, P < .001), fine reticular opacity (56% vs 22%, P < .001), vascular thickening (59% vs 22%, P < .001), and reverse halo sign (5% vs 1%, P = .005). However, it is less likely to have a central and peripheral distribution (14% vs 35%, P < .001), air bronchogram (14% vs 23%, P = .01), pleural thickening (15% vs 33%, P < .001), pleural effusion (4% vs 39%, P < .001), and lymphadenopathy (3% vs 10%, P = .002) (Table 5).

Table 5: Chest CT Features Comparing COVID-19 and Non-COVID-19 Viral Pneumonia

Figure 3 demonstrates example cases in which the majority of the radiologists mistook non-COVID-19 pneumonia for COVID-19 infection or vice versa.

Figure 3a:

Download as PowerPoint

Figure 3b:

Download as PowerPoint

Figure 3c:

Download as PowerPoint

Figure 3d:

Download as PowerPoint

Figure 3e:

Download as PowerPoint

Figure 3f:

Download as PowerPoint

Figure 3g:

Download as PowerPoint

Figure 3h:

Download as PowerPoint

Discussion

Coronavirus disease 2019 (COVID-19) is spreading rapidly worldwide; however, present diagnostic methods for identifying the virus have limitations. Reverse-transcription polymerase chain reaction testing has low sensitivity early on in the disease course. Although chest CT has high sensitivity, it has low specificity (7,11). This low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease at chest CT, such as seasonal flu (5). To optimize patient management, medical care, and disease control, it is important to determine the efficacy of chest CT in distinguishing COVID-19 from pneumonia of other causes by radiologists. This study revealed that radiologists are capable of distinguishing COVID-19 from viral pneumonia at chest CT with high specificity but moderate sensitivity. CT features of COVID-19 pneumonia that were more common than non-COVID viral pneumonia included peripheral distribution (80% vs 57%, P < .001), ground-glass opacity (91% vs 68%, P < .001), fine reticular opacity (56% vs 22%, P < .001), vascular thickening (59% vs 22%, P < .001), and reverse halo sign (5% vs 1%, P = .005).

The ability of health care providers to reliably differentiate COVID-19 from other causes of pneumonia at chest CT would benefit diagnostic work-up of the disease by compensating for the poor sensitivity of RT-PCR, particularly during early disease stages. Although chest CT has demonstrated high sensitivity relative to RT-PCR testing for the diagnosis of COVID-19, it may not reveal distinct patterns for COVID-19 in all cases. This can make it hard to distinguish COVID-19 from other causes of viral pneumonia. For example, influenza and COVID-19 both demonstrate ground-glass opacity and consolidation at chest CT (16). Introducing the possibility of abnormalities with similar chest CT findings to those of COVID-19 ultimately complicates the differential diagnosis.

This study is relevant because it demonstrates that radiologists are capable of distinguishing COVID-19 from other causes of pneumonia at chest CT with high specificity. This suggests that if the differential diagnosis is between COVID-19 and non-COVID-19 pneumonia, a negative diagnosis of COVID-19 by radiologists at chest CT may be sufficient to exclude patients from having the disease with fairly good certainty. Our analysis of specific cases where radiologists were wrong reveals that mistakes were made when the COVID-19 chest CT findings were either subtle (likely reflecting early stage in the disease process) or when COVID-19 demonstrated atypical chest CT findings. It is worth noting that non-COVID-19 pneumonia can also have a typical appearance of COVID-19. This poses a dilemma because mandated quarantine for all patients suspected of having COVID-19 can put a significant strain on medical infrastructure, health care providers, and the lives of patients, but it may need to be followed as a necessary precaution due to variation in presentation with timing of the disease and atypical findings at chest CT. Future direction includes development of an artificial intelligence classifier that can further augment radiologist performance in combination with clinical information.

Our study has several limitations. First, the cohort size was small, especially when it comes to cases reviewed by radiologists in the United States. A selection bias is associated with our screening strategy as well. It remains unclear if diagnostic outcomes would improve in a more well-balanced and larger-scale prospective study of similar design. It is also noteworthy that in this study, the radiologists from the United States had minimal training specific to diagnosing COVID-19 and that radiologists from China practiced in an area with a relatively low prevalence of the disease. It is possible that radiologists from China working near the epicenter of the disease with a higher degree of experience specific to COVID-19 would have performed significantly better than either group in the present study. In addition, although RPP test and chest CT findings within 7 days of symptom presentation were used to enrich our “pneumonia of other cause” cohort with viral pneumonia cases, the cause-and-effect relationship is not 100% accurate. Thus, it is possible that some of the selected patients had mixed viral and bacterial pneumonia or other diseases entirely. Finally, the radiologists were not given clinical information during the evaluation, which could have further improved their performance.

As more research is done, providers may gather information to make this differential easier to navigate. Until that point, it is recommended that individuals with signs of pneumonia at chest CT be quarantined while reverse-transcription polymerase chain reaction testing is performed in conjunction with a thorough medical evaluation, including travel history and disease contacts, to make an accurate coronavirus disease 2019 (COVID-19) diagnosis and to prevent disease spread. In conclusion, radiologists had high specificity but moderate sensitivity in distinguishing COVID-19 from viral pneumonia at chest CT.