CO-RADS: A Categorical CT Assessment Scheme for Patients Suspected of Having COVID-19—Definition and Evaluation

CO-RADS Category 0

CO-RADS category 0 is chosen if none of the five categories can be assigned because scans are incomplete or of insufficient quality, for example because of severe artifacts due to coughing or breathing.

CO-RADS Category 1

CO-RADS category 1 implies a very low level of suspicion for pulmonary involvement by COVID-19 based on either normal CT results or CT findings of unequivocal noninfectious origin. This was modeled on LI-RADS, in which cases that have no nodules or that have nodules with definitely benign features are reported together, as opposed to BI-RADS, in which category 1 refers to normal findings only, we consider this approach more suitable for patients with potential COVID-19; concomitant findings are frequent in the lung, and there is considerable interobserver variability regarding which findings are normal. According to our definition, mild or severe emphysema, perifissural nodules, lung tumors, and fibrosis are classified as CO-RADS category 1 findings. This category is identical to the “negative for pneumonia” category of the RSNA consensus statement (13).

CO-RADS Category 2

CO-RADS category 2 implies a low level of suspicion for pulmonary involvement by COVID-19 based on CT findings in the lungs typical of infectious origin that are considered not compatible with COVID-19. Examples are bronchitis, infectious bronchiolitis, bronchopneumonia, lobar pneumonia, and pulmonary abscess. Features include tree-in-bud sign, a centrilobular nodular pattern, lobar or segmental consolidation, and lung cavitation. These features are similar to the ones in the “atypical appearance” category of the RSNA consensus statement (13). Cases with smooth interlobular septal thickening with pleural effusion, which is also part of this RSNA category, are assigned to CO-RADS category 1 if considered typical for interstitial pulmonary edema or are assigned to CO-RADS category 3 if ground-glass opacities that may mimic pulmonary involvement by COVID-19 are also present. This choice was made because CO-RADS describes the pulmonary, not cardiac, involvement of COVID-19.

CO-RADS Category 3

CO-RADS category 3 implies equivocal findings for pulmonary involvement of COVID-19 based on CT features that can also be found in other viral pneumonias or noninfectious causes. Findings include perihilar ground-glass opacity, homogenous extensive ground-glass opacity with or without sparing of some secondary pulmonary lobules, or ground-glass opacity together with smooth interlobular septal thickening with or without pleural effusion in the absence of other typical CT findings. CO-RADS category 3 also includes small ground-glass opacities that are not centrilobular (otherwise they would be CO-RADS category 2) or not located close to the visceral pleura (otherwise they would be CO-RADS category 4). In addition, it contains patterns of consolidation compatible with organizing pneumonia without other typical findings of COVID-19. This category partially overlaps with the indeterminate appearance category of the RSNA consensus statement but includes those cases with lower likelihood for COVID-19 (13).

CO-RADS Category 4

CO-RADS category 4 implies a high level of suspicion for pulmonary involvement by COVID-19 based on CT findings that are typical for COVID-19 but also show some overlap with other (viral) pneumonias. Findings are similar to those for CO-RADS category 5; however, they are not in contact with the visceral pleura, nor are they located strictly unilaterally in a predominant peribronchovascular distribution or superimposed on severe diffuse preexisting pulmonary abnormalities. CO-RADS category 4 consists of the features of the indeterminate appearance category of the RSNA consensus statement that are associated with a higher likelihood of COVID-19 (13).

CO-RADS Category 5

CO-RADS category 5 implies a very high level of suspicion for pulmonary involvement by COVID-19 based on typical CT findings (Table 2). Mandatory features are ground-glass opacities with or without consolidations in lung regions close to visceral pleural surfaces, including the fissures, and a multifocal bilateral distribution. Other classifications only describe a peripheral location, but we found that the vicinity to the minor or major fissure is also typical. Subpleural sparing can be present. We found that the previously described lower lobe predominance is frequently not present in otherwise typical RT-PCR–positive cases; therefore, lower lobe predominance was excluded as a required feature.

Table 2: Typical Features for Pulmonary Involvement of COVID-19

CO-RADS category 5 requires the presence of at least one confirmatory pattern that aligns with the temporal evolution of the disease (15). The pattern that has been described early in the course of COVID-19 is dominated by multiple ground-glass areas, which often show (half) rounded and unsharp demarcation but can be accompanied by sharply delineated ground-glass areas that outline the shape of multiple adjacent secondary pulmonary lobules. The crazy paving pattern, which has been described as appearing later in the course of the disease, shows visible intralobular lines. As the disease progresses, more consolidations occur within the areas of ground-glass opacity. Finally, opacities that resemble organizing pneumonia occur, such as reverse halo signs or ground-glass opacity with extensive subpleural consolidations and air bronchograms. Subpleural curvilinear bands or bands of ground glass with or without consolidation in a tethered arching pattern with small connections to the pleura are also considered typical findings. Thickened vessels within lung abnormalities are typical and are frequently found in all other confirmatory patterns. CO-RADS category 5 is largely identical to the typical appearance of the RSNA consensus statement (13).

CO-RADS Category 6

CO-RADS category 6, similar to BI-RADS category 6, was introduced to indicate proven COVID-19, as signified by positive RT-PCR test results for virus-specific nucleic acid.

Study Group

An observer study was performed on a set of 105 randomly selected chest CT scans obtained in a group of consecutive patients who presented to the emergency department between March 14, 2020, and March 25, 2020, with suspected SARS-CoV-2 infection in whom RT-PCR was performed. Patient inclusion criteria, CT protocol, and radiation parameters are described in Appendix E2 and Figure E6 (online). Medical ethics committee approval was obtained prior to the study. Informed consent was waived, and data collection and storage were performed in accordance with local guidelines.

Patient characteristics (age, sex, comorbidities); clinical follow-up including a multidisciplinary clinical diagnosis, if applicable; and RT-PCR results were extracted from electronic patient records. These data enabled stratification of all patients into one of the following three groups: patients with at least one positive RT-PCR result for SARS-CoV-2 within 5 days after CT (PCR positive), patients with one or more negative RT-PCR results but a clinical diagnosis of COVID-19 according to clinical records (PCR negative, clinically positive), or patients with one or more negative RT-PCR results and a clinical course not consistent with COVID-19 or consistent with an alternative diagnosis (PCR negative, clinically negative).

CT Scoring Procedure

CT images were extracted from the picture archive and communication system, anonymized, and imported into a browser-based dedicated viewing system for CT scans (CIRRUS Core; https://grand-challenge.org/reader-studies/). The software facilitated reading and scoring of anonymized CT images in the three orthogonal views, providing reading tools such as average or maximum intensity projections, window width–window level adaptation, panning, and zooming.

Eight observers from seven hospitals in the Netherlands participated in the study (B.G., J.K., L.F.B., M.P., H.A.G., J.L.S., C.S., T.R.V.). Four observers (B.G., J.K., J.L.S., T.R.V.) had less than 5 years of experience in reading chest CT scans, whereas the others (L.B., M.P., H.G., C.S.) had 5–27 years. All observers were familiar with the CO-RADS score from clinical experience interpreting at least 30 CT scans. Observers assigned CO-RADS scores using scoring software with drop-down lists, with blinding to RT-PCR results and patient information except for age and sex. In addition, they were blinded for the prevalence of COVID-19 in the selected cohort, medical history, and clinical follow-up.

Statistical Analysis

Statistical analysis was performed using SPSS, version 25 (IBM, Armonk, NY). Data are presented as the mean ± standard deviation or median and interquartile range based on normality of data. A 5 × 5 confusion matrix was made separately per observer, in which the CO-RADS score of the observer was compared with the median CO-RADS score of the remaining seven observers. A similar matrix was calculated using the sum of all individual 5 × 5 tables.

For each observer, a receiver operating characteristics curve was calculated, and the area under the receiver operating characteristics curve (AUC) was used to assess the performance of CO-RADS relative to two reference standards for the diagnosis of COVID-19: a positive RT-PCR test (PCR positive) and a reference that combined the results of the RT-PCR test with a clinical COVID-19 diagnosis (PCR positive and PCR negative but clinically positive). Mean AUC across observers and 95% confidence intervals (CIs) were calculated. In addition, the average percentage of cases assigned to each CO-RADS category, including the 95% CI, were determined for (a) PCR-positive, (b) PCR-negative and clinically positive, and (c) PCR-negative and clinically positive cases.

To quantify interobserver agreement, the Fleiss κ value was determined across observers. The κ values were obtained by comparing the CO-RADS scores of each observer to the median score of the remaining seven observers. Interobserver agreement was considered slight for a κ value of 0.01–0.20, fair for a κ value of 0.21–0.40, moderate for a κ value of 0.41–0.60, substantial for a κ value of 0.61–0.80, and almost perfect for a κ value of 0.81–1.00 (16).

Interobserver Variability of CO-RADS

There was absolute agreement in assigned CO-RADS category in 573 of 840 (68.2%) observations. A discrepancy by one CO-RADS category was seen in 235 of 840 (28.0%) observations, and of these pairs, CO-RADS categories 4 and 5 and CO-RADS categories 1 and 2 occurred in 128 of 840 (15.2%) observations. A difference of two CO-RADS categories was found in 31 (3.7%) observations, and a difference of three categories was found in one (0.1%) observation. The resulting 5 × 5 table is given in Appendix E3 (online). The Fleiss κ value of all observers on CO-RADS was 0.47 (95% CI: 0.45, 0.49). The κ values and 95% CIs for the individual categories were as follows: CO-RADS 1, 0.58 (95% CI: 0.54, 0.62); CO-RADS 2, 0.36 (95% CI: 0.32, 0.40); CO-RADS 3, 0.31 (95% CI: 0.28, 0.35); CO-RADS 4, 0.20 (95% CI: 0.17, 0.24); CO-RADS 5, 0.68 (95% CI: 0.65, 0.72). The κ value for each observer is provided in Table 4. Agreements of individual observers with the median of the remaining observers were either substantial (n = 4) or moderate (n = 4).

Table 4: CO-RADS Interobserver Comparison and Performance

Diagnostic Performance of CO-RADS

CO-RADS was able to distinguish between patients with positive PCR results from those with negative PCR results with an average AUC of 0.91 (95% CI: 0.85, 0.97). Average AUC increased to 0.95 (95% CI: 0.91, 0.99) if a clinical diagnosis of COVID-19 was also accepted (Table 4). The proportion of cases with positive PCR results or clinical diagnosis of COVID-19 increased from CO-RADS 1 to 5, as shown in Figure 1. All five CT scans of patients with negative PCR results but positive clinical findings were assigned CO-RADS categories 3 to 5.

Figure 1:

Download as PowerPoint

False-negative CO-RADS category 1 relative to a combined clinical and RT-PCR reference standard was found in nine of 161 (5.6%; 95% CI: 1.0%, 10%) ratings. This occurred in four patients: In one patient, CT revealed bronchial wall thickening and very subtle ground-glass opacities in both lower lobes (one observer). In one patient, CT revealed multiple ground-glass opacities in only the right lower lobe and a pleural lesion in the right upper lobe (two observers). In two patients, CT revealed concurrent preexisting disease (hypersensitivity pneumonitis, n = 1; silicosis, n = 1) (six observers). CO-RADS category 2 assessments were false-negative for COVID-19 in 22 of 159 ratings (average, 13.8%; 95% CI: 9%, 18%) in 13 cases, including all four false-negative cases with a CO-RADS 1 score. Correspondingly, six of 97 (6.2%; 95% CI: 1.5%, 10%) CO-RADS 4 observations and one of 286 (0.3%; 95% CI: 0%, 1.0%) CO-RADS 5 observations were false-positive, as assessed by six observers for four cases. RT-PCR and clinical diagnosis were almost equally distributed among observations of CO-RADS category 3; 57 of 137 (41.6%; 95% CI: 31%, 54%) observations of CO-RADS category 3 were positive for COVID-19. Figure 2 shows examples of RT-PCR–positive and RT-PCR–negative CO-RADS category 3 observations, and examples of CO-RADS category 4 and CO-RADS category 5 observations are shown in Figures 3 and 4.

Figure 2:

Download as PowerPoint

Figure 3:

Download as PowerPoint

Figure 4:

Download as PowerPoint