Study of Thoracic CT in COVID-19: The STOIC Project

Study Design and Participants

The protocol of this multicenter observational retrospective cohort study can be accessed through ClinicalTrials.gov with identifier NCT04355507. Its design and execution were in accordance with the Standards for Reporting of Diagnostic Accuracy, or STARD, initiative (9).

The STOIC project aimed to build a data set of at least 10 000 CT scans from individuals with suspected COVID-19 pneumonia, evaluated during the first wave of the SARS-CoV-2 pandemic in France. The collected data and list of image annotations are described in Appendix E1 (online), together with the license agreement for data sharing. The project involved 20 university hospitals: 15 from Assistance Publique des Hôpitaux de Paris and five from other cities (Strasbourg, Lyon, Rennes, and Montpellier). Patients were eligible if they had both thoracic CT scans and RTPCR at initial presentation, between March 1st and April 30th, 2020. According to the recommendations of the French Health Authority, CT was not performed as a screening test in patients with no thoracic symptoms but only in those presenting at the hospital with dyspnea and/or desaturation as measured by pulse oximetry (10).

This study was approved by the ethics committee of Cochin Hospital in Paris, which waived the need for written informed consent.

Study Procedures

Thoracic CT.—All participating hospitals were equipped with multidetector CT scanners from different manufacturers, allowing volumetric high-resolution CT acquisitions of the whole thorax. CT acquisitions were performed without contrast material administration except when pulmonary embolism was suspected as a confounding diagnosis to COVID-19 pneumonia at presentation (11).

CT readings.—After anonymized export, CT images were independently read using a dedicated three-dimensional image viewer web application by seven junior (I.S., T.L., A.M., F.B., S.D., C.H., C.J.) and 13 senior (M.P.R., S. Boussouar, C.D.M., D.M., G.C., M.L., S. Bennani, S.M., M.P.D., M.O., S. Bommart, M.E.H., I.P.) chest radiologists who were blinded to initial reports, RT-PCR results, demographic characteristics, clinical symptoms, and outcomes. Senior radiologists had at least 4 years of experience in chest imaging (Table E1 [online]). Readers were asked to classify CT scans as either positive or negative for COVID-19 based on the criteria of the COVID-19 structured report of the French Society of Radiology (12). Positive diagnosis required the presence of predominantly subpleural ground-glass opacities and the absence of mucoid impactions, bronchiolar nodules, or focal consolidation suggesting bacterial infection. Only the initial CT, performed at presentation, was analyzed. CT scans were only read by one radiologist, except those which were unintentionally exported twice and received different anonymization numbers. Again, some CT scans not marked as read were reannotated by the same reader. This provided the opportunity to evaluate inter- and intrareader agreement in a post hoc analysis. For CT scans they considered as positive for COVID-19, readers had to visually quantify lung disease extent using a five-point scale. They also evaluated lung emphysema on a five-point scale (0%, <25%, 25%–50%, 50%–75%, >75%), severity of coronary artery calcifications using a semiquantitative visual method (13), and measured the amount of chest wall fat anterior to the sternum.

Data collected and reference standard.—Patient demographic characteristics were retrieved from the electronic medical records, as was the time elapsed since the onset of symptoms; the need for oxygen supplementation at presentation and any preexisting comorbidities such as diabetes, hypertension, coronary artery disease; and any other preexisting cardiovascular or respiratory disease. Initial (or repeated) RT-PCR results were collected. For patients with positive RT-PCR, the outcome at 1 month was analyzed, with unfavorable outcome being defined as death or intubation.

RT-PCR was used as reference standard. If the first RT-PCR result was negative but turned out to be positive in the following 7 days after CT, then the patient was considered as positive for SARS-CoV-2 infection.

Statistical Analysis

For the power analysis, we considered that with an enrollment target of 10 000 patients, a disease prevalence of at least 50% and expected CT sensitivity and specificity between 80% and 85%, the width of the confidence intervals for both sensitivity and specificity would be less than 1.0%.

CT diagnostic performance was evaluated by its sensitivity, specificity, positive and negative predictive values, accuracy, and area under the curve (AUC) with RT-PCR results as the reference standard. The McNemar test was used for paired comparisons. Cohen κ coefficient was used to measure intrareader agreement for readers who reannotated the same CT scan. Gwet AC1 coefficient for multiple readers was used to evaluate interreader agreement on CT scans annotated by two different readers.

The analysis was conducted on all included patients. Continuous data are presented as means ± standard deviations, while categorical data are summarized as counts and percentages. For modeling purposes, missing clinical data were handled with multiple imputations by chained equations. The risk model for severe outcome was developed on patients positive for both RTPCR and CT, who had unenhanced CT allowing quantifying coronary artery calcifications at CT.

Multivariable logistic regression was used to estimate the risk model. We selected 11 clinical and CT variables to serve as candidate predictors. These included age, sex, need for oxygen supplementation at presentation, comorbidities (diabetes, hypertension, coronary artery disease, respiratory disease), chest wall fat thickness anterior to the sternum, disease extent, coronary artery ordinal calcium score, and emphysema score. Multiple imputations were performed to handle missing data by using the SAS (SAS Institute) procedure for multiple imputations. Severity of the disease (patients who were intubated at one point or deceased were considered as severe cases) was the dependent variable. Stepwise variable selection was applied to select predictors in the final model, with a .20 significance level for entry and a .05 significance level for retention. Discrimination was assessed by calculating the AUC and goodness of fit with the Hosmer-Lemeshow test. A simplified score was obtained by multiplying the regression coefficient by 5 and rounding to the nearest integer (14).

All statistical analyses were performed with SAS software (version 9.4; SAS Institute). A P value of ≤ .05 was considered to indicate statistical significance for all statistical tests conducted.

Study Sample, Initial Characteristics, and Outcome

Among the 10 930 patients who were assessed for eligibility, 195 were excluded either because they did not fulfill the inclusion criteria or were erroneously included twice (Fig 1). A total of 10 735 patients (median age, 65 years; interquartile range, 51–77 years; 6147 men) were finally included. The study demographics are presented in Table 1. RT-PCR was positive for 6448 patients, corresponding to a disease prevalence of 60.0% during the study period.

Figure 1:

Download as PowerPoint

Table 1: Patient Characteristics at Presentation

Among 4557 patients with an initial negative RT-PCR test, repeat tests were performed in 1222 (27%). Repeat RT-PCT testing was positive in 271 patients in the following 7 days, resulting in a sensitivity of first RT-PCR of 95.8% (6176 of 6447) (95% CI: 95.3, 96.3). At 1-month follow up, 84% (5440 of 6448) of the PCR-positive patients were alive and discharged from the hospital, 14% (881 of 6448) had to be intubated at one point, and 15% (988 of 6448) died of complications of their COVID-19 infection.

CT Results and Diagnostic Performance

Among the 6448 patients who had positive RT-PCR results, 5174 were judged positive for COVID-19 at CT, resulting in an overall sensitivity of 80.2% (95% CI: 79.3, 81.2); 80.6% (95% CI: 79.4, 81.7) when based on senior radiologist reading and 79.6% (95% CI: 77.8, 81.3) for junior reading (P = .34). Among the 4287 patients with negative RT-PCR results, 872 were considered positive for COVID-19 at CT. The calculated specificity of CT was 79.7% (95% CI: 78.5, 80.9) overall; 79.5% (95% CI: 78.0, 80.9) for senior and 80.1% (95% CI: 78.0, 82.3) for junior readings, respectively (P = .61). Among the 1041 patients with a first negative PCR result but signs of pneumonia at CT, 430 (41%) had further RT-PCR tests which turned out to be positive in 169 (39%). After 5 days of symptoms, the sensitivity of CT was improved to 88.2% (95% CI: 87.0, 89.4). Excluding patients with positive RT-PCR results but normal CT findings—thus not having pneumonia—increased CT sensitivity to 83.2% (95% CI: 82.3, 84.2).

These results and the corresponding AUCs, accuracy, and predictive values are presented in Table 2.

Table 2: CT Diagnostic Performance

Inter- and Intrareader Agreement

Annotations from different radiologists (Fig 2) were available for 235 CT scans exported twice and had received different anonymization numbers (Table E1 [online]).

Figure 2:

Download as PowerPoint

The Gwet AC1 coefficient was 0.79, indicating strong interreader agreement for the classification of CT scans as either COVID-19 positive or negative. Regarding disease extent, Gwet AC1 coefficient was 0.38 for classification into one of the five categories (<10%, 10%–24%, 25%–49%, 50%–74%, ≥75%), but 0.85 for classification as less than 50% versus greater than or equal to 50% (Fig 3). Different annotations from the same observer were also available for 324 CT scans. This happened when CT scans of a reading list were not labeled as “already read.” Intraobserver agreement was perfect for 16 of the 20 readers (κ = 1.00) and almost perfect for the remaining four (κ ranging from 0.82 to 0.92).

Figure 3:

Download as PowerPoint

The proportion of double readings related to the whole number of readings was homogeneous among the readers (median, 4.3%; interquartile range, 3.9–5.2).

CT Features Reflecting Comorbidities

Emphysema was rarely present, with 86.4% (3663 of 4238) of patients with CT and RT-PCR positivity having no emphysema or, when present, affecting more than 25% of the lung in only 1.8% (79 of 4238).

Conversely, coronary artery calcifications were observed in 59% (2493 of 4238) of patients with CT and RT-PCR positivity (Fig 4). Mean subcutaneous adipose thickness anterior to the sternum was 16.5 mm ± 7.8 (standard deviation) (Fig 5). This parameter showed good correlation with body mass index (ρ = 0.62; P < .001).

Figure 4:

Download as PowerPoint

Figure 5:

Download as PowerPoint

Risk Factors for Severe Outcome at 1-month Follow-up

Risk factors for severe outcome were evaluated for the 4238 patients positive for both RT-PCR and CT, who had unenhanced CT. At 1-month, 24% (1000 of 4238) of patients positive for both CT and RT-PCR had developed severe disease. A comparison of the clinical data and imaging findings for patients with severe and nonsevere outcome is presented in Table 3. There was no evidence of an association between adipose thickness anterior to the sternum with disease severity (P = .10) contrary to emphysema percentage, coronary artery calcifications, and disease extent at CT (P < .001 for all three parameters).

Table 3: Comparison of Characteristics in Severe and Nonsevere COVID-19 Cases

A clinical model selection procedure retained age, sex, oxygen supplementation at presentation, hypertension, and coronary artery disease as clinical risk factors of severe COVID-19 cases across pooled imputed data sets. The model achieved an AUC of 0.64 (95% CI: 0.62, 0.66) and no lack of fit was detected (Hosmer-Lemeshow test: P value ranging from .79 to .96 across imputed data sets). When combining clinical variables and CT annotations, the model selection procedure retained age, sex, oxygen supplementation, hypertension, coronary artery disease, coronary artery calcium score, and disease extent as risk factors. Table 4 displays the adjusted odds ratios of all significant predictors for severe outcome. The final risk model achieved an AUC of 0.69 (95% CI: 0.67, 0.71), without evidence for miscalibration (Hosmer-Lemeshow test: P value ranging from .10 to .96 across imputed data sets) and was better than the clinical model (Likelihood ratio test P < .001) (Fig 6).

Table 4: Adjusted Coefficients and Odds Ratios of Significant Predictors of Severe Outcome (n = 4085)

Figure 6:

Download as PowerPoint

Taking into account the only two significant interactions (coronary artery disease and hypertension, oxygen supplementation and disease extent) did not improve the performance of the model (Table E2; Fig E1 [both online]).

After multiplying the regression coefficient by 5 and rounding up to the nearest integer, a nomogram for severity prediction was derived (Fig E2 [online]), ranging from 0 (lowest risk) to 18 (highest risk), and achieved the same AUC.