CT Angiography Clot Burden Score from Data Mining of Structured Reports for Pulmonary Embolism

Introduction

With annual incidence rates between 39 and 115 per 100 000 people, acute pulmonary embolism (PE) represents the third most frequent cardiovascular disease after acute coronary syndrome and stroke (1). Reported short-term mortality rates up to 58% indicate that PE is a potentially life-threatening condition and emphasize the importance of early diagnosis, appropriate risk stratification, and immediate treatment of the disease (2).

CT pulmonary angiography is the preferred diagnostic imaging modality because of its ability to show endoluminal clots from the main to the subsegmental pulmonary arteries with high sensitivity (3). The cumulative thrombus load in the pulmonary vasculature can be mapped by various pulmonary artery obstruction indexes (PAOIs) for risk stratification, including the Qanadli score (PAOI_Q) as one of the most frequently cited in literature (4). The PAOI_Q is a semiquantitative parameter for PE severity and differentiates between partial and complete obstruction of the pulmonary arteries at main, lobar, and segmental level (4). It was shown that a high PAOI_Q reflects severe PE and correlates with a higher incidence of right heart failure as a fatal complication of PE (5–7). CT pulmonary angiography can also be used to help assess signs of right heart strain by measuring the right-to-left ventricular diameter ratio (RLr) as another finding that correlates with an adverse clinical outcome in patients with acute PE (8). Therefore, reporting of both PE severity and right heart strain is valuable information for the referring physician to perform risk stratification and to follow an appropriate treatment strategy (7).

The radiologic report, especially in a standardized and structured format, can contribute to the clinical assessment of patients with PE and has been shown to outperform conventional narrative reports in terms of accuracy and comprehensibility (9,10). According to the statement paper of the European Society of Radiology, reasons for using structured reports (SRs) include improvement of reporting quality, data acquisition, and accessibility of radiology reports for scientific purposes (11). Although several studies have analyzed the value of SR in improving reporting quality (eg, in terms of oncological staging, surgical planning, and interdisciplinary communication) (12–14), only few studies have used the SR content as a basis for answering scientific questions not primarily related to SR. In this study, we aimed to develop a PE scoring system on the basis of CT angiography mined from structured radiology reports and to compare its diagnostic performance with the PAOI_Q as a conventional clot burden index.

Materials and Methods

This Health Insurance Portability and Accountability Act–compliant retrospective single-center study was conducted with the approval of our institutional review board (S-236/2020). Informed consent was waived.

Patient Selection and Study Design

Consecutive patients suspected of having acute PE and with clinically indicated CT pulmonary angiography who had been assessed by using the SR template of our radiology department at University Hospital Heidelberg between March 2017 and February 2020 were eligible for study inclusion. No exclusion criteria were defined. All patients with PE, without PE, and indeterminate for PE were included for SR data extraction. PE-positive reports were used for calculation of the SR-derived clot burden score. After failure of score calculation in 53 reports, SRs from 304 patients were finally included for score validation.

Structured PE Reporting Template

Structure and content of our PE template are described in detail in Appendix E1 (online) and can be accessed for review (http://www.targetedreporting.com/pe/). Figure 1 outlines an exemplary case of PE assessment showing the PE module and its translation into the final report.

Figure 1:

Download as PowerPoint

Data Extraction of Structured Reports

Following a search query in our radiologic information system database (Centricity RIS-i, version 7; GE Healthcare), a single output file was generated in plain text format containing all SR with a corresponding data header listing additional examination-related parameters. To extract the SR records from the output file, we developed a data mining tool by using Python scripting (Python version 3.4; Python Software Foundation) with a rule-based method of pattern-matching commands called regular expressions. SR content was queried sequentially line by line and translated into an easily accessible data frame. The extracted data were further processed in R version 3.6.2 (R Foundation for Statistical Computing) and prepared for calculation of score variables and subsequent analysis.

Calculation of the Clot Burden Score

The extracted data of the PE module were composed of three main pulmonary artery (MPA)-level options and five lobar-level options and were assigned a numerical value depending on the reported PE occurrence. Because of sufficient description of PE at MPA and lobar levels, we sought to adapt our score to the PAOI_Q (4). Accordingly, PE occurrence at MPA and lobar level was calculated by using the formula Σ (n × d), where n is the proximal clot site equal to the number of segmental branches arising distally and d is the degree of obstruction scored as partial (value of 1) or total (value of 2) occlusion. Because the PE module did not contain information on the number of affected segmental arteries and their degree of obstruction, the thrombus burden was approximated by using the formula SA_n/2, where SA_n is the number of segmental arteries of the respective lobe. For subsegmental PE occurrence, a value of 1 was assigned for all lobes except for the middle lobe, which was assigned a value of 0.5. A pictorial overview of the scoring system is shown in Figure 2. The final clot burden score ranged between 0 and 40 points, and the respective PAOI of the clot burden score (PAOI_CBS) was indicated as percentage of vascular obstruction by using the following formula: where CBS is the clot burden score.

Figure 2:

Download as PowerPoint

Assessment of the Qanadli Score

CT pulmonary angiography scans in patients positive for PE were reviewed independently in random order on a dedicated Picture Archiving and Communication System workstation (Centricity 4.2; GE Healthcare). Image analysis was performed independently and in randomized order by two in-training radiologists (V.L.M. and M.A.F., each with 4 years of experience in body CT) who were blinded to the PAOI_CBS. PE was assessed according to the Qanadli scoring system (4) and indicated by using the following formula: , where Q indicates the Qanadli score.

For each patient, heart measures including RLr were standardly assessed as described previously (15).

Assessment of Scoring Time and Confidence

Scoring time and diagnostic confidence were measured for both scores in a randomly selected subset of 30 patients positive for PE, with 10 patients each having proximal PE at the MPA level, lobar level, and segmental level. Scores were assessed independently by two radiologists (V.L.M. and T.S., who is a trainee with 3 years of experience) blinded to the previous results and with two dedicated SR templates in randomized order. The templates were generated by using an online structured reporting system (Smart Reporting; https://app.smart-radiology.com) differing only in their scoring parameters as clickable decision trees (Fig E1 [online]). Scoring confidence was recorded for both readers by using a five-point Likert scale (1–5, not confident at all through very confident). To avoid recall bias for both assessments, all scans were reviewed with a time interval of at least 3 months.

Statistical Analysis

Score calculations and statistical analyses were performed by an author (M.A.F.) by using software (R version 3.6.2; R Foundation for Statistical Computing). All statistical tests were two sided and statistical significance was considered to be indicated at a P value less than .05. Differences in baseline characteristics and SR items between the PE and non-PE group were compared with the t test for continuous variables and with the χ² test for categorical variables. Correlations for score parameters and RLr were assessed by using Spearman rank correlation. Interobserver agreement was calculated by using the intraclass correlation coefficient (ICC) in a two-way mixed-effects model and tested for absolute agreement (16). ICCs were evaluated as follows: less than 0.50, poor; 0.51–0.75, moderate; 0.76–0.90, good; and greater than 0.90, excellent agreement (16). The Mann-Whitney U test was used for pairwise comparisons. For more than two groups, the Kruskal-Wallis H test with Dunn-Bonferroni post hoc analysis was performed. Area under the receiver operating characteristic curve (AUC) for both scores was calculated to predict right heart strain incidence, as were optimal cutoff values with a maximum sensitivity and specificity model, and a maximum positive and negative predictive value model (15,17). AUCs were tested on significant difference by using the DeLong, DeLong, and Clarke-Pearson method (18).

Results

Patient Characteristics

Structured PE reports from 1248 patients (mean age, 63 years ± 17 [standard deviation]; 639 men) were retrieved from the radiologic information system database. The extracted data yielded 357 patients with PE (28.6%) and 878 patients without PE (70.4%). Interpretations of 13 SRs (1.0%) were indeterminate for PE: eight with suspicion of PE (no thrombus assessment performed) and five with a nondiagnostic CT pulmonary angiography study (Fig 3). Table 1 provides an overview of patient characteristics and extracted SR items on CT pulmonary angiography study quality, PE occurrence, and heart metrics.

Figure 3:

Download as PowerPoint

Data Extraction from the PE Module

Of 357 PE-positive reports, 304 SRs (85.2%) contained adequate descriptions of clot site and degree of pulmonary artery occlusion to perform a script-based calculation of the PAOI_CBS. Table 2 provides an overview of the extracted PE module entries with clot distribution of PE by level on a per-patient basis (n = 304) and by location on a per-level basis (n = 1143) including degrees of occlusion of the main pulmonary and lobar arteries. PAOI_CBS calculation failed in the remaining 53 of 357 (14.8%) SRs. Manual review of these reports revealed deviations from the structure of the PE module because of free-text entries made after the final report was inserted into the radiologic information system text processor. In 45 of 53 (85%) SRs, the free-text records describing PE occurrence did not allow for a post hoc manual calculation of the PAOI_CBS.

Correlation of PAOI_CBS and PAOI_Q

A total of 948 emboli (median per patient, two; range, 1–11) in 304 patients with PE were depicted in the PAOI_Q assessment. Of these, 212 of 948 (22.4%) emboli were found at MPA level (right, 119 of 948 [12.6%]; left, 93 of 948 [9.8%]), 229 of 948 (24.2%) emboli at lobar level (right, 157 of 948 [16.6%]; left, 72 of 948 [7.6%]), 88 of 948 (9.3%) emboli at segmental level (right, 51 of 948 [5.4%]; left, 37 of 948 [3.9%]), and 419 of 948 (44.2%) emboli at subsegmental level (right, 194 of 948 [20.5%]; left, 225 of 948 [23.7%]), with a slight overall dominance of the right lung (521 of 948 [55%]). Figure 4A summarizes the occurrence of the most proximal embolus in the pulmonary artery tree on a per-level basis among the 304 PE-positive reports.

Figure 4:

Download as PowerPoint

Interobserver agreement was excellent for PAOI_Q assessment (ICC, 0.98; P < .001) and RLr measurements (ICC, 0.93; P < .001). The PAOI_CBS correlated strongly with the PAOI_Q (n = 304; r = 0.94; P < .001) (Fig 4B). Subgrouping on the basis of the level of PE occurrence revealed a high positive correlation for PAOI_CBS and PAOI_Q at the MPA level (n = 134; r = 0.90) and high positive correlations at lobar (n = 82; r = 0.83) and segmental and/or subsegmental level (n = 88; r = 0.70) (each P < .001). Furthermore, the SR-derived RLr showed a high correlation with the assessed RLr (n = 302; r = 0.73; P < .001) (Fig 4C).

Diagnostic Performance of PAOI_CBS and PAOI_Q

According to the PAOI_CBS, patients with PE were assigned to one of three PE severity levels (low, <25%; intermediate, 25%–50%; high, >50%), yielding a stepwise increase of the mean RLr from low (RLr, 0.96 ± 0.25) to intermediate (RLr, 1.11 ± 0.36; P = .04) and intermediate to high (RLr, 1.40 ± 0.46; P < .001) PE severity level (Fig 5A). A PAOI of 27.5% was identified as the cutoff for both scores in the maximum sensitivity and specificity model, and PAOIs of 40% and 27.5% for PAOI_CBS and PAOI_Q, respectively, in the maximum positive and negative predictive value model. The respective measures of diagnostic performance are shown in Table 3. AUCs were 0.75 (95% CI: 0.70, 0.81) for PAOI_CBS and 0.77 (95% CI: 0.72, 0.83) for PAOI_Q (Fig 5B). We did not find evidence of differences in AUC between both scores (P = .68).

Figure 5:

Download as PowerPoint

Time and Confidence of Score Assessment

Scoring time and diagnostic confidence are shown in Table 4. Absolute agreement and correlation between PAOI_CBS and PAOI_Q score values were high (ICC, 0.95; Fig 6A). Interobserver agreement was excellent for both scores, yielding ICCs of 0.95 for PAOI_Q (95% CI: 0.91, 0.98; P < .001; Fig 6B) and of 0.99 for PAOI_CBS (95% CI: 0.99, 1.0; P < .001; Fig 6C). Scoring time differed between the two scores regarding proximal PE occurrence, showing mean time savings for PAOI_CBS of 2.47 minutes ± 1.47 (time saving, 56.8% ± 26.9; P < .001) at the lobar level and of 1.93 minutes ± 1.12 (time saving, 50.8% ± 21.5; P < .001) at the segmental level but not at the MPA level, where we observed a trend toward a shortened scoring time for PAOI_CBS by 0.72 minutes ± 0.92 (time saving, 34.2% ± 36.0; P = .24). Confidence in assessing the scores yielded differences at lobar (PAOI_CBS, 4.25 ± 0.44; PAOI_Q, 3.15 ± 0.93; P < .001) and segmental level (PAOI_CBS, 3.95 ± 0.69; PAOI_Q, 3.25 ± 1.07; P = .04), but we did not find evidence of differences at MPA level (PAOI_CBS, 4.40 ± 0.50; PAOI_Q, 4.35 ± 0.59; P = .99).

Figure 6:

Download as PowerPoint

Discussion

We successfully implemented a text mining pipeline to extract key features from 1248 structured reports and established a simplified CT angiography scoring system for pulmonary embolism severity that strongly correlated with the Qanadli score as a conventional clot burden index (r = 0.94, P < .001). Use of the simplified score (ie, the pulmonary artery obstruction index clot burden score) yielded overall time savings (1.3 minutes vs 3.0 minutes; P < .001), higher confidence levels (4.2 vs 3.6; P < .001), and a higher interobserver agreement (intraclass correlation coefficient, 0.99 vs 0.95) compared with the Qanadli score. The simplified and Qanadli scores agreed favorably in predicting right heart strain (area under the curve, 0.75 vs 0.77; P = .68), with cutoff values of 27.5% for each score in the maximum sensitivity and specificity model.

Because of the consistent layout, structure, and vocabulary of the SR, we were able to use regular expressions as a rule-based pattern-matching syntax to mine for SR template items and PE descriptors on clot site and degree of pulmonary artery occlusion. Regular expressions are a powerful tool for efficient and accurate extraction of sequential patterns even from large sets of data sequences but have drawbacks in detecting unexpected semantic variations (19). Consequently, PAOI_CBS calculation failed in 53 of 357 SRs (14.8%) wherein the predefined PE descriptors were replaced by free-text entries. This issue could be overcome by more sophisticated natural language processing techniques (eg, PeFinder or natural language processing libraries such as NILE and pyConText), which rely on controlled vocabulary and grammatical rules to analyze free text in PE reports but also require development of lexical cues and context terms before data extraction (20,21). Deep learning–based natural language processing techniques may organically identify the important words and phrases in the report without previous definition of rules regarding which key elements were important for data extraction (22). However, in 45 of the 53 SRs (85%), even manual score calculation was not possible for lack of adequate PE description, resulting in a method-related failure of PAOI_CBS calculation for eight SRs. Compared with other natural language processing techniques, our data mining approach is easy to implement and may therefore be suitable for sites without advanced natural language processing knowledge to perform analyses for research purposes, provided that data are available in the form of highly structured and preferably disaggregated SRs (23).

Conformity of both scores was evaluated by comparing their impact on right heart strain, represented by the RLr as one of the CT signs shown to be independently associated with clinically severe PE and short-term mortality (24,25). Subgrouping into three severity levels on the basis of PAOI_CBS revealed a stepwise increase of the RLr, which was suggested as marker for right heart strain at cutoff values varying between 1.0 and 1.5 when measurements were performed with a standard axial view of the heart (26–28). In one study, an RLr cutoff value of 1.0 was found to be predictive of short-term mortality (25) whereas other studies failed to prove any association between an increased RLr and mortality (24,27,29). In our study, an RLr greater than or equal to 1.0 was considered to be the cutoff for right heart strain to compare the diagnostic performance of PAOI_CBS and PAOI_Q in predicting right heart strain incidence. We found no evidence of a difference in AUCs between the scores (PAOI_Q vs PAOI_CBS AUC, 0.77 vs 0.75, respectively; P = .68). The PAOI_CBS cutoff value of 27.5% compared favorably with two previous studies that found cutoff values of 27.5% and 23.7% with AUCs of 0.79 and 0.72, respectively, for PAOI_Q in predicting right heart strain (5,30). However, these studies assessed right heart strain by using transthoracic echocardiography. Although some studies have demonstrated a good correlation between CT- and transthoracic echocardiography–derived signs of right heart strain (31,32), the PAOI_CBS warrants further evaluation as a prognostic marker for transthoracic echocardiography–confirmed right heart strain and other outcome-relevant parameters, such as intensive care unit admission and mortality.

Our study had limitations. First, the SR-derived data represent structured but nonstandardized measurements by 67 residents who provided PE interpretations at our radiology department 24 hours per day, 7 days per week. Although all radiologic examinations were subsequently approved by a board-certified radiologist, this could lead to interobserver and shift-dependent interpretive discrepancies and thus inconsistency of recorded data, as was shown for teleradiological reports in a recent study (33). Before study inclusion, SR data were not reviewed for completeness and reporting quality, nor were the PE module entries tested for accuracy of clot reporting. This was primarily to demonstrate that unbiased SR data conform to standardized collected parameters in our study for internal validation and the parameters known from literature for external validation purposes. Nonetheless, our data handling may lead to discrepancies in the calculated PAOI_CBS score values and RLr measurements and, thus, to distortions in the statistical analyses, and should be considered in interpreting the results. Second, we evaluated the PAOI_CBS on the basis of the PAOI_Q as another semiquantitative clot burden index and analyzed both scores in relation to the RLr measured in the same examinations. As mentioned above, external validation of PAOI_CBS with a second modality such as transthoracic echocardiography is needed, as are correlations with clinically relevant primary end points.

In conclusion, our study stresses the role of using structured radiologic report data for research purposes. We used a script-based data mining approach to extract structured report content for the development and validation of a pulmonary embolism (PE) scoring system. This represents a time-saving simplification of the Qanadli score for PE clot burden index calculation and may prove to be beneficial for future research.

Disclosures of Conflicts of Interest: M.A.F. disclosed no relevant relationships. V.L.M. disclosed no relevant relationships. T.S. disclosed no relevant relationships. C.S. disclosed no relevant relationships. R.S. disclosed no relevant relationships. J.K. disclosed no relevant relationships. T.F.W. disclosed no relevant relationships. H.U.K. disclosed payments to institution for grants from Bayer, Siemens, Philips; payment to author for honoraria for lectures, presentations, speakers bureaus, manuscript writing or educational events from AstraZeneca, Philips, Boehringer Ingelheim, Merck Sharp Dohme.

Author Contributions

Author contributions: Guarantor of integrity of entire study, M.A.F.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, M.A.F., V.L.M., J.K.; clinical studies, M.A.F., V.L.M., T.S., H.U.K.; experimental studies, M.A.F., V.L.M., C.S., R.S.; statistical analysis, M.A.F., C.S., J.K.; and manuscript editing, M.A.F., V.L.M., T.S., C.S., J.K., T.F.W., H.U.K.