How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide

Published Online:Mar 14 2023https://doi.org/10.1148/radiol.221437

Abstract

This article provides a practical guide for clinicians and radiologists to critically evaluate diagnostic accuracy systematic reviews when using them to inform clinical decisions.

Systematic reviews of diagnostic accuracy studies can provide the best available evidence to inform decisions regarding the use of a diagnostic test. In this guide, the authors provide a practical approach for clinicians to appraise diagnostic accuracy systematic reviews and apply their results to patient care. The first step is to identify an appropriate systematic review with a research question matching the clinical scenario. The user should evaluate the rigor of the review methods to evaluate its credibility (Did the review use clearly defined eligibility criteria, a comprehensive search strategy, structured data collection, risk of bias and applicability appraisal, and appropriate meta-analysis methods?). If the review is credible, the next step is to decide whether the diagnostic performance is adequate for clinical use (Do sensitivity and specificity estimates exceed the threshold that makes them useful in clinical practice? Are these estimates sufficiently precise? Is variability in the estimates of diagnostic accuracy across studies explained?). Diagnostic accuracy systematic reviews that are judged to be credible and provide diagnostic accuracy estimates with sufficient certainty and relevance are the most useful to inform patient care. This review discusses comparative, noncomparative, and emerging approaches to systematic reviews of diagnostic accuracy using a clinical scenario and examples based on recent publications.

References

1. Salameh JP, Bossuyt PM, McGrath TA, et al. Preferred reporting items for systematic review and meta-analysis of diagnostic test accuracy studies (PRISMA-DTA): explanation, elaboration, and checklist. BMJ 2020;370:m2632. Crossref, Medline, Google Scholar
2. Cohen JF, Deeks JJ, Hooft L, et al. Preferred reporting items for journal and conference abstracts of systematic reviews and meta-analyses of diagnostic test accuracy studies (PRISMA-DTA for Abstracts): checklist, explanation, and elaboration. BMJ 2021;372:n265. Crossref, Medline, Google Scholar
3. Frank RA, McInnes MDF, Levine D, et al. Are Study and Journal Characteristics Reliable Indicators of “Truth” in Imaging Research? Radiology 2018;287(1):215–223. Link, Google Scholar
4. Higgins JPT, Thomas J. Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.0. (updated July 2019). Cochrane, 2019. https://training.cochrane.org/handbook. Google Scholar
5. Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA 2005;293(19):2362–2366. Crossref, Medline, Google Scholar
6. Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines; Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press, 2011. https://www.ncbi.nlm.nih.gov/books/NBK209539/. Crossref, Google Scholar
7. Murad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA 2014;312(2):171–179. Crossref, Medline, Google Scholar
8. Halligan S, Altman DG. Evidence-based practice in radiology: steps 3 and 4--appraise and apply systematic reviews and meta-analyses. Radiology 2007;243(1):13–27. Link, Google Scholar
9. Deeks J, Bossuyt P, Leeflang M, Takwoingi Y, Flemyng E. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 2.0. London: Cochrane, 2022. https://training.cochrane.org/handbook-diagnostic-test-accuracy. Accessed September 2022. Google Scholar
10. University of Oxford. Systematic Reviews - Centre for Evidence-Based Medicine. 2022. https://www.cebm.ox.ac.uk/resources/systematic-reviews?07d56b76-140c-11ed-8e22-0a442fc5b724. Accessed September 2022. Google Scholar
11. Balogh EP, Miller BT, Ball JR, eds. Committee on Diagnostic Error in Health Care; Board on Health Care Services; Institute of Medicine; The National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press, 2015. https://www.ncbi.nlm.nih.gov/books/NBK338596/. Google Scholar
12. Singh H, Graber ML. Improving Diagnosis in Health Care--The Next Imperative for Patient Safety. N Engl J Med 2015;373(26):2493–2495. Crossref, Medline, Google Scholar
13. McGrath TA, McInnes MD, Korevaar DA, Bossuyt PM. Meta-Analyses of Diagnostic Accuracy in Imaging Journals: Analysis of Pooling Techniques and Their Effect on Summary Estimates of Diagnostic Accuracy. Radiology 2016;281(1):78–85. Link, Google Scholar
14. McGrath TA, Alabousi M, Skidmore B, et al. Recommendations for reporting of systematic reviews and meta-analyses of diagnostic test accuracy: a systematic review. Syst Rev 2017;6(1):194. Crossref, Medline, Google Scholar
15. McGrath TA, McInnes MDF, van Es N, Leeflang MMG, Korevaar DA, Bossuyt PMM. Overinterpretation of Research Findings: Evidence of “Spin” in Systematic Reviews of Diagnostic Accuracy Studies. Clin Chem 2017;63(8):1353–1362. Crossref, Medline, Google Scholar
16. McGrath TA, McInnes MDF, Langer FW, Hong J, Korevaar DA, Bossuyt PMM. Treatment of multiple test readers in diagnostic accuracy systematic reviews-meta-analyses of imaging studies. Eur J Radiol 2017;93:59–64. Crossref, Medline, Google Scholar
17. McInnes MD, Bossuyt PM. Pitfalls of Systematic Reviews and Meta-Analyses in Imaging Research. Radiology 2015;277(1):13–21. Link, Google Scholar
18. McInnes MDF, Moher D, Thombs BD, et althe PRISMA-DTA Group. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA 2018;319(4):388–396 [Published correction appears in JAMA 2019;322(20):2026.]. Crossref, Medline, Google Scholar
19. Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM; Cochrane Diagnostic Test Accuracy Working Group. Systematic reviews of diagnostic test accuracy. Ann Intern Med 2008;149(12):889–897. Crossref, Medline, Google Scholar
20. Tunis AS, McInnes MD, Hanna R, Esmail K. Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement? Radiology 2013;269(2):413–426. Link, Google Scholar
21. Willis BH, Quigley M. Uptake of newer methodological developments and the deployment of meta-analysis in diagnostic test research: a systematic review. BMC Med Res Methodol 2011;11(1):27. Crossref, Medline, Google Scholar
22. Willis BH, Quigley M. The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol 2011;11(1):163. Crossref, Medline, Google Scholar
23. McGrath TA, Bowdridge JC, Prager R, et al. Overinterpretation of Research Findings: Evaluation of “Spin” in Systematic Reviews of Diagnostic Accuracy Studies in High-Impact Factor Journals. Clin Chem 2020;66(7):915–924. Crossref, Medline, Google Scholar
24. Duke E, Kalb B, Arif-Tiwari H, et al. A Systematic Review and Meta-Analysis of Diagnostic Performance of MRI for Evaluation of Acute Appendicitis. AJR Am J Roentgenol 2016;206(3):508–517. Crossref, Medline, Google Scholar
25. Salameh JP, McInnes MDF, Moher D, et al. Completeness of Reporting of Systematic Reviews of Diagnostic Test Accuracy Based on the PRISMA-DTA Reporting Guideline. Clin Chem 2019;65(2):291–301. Crossref, Medline, Google Scholar
26. Hong PJ, Korevaar DA, McGrath TA, et al. Reporting of imaging diagnostic accuracy studies with focus on MRI subgroup: Adherence to STARD 2015. J Magn Reson Imaging 2018;47(2):523–544. Crossref, Medline, Google Scholar
27. Whiting P, Savović J, Higgins JP, et alROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225–234. Crossref, Medline, Google Scholar
28. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282(11):1061–1066. Crossref, Medline, Google Scholar
29. de Vet HCW, Eisinga A, Riphagen II, Aertgeerts B, Pewsner D. Searching for Studies. In: Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 0.4, Chapter 7. London, UK: The Cochrane Collaboration, 2008. [updated September 2008.] Google Scholar
30. Whiting P, Westwood M, Beynon R, Burke M, Sterne JA, Glanville J. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clin Epidemiol 2011;64(6):602–607. Crossref, Medline, Google Scholar
31. Beynon R, Leeflang MM, McDonald S, et al. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev 2013;2013(9):MR000022. Medline, Google Scholar
32. Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess 2003;7(41):1–90. Crossref, Medline, Google Scholar
33. Whiting PF, Rutjes AW, Westwood ME, et alQUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155(8):529–536. Crossref, Medline, Google Scholar
34. Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005;58(10):982–990. Crossref, Medline, Google Scholar
35. Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med 2001;20(19):2865–2884. Crossref, Medline, Google Scholar
36. Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies. BMC Med Res Methodol 2016;16(1):97. Crossref, Medline, Google Scholar
37. Bossuyt PM, Takwoingi Y, Deeks J, Gatsonis C. Chapter 10: Understanding meta-analysis Draft version. (March 28, 2022). In: Deeks JJ, Bossuyt PM, Leeflang MM, Takwoingi Y, eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 2. London, England: Cochrane. Google Scholar
38. Ebrahimzadeh S, Islam N, Dawit H, et al; Cochrane COVID-19 Diagnostic Test Accuracy Group. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev 2022;5:CD013639. Medline, Google Scholar
39. Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol 2008;8(1):79. Crossref, Medline, Google Scholar
40. Zhou Y, Dendukuri N. Statistics for quantifying heterogeneity in univariate and bivariate meta-analyses of binary data: the case of meta-analyses of diagnostic accuracy. Stat Med 2014;33(16):2701–2717. Crossref, Medline, Google Scholar
41. Kuo RYL, Harrison C, Curran TA, et al. Artificial Intelligence in Fracture Detection: A Systematic Review and Meta-Analysis. Radiology 2022;304(1):50–62. Link, Google Scholar
42. Ghazi Sherbaf F, Sair HI, Shakoor D, et al. DECT in Detection of Vertebral Fracture-associated Bone Marrow Edema: A Systematic Review and Meta-Analysis with Emphasis on Technical and Imaging Interpretation Parameters. Radiology 2021;300(1):110–119. Link, Google Scholar
43. Eng KA, Abadeh A, Ligocki C, et al. Acute Appendicitis: A Meta-Analysis of the Diagnostic Accuracy of US, CT, and MRI as Second-Line Imaging Tests after an Initial US. Radiology 2018;288(3):717–727. Link, Google Scholar
44. Leeflang MMG, Reitsma JB. Systematic reviews and meta-analyses addressing comparative test accuracy questions. Diagn Progn Res 2018;2(1):17. Crossref, Medline, Google Scholar
45. Takwoingi Y, Partlett C, Riley RD, Hyde C, Deeks JJ. Methods and reporting of systematic reviews of comparative accuracy were deficient: a methodological survey and proposed guidance. J Clin Epidemiol 2020;121:1–14. Crossref, Medline, Google Scholar
46. Dehmoobad Sharifabadi A, Leeflang M, Treanor L, et al. Comparative reviews of diagnostic test accuracy in imaging research: evaluation of current practices. Eur Radiol 2019;29(10):5386–5394. Crossref, Medline, Google Scholar
47. Yang B, Vali Y, Dehmoobad Sharifabadi A, et al. Risk of bias assessment of test comparisons was uncommon in comparative accuracy systematic reviews: an overview of reviews. J Clin Epidemiol 2020;127:167–174. Crossref, Medline, Google Scholar
48. Takwoingi Y, Leeflang MM, Deeks JJ. Empirical evidence of the importance of comparative studies of diagnostic test accuracy. Ann Intern Med 2013;158(7):544–554. Crossref, Medline, Google Scholar
49. Yang B, Olsen M, Vali Y, et al. Study designs for comparative diagnostic test accuracy: A methodological review and classification scheme. J Clin Epidemiol 2021;138:128–138. Crossref, Medline, Google Scholar
50. Yang B, Mallett S, Takwoingi Y, et alQUADAS-C Group†. QUADAS-C: A Tool for Assessing Risk of Bias in Comparative Diagnostic Accuracy Studies. Ann Intern Med 2021;174(11):1592–1599. Crossref, Medline, Google Scholar
51. Chan KK, Joo DA, McRae AD, et al. Chest ultrasonography versus supine chest radiography for diagnosis of pneumothorax in trauma patients in the emergency department. Cochrane Database Syst Rev 2020;7:CD013031. Medline, Google Scholar
52. Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Analysing and Presenting Results. In: Deeks JJ, Bossuyt PM, Gatsonis C, eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0, Chapter 10. London, UK: The Cochrane Collaboration, 2010. http://srdta.cochrane.org/. Google Scholar
53. Takwoingi Y, Riley RD, Deeks JJ. Meta-analysis of diagnostic accuracy studies in mental health. Evid Based Ment Health 2015;18(4):103–109. Crossref, Medline, Google Scholar
54. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004;23(20):3105–3124. Crossref, Medline, Google Scholar
55. Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015;162(11):777–784. Crossref, Medline, Google Scholar
56. Zarin W, Veroniki AA, Nincic V, et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med 2017;15(1):3 [Published correction appears in BMC Med 2017;15(1):61.]. Crossref, Medline, Google Scholar
57. Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. Med Decis Making 2013;33(5):607–617. Crossref, Medline, Google Scholar
58. Ades AE, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. Evidence synthesis for decision making 7: a reviewer’s checklist. Med Decis Making 2013;33(5):679–691. Crossref, Medline, Google Scholar
59. Dias S, Sutton AJ, Welton NJ, Ades AE. Evidence synthesis for decision making 3: heterogeneity--subgroups, meta-regression, bias, and bias-adjustment. Med Decis Making 2013;33(5):618–640. Crossref, Medline, Google Scholar
60. Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades AE. Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials. Med Decis Making 2013;33(5):641–656. Crossref, Medline, Google Scholar
61. Puhan MA, Schünemann HJ, Murad MH, et alGRADE Working Group. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 2014;349:g5630. Crossref, Medline, Google Scholar
62. Brignardello-Petersen R, Bonner A, Alexander PE, et alGRADE Working Group. Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol 2018;93:36–44 [Published correction appears in J Clin Epidemiol 2018;98:162.]. Crossref, Medline, Google Scholar
63. Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health 2014;17(2):157–173. Crossref, Medline, Google Scholar
64. Filipe MD, Patuleia SIS, de Jong VMT, Vriens MR, van Diest PJ, Witkamp AJ. Network Meta-analysis for the Diagnostic Approach to Pathologic Nipple Discharge. Clin Breast Cancer 2020;20(6):e723–e748. Crossref, Medline, Google Scholar
65. Veroniki AA, Tsokani S, Rücker G, Mavridis D, Takwoingi Y. Challenges in Comparative Meta-Analysis of the Accuracy of Multiple Diagnostic Tests. Methods Mol Biol 2022;2345:299–316. Crossref, Medline, Google Scholar
66. Veroniki A, Tsokani S, Paraskevaidas E, Mavridis D. Evaluating multiple diagnostic tests: an application to cervical cancer. HJOG 2021;20(1):11–24. Crossref, Google Scholar
67. Owen RK, Cooper NJ, Quinn TJ, Lees R, Sutton AJ. Network meta-analysis of diagnostic test accuracy studies identifies and ranks the optimal diagnostic tests and thresholds for health care policy and decision-making. J Clin Epidemiol 2018;99:64–74. Crossref, Medline, Google Scholar
68. Menten J, Lesaffre E. A general framework for comparative Bayesian meta-analysis of diagnostic studies. BMC Med Res Methodol 2015;15(1):70. Crossref, Medline, Google Scholar
69. Trikalinos TA, Hoaglin DC, Small KM, Terrin N, Schmid CH. Methods for the joint meta-analysis of multiple tests. Res Synth Methods 2014;5(4):294–312. Crossref, Medline, Google Scholar
70. Nyaga VN, Aerts M, Arbyn M. ANOVA model for network meta-analysis of diagnostic test accuracy data. Stat Methods Med Res 2018;27(6):1766–1784. Crossref, Medline, Google Scholar
71. Veroniki AA, Tsokani S, Agarwal R, et al. Diagnostic test accuracy network meta-analysis methods: A scoping review and empirical assessment. J Clin Epidemiol 2022;146:86–96. Crossref, Medline, Google Scholar
72. van der Pol CB, McInnes MDF, Salameh JP, et al. CT/MRI and CEUS LI-RADS Major Features Association with Hepatocellular Carcinoma: Individual Patient Data Meta-Analysis. Radiology 2022;302(2):326–335. Link, Google Scholar
73. van der Pol CB, McInnes MDF, Salameh JP, et al; LI-RADS IPD Group Collaborators. Impact of Reference Standard on CT, MRI, and Contrast-enhanced US LI-RADS Diagnosis of Hepatocellular Carcinoma: A Meta-Analysis. Radiology 2022;303(3):544–545. Link, Google Scholar
74. van der Pol CB, Lim CS, Sirlin CB, et al. Accuracy of the Liver Imaging Reporting and Data System in Computed Tomography and Magnetic Resonance Image Analysis of Hepatocellular Carcinoma or Overall Malignancy-A Systematic Review. Gastroenterology 2019;156(4):976–986. Crossref, Medline, Google Scholar
75. Levis B, Benedetti A, Levis AW, et al. Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool. Am J Epidemiol 2017;185(10):954–964. Crossref, Medline, Google Scholar
76. Singh S, Venkatesh SK, Wang Z, et al. Diagnostic performance of magnetic resonance elastography in staging liver fibrosis: a systematic review and meta-analysis of individual participant data. Clin Gastroenterol Hepatol 2015;13(3):440–451.e6. Crossref, Medline, Google Scholar
77. Petroff D, Blank V, Newsome PN, et al. Assessment of hepatic steatosis by controlled attenuation parameter using the M and XL probes: an individual patient data meta-analysis. Lancet Gastroenterol Hepatol 2021;6(3):185–198. Crossref, Medline, Google Scholar
78. Stewart LA, Clarke M, Rovers M, et alPRISMA-IPD Development Group. Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. JAMA 2015;313(16):1657–1665. Crossref, Medline, Google Scholar
79. Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010;340(feb05 1):c221. Crossref, Medline, Google Scholar
80. Living systematic reviews. Cochrane Community. https://community.cochrane.org/review-production/production-resources/living-systematic-reviews. Accessed March 22, 2022. Google Scholar
81. Islam N, Salameh JP, Leeflang MM, et al; Cochrane COVID-19 Diagnostic Test Accuracy Group. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev 2020;11:CD013639. Medline, Google Scholar
82. Islam N, Ebrahimzadeh S, Salameh JP, et al; Cochrane COVID-19 Diagnostic Test Accuracy Group. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev 2021;3:CD013639. Medline, Google Scholar
83. Salameh JP, Leeflang MM, Hooft L, et al; Cochrane COVID-19 Diagnostic Test Accuracy Group. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev 2020;9:CD013639. Medline, Google Scholar
84. Kim HY, Park JH, Lee YJ, Lee SS, Jeon JJ, Lee KH. Systematic Review and Meta-Analysis of CT Features for Differentiating Complicated and Uncomplicated Appendicitis. Radiology 2018;287(1):104–115. Link, Google Scholar

Article History

Received: June 19 2022
Revision requested: July 19 2022
Revision received: Oct 25 2022
Accepted: Oct 28 2022
Published online: Mar 14 2023

Tweetorial »

Recently Published

Abbreviations

Abbreviations:
IPD	individual participant data
NMA	network meta-analysis
QUADAS	Quality Assessment of Diagnostic Accuracy Studies
QUADAS-2	Quality Assessment of Diagnostic Accuracy Studies 2
ROC	receiver operating characteristic

Metrics

Altmetric Score

PDF download