Development and Validation of Electronic Health Record–based Triggers to Detect Delays in Follow-up of Abnormal Lung Imaging Findings

Published Online:

An electronic health record–based trigger algorithm was developed to identify patients who are at risk for delayed follow-up of abnormal imaging findings. The trigger achieved a positive predictive value of 57.3% and shows promise in detecting diagnostic delays in patients with imaging findings suspicious for lung cancer.


To develop an electronic health record (EHR)–based trigger algorithm to identify delays in follow-up of patients with imaging results that are suggestive of lung cancer and to validate this trigger on retrospective data.

Materials and Methods

The local institutional review board approved the study. A “trigger” algorithm was developed to automate the detection of delays in diagnostic evaluation of chest computed tomographic (CT) images and conventional radiographs that were electronically flagged by reviewing radiologists as being “suspicious for malignancy.” The trigger algorithm was developed through literature review and expert input. It included patients who were alive and 40–70 years old, and it excluded instances in which appropriate timely follow-up (defined as occurring within 30 days) was detected (eg, pulmonary visit) or when follow-up was unnecessary (eg, in patients with a terminal illness). The algorithm was iteratively applied to a retrospective test cohort in an EHR data warehouse at a large Veterans Affairs facility, and manual record reviews were used to validate each individual criterion. The final algorithm aimed at detecting an absence of timely follow-up was retrospectively applied to an independent validation cohort to determine the positive predictive value (PPV). Trigger performance, time to follow-up, reasons for lack of follow-up, and cancer outcomes were analyzed and reported by using descriptive statistics.


The trigger algorithm was retrospectively applied to the records of 89 168 patients seen between January 1, 2009, and December 31, 2009. Of 538 records with an imaging report that was flagged as suspicious for malignancy, 131 were identified by the trigger as being high risk for delayed diagnostic evaluation. Manual chart reviews confirmed a true absence of follow-up in 75 cases (trigger PPV of 57.3% for detecting evaluation delays), of which four received a diagnosis of primary lung cancer within the subsequent 2 years.


EHR-based triggers can be used to identify patients with suspicious imaging findings in whom follow-up diagnostic evaluation was delayed.

© RSNA, 2015


  • 1. Singh H, Sethi S, Raber M, Petersen LA. Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol 2007;25(31):5009–5018. Crossref, MedlineGoogle Scholar
  • 2. Singh H, Hirani K, Kadiyala H, et al. Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study. J Clin Oncol 2010;28(20):3307–3315. Crossref, MedlineGoogle Scholar
  • 3. Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med 2006;145(7):488–496. Crossref, MedlineGoogle Scholar
  • 4. Phillips RL Jr, Bartholomew LA, Dovey SM, Fryer GE Jr, Miyoshi TJ, Green LA. Learning from malpractice claims about negligent, adverse events in primary care in the United States. Qual Saf Health Care 2004;13(2):121–126. Crossref, MedlineGoogle Scholar
  • 5. Brenner RJ, Lucey LL, Smith JJ, Saunders R. Radiology and medical malpractice claims: a report on the practice standards claims survey of the Physician Insurers Association of America and the American College of Radiology. AJR Am J Roentgenol 1998;171(1):19–22. Crossref, MedlineGoogle Scholar
  • 6. Berlin L, Murphy DR, Singh H. Breakdowns in communication of radiological findings: an ethical and medico-legal conundrum. Diagnosis 2014;1(4):263–268. CrossrefGoogle Scholar
  • 7. Poon EG, Kachalia A, Puopolo AL, Gandhi TK, Studdert DM. Cognitive errors and logistical breakdowns contributing to missed and delayed diagnoses of breast and colorectal cancers: a process analysis of closed malpractice claims. J Gen Intern Med 2012;27(11):1416–1423. Crossref, MedlineGoogle Scholar
  • 8. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med 2005;165(13):1493–1499. Crossref, MedlineGoogle Scholar
  • 9. Singh H, Giardina TD, Petersen LA, et al. Exploring situational awareness in diagnostic errors in primary care. BMJ Qual Saf 2012;21(1):30–38. Crossref, MedlineGoogle Scholar
  • 10. Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012;125(2):209.e1–e7. Crossref, MedlineGoogle Scholar
  • 11. Murphy DR, Reis B, Kadiyala H, et al. Electronic health record-based messages to primary care providers: valuable information or just noise? Arch Intern Med 2012;172(3):283–285. Crossref, MedlineGoogle Scholar
  • 12. McDonald CJ, McDonald MH. Electronic medical records and preserving primary care physicians’ time: comment on “electronic health record-based messages to primary care providers”. Arch Intern Med 2012;172(3):285–287. Crossref, MedlineGoogle Scholar
  • 13. Murphy DR, Singh H, Berlin L. Communication breakdowns and diagnostic errors: a radiology perspective. Diagnosis 2014;1(4):253–261. CrossrefGoogle Scholar
  • 14. Menon S, Smith MW, Sittig DF, et al. How context affects electronic health record-based test result follow-up: a mixed-methods evaluation. BMJ Open 2014;4(11):e005985. CrossrefGoogle Scholar
  • 15. Al-Mutairi A, Meyer AND, Chang P, Singh H. Lack of timely follow-up of abnormal imaging results and radiologists’ recommendations. J Am Coll Radiol 2015;12(4):385–389. Crossref, MedlineGoogle Scholar
  • 16. Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med 2011;365(19):1758–1759. Crossref, MedlineGoogle Scholar
  • 17. Szekendi MK, Sullivan C, Bobb A, et al. Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care 2006;15(3):184–190. Crossref, MedlineGoogle Scholar
  • 18. Classen DC, Resar R, Griffin F, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood) 2011;30(4):581–589. Crossref, MedlineGoogle Scholar
  • 19. Jha AK, Classen DC. Getting moving on patient safety: harnessing electronic data for safer care. N Engl J Med 2011;365(19):1756–1758. Crossref, MedlineGoogle Scholar
  • 20. Zalis M, Harris M. Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol 2010;7(8):625–633. Crossref, MedlineGoogle Scholar
  • 21. Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003;12(Suppl 2):ii39–ii45. MedlineGoogle Scholar
  • 22. Honigman B, Light P, Pulling RM, Bates DW. A computerized method for identifying incidents associated with adverse drug events in outpatients. Int J Med Inform 2001;61(1):21–32. Crossref, MedlineGoogle Scholar
  • 23. Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients: 1991. Qual Saf Health Care 2005;14(3):221–225; discussion 225–226. Crossref, MedlineGoogle Scholar
  • 24. Mull HJ, Nebeker JR. Informatics tools for the development of action-oriented triggers for outpatient adverse drug events. AMIA Annu Symp Proc 2008 Nov 6:505–509. MedlineGoogle Scholar
  • 25. Knirsch CA, Jain NL, Pablos-Mendez A, Friedman C, Hripcsak G. Respiratory isolation of tuberculosis patients using clinical guidelines and an automated clinical decision support system. Infect Control Hosp Epidemiol 1998;19(2):94–100. Crossref, MedlineGoogle Scholar
  • 26. Muething SE, Conway PH, Kloppenborg E, et al. Identifying causes of adverse events detected by an automated trigger tool through in-depth analysis. Qual Saf Health Care 2010;19(5):435–439. MedlineGoogle Scholar
  • 27. Murphy DR, Laxmisan A, Reis BA, et al. Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf 2014;23(1):8–16. Crossref, MedlineGoogle Scholar
  • 28. Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med 2009;169(17):1578–1586. Crossref, MedlineGoogle Scholar
  • 29. Singh H, Arora HS, Vij MS, Rao R, Khan MM, Petersen LA. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14(4):459–466. Crossref, MedlineGoogle Scholar
  • 30. Veteran Population. National Center for Veterans Analysis and Statistics. Accessed December 11, 2014. Google Scholar
  • 31. Giardina TD, King BJ, Ignaczak AP, et al. Root cause analysis reports help identify common factors in delayed diagnosis and treatment of outpatients. Health Aff (Millwood) 2013;32(8):1368–1375. Crossref, MedlineGoogle Scholar
  • 32. Poon EG, Haas JS, Louise Puopolo A, et al. Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med 2004;19(4):316–323. Crossref, MedlineGoogle Scholar
  • 33. Gordon JRS, Wahls T, Carlos RC, Pipinos II, Rosenthal GE, Cram P. Failure to recognize newly identified aortic dilations in a health care system with an advanced electronic medical record. Ann Intern Med 2009;151(1):21–27, W5. Crossref, MedlineGoogle Scholar
  • 34. Schiff GD, Kim S, Krosnjar N, et al. Missed hypothyroidism diagnosis uncovered by linking laboratory and pharmacy data. Arch Intern Med 2005;165(5):574–577. Crossref, MedlineGoogle Scholar
  • 35. Hunnibell LS, Rose MG, Connery DM, et al. Using nurse navigation to improve timeliness of lung cancer care at a veterans hospital. Clin J Oncol Nurs 2012;16(1):29–36. Crossref, MedlineGoogle Scholar
  • 36. Hunnibell LS, Slatore CG, Ballard EA. Foundations for lung nodule management for nurse navigators. Clin J Oncol Nurs 2013;17(5):525–531. Crossref, MedlineGoogle Scholar
  • 37. McMullen L. Oncology nurse navigators and the continuum of cancer care. Semin Oncol Nurs 2013;29(2):105–117. Crossref, MedlineGoogle Scholar
  • 38. Choksi VR, Marn CS, Bell Y, Carlos R. Efficiency of a semiautomated coding and review process for notification of critical findings in diagnostic imaging. AJR Am J Roentgenol 2006;186(4):933–936. Crossref, MedlineGoogle Scholar
  • 39. Powell AA, Schultz EM, Ordin DL, et al. Timeliness across the continuum of care in veterans with lung cancer. J Thorac Oncol 2008;3(9):951–957. Crossref, MedlineGoogle Scholar
  • 40. Gould MK, Ghaus SJ, Olsson JK, Schultz EM. Timeliness of care in veterans with non-small cell lung cancer. Chest 2008;133(5):1167–1173. Crossref, MedlineGoogle Scholar
  • 41. Smith M, Murphy D, Laxmisan A, et al. Developing software to “track and catch” missed follow-up of abnormal test results in a complex sociotechnical environment. Appl Clin Inform 2013;4(3):359–375. Crossref, MedlineGoogle Scholar
  • 42. Singh H, Vij MS. Eight recommendations for policies for communicating abnormal test results. Jt Comm J Qual Patient Saf 2010;36(5):226–232. MedlineGoogle Scholar
  • 43. Nebeker J, Stoddard G, Rosen A. Considering Sensitivity and Positive Predictive Value in Comparing the Performance of Triggers Systems for Iatrogenic Adverse Events: Triggers and Targeted Injury Detection Systems (TIDS) Expert Panel Meeting. Rockville, Md: Agency for Healthcare Research and Quality, 2009. Google Scholar

Article History

Received October 30, 2014; revision requested December 11; revision received January 19, 2015; accepted January 30; final version accepted February 19.
Published online: May 11 2015
Published in print: Oct 2015