Development and Validation of Electronic Health Record–based Triggers to Detect Delays in Follow-up of Abnormal Lung Imaging Findings
Abstract
An electronic health record–based trigger algorithm was developed to identify patients who are at risk for delayed follow-up of abnormal imaging findings. The trigger achieved a positive predictive value of 57.3% and shows promise in detecting diagnostic delays in patients with imaging findings suspicious for lung cancer.
Purpose
To develop an electronic health record (EHR)–based trigger algorithm to identify delays in follow-up of patients with imaging results that are suggestive of lung cancer and to validate this trigger on retrospective data.
Materials and Methods
The local institutional review board approved the study. A “trigger” algorithm was developed to automate the detection of delays in diagnostic evaluation of chest computed tomographic (CT) images and conventional radiographs that were electronically flagged by reviewing radiologists as being “suspicious for malignancy.” The trigger algorithm was developed through literature review and expert input. It included patients who were alive and 40–70 years old, and it excluded instances in which appropriate timely follow-up (defined as occurring within 30 days) was detected (eg, pulmonary visit) or when follow-up was unnecessary (eg, in patients with a terminal illness). The algorithm was iteratively applied to a retrospective test cohort in an EHR data warehouse at a large Veterans Affairs facility, and manual record reviews were used to validate each individual criterion. The final algorithm aimed at detecting an absence of timely follow-up was retrospectively applied to an independent validation cohort to determine the positive predictive value (PPV). Trigger performance, time to follow-up, reasons for lack of follow-up, and cancer outcomes were analyzed and reported by using descriptive statistics.
Results
The trigger algorithm was retrospectively applied to the records of 89 168 patients seen between January 1, 2009, and December 31, 2009. Of 538 records with an imaging report that was flagged as suspicious for malignancy, 131 were identified by the trigger as being high risk for delayed diagnostic evaluation. Manual chart reviews confirmed a true absence of follow-up in 75 cases (trigger PPV of 57.3% for detecting evaluation delays), of which four received a diagnosis of primary lung cancer within the subsequent 2 years.
Conclusion
EHR-based triggers can be used to identify patients with suspicious imaging findings in whom follow-up diagnostic evaluation was delayed.
© RSNA, 2015
References
- 1. . Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol 2007;25(31):5009–5018. Crossref, Medline, Google Scholar
- 2. . Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study. J Clin Oncol 2010;28(20):3307–3315. Crossref, Medline, Google Scholar
- 3. . Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med 2006;145(7):488–496. Crossref, Medline, Google Scholar
- 4. . Learning from malpractice claims about negligent, adverse events in primary care in the United States. Qual Saf Health Care 2004;13(2):121–126. Crossref, Medline, Google Scholar
- 5. . Radiology and medical malpractice claims: a report on the practice standards claims survey of the Physician Insurers Association of America and the American College of Radiology. AJR Am J Roentgenol 1998;171(1):19–22. Crossref, Medline, Google Scholar
- 6. . Breakdowns in communication of radiological findings: an ethical and medico-legal conundrum. Diagnosis 2014;1(4):263–268. Crossref, Google Scholar
- 7. . Cognitive errors and logistical breakdowns contributing to missed and delayed diagnoses of breast and colorectal cancers: a process analysis of closed malpractice claims. J Gen Intern Med 2012;27(11):1416–1423. Crossref, Medline, Google Scholar
- 8. . Diagnostic error in internal medicine. Arch Intern Med 2005;165(13):1493–1499. Crossref, Medline, Google Scholar
- 9. . Exploring situational awareness in diagnostic errors in primary care. BMJ Qual Saf 2012;21(1):30–38. Crossref, Medline, Google Scholar
- 10. . Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012;125(2):209.e1–e7. Crossref, Medline, Google Scholar
- 11. . Electronic health record-based messages to primary care providers: valuable information or just noise? Arch Intern Med 2012;172(3):283–285. Crossref, Medline, Google Scholar
- 12. . Electronic medical records and preserving primary care physicians’ time: comment on “electronic health record-based messages to primary care providers”. Arch Intern Med 2012;172(3):285–287. Crossref, Medline, Google Scholar
- 13. . Communication breakdowns and diagnostic errors: a radiology perspective. Diagnosis 2014;1(4):253–261. Crossref, Google Scholar
- 14. . How context affects electronic health record-based test result follow-up: a mixed-methods evaluation. BMJ Open 2014;4(11):e005985. Crossref, Google Scholar
- 15. . Lack of timely follow-up of abnormal imaging results and radiologists’ recommendations. J Am Coll Radiol 2015;12(4):385–389. Crossref, Medline, Google Scholar
- 16. . Evidence-based medicine in the EMR era. N Engl J Med 2011;365(19):1758–1759. Crossref, Medline, Google Scholar
- 17. . Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care 2006;15(3):184–190. Crossref, Medline, Google Scholar
- 18. . ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood) 2011;30(4):581–589. Crossref, Medline, Google Scholar
- 19. . Getting moving on patient safety: harnessing electronic data for safer care. N Engl J Med 2011;365(19):1756–1758. Crossref, Medline, Google Scholar
- 20. . Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol 2010;7(8):625–633. Crossref, Medline, Google Scholar
- 21. . Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003;12(Suppl 2):ii39–ii45. Medline, Google Scholar
- 22. . A computerized method for identifying incidents associated with adverse drug events in outpatients. Int J Med Inform 2001;61(1):21–32. Crossref, Medline, Google Scholar
- 23. . Computerized surveillance of adverse drug events in hospital patients: 1991. Qual Saf Health Care 2005;14(3):221–225; discussion 225–226. Crossref, Medline, Google Scholar
- 24. . Informatics tools for the development of action-oriented triggers for outpatient adverse drug events. AMIA Annu Symp Proc 2008 Nov 6:505–509. Medline, Google Scholar
- 25. . Respiratory isolation of tuberculosis patients using clinical guidelines and an automated clinical decision support system. Infect Control Hosp Epidemiol 1998;19(2):94–100. Crossref, Medline, Google Scholar
- 26. . Identifying causes of adverse events detected by an automated trigger tool through in-depth analysis. Qual Saf Health Care 2010;19(5):435–439. Medline, Google Scholar
- 27. . Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf 2014;23(1):8–16. Crossref, Medline, Google Scholar
- 28. . Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med 2009;169(17):1578–1586. Crossref, Medline, Google Scholar
- 29. . Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14(4):459–466. Crossref, Medline, Google Scholar
- 30. . National Center for Veterans Analysis and Statistics. http://www.va.gov/vetdata/Veteran_Population.asp. Accessed December 11, 2014. Google Scholar
- 31. . Root cause analysis reports help identify common factors in delayed diagnosis and treatment of outpatients. Health Aff (Millwood) 2013;32(8):1368–1375. Crossref, Medline, Google Scholar
- 32. . Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med 2004;19(4):316–323. Crossref, Medline, Google Scholar
- 33. . Failure to recognize newly identified aortic dilations in a health care system with an advanced electronic medical record. Ann Intern Med 2009;151(1):21–27, W5. Crossref, Medline, Google Scholar
- 34. . Missed hypothyroidism diagnosis uncovered by linking laboratory and pharmacy data. Arch Intern Med 2005;165(5):574–577. Crossref, Medline, Google Scholar
- 35. . Using nurse navigation to improve timeliness of lung cancer care at a veterans hospital. Clin J Oncol Nurs 2012;16(1):29–36. Crossref, Medline, Google Scholar
- 36. . Foundations for lung nodule management for nurse navigators. Clin J Oncol Nurs 2013;17(5):525–531. Crossref, Medline, Google Scholar
- 37. . Oncology nurse navigators and the continuum of cancer care. Semin Oncol Nurs 2013;29(2):105–117. Crossref, Medline, Google Scholar
- 38. . Efficiency of a semiautomated coding and review process for notification of critical findings in diagnostic imaging. AJR Am J Roentgenol 2006;186(4):933–936. Crossref, Medline, Google Scholar
- 39. . Timeliness across the continuum of care in veterans with lung cancer. J Thorac Oncol 2008;3(9):951–957. Crossref, Medline, Google Scholar
- 40. . Timeliness of care in veterans with non-small cell lung cancer. Chest 2008;133(5):1167–1173. Crossref, Medline, Google Scholar
- 41. . Developing software to “track and catch” missed follow-up of abnormal test results in a complex sociotechnical environment. Appl Clin Inform 2013;4(3):359–375. Crossref, Medline, Google Scholar
- 42. . Eight recommendations for policies for communicating abnormal test results. Jt Comm J Qual Patient Saf 2010;36(5):226–232. Medline, Google Scholar
- 43. . Considering Sensitivity and Positive Predictive Value in Comparing the Performance of Triggers Systems for Iatrogenic Adverse Events: Triggers and Targeted Injury Detection Systems (TIDS) Expert Panel Meeting. Rockville, Md: Agency for Healthcare Research and Quality, 2009. Google Scholar
Article History
Received October 30, 2014; revision requested December 11; revision received January 19, 2015; accepted January 30; final version accepted February 19.Published online: May 11 2015
Published in print: Oct 2015