Lung Cancer: Interobserver Agreement on Interpretation of Pulmonary Findings at Low-Dose CT Screening

Purpose: To evaluate agreement among radiologists on the interpretation of pulmonary findings at low-dose computed tomographic (CT) screening examinations for lung cancer.

Materials and Methods: Institutional review board approval and informed consent were obtained. HIPAA guidelines were followed. Sixteen radiologists from the 10 National Lung Screening Trial screening centers of the National Cancer Institute's Lung Screening Study network reviewed image subsets from 135 baseline low-dose screening CT examinations in 135 trial participants (89 men, 46 women; mean age, 62.7 years ± 5.4 [standard deviation]). Interpretations were classified into one of four of the following categories: noncalcified nodule 4 mm or larger in greatest transverse dimension (positive screening result); noncalcified nodule smaller than 4 mm in greatest transverse dimension (negative screening result); calcified, benign nodule (negative screening result); or no nodule (negative screening result). A recommendation for follow-up evaluation was obtained for each case. Interobserver agreement was evaluated by using the multirater κ statistic and by using response frequencies and descriptive statistics.

Results: Multirater κ values ranged from 0.58 (for agreement among all four classifications; 95% confidence interval: 0.55, 0.61) to 0.64 (for agreement on classification as a positive or negative screening result; 95% confidence interval: 0.62, 0.65). The average percentage of reader pairs in agreement on the screening result per case (percentage agreement) was 82%. There was wide variation in the total number of abnormalities detected and classified as pulmonary nodules, with differences of up to more than twofold among radiologists. For cases classified as positive, multirater κ for follow-up recommendations was 0.35.

Conclusion: Interobserver agreement was moderate to substantial; potential for considerable improvement exists.

© RSNA, 2007

Clinical trial registration no. NCT00047385


  • 1 Kaneko M, Eguchi K, Ohmatsu H, et al. Peripheral lung cancer: screening and detection with low-dose spiral CT versus radiography. Radiology 1996; 201: 798–802. LinkGoogle Scholar
  • 2 Henschke CI, Lee IJ, Wu N, et al. CT screening for lung cancer: prevalence and incidence of mediastinal masses. Radiology 2006;239:586–590. LinkGoogle Scholar
  • 3 Sone S, Li F, Yang ZG, et al. Results of three-year mass screening programme for lung cancer using mobile low-dose spiral computed tomography scanner. Br J Cancer 2001;84:25–32. Crossref, MedlineGoogle Scholar
  • 4 Swensen SJ, Jett JR, Hartman TE, et al. CT screening for lung cancer: five-year prospective experience. Radiology 2005;235:259–265. LinkGoogle Scholar
  • 5 Diederich S, Thomas M, Semik M, et al. Screening for early lung cancer with low-dose spiral computed tomography: results of annual follow-up examinations in asymptomatic smokers. Eur Radiol 2004;14:691–702. Crossref, MedlineGoogle Scholar
  • 6 Pastorino U, Bellomi M, Landoni C, et al. Early lung-cancer detection with spiral CT and positron emission tomography in heavy smokers: 2-year results. Lancet 2003;362:593–597. Crossref, MedlineGoogle Scholar
  • 7 Gohagan J, Marcus P, Fagerstrom R, Pinsky P, Kramer B, Prorok P. Baseline findings of a randomized feasibility trial of lung cancer screening with spiral CT scan vs chest radiograph: the Lung Screening Study of the National Cancer Institute. Chest 2004;126:114–121. Crossref, MedlineGoogle Scholar
  • 8 Gohagan JK, Marcus PM, Fagerstrom RM, et al. Final results of the Lung Screening Study, a randomized feasibility study of spiral CT versus chest X-ray screening for lung cancer. Lung Cancer 2005;47:9–15. Crossref, MedlineGoogle Scholar
  • 9 MacRedmond R, Logan PM, Lee M, Kenny D, Foley C, Costello RW. Screening for lung cancer using low dose CT scanning. Thorax 2004;59:237–241. Crossref, MedlineGoogle Scholar
  • 10 Humphrey LL, Teutsch S, Johnson M. Lung cancer screening with sputum cytologic examination, chest radiography, and computed tomography: an update for the U.S. Preventive Services Task Force. Ann Intern Med 2004;140:740–753. Google Scholar
  • 11 Henschke CI, Naidich DP, Yankelevitz DF, et al. Early lung cancer action project: initial findings on repeat screenings. Cancer 2001;92:153–159. Crossref, MedlineGoogle Scholar
  • 12 Swensen SJ, Jett JR, Sloan JA, et al. Screening for lung cancer with low-dose spiral computed tomography. Am J Respir Crit Care Med 2002;165:508–513. Crossref, MedlineGoogle Scholar
  • 13 Church TR. Chest radiography as the comparison for spiral CT in the National Lung Screening Trial. Acad Radiol 2003;10:713–715. Crossref, MedlineGoogle Scholar
  • 14 Hillman BJ. Economic, legal, and ethical rationales for the ACRIN national lung screening trial of CT screening for lung cancer. Acad Radiol 2003;10:349–350. Crossref, MedlineGoogle Scholar
  • 15 Moore SM, Gierada DS, Clark KW, Blaine GJ. Image quality assurance in the prostate, lung, colorectal, and ovarian cancer screening trial network of the National Lung Screening Trial. J Digit Imaging 2005;18:242–250. Crossref, MedlineGoogle Scholar
  • 16 Kundel HL, Polansky M. Measurement of observer agreement. Radiology 2003;228:303–308. LinkGoogle Scholar
  • 17 Crewson PE. Reader agreement studies. AJR Am J Roentgenol 2005;184:1391–1397. Crossref, MedlineGoogle Scholar
  • 18 Hansell DM, Wells AU. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Clin Radiol 2003;58:573–574. Crossref, MedlineGoogle Scholar
  • 19 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174. Crossref, MedlineGoogle Scholar
  • 20 Erasmus JJ, Gladish GW, Broemeling L, et al. Interobserver and intraobserver variability in measurement of non-small-cell carcinoma lung lesions: implications for assessment of tumor response. J Clin Oncol 2003;21:2574–2582. Crossref, MedlineGoogle Scholar
  • 21 Revel MP, Bissery A, Bienvenu M, Aycard L, Lefort C, Frija G. Are two-dimensional CT measurements of small noncalcified pulmonary nodules reliable? Radiology 2004;231:453–458. LinkGoogle Scholar
  • 22 Bogot NR, Kazerooni EA, Kelly AM, Quint LE, Desjardins B, Nan B. Interobserver and intraobserver variability in the assessment of pulmonary nodule size on CT using film and computer display methods. Acad Radiol 2005;12:948–956. Crossref, MedlineGoogle Scholar
  • 23 Henschke CI, McCauley DI, Yankelevitz DF, et al. Early Lung Cancer Action Project: overall design and findings from baseline screening. Lancet 1999;354:99–105. Crossref, MedlineGoogle Scholar
  • 24 Diederich S, Wormanns D, Semik M, et al. Screening for early lung cancer with low-dose spiral CT: prevalence in 817 asymptomatic smokers. Radiology 2002;222:773–781. LinkGoogle Scholar
  • 25 Leader JK, Warfel TE, Fuhrman CR, et al. Pulmonary nodule detection with low-dose CT of the lung: agreement among radiologists. AJR Am J Roentgenol 2005;185:973–978. Crossref, MedlineGoogle Scholar
  • 26 Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med 1994;331:1493–1499. Crossref, MedlineGoogle Scholar
  • 27 Kerlikowske K, Grady D, Barclay J, et al. Variability and accuracy in mammographic interpretation using the American College of Radiology Breast Imaging Reporting and Data System. J Natl Cancer Inst 1998;90:1801–1809. Crossref, MedlineGoogle Scholar
  • 28 Siegel CL, McFarland EG, Brink JA, Fisher AJ, Humphrey P, Heiken JP. CT of cystic renal masses: analysis of diagnostic performance and interobserver variation. AJR Am J Roentgenol 1997;169:813–818. Crossref, MedlineGoogle Scholar
  • 29 Aziz ZA, Wells AU, Hansell DM, et al. HRCT diagnosis of diffuse parenchymal lung disease: inter-observer variation. Thorax 2004;59:506–511. Crossref, MedlineGoogle Scholar
  • 30 Garg K, Kemp JL, Wojcik D, et al. Thromboembolic disease: comparison of combined CT pulmonary angiography and venography with bilateral leg sonography in 70 patients. AJR Am J Roentgenol 2000;175:997–1001. Crossref, MedlineGoogle Scholar
  • 31 Cham MD, Yankelevitz DF, Shaham D, et al. Deep venous thrombosis: detection by using indirect CT venography. The Pulmonary Angiography-Indirect CT Venography Cooperative Group. Radiology 2000;216:744–751. Google Scholar
  • 32 Patel S, Kazerooni EA. Helical CT for the evaluation of acute pulmonary embolism. AJR Am J Roentgenol 2005;185:135–149. Crossref, MedlineGoogle Scholar
  • 33 Aberle DR, Gamsu G, Henschke CI, Naidich DP, Swensen SJ. A consensus statement of the Society of Thoracic Radiology: screening for lung cancer with helical computed tomography. J Thorac Imaging 2001;16:65–68. Crossref, MedlineGoogle Scholar
  • 34 MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 2005;237:395–400. LinkGoogle Scholar
  • 35 Henschke CI, Yankelevitz DF, Naidich DP, et al. CT screening for lung cancer: suspiciousness of nodules according to size on baseline scans. Radiology 2004;231:164–168. LinkGoogle Scholar
  • 36 Shah SK, McNitt-Gray MF, De Zoysa KR, et al. Solitary pulmonary nodule diagnosis on CT: results of an observer study. Acad Radiol 2005;12:496–501. Crossref, MedlineGoogle Scholar
  • 37 Brown MS, Goldin JG, Rogers S, et al. Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol 2005;12:681–686. Crossref, MedlineGoogle Scholar
  • 38 Kostis WJ, Yankelevitz DF, Reeves AP, Fluture SC, Henschke CI. Small pulmonary nodules: reproducibility of three-dimensional volumetric measurement and estimation of time to follow-up CT. Radiology 2004;231:446–452. LinkGoogle Scholar
  • 39 Goodman LR, Gulsun M, Washington L, Nagy PG, Piacsek KL. Inherent variability of CT lung nodule measurements in vivo using semiautomated volumetric measurements. AJR Am J Roentgenol 2006;186:989–994. Crossref, MedlineGoogle Scholar
  • 40 Takashima S, Sone S, Li F, et al. Small solitary pulmonary nodules (< or =1 cm) detected at population-based CT screening for lung cancer: reliable high-resolution CT features of benign lesions. AJR Am J Roentgenol 2003;180:955–964. Crossref, MedlineGoogle Scholar
  • 41 Shah SK, McNitt-Gray MF, Rogers SR, et al. Computer-aided diagnosis of the solitary pulmonary nodule. Acad Radiol 2005;12:570–575. Crossref, MedlineGoogle Scholar

Article History

Published in print: 2008