"skipMainNavigation"
closeDrawerMenuopenDrawerMenu
Search
Quick Search in Journals
Quick Search anywhere
Advanced Search
Radiology
Menu
  • Information
    • Previous
    Next
    Original Research

    Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography

    Published Online:Mar 1 2012https://doi.org/10.1148/radiol.11110938

    Abstract

    The performance of distributed human intelligence is not significantly different from that of computer-aided detection (CAD) for colonic polyp classification; the use of distributed human intelligence may provide insights that guide future CAD development.

    Purpose

    To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography.

    Materials and Methods

    This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification.

    Results

    The detection-level AUC for KWs was 0.845 ± 0.045 (standard error) in trial 1 and 0.855 ± 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 ± 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 ± 0.032 in trial 1, 0.966 ± 0.027 in trial 2, and 0.877 ± 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 ± 0.052 in trial 1 and 0.839 ± 0.046 in trial 2 (P = .041).

    Conclusion

    The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.

    © RSNA, 2012

    Supplemental material: http://radiology.rsna.org/lookup/suppl/doi:10.1148/radiol.11110938/-/DC1

    References

    • 1 Jemal A, Siegel R, Xu J, Ward E. Cancer statistics, 2010. CA Cancer J Clin 2010;60(5):277–300. Crossref, Medline, Google Scholar
    • 2 Smith RA, Cokkinides V, Brooks D, Saslow D, Brawley OW. Cancer screening in the United States, 2010: a review of current American Cancer Society guidelines and issues in cancer screening. CA Cancer J Clin 2010;60(2):99–119. Crossref, Medline, Google Scholar
    • 3 Dachman AH, Obuchowski NA, Hoffmeister JW, et al.. Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology 2010;256(3):827–835. Link, Google Scholar
    • 4 Taylor SA, Charman SC, Lefere P, et al.. CT colonography: investigation of the optimum reader paradigm by using computer-aided detection software. Radiology 2008;246(2):463–471. Link, Google Scholar
    • 5 Hock D, Ouhadi R, Materne R, et al.. Virtual dissection CT colonography: evaluation of learning curves and reading times with and without computer-aided detection. Radiology 2008;248(3):860–868. Link, Google Scholar
    • 6 Petrick N, Haider M, Summers RM, et al.. CT colonography with computer-aided detection as a second reader: observer performance study. Radiology 2008;246(1):148–156. Link, Google Scholar
    • 7 Summers RM. Improving the accuracy of CTC interpretation: computer-aided detection. Gastrointest Endosc Clin N Am 2010;20(2):245–257. Crossref, Medline, Google Scholar
    • 8 Summers RM. How perceptual factors affect the use and accuracy of CAD for interpretation of CT images. In: Samei EKrupinski E, eds. The handbook of medical image perception and techniques. Cambridge, England: Cambridge University Press, 2009; 311–319. Google Scholar
    • 9 Pickhardt PJ, Choi JR, Hwang I, et al.. Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults. N Engl J Med 2003;349(23):2191–2200. Crossref, Medline, Google Scholar
    • 10 Pickhardt PJ, Choi JH. Electronic cleansing and stool tagging in CT colonography: advantages and pitfalls with primary three-dimensional evaluation. AJR Am J Roentgenol 2003;181(3):799–805. Crossref, Medline, Google Scholar
    • 11 Summers RM, Yao J, Pickhardt PJ, et al.. Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology 2005;129(6):1832–1844. Crossref, Medline, Google Scholar
    • 12 Summers RM, Beaulieu CF, Pusanik LM, et al.. Automated polyp detector for CT colonography: feasibility study. Radiology 2000;216(1):284–290. Link, Google Scholar
    • 13 Summers RM, Johnson CD, Pusanik LM, Malley JD, Youssef AM, Reed JE. Automated polyp detection at CT colonography: feasibility assessment in a human population. Radiology 2001;219(1):51–59. Link, Google Scholar
    • 14 Summers RM, Jerebko AK, Franaszek M, Malley JD, Johnson CD. Colonic polyps: complementary role of computer-aided detection in CT colonography. Radiology 2002;225(2):391–399. Link, Google Scholar
    • 15 Jerebko AK, Malley JD, Franaszek M, Summers RM. Computer-aided polyp detection in CT colonography using an ensemble of support vector machines. In: Lemke HUVannier MWInamura KFarman AGDoi KReiber JHC, eds. CARS 2003. Computer Assisted Radiology and Surgery. Proceedings of the 17th International Congress and Exhibition. London, England: Elsevier, 2003;1019–1024. Crossref, Google Scholar
    • 16 Malley JD, Jerebko AK, Miller MT, Summers RM. Variance reduction for error estimation when classifying colon polyps from CT colonography. In: Clough AVAmini AA, eds. SPIE medical imaging. San Diego, Calif: SPIE, 2003; 570–578. Crossref, Google Scholar
    • 17 Yao J, Summers RM. 3D colonic polyp segmentation using dynamic deformable surfaces. In: Amini AAManduca A, eds. SPIE medical imaging. San Diego, Calif: SPIE, 2004; 280–289. Crossref, Google Scholar
    • 18 Iordanescu G, Pickhardt PJ, Choi JR, Summers RM. Automated seed placement for colon segmentation in computed tomography colonography. Acad Radiol 2005;12(2):182–190. Crossref, Medline, Google Scholar
    • 19 Summers RM, Franaszek M, Miller MT, Pickhardt PJ, Choi JR, Schindler WR. Computer-aided detection of polyps on oral contrast-enhanced CT colonography. AJR Am J Roentgenol 2005;184(1):105–108. Crossref, Medline, Google Scholar
    • 20 Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng 1978;4(4):166–171. Google Scholar
    • 21 Chakraborty DP. Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008;15(12):1554–1566. Crossref, Medline, Google Scholar
    • 22 Li Q, Doi K. Reduction of bias and variance for evaluation of computer-aided diagnostic schemes. Med Phys 2006;33(4):868–875. Crossref, Medline, Google Scholar
    • 23 Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998;17(9):1033–1053. Crossref, Medline, Google Scholar
    • 24 Shalizi C. The bootstrap. Am Sci 2010;98:186–190. Crossref, Google Scholar
    • 25 Howe J. The rise of crowdsourcing. Wired 2006; 14(6). http://www.wired.com/wired/archive/14.06/crowds.html. Accessed November 23, 2010. Google Scholar
    • 26 Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 2010;5(5):411–419. Google Scholar
    • 27 Brabham DC. Crowdsourcing as a model for problem solving: an introduction and cases. Convergence 2008;14(1):75–90. Crossref, Google Scholar
    • 28 Oprea TI, Bologa CG, Boyer S, et al.. A crowdsourcing evaluation of the NIH chemical probes. Nat Chem Biol 2009;5(7):441–447. Crossref, Medline, Google Scholar
    • 29 Johnston SC, Hauser SL. Crowdsourcing scientific innovation. Ann Neurol 2009;65(6):A7–A8. Crossref, Medline, Google Scholar
    • 30 Raykar VC, Yu S, Zhao LH, et al.. Learning from crowds. J Mach Learn Res 2010;11:1297–1322. Google Scholar
    • 31 Wang S, Anugu V, Nguyen TB, et al.. Fusion of machine intelligence and human intelligence for colonic polyp detection in CT colonography. In: Biomedical imaging: from nano to macro, 2011 IEEE International Symposium on. Chicago, Ill: IEEE, 2011; 160–164. Crossref, Google Scholar

    Article History

    Received May 11, 2011; revision requested June 15; final revision received July 26; accepted September 13; final version accepted September 26.
    Published online: Mar 2012
    Published in print: Mar 2012