The performance of distributed human intelligence is not significantly different from that of computer-aided detection (CAD) for colonic polyp classification; the use of distributed human intelligence may provide insights that guide future CAD development.
To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography.
Materials and Methods
This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification.
The detection-level AUC for KWs was 0.845 ± 0.045 (standard error) in trial 1 and 0.855 ± 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 ± 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 ± 0.032 in trial 1, 0.966 ± 0.027 in trial 2, and 0.877 ± 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 ± 0.052 in trial 1 and 0.839 ± 0.046 in trial 2 (P = .041).
The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.
© RSNA, 2012
Supplemental material: http://radiology.rsna.org/lookup/suppl/doi:10.1148/radiol.11110938/-/DC1
- 1 . Cancer statistics, 2010. CA Cancer J Clin 2010;60(5):277–300. Crossref, Medline, Google Scholar
- 2 . Cancer screening in the United States, 2010: a review of current American Cancer Society guidelines and issues in cancer screening. CA Cancer J Clin 2010;60(2):99–119. Crossref, Medline, Google Scholar
- 3 . Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology 2010;256(3):827–835. Link, Google Scholar
- 4 . CT colonography: investigation of the optimum reader paradigm by using computer-aided detection software. Radiology 2008;246(2):463–471. Link, Google Scholar
- 5 . Virtual dissection CT colonography: evaluation of learning curves and reading times with and without computer-aided detection. Radiology 2008;248(3):860–868. Link, Google Scholar
- 6 . CT colonography with computer-aided detection as a second reader: observer performance study. Radiology 2008;246(1):148–156. Link, Google Scholar
- 7 . Improving the accuracy of CTC interpretation: computer-aided detection. Gastrointest Endosc Clin N Am 2010;20(2):245–257. Crossref, Medline, Google Scholar
- 8 . How perceptual factors affect the use and accuracy of CAD for interpretation of CT images. In: Samei EKrupinski E, eds. The handbook of medical image perception and techniques. Cambridge, England: Cambridge University Press, 2009; 311–319. Google Scholar
- 9 . Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults. N Engl J Med 2003;349(23):2191–2200. Crossref, Medline, Google Scholar
- 10 . Electronic cleansing and stool tagging in CT colonography: advantages and pitfalls with primary three-dimensional evaluation. AJR Am J Roentgenol 2003;181(3):799–805. Crossref, Medline, Google Scholar
- 11 . Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology 2005;129(6):1832–1844. Crossref, Medline, Google Scholar
- 12 . Automated polyp detector for CT colonography: feasibility study. Radiology 2000;216(1):284–290. Link, Google Scholar
- 13 . Automated polyp detection at CT colonography: feasibility assessment in a human population. Radiology 2001;219(1):51–59. Link, Google Scholar
- 14 . Colonic polyps: complementary role of computer-aided detection in CT colonography. Radiology 2002;225(2):391–399. Link, Google Scholar
- 15 . Computer-aided polyp detection in CT colonography using an ensemble of support vector machines. In: Lemke HUVannier MWInamura KFarman AGDoi KReiber JHC, eds. CARS 2003. Computer Assisted Radiology and Surgery. Proceedings of the 17th International Congress and Exhibition. London, England: Elsevier, 2003;1019–1024. Crossref, Google Scholar
- 16 . Variance reduction for error estimation when classifying colon polyps from CT colonography. In: Clough AVAmini AA, eds. SPIE medical imaging. San Diego, Calif: SPIE, 2003; 570–578. Crossref, Google Scholar
- 17 . 3D colonic polyp segmentation using dynamic deformable surfaces. In: Amini AAManduca A, eds. SPIE medical imaging. San Diego, Calif: SPIE, 2004; 280–289. Crossref, Google Scholar
- 18 . Automated seed placement for colon segmentation in computed tomography colonography. Acad Radiol 2005;12(2):182–190. Crossref, Medline, Google Scholar
- 19 . Computer-aided detection of polyps on oral contrast-enhanced CT colonography. AJR Am J Roentgenol 2005;184(1):105–108. Crossref, Medline, Google Scholar
- 20 . A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng 1978;4(4):166–171. Google Scholar
- 21 . Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008;15(12):1554–1566. Crossref, Medline, Google Scholar
- 22 . Reduction of bias and variance for evaluation of computer-aided diagnostic schemes. Med Phys 2006;33(4):868–875. Crossref, Medline, Google Scholar
- 23 . Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998;17(9):1033–1053. Crossref, Medline, Google Scholar
- 24 . The bootstrap. Am Sci 2010;98:186–190. Crossref, Google Scholar
- 25 . The rise of crowdsourcing. Wired 2006; 14(6). http://www.wired.com/wired/archive/14.06/crowds.html. Accessed November 23, 2010. Google Scholar
- 26 . Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 2010;5(5):411–419. Google Scholar
- 27 . Crowdsourcing as a model for problem solving: an introduction and cases. Convergence 2008;14(1):75–90. Crossref, Google Scholar
- 28 . A crowdsourcing evaluation of the NIH chemical probes. Nat Chem Biol 2009;5(7):441–447. Crossref, Medline, Google Scholar
- 29 . Crowdsourcing scientific innovation. Ann Neurol 2009;65(6):A7–A8. Crossref, Medline, Google Scholar
- 30 . Learning from crowds. J Mach Learn Res 2010;11:1297–1322. Google Scholar
- 31 . Fusion of machine intelligence and human intelligence for colonic polyp detection in CT colonography. In: Biomedical imaging: from nano to macro, 2011 IEEE International Symposium on. Chicago, Ill: IEEE, 2011; 160–164. Crossref, Google Scholar
Article HistoryReceived May 11, 2011; revision requested June 15; final revision received July 26; accepted September 13; final version accepted September 26.
Published online: Mar 2012
Published in print: Mar 2012