Published Online:

Machine learning is a powerful technique for recognizing patterns on medical images; however, it must be used with caution because it can be misused if the strengths and weaknesses of this technology are not understood.

Machine learning is a technique for recognizing patterns that can be applied to medical images. Although it is a powerful tool that can help in rendering medical diagnoses, it can be misapplied. Machine learning typically begins with the machine learning algorithm system computing the image features that are believed to be of importance in making the prediction or diagnosis of interest. The machine learning algorithm system then identifies the best combination of these image features for classifying the image or computing some metric for the given image region. There are several methods that can be used, each with different strengths and weaknesses. There are open-source versions of most of these machine learning methods that make them easy to try and apply to images. Several metrics for measuring the performance of an algorithm exist; however, one must be aware of the possible associated pitfalls that can result in misleading metrics. More recently, deep learning has started to be used; this method has the benefit that it does not require image feature identification and calculation as a first step; rather, features are identified as part of the learning process. Machine learning has been used in medical imaging and will have a greater influence in the future. Those working in medical imaging must be aware of how machine learning works.

©RSNA, 2017


  • 1. From $600 M to $6 billion, artificial intelligence systems poised for dramatic market expansion in healthcare. Frost & Sullivan website. Accessed September 2, 2016. Google Scholar
  • 2. Schoepf UJ, Costello P. CT angiography for diagnosis of pulmonary embolism: state of the art. Radiology 2004;230(2):329–337. LinkGoogle Scholar
  • 3. Schoepf UJ, Schneider AC, Das M, Wood SA, Cheema JI, Costello P. Pulmonary embolism: computer-aided detection at multidetector row spiral computed tomography. J Thorac Imaging 2007;22(4):319–323. Crossref, MedlineGoogle Scholar
  • 4. Dundar MM, Fung G, Krishnapuram B, Rao RB. Multiple-instance learning algorithms for computer-aided detection. IEEE Trans Biomed Eng 2008;55(3):1015–1021. Crossref, MedlineGoogle Scholar
  • 5. Summers RM. Improving the accuracy of CTC interpretation: computer-aided detection. Gastrointest Endosc Clin N Am 2010;20(2):245–257. Crossref, MedlineGoogle Scholar
  • 6. Yoshida H, Näppi J. CAD in CT colonography without and with oral contrast agents: progress and challenges. Comput Med Imaging Graph 2007;31(4-5):267–284. Crossref, MedlineGoogle Scholar
  • 7. Chan HP, Lo SC, Sahiner B, Lam KL, Helvie MA. Computer-aided detection of mammographic microcalcifications: pattern recognition with an artificial neural network. Med Phys 1995;22(10):1555–1567. Crossref, MedlineGoogle Scholar
  • 8. Bauer S, Wiest R, Nolte LP, Reyes M. A survey of MRI-based medical image analysis for brain tumor studies. Phys Med Biol 2013;58(13):R97–R129. Crossref, MedlineGoogle Scholar
  • 9. Mitchell TM, Shinkareva SV, Carlson A, et al. Predicting human brain activity associated with the meanings of nouns. Science 2008;320(5880):1191–1195. Crossref, MedlineGoogle Scholar
  • 10. Davatzikos C, Fan Y, Wu X, Shen D, Resnick SM. Detection of prodromal Alzheimer’s disease via pattern classification of magnetic resonance imaging. Neurobiol Aging 2008;29(4):514–523. Crossref, MedlineGoogle Scholar
  • 11. Kim D, Burge J, Lane T, Pearlson GD, Kiehl KA, Calhoun VD. Hybrid ICA-Bayesian network approach reveals distinct effective connectivity differences in schizophrenia. Neuroimage 2008;42(4):1560–1568. Crossref, MedlineGoogle Scholar
  • 12. Suzuki K. Pixel-based machine learning in medical imaging. Int J Biomed Imaging 2012;2012:792079 . MedlineGoogle Scholar
  • 13. Jalalian A, Mashohor SB, Mahmud HR, Saripan MI, Ramli AR, Karasfi B. Computer-aided detection/diagnosis of breast cancer in mammography and ultrasound: a review. Clin Imaging 2013;37(3):420–426. Crossref, MedlineGoogle Scholar
  • 14. Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med 2001;23(1):89–109. Crossref, MedlineGoogle Scholar
  • 15. Flach P. Machine learning: the art and science of algorithms that make sense of data. Cambridge, England: Cambridge University Press, 2012. CrossrefGoogle Scholar
  • 16. Cristianini N, Shawe-Taylor J. An introduction to support vector machines and other kernel-based learning methods. Cambridge, England: Cambridge University Press, 2000. CrossrefGoogle Scholar
  • 17. Quinlan JR. Induction of decision trees. Mach Learn 1986;1(1):81–106. CrossrefGoogle Scholar
  • 18. Seber GAF, Lee AJ. Linear regression analysis. 2nd ed. New York, NY: Wiley, 2012. Google Scholar
  • 19. Hosmer DW, Stanley L. Applied logistic regression. 2nd ed. New York, NY: Wiley, 2000. CrossrefGoogle Scholar
  • 20. Lowd D, Daniel L, Pedro D. Naive Bayes models for probability estimation. Proceedings of the 22nd International Conference on Machine Learning - ICML ’05. New York, NY: Association for Computing Machinery, 2005. CrossrefGoogle Scholar
  • 21. Zhou CY, Chen YQ. Improving nearest neighbor classification with cam weighted distance. Pattern Recognit 2006;39(4):635–645. CrossrefGoogle Scholar
  • 22. Breiman L. Random forests. Mach Learn 2001;45(1):5–32. CrossrefGoogle Scholar
  • 23. Hornik K, Kurt H, Maxwell S, Halbert W. Multilayer feedforward networks are universal approximators. Neural Netw 1989;2(5):359–366. CrossrefGoogle Scholar
  • 24. Krishna K, Narasimha Murty M. Genetic K-means algorithm. IEEE Trans Syst Man Cybern B Cybern 1999;29(3):433–439. Crossref, MedlineGoogle Scholar
  • 25. Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 2002;24(5):603–619. CrossrefGoogle Scholar
  • 26. Dueck D, Frey BJ. Non-metric affinity propagation for unsupervised image categorization. IEEE 11th International Conference on Computer Vision. New York, NY: Institute of Electrical and Electronics Engineers, 2007;1–8. Google Scholar
  • 27. Johnson SC. Hierarchical clustering schemes. Psychometrika 1967;32(3):241–254. Crossref, MedlineGoogle Scholar
  • 28. Birant D, Kut A. ST-DBSCAN: an algorithm for clustering spatial-temporal data. Data Knowl Eng 2007;60(1):208–221. CrossrefGoogle Scholar
  • 29. Roberts SJ, Husmeier D, Rezek I, Penny W. Bayesian approaches to Gaussian mixture modeling. IEEE Trans Pattern Anal Mach Intell 1998;20(11):1133–1142. CrossrefGoogle Scholar
  • 30. Chellappa R, Jain AK. Markov random fields: theory and application. Boston, Mass: Academic Press, 1993. Google Scholar
  • 31. Dunn JC. A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters. J Cybern 1973;3(3):32–57. CrossrefGoogle Scholar
  • 32. Bezdek JC, Ehrlich R, Full W. FCM: the fuzzy C-means clustering algorithm. Comput Geosci 1984;10(2-3):191–203. CrossrefGoogle Scholar
  • 33. Nixon M, Nixon MS, Aguado AS. Feature extraction & image processing for computer vision. London, England: Academic Press, 2012. Google Scholar
  • 34. Wang Y, Yuhang W. Spatial feature extraction algorithms (master’s thesis). Hanover, NH: Dartmouth College, 2005. Google Scholar
  • 35. Way TW, Sahiner B, Hadjiiski LM, Chan HP. Effect of finite sample size on feature selection and classification: a simulation study. Med Phys 2010;37(2):907–920. Crossref, MedlineGoogle Scholar
  • 36. Saeys Y, Inza I, Larrañaga P. A review of feature selection techniques in bioinformatics. Bioinformatics 2007;23(19):2507–2517. Crossref, MedlineGoogle Scholar
  • 37. Kuhn M, Johnson K. An introduction to feature selection. In: Applied predictive modeling. New York, NY: Springer, 2013; 487–519. CrossrefGoogle Scholar
  • 38. Kohavi R, Ron K, John GH. Automatic parameter selection by minimizing estimated error. Proceedings of the Twelfth International Conference on machine learning, Tahoe City, Calif. New York, NY: Elsevier, 1995; 304–312. CrossrefGoogle Scholar
  • 39. Arlot S, Celisse A. A survey of cross-validation procedures for model selection. Stat Surv 2010;4:40–79. CrossrefGoogle Scholar
  • 40. Zhang T, Tong Z. A leave-one-out cross validation bound for kernel methods with applications in learning. Lect Notes Comput Sci 2001;2111:427–443. CrossrefGoogle Scholar
  • 41. Cover T, Hart P. Nearest neighbor pattern classification. IEEE Trans Inf Theory 1967;13(1):21–27. CrossrefGoogle Scholar
  • 42. Vapnik V, Lerner A. Pattern recognition using generalized portrait method. Autom Remote Control 1963;24(6):774–780. Google Scholar
  • 43. Buhmann MD. Radial basis functions with compact support. In: Radial basis functions: theory and implementations. Cambridge, England: Cambridge University Press, 147–162. Google Scholar
  • 44. Prajapati GL, Patle A. On performing classification using SVM with radial basis and polynomial kernel functions: 2010 3rd International Conference on Emerging Trends in Engineering and Technology. New York, NY: Institute of Electrical and Electronics Engineers, 2010. CrossrefGoogle Scholar
  • 45. Rokach L, Maimon O. Data mining with decision trees: theory and applications. 2nd ed. Singapore: World Scientific Publishing Company, 2014. CrossrefGoogle Scholar
  • 46. Breiman L, Leo B. Bagging predictors. Mach Learn 1996;24(2):123–140. CrossrefGoogle Scholar
  • 47. Duda R, Hart P. Pattern classification and scene analysis. New York, NY: Wiley, 1973; 139–143. Google Scholar
  • 48. Hand DJ, Keming Y. Idiot’s Bayes: not so stupid after all? Int Stat Rev 2001;69(3):385–398. Google Scholar
  • 49. Szegedy C, Toshev A, Erhan D. Deep neural networks for object detection. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, eds. Advances in neural information processing systems. Red Hook, NY: Curran Associates, 2013; 2553–2561. Google Scholar
  • 50. Brosch T. Efficient deep learning of 3D structural brain MRIs for manifold learning and lesion segmentation with application to multiple sclerosis. The University of British Columbia Library website. Published 2016. Accessed October 2016. Google Scholar
  • 51. Abadi M, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Cornell University Library website. Published 2016. Accessed October 2016. Google Scholar
  • 52. Lee H, Hansung L, Yunsu C, Jeongnyeo K, Daihee P. Face image retrieval using sparse representation classifier with Gabor-LBP histogram. Lect Notes Comput Sci 2011;6513:273–280. CrossrefGoogle Scholar
  • 53. Le Cun Y, Jackel LD, Boser B, et al. Handwritten digit recognition: applications of neural net chips and automatic learning. Neurocomputing 1990;68:303–318. CrossrefGoogle Scholar
  • 54. Dahl GE, Sainath TN, Hinton GE. Improving deep neural networks for LVCSR using rectified linear units and dropout. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. New York, NY: Institute of Electrical and Electronics Engineers, 2013. Google Scholar
  • 55. Zhou YT, Chellappa R, Vaid A, Jenkins BK. Image restoration using a neural network. IEEE Trans Acoust 1988; 36(7):1141–1151. CrossrefGoogle Scholar
  • 56. Srivastava N, Hinton GR, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014;15(7):1929–1958. Google Scholar
  • 57. Russakovsky O, Olga R, Jia D, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis 2015;115(3):211–252. CrossrefGoogle Scholar
  • 58. Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998;86(11):2278–2324. CrossrefGoogle Scholar
  • 59. Szegedy C, Christian S, Wei L, et al. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, NY: Institute of Electrical and Electronics Engineers, 2015. Google Scholar
  • 60. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ, eds. Advances in neural information processing systems. Red Hook, NY: Curran Associates, 2012; 1097–1105. Google Scholar
  • 61. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Cornell University Library website. Published 2014. Accessed October 2016. Google Scholar
  • 62. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Cornell University Library website. Published 2015. Accessed October 2016. Google Scholar
  • 63. Scikit-learn: machine learning in Python. Scikit Learn website. Accessed May 2, 2016. Google Scholar
  • 64. Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization. CLiPS Resources website. Accessed October 2016. Google Scholar
  • 65. Coelho LP. Milk: machine learning toolkit. Luis Predo Coelho website. Accessed May 2, 2016. Google Scholar
  • 66. Pylearn2. Pylearn2 dev documentation website. Accessed October 2016. Google Scholar
  • 67. Keras: Deep learning library for Theano and TensorFlow. Keras Documentation website. Accessed October 2016. Google Scholar
  • 68. DeePy: a highly extensible deep learning framework. DeePy website. Accessed October 2016. Google Scholar
  • 69. The Caret package. Caret website. Accessed May 2, 2016. Google Scholar
  • 70. Boruta: wrapper algorithm for all-relevant feature selection. Boruta website. Accessed May 2, 2016. Google Scholar
  • 71. GMMBoost: likelihood-based boosting for generalized mixed models. GMMBoost website. Accessed October 2016. Google Scholar
  • 72. H2O: R interface for H2O. H2O website. Accessed October 2016. Google Scholar
  • 73. Shogun: unified and efficient machine learning. Shogun website. Accessed October 2016. Google Scholar
  • 74. EBLearn: open source C++ machine learning library. EBLearn website. Accessed October 2016. Google Scholar
  • 75. Intel deep learning framework (IDLF). Intel Software Developer Zone website. Accessed October 2016. Google Scholar
  • 76. Torch: a scientific computing framework for Luajit. Torch website. Accessed October 2016. Google Scholar
  • 77. Encog machine learning framework. Heaton Research website. Accessed October 2016. Google Scholar
  • 78. Apache Spark: lightning-fast cluster computing. Apache Spark website. Accessed October 2016. Google Scholar
  • 79. What is Apache Mahout? Mahout website. Accessed October 2016. Google Scholar
  • 80. MALLET: machine learning for language toolkit. MALLET website. Accessed October 2016. Google Scholar
  • 81. Deep learning for Java: open-source, distributed, deep learning library for the JVM. Deeplearning4j website. Accessed October 2016. Google Scholar
  • 82. Clusterfck. GitHub Harthur/clusterfck website. Accessed October 2016. Google Scholar
  • 83. LDA topic modeling for node.js. GitHub Primaryobjects/Ida website Accessed October 2016. Google Scholar
  • 84. Node-SVM: support vector machine (SVM) library for nodejs. Not Pictured: Mangoes website. Accessed DATE. Google Scholar
  • 85. ConvNetJS: deep learning in your browser. ConvNetJS website. Accessed DATE. Google Scholar

Article History

Received: May 5 2016
Revision requested: July 27 2016
Revision received: Oct 28 2016
Accepted: Nov 4 2016
Published online: Feb 17 2017
Published in print: Mar 2017