Measuring and Establishing the Accuracy and Reproducibility of 3D Printed Medical Models

Published Online:https://doi.org/10.1148/rg.2017160165

Although current 3D printing technology, when optimally used, enables creation of medical models with accuracies within the range of clinical imaging spatial resolutions, a thorough understanding of factors affecting the accuracy and reproducibility of these models and the techniques used to measure these parameters is required to ensure appropriate medical practice and establish future guidelines.

Despite the rapid growth of three-dimensional (3D) printing applications in medicine, the accuracy and reproducibility of 3D printed medical models have not been thoroughly investigated. Although current technologies enable 3D models to be created with accuracy within the limits of clinical imaging spatial resolutions, this is not always achieved in practice. Inaccuracies are due to errors that occur during the imaging, segmentation, postprocessing, and 3D printing steps. Radiologists’ understanding of the factors that influence 3D printed model accuracy and the metrics used to measure this accuracy is key in directing appropriate practices and establishing reference standards and validation procedures. The authors review the various factors in each step of the 3D model printing process that contribute to model inaccuracy, including the intrinsic limitations of each printing technology. In addition, common sources of model inaccuracy are illustrated. Metrics involving comparisons of model dimensions and morphology that have been developed to quantify differences between 3D models also are described and illustrated. These metrics can be used to define the accuracy of a model, as compared with the reference standard, and to measure the variability of models created by different observers or using different workflows. The accuracies reported for specific indications of 3D printing are summarized, and potential guidelines for quality assurance and workflow assessment are discussed.

Online supplemental material is available for this article.

©RSNA, 2017

References

  • 1. Mitsouras D, Liacouras P, Imanzadeh A, et al. Medical 3D printing for the radiologist. RadioGraphics 2015;35(7):1965–1988.
  • 2. Giannopoulos AA, Steigner ML, George E, et al. Cardiothoracic applications of 3-dimensional printing. J Thorac Imaging 2016;31(5):253–272.
  • 3. Matsumoto JS, Morris JM, Foley TA, et al. Three-dimensional physical modeling: applications and experience at Mayo Clinic. RadioGraphics 2015;35(7):1989–2006.
  • 4. Namba K, Higaki A, Kaneko N, Mashiko T, Nemoto S, Watanabe E. Microcatheter shaping for intracranial aneurysm coiling using the 3-dimensional printing rapid prototyping technology: preliminary result in the first 10 consecutive cases. World Neurosurg 2015;84(1):178–186.
  • 5. The world’s first 3D printed total jaw reconstruction. Xilloc website. http://www.xilloc.com/patients/stories/total-mandibular-implant/. Accessed September 20, 2016.
  • 6. Wong KC, Kumta SM, Geel NV, Demol J. One-step reconstruction with a 3D-printed, biomechanically evaluated custom implant after complex pelvic tumor resection. Comput Aided Surg 2015;20(1):14–23.
  • 7. Aranda JL, Jiménez MF, Rodríguez M, Varela G. Tridimensional titanium-printed custom-made prosthesis for sternocostal reconstruction. Eur J Cardiothorac Surg 2015;48(4):e92–e94.
  • 8. Morrison RJ, Hollister SJ, Niedner MF, et al. Mitigation of tracheobronchomalacia with 3D-printed personalized medical devices in pediatric patients. Sci Transl Med 2015;7(285):285ra64.
  • 9. Parthasarathy J. 3D modeling, custom implants and its future perspectives in craniofacial surgery. Ann Maxillofac Surg 2014;4(1):9–18.
  • 10. Young BW. Major first!: FDA grants first 3D printed implant clearance. OPM in the News website. http://www.oxfordpm.com/major-first-fda-grants-first-3d-printed-implant-clearance. Accessed September 20, 2016.
  • 11. Erickson DM, Chance D, Schmitt S, Mathis J. An opinion survey of reported benefits from the use of stereolithographic models. J Oral Maxillofac Surg 1999;57(9):1040–1043.
  • 12. Martelli N, Serrano C, van den Brink H, et al. Advantages and disadvantages of 3-dimensional printing in surgery: a systematic review. Surgery 2016;159(6):1485–1500.
  • 13. Liu YF, Xu LW, Zhu HY, Liu SS. Technical procedures for template-guided surgery for mandibular reconstruction based on digital design and manufacturing. Biomed Eng Online 2014;13:63.
  • 14. Weinstock P, Prabhu SP, Flynn K, Orbach DB, Smith E. Optimizing cerebrovascular surgical and endovascular procedures in children via personalized 3D printing. J Neurosurg Pediatr doi: 10.3171/2015.3.PEDS14677. Published online July 31, 2015.
  • 15. Yang M, Li C, Li Y, et al. Application of 3D rapid prototyping technology in posterior corrective surgery for Lenke 1 adolescent idiopathic scoliosis patients. Medicine (Baltimore) 2015;94(8):e582.
  • 16. Fasel JH, Beinemann J, Schaller K, Gailloud P. A critical inventory of preoperative skull replicas. Ann R Coll Surg Engl 2013;95(6):401–404.
  • 17. Chang PS, Parker TH, Patrick CW Jr, Miller MJ. The accuracy of stereolithography in planning craniofacial bone replacement. J Craniofac Surg 2003;14(2):164–170.
  • 18. Kondo K, Nemoto M, Masuda H, et al. Anatomical reproducibility of a head model molded by a three-dimensional printer. Neurol Med Chir (Tokyo) 2015;55(7):592–598.
  • 19. Frölich AM, Spallek J, Brehmer L, et al. 3D printing of intracranial aneurysms using fused deposition modeling offers highly accurate replications. AJNR Am J Neuroradiol 2016;37(1):120–124.
  • 20. Mathur M, Patil P, Bove A. The role of 3D printing in structural heart disease: all that glitters is not gold. JACC Cardiovasc Imaging 2015;8(8):987–988.
  • 21. Stumpel LJ. Deformation of stereolithographically produced surgical guides: an observational case series report. Clin Implant Dent Relat Res 2012;14(3):442–453.
  • 22. Russon M. 12 NHS hospitals using 3D-printed models to test implants before surgery. International Business Times website. http://www.ibtimes.co.uk/12-nhs-hospitals-using-3d-printed-models-test-implants-before-surgery-1462862. Published August 27, 2014. Accessed September 20, 2016.
  • 23. Schweizer A, Mauler F, Vlachopoulos L, Nagy L, Fürnstahl P. Computer-assisted 3-dimensional reconstructions of scaphoid fractures and nonunions with and without the use of patient-specific guides: early clinical outcomes and postoperative assessments of reconstruction accuracy. J Hand Surg Am 2016;41(1):59–69.
  • 24. Gan Y, Ding J, Xu Y, Hou C. Accuracy and efficacy of osteotomy in total knee arthroplasty with patient-specific navigational template. Int J Clin Exp Med 2015;8(8): 12192–12201.
  • 25. Hu Y, Yuan ZS, Spiker WR, et al. A comparative study on the accuracy of pedicle screw placement assisted by personalized rapid prototyping template between pre- and post-operation in patients with relatively normal mid-upper thoracic spine. Eur Spine J 2016;25(6):1706–1715.
  • 26. Teeter MG, Kopacz AJ, Nikolov HN, Holdsworth DW. Metrology test object for dimensional verification in additive manufacturing of metals for biomedical applications. Proc Inst Mech Eng H 2015;229(1):20–27.
  • 27. 3D Stereolithography Printers Tech Specs. 3D Systems website. https://www.3dsystems.com/sites/default/files/2017-05/3D-Systems_SLA_TECH_SPECS_USEN_2017.05.01_WEB.pdf. Accessed June 20, 2017.
  • 28. O’Hara RP, Chand A, Vidilaya S, et al. Advanced 3D mesh manipulation in stereolithographic files and post-print processing for the manufacturing of patient-specific vascular flow phantoms. In: Zhang J, Cook TS, eds. Proceedings of SPIE: medical imaging 2016—PACS and imaging informatics: next generation and innovations. Vol 9789. Bellingham, Wash: International Society for Optics and Photonics, 2016; 978909.
  • 29. El-Katatny I, Masood SH, Morsi YS. Error analysis of FDM fabricated medical replicas. Rapid Prototyp J 2010;16(1):36–43.
  • 30. Harris BD, Nilsson S, Poole CM. A feasibility study for using ABS plastic and a low-cost 3D printer for patient-specific brachytherapy mould design. Australas Phys Eng Sci Med 2015;38(3):399–412.
  • 31. Ishida Y, Miyasaka T. Dimensional accuracy of dental casting patterns created by 3D printers. Dent Mater J 2016;35(2):250–256.
  • 32. Salmi M, Paloheimo KS, Tuomi J, Wolff J, Mäkitie A. Accuracy of medical models made by additive manufacturing (rapid manufacturing). J Craniomaxillofac Surg 2013;41(7):603–609.
  • 33. Braian M, Jimbo R, Wennerberg A. Production tolerance of additive manufactured polymeric objects for clinical applications. Dent Mater 2016;32(7):853–861.
  • 34. Taft RM, Kondor S, Grant GT. Accuracy of rapid prototype models for head and neck reconstruction. J Prosthet Dent 2011;106(6):399–408.
  • 35. Petropolis C, Kozan D, Sigurdson L. Accuracy of medical models made by consumer-grade fused deposition modelling printers. Plast Surg (Oakv) 2015;23(2):91–94.
  • 36. Pang TH, Guertin MD, Nguyen HD. Accuracy of stereolithography parts: mechanism and modes of distortion for a “letter-H” diagnostic part. In: Proceedings of the Solid Freeform Fabrication Symposium, Austin, Texas, August 7–9, 1995; 170–180.
  • 37. Doran SJ, Charles-Edwards L, Reinsberg SA, Leach MO. A complete distortion correction for MR images. I. Gradient warp correction. Phys Med Biol 2005;50(7):1343–1361.
  • 38. Martorelli M, Gerbino S, Giudice M, Ausiello P. A comparison between customized clear and removable orthodontic appliances manufactured using RP and CNC techniques. Dent Mater 2013;29(2):e1–e10.
  • 39. Mitsouras D, Lee TC, Liacouras P, et al. Three-dimensional printing of MRI-visible phantoms and MR image-guided therapy simulation. Magn Reson Med 2017;77(2):613–622. doi: 10.1002/mrm.26136. Published online February 11, 2016.
  • 40. Engelke K, Libanati C, Liu Y, et al. Quantitative computed tomography (QCT) of the forearm using general purpose spiral whole-body CT scanners: accuracy, precision and comparison with dual-energy X-ray absorptiometry (DXA). Bone 2009;45(1):110–118.
  • 41. Whyms BJ, Vorperian HK, Gentry LR, Schimek EM, Bersu ET, Chung MK. The effect of computed tomographic scanner parameters and 3-dimensional volume rendering techniques on the accuracy of linear, angular, and volumetric measurements of the mandible. Oral Surg Oral Med Oral Pathol Oral Radiol 2013;115(5):682–691.
  • 42. 3D point cloud and mesh processing software open source project. CloudCompare website. http://www.danielgm.net/cc/. Accessed June 11, 2016.
  • 43. Di Prima M, Coburn J, Hwang D, Kelly J, Khairuzzaman A, Ricles L. Additively manufactured medical products: the FDA perspective. 3D Print Med 2016;2(1).
  • 44. Barker TM, Earwaker WJ, Lisle DA. Accuracy of stereolithographic models of human anatomy. Australas Radiol 1994;38(2):106–111.
  • 45. Ibrahim D, Broilo TL, Heitz C, et al. Dimensional error of selective laser sintering, three-dimensional printing and PolyJet models in the reproduction of mandibular anatomy. J Craniomaxillofac Surg 2009;37(3):167–173.
  • 46. Choi JY, Choi JH, Kim NK, et al. Analysis of errors in medical rapid prototyping models. Int J Oral Maxillofac Surg 2002;31(1):23–32.
  • 47. Silva DN, Gerhardt de Oliveira M, Meurer E, Meurer MI, Lopes da Silva JV, Santa-Bárbara A. Dimensional error in selective laser sintering and 3D-printing of models for craniomaxillary anatomy reconstruction. J Craniomaxillofac Surg 2008;36(8):443–449.
  • 48. van Eijnatten M, Rijkhorst EJ, Hofman M, Forouzanfar T, Wolff J. The accuracy of ultrashort echo time MRI sequences for medical additive manufacturing. Dentomaxillofac Radiol 2016;45(5):20150424.
  • 49. Ogden KM, Aslan C, Ordway N, Diallo D, Tillapaugh-Fay G, Soman P. Factors affecting dimensional accuracy of 3-D printed anatomical structures derived from CT data. J Digit Imaging 2015;28(6):654–663.
  • 50. Wu XB, Wang JQ, Zhao CP, et al. Printed three-dimensional anatomic templates for virtual preoperative planning before reconstruction of old pelvic injuries: initial results. Chin Med J (Engl) 2015;128(4):477–482.
  • 51. Pinto JM, Arrieta C, Andia ME, et al. Sensitivity analysis of geometric errors in additive manufacturing medical models. Med Eng Phys 2015;37(3):328–334.
  • 52. Fourie Z, Damstra J, Schepers RH, Gerrits PO, Ren Y. Segmentation process significantly influences the accuracy of 3D surface models derived from cone beam computed tomography. Eur J Radiol 2012;81(4):e524–e530.
  • 53. Santana RR, Lozada J, Kleinman A, Al-Ardah A, Herford A, Chen JW. Accuracy of cone beam computerized tomography and a three-dimensional stereolithographic model in identifying the anterior loop of the mental nerve: a study on cadavers. J Oral Implantol 2012;38(6):668–676.
  • 54. Bowers ME, Trinh N, Tung GA, Crisco JJ, Kimia BB, Fleming BC. Quantitative MR imaging using “LiveWire” to measure tibiofemoral articular cartilage thickness. Osteoarthritis Cartilage 2008;16(10):1167–1173.
  • 55. Gelaude F, Vander Sloten J, Lauwers B. Accuracy assessment of CT-based outer surface femur meshes. Comput Aided Surg 2008;13(4):188–199.
  • 56. Liang X, Lambrichts I, Sun Y, et al. A comparative evaluation of cone beam computed tomography (CBCT) and multi-slice CT (MSCT). II. On 3D model accuracy. Eur J Radiol 2010;75(2):270–274.
  • 57. Berry E, Brown JM, Connell M, et al. Preliminary experience with medical applications of rapid prototyping by selective laser sintering. Med Eng Phys 1997;19(1):90–96.
  • 58. Frühwald J, Schicho KA, Figl M, Benesch T, Watzinger F, Kainberger F. Accuracy of craniofacial measurements: computed tomography and three-dimensional computed tomography compared with stereolithographic models. J Craniofac Surg 2008;19(1):22–26.
  • 59. Wu AM, Shao ZX, Wang JS, et al. The accuracy of a method for printing three-dimensional spinal models. PLoS One 2015;10(4):e0124291.
  • 60. Waran V, Devaraj P, Hari Chandran T, et al. Three-dimensional anatomical accuracy of cranial models created by rapid prototyping techniques validated using a neuronavigation station. J Clin Neurosci 2012;19(4):574–577.
  • 61. Ionita CN, Mokin M, Varble N, et al. Challenges and limitations of patient-specific vascular phantom fabrication using 3D Polyjet printing. Proc SPIE Int Soc Opt Eng 2014;9038:90380M.
  • 62. Xu WH, Liu J, Li ML, Sun ZY, Chen J, Wu JH. 3D printing of intracranial artery stenosis based on the source images of magnetic resonance angiograph. Ann Transl Med 2014;2(8):74.
  • 63. Biglino G, Verschueren P, Zegels R, Taylor AM, Schievano S. Rapid prototyping compliant arterial phantoms for in-vitro studies and device testing. J Cardiovasc Magn Reson 2013;15:2.
  • 64. Baeck K, Lopes P, Biglino G, Capelli C, Verschueren P. State of the art in 3D printing of compliant cardiovascular models: HeartPrint—material characterization of HeartPrint models and comparison with arterial tissue properties. In: Proceedings of the 3rd Joint Workshop on New Technologies for Computer/Robot Assisted Surgery. Verona, Italy, September 11–13, 2013.
  • 65. Ripley B, Kelil T, Cheezum MK, et al. 3D printing based on cardiac CT assists anatomic visualization prior to transcatheter aortic valve replacement. J Cardiovasc Comput Tomogr 2016;10(1):28–36.
  • 66. Cai T, Ripley BA, Cheezum M, et al. Accuracy of 3D printed models of the aortic valve complex for transcatheter aortic valve replacement (TAVR) planning: comparison to computed tomographic angiography (CTA) [abstr]. In: Radiological Society of North America Scientific Assembly and Annual Meeting Program. Oak Brook, Ill: Radiological Society of North America, 2015; 281.
  • 67. Anderson JR, Thompson WL, Alkattan AK, et al. Three-dimensional printing of anatomically accurate, patient specific intracranial aneurysm models. J Neurointerv Surg 2016;8(5):517–520.
  • 68. Håkansson A, Rantatalo M, Hansen T, Wanhainen A. Patient specific biomodel of the whole aorta: the importance of calcified plaque removal. Vasa 2011;40(6):453–459.
  • 69. Koleilat I, Jaeggli M, Ewing JA, Androes M, Simionescu DT, Eidt J. Interobserver variability in physician-modified endograft planning by comparison with a three-dimensional printed aortic model. J Vasc Surg 2016;64(6):1789–1796.
  • 70. Maragiannis D, Jackson MS, Igo SR, et al. Replicating patient-specific severe aortic valve stenosis with functional 3D modeling. Circ Cardiovasc Imaging 2015;8(10):e003626.
  • 71. Olivieri LJ, Krieger A, Loke YH, Nath DS, Kim PC, Sable CA. Three-dimensional printing of intracardiac defects from three-dimensional echocardiographic images: feasibility and relative accuracy. J Am Soc Echocardiogr 2015;28(4):392–397.
  • 72. Huotilainen E, Jaanimets R, Valášek J, et al. Inaccuracies in additive manufactured medical skull models caused by the DICOM to STL conversion process. J Craniomaxillofac Surg 2014;42(5):e259–e265.
  • 73. Naitoh M, Kubota Y, Katsumata A, Ohsaki C, Ariji E. Dimensional accuracy of a binder jet model produced from computerized tomography data for dental implants. J Oral Implantol 2006;32(6):273–276.
  • 74. Arrieta C, Uribe S, Ramos-Grez J, et al. Quantitative assessments of geometric errors for rapid prototyping in medical applications. Rapid Prototyp J 2012;18(6):431–442.
  • 75. Cai T, Rybicki FJ, Giannopoulos A, et al. The residual STL volume as a metric to evaluate accuracy and reproducibility of anatomic models for 3D printing: application in the validation of 3D printable models of maxillofacial bone from reduced radiation dose CT images. 3D Print Med 2015;1(2).
  • 76. Dickinson KJ, Matsumoto J, Cassivi SD, et al. Individualizing management of complex esophageal pathology using three-dimensional printed models. Ann Thorac Surg 2015;100(2):692–697.
  • 77. Soga S, Pomahac B, Wake N, et al. CT angiography for surgical planning in face transplantation candidates. AJNR Am J Neuroradiol 2013;34(10):1873–1881.
  • 78. OsiriX: the world famous DICOM viewer. Osirix website. http://www.osirix-viewer.com/. Accessed June 12, 2016.
  • 79. Slicer 4.6 released. 3D Slicer website. https://www.slicer.org/. Accessed June 12, 2016.
  • 80. Cheng GZ, San Jose Estepar R, Folch E, Onieva J, Gangadharan S, Majid A. Three-dimensional printing and 3D slicer: powerful tools in understanding and treating structural lung disease. Chest 2016;149(5):1136–1142.

Article History

Received: June 18 2016
Revision requested: Aug 25 2016
Revision received: Sept 25 2016
Accepted: Dec 21 2016
Published online: Aug 11 2017
Published in print: Sept 2017