Nextmed: Automatic Imaging Segmentation, 3D Reconstruction, and 3D Model Visualization Platform Using Augmented and Virtual Reality
Abstract
:1. Introduction
2. Materials and Methods
2.1. Development Environment for Segmentation Module
2.2. Acquisition of Medical Images
2.3. Computer Vision Algorithms
- Processed by thresholding: based on keeping the voxels whose intensities are within a fixed range, and setting the rest to a fixed value that is interpreted as the background of the image. This process is useful for separating structures when no different regions with similar intensities exist.
- Morphological processing: based on form, such as erosion and dilation.
- Geometric/positional processing: based on the relative or absolute position of elements.
2.4. 3D Reconstruction and Storage
2.5. 3D Visualization Platforms
3. Results
3.1. From dicom to 3D Models with Automatic Segmentation
3.2. Results Visualization
4. Discussion
- (1)
- Different anatomical structures can be automatically segmented and a 3D model can be generated in any computer (a workstation is not necessary).
- (2)
- A tool is offered to physicians to visualize medical images in 3D with three different versions: augmented, virtual reality, and computer.
- (3)
- All algorithms were tested using more than 1000 dicom images from computed tomography.
- (4)
- This technology was prepared for its implementation in the daily work of radiologists and specialists, as the entire process is automated.
- (5)
- This work has given rise to the Nextmed project, which is currently still in progress.
Future Work
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Doi, K. Diagnostic imaging over the last 50 years: Research and development in medical imaging science and technology. Phys. Med. Biol. 2006, 51, R5. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kirchgeorg, A.M.; Prokop, M. Increasing spiral ct benefits with postprocessing applications. Eur. J. Radiol. 1998, 28, 39–54. [Google Scholar] [CrossRef]
- Hill, M.; Thornham, A.; Taylor, C.J. Model-Based Interpretation of 3d Medical Images. In Proceedings of the 4th British Machine Vision Conference, Guildford, UK, 21–23 September 1993; pp. 1–10. [Google Scholar] [CrossRef] [Green Version]
- Sharma, M.N.; Aggarwal, L. Automated medical image segmentation techniques. J. Med. Phys. Assoc. Med. Phys. India 2010, 35, 3. [Google Scholar] [CrossRef] [PubMed]
- Cid, Y.D.; del Toro, O.A.J.; Depeursinge, A.; Müller, H. Efficient and fully automatic segmentation of the lungs in CT volumes. In Proceedings of the VISCERAL Anatomy Grand Challenge at the 2015 IEEE International Symposium on Biomedical Imaging (ISBI), New York, NY, USA, 16 April 2015; Goksel, O., del Toro, O.A.J., Foncubierta-Rodríguez, A., Müller, H., Eds.; CEUR Workshop Proceedings Series, no. 1390; CEUR-WS.org: Aachen, Germany; pp. 31–35.
- Ecabert, O.; Peters, J.; Schramm, H.; Lorenz, C.; von Berg, J.J.; Walker, M.; Vembar, M.; Olszewski, M.E.; Subramanyan, K.; Lavi, G.; et al. Automatic model-based segmentation of the heart in ct images. IEEE Trans. Med. Imaging 2008, 27, 1189–1201. [Google Scholar] [CrossRef]
- Petitjean, C.; Zuluaga, M.A.; Bai, W.; Dacher, J.; Grosgeorge, D.; Caudron, J.; Ruan, S.; Ben Ayed, I.; Cardoso, J.; Chen, H.; et al. Right ventricle segmentation from cardiac MRI: A collation study. Med Image Anal. 2015, 19, 187–202. [Google Scholar] [CrossRef] [Green Version]
- Ruskó, L.; Bekes, G.; Fidrich, M. Automatic segmentation of the liver from multi-and single- phase contrast-enhanced CT images. Med Image Anal. 2009, 13, 871–882. [Google Scholar] [CrossRef]
- Santarelli, M.F.; Positano, V.; Michelassi, C.; Lombardi, M.; Landini, L. Automated cardiac mr image segmentation: Theory and measurement evaluation. Med Eng. Phys. 2003, 25, 149–159. [Google Scholar] [CrossRef]
- Sargent, D.; Park, S.Y. Automatic segmentation of mammogram and tomosynthesis images. Proc. SPIE 2016, 9784, 978437. [Google Scholar] [CrossRef]
- Shrivastava, D.; Kandar, D.; Maji, A.K. Automated segmentation of bone computed tomography image using deep convolution neural network. J. Comput. Theor. Nanosci. 2018, 15, 2036–2039. [Google Scholar] [CrossRef]
- Stein, T.; Metzger, J.; Scherer, J.; Isensee, F.; Norajitra, T.; Kleesiek, J.; Maier, K.-H.; Nolden, M. Efficient web-based review for automatic segmentation of volumetric dicom images. Bildverarb. Medizin. 2019, 158–163. [Google Scholar] [CrossRef]
- González Izard, S.; Juanes Méndez, J.A.; García-Peñalvo, F.J. Nextmed: How to enhance 3d radiological images with augmented and virtual reality. In Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality, Salamanca, Spain, 24–26 October 2018; pp. 397–404. [Google Scholar] [CrossRef]
- González Izard, S.; Alonso Plaza, O.; Sánchez Torres, R.; Juanes Méndez, J.A.; García-Peñalvo, F.J. Nextmed, augmented and virtual reality platform for 3d medical imaging visualization: Explanation of the software plat- form developed for 3d models visualization related with medical images using augmented and virtual reality technology. In Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality, Leon, Spain, 16–18 October 2019; pp. 459–467. [Google Scholar] [CrossRef] [Green Version]
- Shuhaiber, J.H. Augmented reality in surgery. Arch. Surg. 2004, 139, 170–174. [Google Scholar] [CrossRef] [Green Version]
- McCloy, R.; Stone, R. Virtual reality in surgery. BMJ 2001, 323, 912–915. [Google Scholar] [CrossRef] [PubMed]
- Gibson, E.; Li, W.; Sudre, C.; Fidon, L.; Shakir, D.I.; Wang, G.; Eaton-Rosen, Z.; Gray, R.; Doel, T.; Hu, Y.; et al. Niftynet: A deep-learning platform for medical imaging. Comput. Methods Programs Biomed. 2018, 158, 113–122. [Google Scholar] [CrossRef] [PubMed]
- Milletari, F.; Navab, N.; Ahmadi, S.A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In Proceedings of the Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, 25–28 October 2016; pp. 565–571. [Google Scholar] [CrossRef] [Green Version]
- Sun, L.; Peng, Z.; Wang, Z.; Pu, H.; Guo, L.; Yuan, G.; Yin, F.; Pu, T. Automatic lung segmentation in chest CT image using morphology. In Proceedings of the 9th International Symposium on Advanced Optical Manufacturing and Testing Technologies: Optoelectronic Materials and Devices for Sensing and Imaging, Chengdu, China, 26–29 June 2018; p. 108431D. [Google Scholar]
- Image Segmentation Using Deep Learning: A Survey. Available online: https://arxiv.org/abs/2001.05566 (accessed on 22 May 2020).
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Zhou, Z.; Siddiquee, M.M.R.; Tajbakhsh, N.; Liang, J. UNet++: A Nested U-Net Architecture for Medical Image Segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2018; Volume 11045. [Google Scholar] [CrossRef] [Green Version]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Lecture Notes in Computer Science; Navab, N., Hornegger, J., Frangi, A.W., Eds.; Springer: Berlin/Heidelberg, Germany, 2015; Volume 9351. [Google Scholar] [CrossRef] [Green Version]
- Catà, M.; Casamitjana, A.; Sanchez, I.; Combalia, M.; Vilaplana, V. Masked V-Net: An approach to brain tumor segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention. MICCAI BraTS Challenge, Quebec, QC, Canada, 11–13 September 2017; pp. 42–49. [Google Scholar]
- Xue, Y.; Xu, T.; Zhang, H.; Long, L.R.; Huang, X. SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation. Neuroinform 2018, 16, 383–392. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhou, Y.; Xie, L.; Shen, W.; Wang, Y.; Fishman, E.K.; Yuille, A.L. A Fixed-Point Model for Pancreas Segmentation in Abdominal CT Scans. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI BraTS Challenge, Quebec, QC, Canada, 11–13 September 2017; pp. 693–701. [Google Scholar] [CrossRef] [Green Version]
- Izard, S.G.; Méndez, J.A.J. Virtual reality medical training system. In Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality, Salamanca, Spain, 2–4 November 2016; pp. 479–485. [Google Scholar] [CrossRef]
- Izard, S.G.; Méndez, J.A.J.; García-Peñalvo, F.J.; López, M.J.; Vázquez, F.P.; Ruisoto, P. 360 vision applications for medical training. In Proceedings of the Fifth International Conference on Technological Ecosystems for Enhancing Multiculturality, Cádiz, Spain, 18–20 October 2017; ACM: New York, NY, USA, 2017; pp. 1–7. [Google Scholar] [CrossRef]
- Izard, S.G.; Méndez, J.A.J.; Estella, J.M.G.; Ledesma, M.J.S.; García-Peñalvo, F.J.; Ruisoto, P. Virtual simulation for scoliosis surgery. In Proceedings of the Fifth International Conference on Technological Ecosystems for Enhancing Multiculturality, Cádiz, Spain, 18–20 October 2017; Association for Computing Machinery: New York, NY, USA, 2017; pp. 1–8. [Google Scholar]
- Izard, S.G.; Juanes, J.A.; Peñalvo, F.J.G.; Estella, J.M.G.; Ledesma, M.J.S.; Ruisoto, P. Virtual reality as an educational and training tool for medicine. J. Med. Syst. 2018, 42, 50. [Google Scholar] [CrossRef]
- Izard, S.G.; Méndez, J.A.J.; Palomera, P.R. Virtual reality educational tool for human anatomy. J. Med. Syst. 2017, 41, 76. [Google Scholar] [CrossRef]
- Izard, S.G.; Méndez, J.A.J.; Palomera, P.R.; García-Peñalvo, F.J. Applications of virtual and augmented reality in biomedical imaging. J. Med. Syst. 2019, 43, 102. [Google Scholar] [CrossRef]
- Ibáñez, L.; Schroeder, W.; Ng, L.; Cates, J.; The Insight Software Consortium. The ITK Software Guide Second Edition Updated for ITK Version 2.4; Kitware, Inc.: New York, NY, USA, 2005. [Google Scholar]
- Yoo, T.S.; Ackerman, M.J.; Lorensen, W.E.; Schroeder, W.; Chalana, V.; Aylward, S.; Metaxas, D.; Whitaker, R. Engineering and algorithm design for an image processing API: A technical report on ITK the insight toolkit. In Studies in Health Technology and Informatics; IOS Press: Amsterdam, The Netherlands, 2002; pp. 586–592. [Google Scholar]
- Perkel, J. Why JUPYTER is Data Scientists’ Computational Notebook of Choice. Nature 2018, 563, 145–146. [Google Scholar] [CrossRef] [Green Version]
- Armato, S.G., III; McLennan, G.; Bidaut, L.; McNitt-Gray, M.F.; Meyer, C.R.; Reeves, A.P.; Clarke, L.P. Data from lidc-idri. Cancer Imaging Arch 2015, 9. [Google Scholar] [CrossRef]
- Armato, S.G., III; McLennan, G.; Bidaut, L.; McNitt-Gray, M.F.; Meyer, C.R.; Reeves, A.P.; Zhao, B.; Aberle, D.R.; Henschke, C.I.; Hoffman, E.A.; et al. The lung image database consortium (LIDC) and image database resource initiative (IDRI): A completed reference database of lung nodules on CT scans. Med Phys. 2011, 38, 915–931. [Google Scholar] [CrossRef]
- Bhagat, A.P.; Atique, M. Medical images: Formats, compression techniques and dicom image retrieval a survey. In Proceedings of the 2012 International Conference on Devices, Circuits and Systems (ICDCS), Coimbatore, India, 15–16 March 2012; pp. 172–176. [Google Scholar] [CrossRef]
- Mildenberger, P.; Eichelberg, M.; Martin, E. Introduction to the dicom standard. Eur. Radiol. 2002, 12, 920–927. [Google Scholar] [CrossRef]
- Geveci, B.; Schroeder, W.; Brown, A.; Wilson, G. The Architecture of Open Source Applications; Kitware, Inc.: New York, NY, USA, 2012. [Google Scholar]
- Schroeder, W.J.; Avila, L.S.; Hoffman, W. Visualizing with vtk: A tutorial. IEEE Comput. Graph. Appl. 2000, 20, 20–27. [Google Scholar] [CrossRef] [Green Version]
- Yagou, H.; Ohtake, Y.; Belyaev, A. Mesh smoothing via mean and median filtering applied to face normals. In Proceedings of the Geometric Modeling and Processing. Theory and Applications (GMPC 2002), Saitama, Japan, 10–12 July 2002; pp. 124–131. [Google Scholar] [CrossRef]
- Fain, Y.; Moiseev, A. Angular 2 Development with TypeScript; Manning Publications Co.: Shelter Island, NY, USA, 2016. [Google Scholar]
- Rubio, D. Beginning Django; Apress: Berkeley, CA, USA, 2017. [Google Scholar]
- Schmitz, G.; Fett, D.; Küsters, R. A comprehensive formal security analysis of oAuth 2.0. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 24–28 October 2016; pp. 1204–1215. [Google Scholar]
- Tommasi, E.; Agrusti, A.; Cataldo, R.; De Mitri, I.; Favetta, M.; Maglio, S.; Massafra, A.; Quarta, M.; Torsello, M.; Zecca, I.; et al. Automatic lung segmentation in CT images with accurate handling of the hilar region. J. Digit. Imaging 2011, 24, 11–27. [Google Scholar]
- Singadkar, G.; Mahajan, A.; Thakur, M.; Talbar, S. Automatic lung segmentation for the inclusion of juxtapleural nodules and pulmonary vessels using curvature based border correction. J. King Saud Univ. Comput. Inf. Sci. 2018. [Google Scholar] [CrossRef]
- Kroes, T.; Post, F.H.; Botha, C. Exposure render: An interactive photo-realistic volume rendering framework. PLoS ONE 2012, 7, e38586. [Google Scholar] [CrossRef] [PubMed]
- Sanders, J.; Kandrot, E. CUDA by Example: An Introduction to General-Purpose GPU Programming; Addison-Wesley Professional: Boston, MA, USA, 2010. [Google Scholar]
- Birkfellner, W.; Figl, M.; Huber, K.; Watzinger, F.; Wanschitz, F.; Hummel, J.; Hanel, R.; Greimel, W.; Homolkam, P.; Ewers, R.; et al. A head-mounted operating binocular for augmented reality visualization in medicine—Design and initial evaluation. IEEE Trans. Med. Imaging 2002, 21, 991–997. [Google Scholar] [CrossRef] [PubMed]
- Nikolaou, V.S.; Chytas, D.; Malahias, M.A. Augmented reality in orthopedics: Current state and future directions. Front. Surg. 2019, 6, 38. [Google Scholar]
- Gerard, I.J.; Kersten-Oertel, M.; Drouin, S.; Hall, J.A.; Petrecca, K.; De Nigris, D.; Collins, D.L.; Ian, J.; Kersten-Oertel, M. Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: A pilot study of eight cases. J. Med. Imaging 2018, 5, 021210. [Google Scholar] [CrossRef]
- Jiang, H.; Xu, S.; State, A.; Feng, F.; Fuchs, H.; Hong, M.; Rozenblit, J. Enhancing a laparoscopy training system with augmented reality visualization. In Proceedings of the Spring Simulation Conference (SpringSim), Tucson, AZ, USA, 29 April–2 May 2019; pp. 1–12. [Google Scholar]
- Navab, N.; Blum, T.; Wang, L.; Okur, A.; Wendler, T. First deployments of augmented reality in operating rooms. Computer 2012, 45, 48–55. [Google Scholar] [CrossRef]
- Navab, N.; Traub, J.; Sielhorst, T.; Feuerstein, M.; Bichlmeier, C. Action- and workflow-driven aug- mented reality for computer-aided medical procedures. IEEE Comput. Graph. Appl. 2007, 27, 10–14. [Google Scholar] [CrossRef]
- Michael Müller Carsten, N.; Gutt Hans-Peter Meinzer Jens, J.; Rassweiler Selcuk, G.; Simpfendörfer, T.; Baumhauer, M.; Teber, D. Augmented reality visualization during laparoscopic radical prostatectomy. J. Endourol. 2011, 25, 1841–1845. [Google Scholar]
- Chugh, H.; Mehta, V.; Banerjee, D.; Banerjee, P. Applications of augmented reality in emerging health diagnostics: A survey. In Proceedings of the International Conference on Automation and Computational Engineering (ICACE), Greater Noida, India, 3–4 October 2018; pp. 45–51. [Google Scholar] [CrossRef]
- InterestingEngineering. Google’s Medical AI Detects Lung Cancer with 94 Percent Accuracy. Available online: https://interestingengineering.com/googles-medical-ai-detects-lung-cancer-with-94-accuracy (accessed on 21 December 2019).
- Liu, Y.; Gadepalli, K.; Norouzi, M.; Dahl, G.E.; Kohlberger, T.; Boyko, A.; Hipp, J.D.; Timofeev, A.; Nelson, P.; Corrado, G.; et al. Detecting cancer metastases on gigapixel pathology images. arXiv 2017, arXiv:1703.02442. [Google Scholar]
- Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M.C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016. [Google Scholar] [CrossRef] [PubMed]
- Poplin, R.; Varadarajan, A.V.; Blumer, K.; Liu, Y.; McConnell, M.; Corrado, G.; Peng, L.; Webster, D. Predicting cardiovascular risk factors in retinal fundus photographs using deep learning. Nat. Biomed. Eng. 2018, 2, 158–164. [Google Scholar] [CrossRef] [PubMed]
- García-Peñalvo, F.J. Formación en la sociedad del conocimiento, un programa de doctorado con una perspectiva interdisciplinar. Educ. Knowl. Soc. 2014, 15, 4–9. [Google Scholar] [CrossRef]
- García-Peñalvo, F.J. Engineering contributions to a multicultural perspective of the knowledge society. IEEE Rev. Iberoam. Tecnol. Aprendiz. 2015, 10, 17–18. [Google Scholar] [CrossRef]
CT Scan | Region Segmented | Execution Time | Image Resolution | Number of Slices |
---|---|---|---|---|
LIDC-IDRI-0001 | lungs | 6.94 s | 512 × 512 | 133 |
lung vessels | 1 min 23 s | |||
trachea | 11.4 s | |||
spine | 1 min 42 s | |||
heart | 2 min 18 s | |||
LIDC-IDRI-0002 | lungs | 13.3 s | 512 × 512 | 261 |
lung vessels | 1 min 56 s | |||
trachea | 31 s | |||
spine | 3 min 25 s | |||
heart | 4 min 40 s | |||
LIDC-IDRI-1004 | lungs | 25.3 s | 512 × 512 | 529 |
lung vessels | 3 min 16 s | |||
trachea | 1 min 28 s | |||
spine | 6 min 54 s | |||
heart | 10 min 2 s |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
González Izard, S.; Sánchez Torres, R.; Alonso Plaza, Ó.; Juanes Méndez, J.A.; García-Peñalvo, F.J. Nextmed: Automatic Imaging Segmentation, 3D Reconstruction, and 3D Model Visualization Platform Using Augmented and Virtual Reality. Sensors 2020, 20, 2962. https://doi.org/10.3390/s20102962
González Izard S, Sánchez Torres R, Alonso Plaza Ó, Juanes Méndez JA, García-Peñalvo FJ. Nextmed: Automatic Imaging Segmentation, 3D Reconstruction, and 3D Model Visualization Platform Using Augmented and Virtual Reality. Sensors. 2020; 20(10):2962. https://doi.org/10.3390/s20102962
Chicago/Turabian StyleGonzález Izard, Santiago, Ramiro Sánchez Torres, Óscar Alonso Plaza, Juan Antonio Juanes Méndez, and Francisco José García-Peñalvo. 2020. "Nextmed: Automatic Imaging Segmentation, 3D Reconstruction, and 3D Model Visualization Platform Using Augmented and Virtual Reality" Sensors 20, no. 10: 2962. https://doi.org/10.3390/s20102962