Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy
Abstract
:1. Introduction
2. Background of IIM and Super Resolution
2.1. Integral Imaging Microscopy
2.2. Deep Learning-Based Super-Resolution Algorithm
3. Proposed Method for IIM Super-Resolution
3.1. The Generator Network
3.2. The Discriminator Network
4. Experimental Setup and Quality Measurement Metrics
4.1. PSNR
4.2. SSIM
4.3. PSD
5. Results and Discussion of the Proposed Resolution Enhancement Method
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
C–B–L | Convolution, batch normalization, leaky ReLU |
CL | Camera lens |
DOF | Depth of field |
EI | Elemental image |
EIA | Elemental image array |
EL | Elemental lens |
GAN | Generative adversarial network |
GPU | Graphic processing unit |
HR | High resolution |
HVS | Human visual system |
IIM | Integral imaging microscopy |
IQM | Image quality measurement |
IVEI | Intermediate view elemental image |
LA | Lens array |
LFM | Light field microscopy |
LR | Low resolution |
MLA | Micro lens array |
OVI | Orthographic view image |
PCB | Printed circuit board |
PSD | power spectral density |
PSNR | Peak signal-to-noise ratio |
ReLU | Rectified linear unit |
PReLU | Parametric rectified linear unit |
SISR | Single image super-resolution |
SR | Super resolution |
SSIM | Structure similarity index |
References
- Belthangady, C.; Royer, L.A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 2019, 16, 1215–1225. [Google Scholar] [CrossRef]
- Palmieri, L.; Scrofani, G.; Incardona, N.; Saavedra, G.; Martínez-Corral, M.; Koch, R. Robust Depth Estimation for Light Field Microscopy. Sensors 2019, 19, 500. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Schlafer, S.; Meyer, R.L. Confocal microscopy imaging of the biofilm matrix. J. Microbiol. Methods 2017, 138, 50–59. [Google Scholar] [CrossRef] [PubMed]
- Wu, Y.; Rivenson, Y.; Wang, H.; Luo, Y.; Ben-David, E.; Bentolila, L.A.; Pritz, C.; Ozcan, A. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods 2019, 16, 1323–1331. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kwon, K.-C.; Kwon, K.H.; Erdenebat, M.-U.; Piao, Y.-L.; Lim, Y.-T.; Kim, M.Y.; Kim, N. Resolution-Enhancement for an Integral Imaging Microscopy Using Deep Learning. IEEE Photonics J. 2019, 11, 1–12. [Google Scholar] [CrossRef]
- Kim, J.; Jung, J.-H.; Jeong, Y.; Hong, K.; Lee, B. Real-time integral imaging system for light field microscopy. Opt. Express 2014, 22, 10210–10220. [Google Scholar] [CrossRef] [PubMed]
- Kim, N.; Alam, M.A.; Bang, L.T.; Phan, A.H.; Piao, M.L.; Erdenebat, M.U. Advances in the light field displays based on integral imaging and holographic techniques (Invited Paper). Chin. Opt. Lett. 2014, 12, 060005. [Google Scholar] [CrossRef]
- Jang, J.-S.; Javidi, B. Three-dimensional integral imaging of micro-objects. Opt. Lett. 2004, 29, 1230. [Google Scholar] [CrossRef] [Green Version]
- Levoy, M.; Ng, R.; Adams, A.; Footer, M.; Horowitz, M. Light field microscopy. In Proceedings of the ACM SIGGRAPH 2006 Papers, SIGGRAPH’06, Boston, MA, USA, 30 July–3 August 2006; ACM Press: New York, NY, USA, 2006; pp. 924–934. [Google Scholar]
- Lim, Y.-T.; Park, J.-H.; Kwon, K.-C.; Kim, N. Resolution-enhanced integral imaging microscopy that uses lens array shifting. Opt. Express 2009, 17, 19253. [Google Scholar] [CrossRef]
- Kwon, K.-C.; Erdenebat, M.-U.; Lim, Y.-T.; Joo, K.-I.; Park, M.-K.; Park, H.; Jeong, J.-R.; Kim, H.-R.; Kim, N. Enhancement of the depth-of-field of integral imaging microscope by using switchable bifocal liquid-crystalline polymer micro lens array. Opt. Express 2017, 25, 30503. [Google Scholar] [CrossRef]
- Kwon, K.-C.; Jeong, J.-S.; Erdenebat, M.-U.; Lim, Y.-T.; Yoo, K.-H.; Kim, N. Real-time interactive display for integral imaging microscopy. Appl. Opt. 2014, 53, 4450. [Google Scholar] [CrossRef]
- Jang, J.-S.; Javidi, B. Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics. Opt. Lett. 2002, 27, 324. [Google Scholar] [CrossRef] [PubMed]
- Kishk, S.; Javidi, B. Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging. Opt. Express 2003, 11, 3528. [Google Scholar] [CrossRef]
- Rivenson, Y.; Göröcs, Z.; Günaydin, H.; Zhang, Y.; Wang, H.; Ozcan, A. Deep learning microscopy. Optica 2017, 4, 1437. [Google Scholar] [CrossRef] [Green Version]
- Martinez-Corral, M.; Dorado, A.; Barreiro, J.C.; Saavedra, G.; Javidi, B. Recent Advances in the Capture and Display of Macroscopic and Microscopic 3-D Scenes by Integral Imaging. Proc. IEEE 2017, 105, 825–836. [Google Scholar] [CrossRef] [Green Version]
- Alam, S.; Kwon, K.-C.; Erdenebat, M.-U.; Lim, Y.-T.; Imtiaz, S.; Sufian, M.A.; Jeon, S.-H.; Kim, N. Resolution Enhancement of an Integral Imaging Microscopy Using Generative Adversarial Network. In Proceedings of the 14th Pacific Rim Conference on Lasers and Electro-Optics (CLEO PR 2020) (2020), paper C3G_4, The Optical Society, Sydney, Australia, 2–6 August 2020. [Google Scholar]
- Erdmann, L.; Gabriel, K.J. High-resolution digital integral photography by use of a scanning microlens array. Appl. Opt. 2001, 40, 5592. [Google Scholar] [CrossRef] [PubMed]
- Kwon, K.-C.; Jeong, J.-S.; Erdenebat, M.-U.; Piao, Y.-L.; Yoo, K.-H.; Kim, N. Resolution-enhancement for an orthographic-view image display in an integral imaging microscope system. Biomed. Opt. Express 2015, 6, 736–746. [Google Scholar] [CrossRef] [Green Version]
- Kwon, H.; Yoon, H.; Park, K.-W. CAPTCHA Image Generation: Two-Step Style-Transfer Learning in Deep Neural Networks. Sensors 2020, 20, 1495. [Google Scholar] [CrossRef] [Green Version]
- Kwon, H.; Yoon, H.; Park, K.W. Robust CAPTCHA image generation enhanced with adversarial example methods. IEICE Trans. Inf. Syst. 2020, 103, 879–882. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Q.; Yang, L.T.; Chen, Z.; Li, P. A survey on deep learning for big data. Inf. Fusion 2018, 42, 146–157. [Google Scholar] [CrossRef]
- Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.A.W.M.; van Ginneken, B.; Sánchez, C.I. A survey on deep learning in medical image analysis. Med. Image Anal. 2017, 42, 60–88. [Google Scholar] [CrossRef] [Green Version]
- Alam, M.S.; Kwon, K.-C.; Alam, M.A.; Abbass, M.Y.; Imtiaz, S.M.; Kim, N. Trajectory-Based Air-Writing Recognition Using Deep Neural Network and Depth Sensor. Sensors 2020, 20, 376. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Xie, J.; Girshick, R.; Farhadi, A. Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks; European Conference on Computer Vision; Springer: Cham, Switzerland, 2016. [Google Scholar]
- Nguyen-Phuoc, T.; Li, C.; Theis, L.; Richardt, C.; Yang, Y.L. HoloGAN: Unsupervised learning of 3D representations from natural images. In Proceedings of the 2019 International Conference on Computer Vision Workshop, ICCVW 2019, Seoul, Korea, 27 October–2 November 2019; pp. 2037–2040. [Google Scholar]
- Yang, W.; Zhang, X.; Tian, Y.; Wang, W.; Xue, J.-H.; Liao, Q. Deep Learning for Single Image Super-Resolution: A Brief Review. IEEE Trans. Multimed. 2019, 21, 3106–3121. [Google Scholar] [CrossRef] [Green Version]
- Abbass, M.Y.; Kwon, K.C.; Alam, M.S.; Piao, Y.L.; Lee, K.Y.; Kim, N. Image super resolution based on residual dense CNN and guided filters. Multimed. Tools Appl. 2020, 80, 1–19. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Mei, Y.; Fan, Y.; Zhou, Y.; Huang, L.; Huang, T.S.; Shi, H. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; IEEE Computer Society: Piscataway, NJ, USA, 2020; pp. 5689–5698. [Google Scholar]
- Jiang, K.; Wang, Z.; Yi, P.; Jiang, J. Hierarchical dense recursive network for image super-resolution. Pattern Recognit. 2020, 107, 107475. [Google Scholar] [CrossRef]
- Dong, C.; Loy, C.C.; He, K.; Tang, X. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 38, 295–307. [Google Scholar] [CrossRef] [Green Version]
- Lai, W.-S.; Huang, J.-B.; Ahuja, N.; Yang, M.-H. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; IEEE: Piscataway, NJ, USA, 2017; Volume 2017, pp. 5835–5843. [Google Scholar]
- Zhang, K.; Zuo, W.; Zhang, L. Learning a Single Convolutional Super-Resolution Network for Multiple Degradations. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 3262–3271. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Proceedings of the International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 2672–2680. [Google Scholar]
- Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; Abbeel, P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv 2016, arXiv:1606.03657. (preprint). [Google Scholar]
- Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv 2015, arXiv:1511.06434. (preprint). [Google Scholar]
- Zhu, J.-Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. Proc. IEEE Int. Conf. Comput. Vis. 2017, 2223–2232. [Google Scholar]
- Ledig, C.; Theis, L.; Huszar, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 105–114. [Google Scholar]
- Nair, V.; Hinton, G.E. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel, 21–24 June 2010; Omnipress: Madison, WI, USA, 2010; pp. 807–814. [Google Scholar]
- Xu, B.; Wang, N.; Chen, T.; Li, M. Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv 2015, arXiv:1505.00853. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; Volume 2015, pp. 1026–1034. [Google Scholar]
- Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6–11 July 2015; International Machine Learning Society (IMLS): Baltimore, ML, USA, 2015; Volume 1, pp. 448–456. [Google Scholar]
- Shi, W.; Caballero, J.; Huszar, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; IEEE Computer Society: Washington, DC, USA, 2016; Volume 2016, pp. 1874–1883. [Google Scholar]
- Maas, A.L.; Hannun, A.Y.; Ng, A.Y. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, Atlanta, GA, USA, 16–21 June 2013. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- The PASCAL Visual Object Classes Challenge (VOC2012). 2012. Available online: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/ (accessed on 15 June 2020).
- Mandal, J.K.; Satapathy, S.C.; Sanyal, M.K.; Sarkar, P.P.; Mukhopadhyay, A. Analysis and Evaluation of Image Quality Metrics. Adv. Intell. Syst. Comput. 2015, 340, 369–378. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wang, Z.; Chen, J.; Hoi, S.C.H. Deep Learning for Image Super-resolution: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 1. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Honeybee | Z. Mays | Hydra | Chip | PCB | ||
---|---|---|---|---|---|---|
SRCNN | PSNR | 15.74 | 18.56 | 16.33 | 12.82 | 12.32 |
SSIM | 0.63 | 0.84 | 0.89 | 0.39 | 0.64 | |
PSD | 4.82 | 4.53 | 4.28 | 5.01 | 4.82 | |
LapSRN | PSNR | 29.11 | 29.34 | 37.43 | 30.63 | 29.85 |
SSIM | 0.97 | 0.97 | 0.99 | 0.98 | 0.98 | |
PSD | 5.10 | 4.90 | 4.38 | 5.17 | 5.06 | |
SRMD (general) | PSNR | 23.54 | 27.19 | 19.35 | 22.83 | 23.41 |
SSIM | 0.83 | 0.92 | 0.97 | 0.98 | 0.91 | |
PSD | 4.27 | 4.18 | 3.90 | 4.51 | 4.56 | |
SRMD (bicubic) | PSNR | 29.23 | 32.57 | 37.46 | 30.97 | 33.99 |
SSIM | 0.96 | 0.97 | 0.99 | 0.98 | 0.98 | |
PSD | 5.23 | 4.98 | 4.36 | 5.25 | 5.01 | |
SRMDNF (general) | PSNR | 27.74 | 31.34 | 36.68 | 28.80 | 30.15 |
SSIM | 0.93 | 0.95 | 0.99 | 0.97 | 0.97 | |
PSD | 4.81 | 4.76 | 4.18 | 5.17 | 4.81 | |
SRMDNF (bicubic) | PSNR | 32.38 | 31.86 | 37.25 | 31.49 | 34.86 |
SSIM | 0.97 | 0.92 | 0.98 | 0.98 | 0.98 | |
PSD | 5.18 | 4.17 | 4.39 | 5.32 | 4.81 | |
SRGAN | PSNR | 32.68 | 32.33 | 37.33 | 31.47 | 31.53 |
SSIM | 0.98 | 0.98 | 0.99 | 0.98 | 0.98 | |
PSD | 5.49 | 5.24 | 4.74 | 5.46 | 5.16 | |
Proposed | PSNR | 33.37 | 33.19 | 37.84 | 32.14 | 32.60 |
SSIM | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | |
PSD | 5.75 | 5.75 | 5.06 | 5.79 | 5.57 |
Honeybee | Z. Mays | Hydra | Chip | PCB | ||
---|---|---|---|---|---|---|
SRCNN | PSNR | 16.60 | 19.57 | 16.61 | 13.73 | 12.97 |
SSIM | 0.67 | 0.86 | 0.89 | 0.47 | 0.66 | |
PSD | 4.43 | 3.87 | 3.97 | 4.44 | 4.43 | |
LapSRN | PSNR | 23.29 | 25.53 | 30.70 | 22.81 | 22.86 |
SSIM | 0.87 | 0.93 | 0.98 | 0.87 | 0.93 | |
PSD | 4.45 | 4.28 | 4.14 | 4.81 | 4.73 | |
SRMD (general) | PSNR | 21.00 | 25.58 | 27.43 | 21.10 | 21.57 |
SSIM | 0.79 | 0.90 | 0.97 | 0.72 | 0.88 | |
PSD | 3.93 | 3.99 | 3.71 | 4.21 | 4.38 | |
SRMD (bicubic) | PSNR | 24.99 | 29.07 | 33.66 | 25.62 | 25.64 |
SSIM | 0.88 | 0.92 | 0.98 | 0.89 | 0.94 | |
PSD | 4.44 | 4.33 | 4.01 | 4.65 | 4.48 | |
SRMDNF (general) | PSNR | 25.12 | 29.10 | 33.70 | 25.96 | 26.02 |
SSIM | 0.88 | 0.92 | 0.99 | 0.90 | 0.94 | |
PSD | 4.50 | 4.46 | 4.01 | 4.87 | 4.65 | |
SRMDNF (bicubic) | PSNR | 25.22 | 29.25 | 34.17 | 25.96 | 26.08 |
SSIM | 0.89 | 0.92 | 0.99 | 0.90 | 0.94 | |
PSD | 4.57 | 4.44 | 4.01 | 4.87 | 4.69 | |
SRGAN | PSNR | 29.58 | 30.46 | 34.28 | 28.06 | 29.72 |
SSIM | 0.98 | 0.98 | 0.99 | 0.98 | 0.98 | |
PSD | 4.98 | 4.63 | 4.59 | 5.14 | 4.87 | |
Proposed | PSNR | 31.63 | 31.79 | 35.14 | 30.98 | 31.95 |
SSIM | 0.99 | 0.98 | 0.99 | 0.98 | 0.99 | |
PSD | 5.76 | 5.76 | 5.07 | 5.78 | 5.56 |
Honeybee | Z. Mays | Hydra | Chip | PCB | |
---|---|---|---|---|---|
PSNR | 31.71 | 31.89 | 35.59 | 30.48 | 31.81 |
SSIM | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 |
PSD | 5.74 | 5.74 | 5.04 | 5.79 | 5.18 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Alam, M.S.; Kwon, K.-C.; Erdenebat, M.-U.; Y. Abbass, M.; Alam, M.A.; Kim, N. Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy. Sensors 2021, 21, 2164. https://doi.org/10.3390/s21062164
Alam MS, Kwon K-C, Erdenebat M-U, Y. Abbass M, Alam MA, Kim N. Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy. Sensors. 2021; 21(6):2164. https://doi.org/10.3390/s21062164
Chicago/Turabian StyleAlam, Md. Shahinur, Ki-Chul Kwon, Munkh-Uchral Erdenebat, Mohammed Y. Abbass, Md. Ashraful Alam, and Nam Kim. 2021. "Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy" Sensors 21, no. 6: 2164. https://doi.org/10.3390/s21062164
APA StyleAlam, M. S., Kwon, K.-C., Erdenebat, M.-U., Y. Abbass, M., Alam, M. A., & Kim, N. (2021). Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy. Sensors, 21(6), 2164. https://doi.org/10.3390/s21062164