Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages
Abstract
1. Introduction
2. Methods
2.1. Frame Sorting
2.1.1. Disc and Fovea Detection and Frame Categorization
2.1.2. Montage Center Frame Selection and Per-Frame Sorting
2.2. Frame Integration
2.2.1. Vessel Segmentation and Frame Preprocessing
2.2.2. Keypoint Matching Based Rigid Registration
2.2.3. Non-Rigid Registration
2.2.4. Blending
2.3. Algorithm Summary
3. Experimental Results
3.1. Dataset and Experimental Environment
3.2. Quantitative Evaluation
Algorithm 1: Retinal Fundus Photomontage Construction Using Deep Learning. |
Input: Set of fundus image frames , Trained Faster R-CNN for detecting optic disc and fovea Trained SSANet for vessel segmentation Output: Constructed photomontage , vessel map montage |
3.3. Qualitative Evaluation
4. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Cunha-Vaz, J.G. Pathophysiology of diabetic retinopathy. Br. J. Ophthalmol. 1978, 62, 351–355. [Google Scholar] [CrossRef]
- Ikram, M.K.; De Jong, F.J.; Van Dijk, E.J.; Prins, N.D.; Hofman, A.; Breteler, M.M.B.; De Jong, P.T.V.M. Retinal vessel diameters and cerebral small vessel disease: The Rotterdam Scan Study. Brain 2005, 129, 182–188. [Google Scholar] [CrossRef]
- Ritt, M.; Schmieder, R.E. Wall-to-Lumen ratio of retinal arterioles as a tool to assess vascular changes. Hypertension 2009, 54, 384–387. [Google Scholar] [CrossRef] [PubMed]
- Son, J.; Shin, J.Y.; Chun, E.J.; Jung, K.H.; Park, K.H.; Park, S.J. Predicting high coronary artery calcium score from retinal fundus images with deep learning algorithms. Transl. Vis. Sci. Technol. 2020, 9, 28. [Google Scholar] [CrossRef] [PubMed]
- Son, J.; Shin, J.Y.; Kim, H.D.; Jung, K.H.; Park, K.H.; Park, S.J. Development and Validation of Deep Learning Models for Screening Multiple Abnormal Findings in Retinal Fundus Images. Ophthalmology 2020, 127, 85–94. [Google Scholar] [CrossRef]
- Wykoff, C.C.; Eichenbaum, D.A.; Roth, D.B.; Hill, L.; Fung, A.E.; Haskova, Z. Ranibizumab induces regression of diabetic retinopathy in most patients at high risk of progression to proliferative diabetic retinopathy. Ophthalmol. Retin. 2018, 2, 997–1009. [Google Scholar] [CrossRef]
- Mahurkar, A.A.; Vivino, M.A.; Trus, B.L.; Kuehl, E.M.; Datiles, M.B., 3rd; Kaiser-Kupfer, M.I. Constructing retinal fundus photomontages. A new computer-based method. Invest. Ophthalmol. Vis. Sci. 1996, 37, 1675–1683. [Google Scholar] [PubMed]
- Can, A.; Stewart, C.V.; Roysam, B. Robust hierarchical algorithm for constructing a mosaic from images of the curved human retina. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Fort Collins, CO, USA, 23–25 June 1999; Volume 2. [Google Scholar]
- Cattin, P.C.; Bay, H.; Van Gool, L.; Székely, G. Retina mosaicing using local features. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI); Springer: New York, NY, USA, 2006; pp. 185–192. [Google Scholar]
- Lee, S.; Reinhardt, J.M.; Cattin, P.C.; Abràmoff, M.D. Objective and expert-independent validation of retinal image registration algorithms by a projective imaging distortion model. Med. Image Anal. 2010, 14, 539–549. [Google Scholar] [CrossRef] [PubMed]
- Feng, X.; Cai, G.; Gou, X.; Yun, Z.; Wang, W.; Yang, W. Retinal Mosaicking with Vascular Bifurcations Detected on Vessel Mask by a Convolutional Network. J. Healthc. Eng. 2020, 2020. [Google Scholar] [CrossRef] [PubMed]
- Bay, H.; Tuytelaars, T.; Van Gool, L. Surf: Speeded up robust features. In Proceedings of the European Conference on Computer Vision (ECCV); Springer: New York, NY, USA, 2006; pp. 404–417. [Google Scholar]
- Hernandez-Matas, C.; Zabulis, X.; Argyros, A.A. An experimental evaluation of the accuracy of keypoints-based retinal image registration. In Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Korea, 11–15 July 2017; pp. 377–381. [Google Scholar] [CrossRef]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object eetection with region proposal networks. In Advances in Neural Information Processing Systems; Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R., Eds.; IEEE Computer Society: Washington, DC, USA, 2015; Volume 28, pp. 91–99. [Google Scholar]
- Noh, K.J.; Park, S.J.; Lee, S. Fine-Scale vessel extraction in fundus images by registration with fluorescein angiography. In Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI); Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 779–787. [Google Scholar]
- Noh, K.J.; Kim, J.; Park, S.J.; Lee, S. Multimodal registration of fundus images With fluorescein angiography for fine-scale vessel segmentation. IEEE Access 2020, 8, 63757–63769. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, 20–25 June 2009. [Google Scholar]
- Noh, K.J.; Park, S.J.; Lee, S. Scale-space approximated convolutional neural networks for retinal vessel segmentation. Comput. Methods Programs Biomed. 2019, 178, 237–246. [Google Scholar] [CrossRef] [PubMed]
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision; Cambridge University Press: Cambridge, MA, USA, 2003. [Google Scholar]
- Byrd, R.H.; Lu, P.; Nocedal, J.; Zhu, C. A limited memory algorithm for bound constrained optimization. Siam J. Sci. Comput. 1995, 16, 1190–1208. [Google Scholar] [CrossRef]
- Burt, P.J.; Adelson, E.H. A multiresolution spline with application to image mosaics. Acm Trans. Graph. (TOG) 1983, 2, 217–236. [Google Scholar] [CrossRef]
- Park, S.J.; Shin, J.Y.; Kim, S.; Son, J.; Jung, K.H.; Park, K.H. A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training. J. Korean Med. Sci. 2018, 33, e239. [Google Scholar] [CrossRef] [PubMed]
- Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32; Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2019; pp. 8024–8035. [Google Scholar]
- Bradski, G. The opencv library. Dr Dobb’s J. Softw. Tools 2000, 25, 120–125. [Google Scholar]
- Lowekamp, B.C.; Chen, D.T.; Ibáñez, L.; Blezek, D. The design of SimpleITK. Front. Neuroinform. 2013, 7, 45. [Google Scholar] [CrossRef] [PubMed]
- Gehan, M.A.; Fahlgren, N.; Abbasi, A.; Berry, J.C.; Callen, S.T.; Chavez, L.; Doust, A.N.; Feldman, M.J.; Gilbert, K.B.; Hodge, J.G.; et al. PlantCV v2: Image analysis software for high-throughput plant phenotyping. PeerJ 2017, 5, e4088. [Google Scholar] [CrossRef] [PubMed]
- Li, D.; Zhang, L.; Sun, C.; Yin, T.; Liu, C.; Yang, J. Robust Retinal Image Enhancement via Dual-Tree Complex Wavelet Transform and Morphology-Based Method. IEEE Access 2019, 7, 47303–47316. [Google Scholar] [CrossRef]
- Kowa American Corporation. KOWA VK-2 Image Filing System: Features; 2021. Available online: https://ophthalmic.kowa-usa.com/products/software/vk-2-image-filing-system-features (accessed on 12 January 2021).
- Brown, M.; Lowe, D.G. Automatic Panoramic Image Stitching using Invariant Features. Int. J. Comput. Vis. 2007, 74, 59–73. [Google Scholar] [CrossRef]
Preprocessing | Frame Sorting Criterion | Avg. Frames (std) | p-Value | % of Frames | Avg. TRE (std) |
---|---|---|---|---|---|
Min/max norm. | Number of keypoint matches | 3.83 (1.48) | 51.87% | 26.16 (30.93) | |
Min/max norm. | Optic disc/fovea detection | 3.62 (1.41) | 49.28% | 24.97 (33.23) | |
Modified top-hat [28] | Optic disc/fovea detection | 5.1 (1.98) | 69.33% | 24.22 (30.1) | |
Vessel contrast | Number of keypoint matches | 6.04 (1.31) | 0.24 | 82.82% | 23.98 (30.41) |
Vessel contrast | Optic disc/fovea detection | 6.34 (1.46) | – | 86.25% | 23.67 (31.1) |
† P-value: the p-value of the null hypothesis for the number of frames measured by the paired t-test of comparative methods and the proposed method in last row. ‡ This row refers to the proposed method. |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kim, J.; Go, S.; Noh, K.J.; Park, S.J.; Lee, S. Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages. Appl. Sci. 2021, 11, 1754. https://doi.org/10.3390/app11041754
Kim J, Go S, Noh KJ, Park SJ, Lee S. Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages. Applied Sciences. 2021; 11(4):1754. https://doi.org/10.3390/app11041754
Chicago/Turabian StyleKim, Jooyoung, Sojung Go, Kyoung Jin Noh, Sang Jun Park, and Soochahn Lee. 2021. "Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages" Applied Sciences 11, no. 4: 1754. https://doi.org/10.3390/app11041754
APA StyleKim, J., Go, S., Noh, K. J., Park, S. J., & Lee, S. (2021). Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages. Applied Sciences, 11(4), 1754. https://doi.org/10.3390/app11041754