Automatic Diabetic Retinopathy Grading via Self-Knowledge Distillation
Abstract
:1. Introduction
- (1)
- A novel self-knowledge distillation framework is proposed for diabetic retinopathy image grading. It can customize the pruning of the model according to the actual application scenario, which reduces the time delay while not significantly degrading the accuracy.
- (2)
- The introduction of CAM-Attention promotes the model to focus on pathological regions, and the Mimicking Module enables the model to maintain its original hierarchy while pruning. Experimental results confirm that the two proposed modules have a positive effect on the results.
- (3)
- The quantitative and qualitative results on the Messidor and IDRID datasets confirm the effectiveness of the methodology in this paper.
2. Methodology
2.1. CAM-Attention Module
2.2. Self-Knowledge Distillation
2.3. Mimicking Module
3. Experiments
3.1. Datasets Descriptions
3.2. Experimental Setup
3.3. Results on Messidor Dataset
3.4. Results on IDRID Dataset
4. Analysis and Discussion
4.1. Ablation Studies
4.2. Efficiency of the Network
4.3. Discussion on Free Parameters Selection
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
Abbreviations
DR | Diabetic retinopathy |
DME | Diabetic macular edema |
KD | Knowledge distillation |
MM | The Mimicking Module |
CNN | Convolution neural network |
KL | Kullback–Leibler divergence |
GNN | Graph neural network |
CI | Confidence interval |
FLOPs | Floating-point operations |
Appendix A
Appendix A.1. Results on CIFAR-100
Method | Main Branch | Side Branch 3 | Side Branch 2 | Side Branch 1 |
---|---|---|---|---|
baseline | 77.09/75.61 | - | - | - |
SKD [18] | 78.64 | 78.23 | 74.57 | 67.85 |
ours | 79.01 | 78.79 | 77.85 | 76.46 |
References
- Li, T.; Gao, Y.; Wang, K.; Guo, S.; Liu, H.; Kang, H. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Inf. Sci. 2019, 501, 511–522. [Google Scholar] [CrossRef]
- Zheng, Y.; He, M.; Congdon, N. The worldwide epidemic of diabetic retinopathy. Indian J. Ophthalmol. 2012, 60, 428. [Google Scholar] [PubMed]
- Adarsh, P.; Jeyakumari, D. Multiclass SVM-based automated diagnosis of diabetic retinopathy. In Proceedings of the International Conference on Communication and Signal Processing, Melmaruvathur, India, 3–5 April 2013; pp. 206–210. [Google Scholar]
- Antal, B.; Hajdu, A. An ensemble-based system for automatic screening of diabetic retinopathy. Knowl.-Based Syst. 2014, 60, 20–27. [Google Scholar] [CrossRef] [Green Version]
- Seoud, L.; Hurtut, T.; Chelbi, J.; Cheriet, F.; Langlois, J.P. Red lesion detection using dynamic shape features for diabetic retinopathy screening. IEEE Trans. Med. Imaging 2015, 35, 1116–1126. [Google Scholar] [CrossRef] [PubMed]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, LV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Luo, L.; Xue, D.; Feng, X. EHANet: An Effective Hierarchical Aggregation Network for Face Parsing. Appl. Sci. 2020, 10, 3135. [Google Scholar] [CrossRef]
- Bochkovskiy, A.; Wang, C.Y.; Liao, H.Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv 2020, arXiv:2004.10934. [Google Scholar]
- Zhao, Z.; Zhang, K.; Hao, X.; Tian, J.; Chua, M.C.H.; Chen, L.; Xu, X. Bira-net: Bilinear attention net for diabetic retinopathy grading. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–25 September 2019; pp. 1385–1389. [Google Scholar]
- Ni, J.; Chen, Q.; Liu, C.; Wang, H.; Cao, Y.; Liu, B. An Effective CNN Approach for Diabetic Retinopathy Stage Classification with Dual Inputs and Selective Data Sampling. In Proceedings of the IEEE International Conference on Machine Learning And Applications (ICMLA), Boca Raton, FL, USA, 16–19 December 2019; pp. 1578–1584. [Google Scholar]
- Wang, Z.; Yang, J. Diabetic retinopathy detection via deep convolutional networks for discriminative localization and visual explanation. In Proceedings of the Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, NO, USA, 2–7 February 2018; pp. 514–521. [Google Scholar]
- Li, X.; Hu, X.; Yu, L.; Zhu, L.; Fu, C.W.; Heng, P.A. CANet: Cross-Disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Grading. IEEE Trans. Med. Imaging 2020, 39, 1483–1493. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Porwal, P.; Pachade, S.; Kamble, R.; Kokare, M.; Deshmukh, G.; Sahasrabuddhe, V.; Meriaudeau, F. Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data 2018, 3, 25. [Google Scholar] [CrossRef] [Green Version]
- Foo, A.; Hsu, W.; Lee, M.L.; Lim, G.; Wong, T.Y. Multi-Task Learning for Diabetic Retinopathy Grading and Lesion Segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–20 February 2020; pp. 13267–13272. [Google Scholar]
- Zhao, M.; Hamarneh, G. Retinal image classification via vasculature-guided sequential attention. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea, 27 October–3 November 2019. [Google Scholar]
- Zhou, Y.; He, X.; Huang, L.; Liu, L.; Zhu, F.; Cui, S.; Shao, L. Collaborative learning of semi-supervised segmentation and classification for medical images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, LA, USA, 16–19 June 2019; pp. 2079–2088. [Google Scholar]
- Decencière, E.; Zhang, X.; Cazuguel, G.; Lay, B.; Cochener, B.; Trone, C.; Gaun, P.; Ordonez, R.; Massin, P.; Erginay, A.; et al. Feedback on a publicly distributed image database: The Messidor database. Image Anal. Stereol. 2014, 33, 231–234. [Google Scholar] [CrossRef] [Green Version]
- Zhang, L.; Song, J.; Gao, A.; Chen, J.; Bao, C.; Ma, K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–3 November 2019; pp. 3713–3722. [Google Scholar]
- Li, D.; Chen, Q. Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 7642–7651. [Google Scholar]
- Fukui, H.; Hirakawa, T.; Yamashita, T.; Fujiyoshi, H. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, LA, USA, 16–19 June 2019; pp. 10705–10714. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 7132–7141. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.Y.; So Kweon, I. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, LV, USA, 26 June–1 July 2016; pp. 2921–2929. [Google Scholar]
- Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv 2015, arXiv:1503.02531. [Google Scholar]
- Buciluǎ, C.; Caruana, R.; Niculescu-Mizil, A. Model compression. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, 20–23 August 2006; pp. 535–541. [Google Scholar]
- Kim, I.; Baek, W.; Kim, S. Spatially Attentive Output Layer for Image Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 9533–9542. [Google Scholar]
- Lee, C.Y.; Xie, S.; Gallagher, P.; Zhang, Z.; Tu, Z. Deeply-supervised nets. In Proceedings of the Artificial Intelligence and Statistics, San Diego, CA, USA, 9–12 May 2015; pp. 562–570. [Google Scholar]
- Romero, A.; Ballas, N.; Kahou, S.E.; Chassang, A.; Gatta, C.; Bengio, Y. Fitnets: Hints for thin deep nets. arXiv 2014, arXiv:1412.6550. [Google Scholar]
- Bengio, Y.; Courville, A.; Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1798–1828. [Google Scholar] [CrossRef] [PubMed]
- Pires, R.; Avila, S.; Jelinek, H.F.; Wainer, J.; Valle, E.; Rocha, A. Beyond lesion-based diabetic retinopathy: A direct approach for referral. IEEE J. Biomed. Health Inform. 2015, 21, 193–200. [Google Scholar] [CrossRef] [PubMed]
- Wang, Z.; Yin, Y.; Shi, J.; Fang, W.; Li, H.; Wang, X. Zoom-in-net: Deep mining lesions for diabetic retinopathy detection. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada, 6–11 October 2017; pp. 267–275. [Google Scholar]
- Vo, H.H.; Verma, A. New deep neural nets for fine-grained diabetic retinopathy recognition on hybrid color space. In Proceedings of the IEEE International Symposium on Multimedia (ISM), San Jose, CA, USA, 11–13 December 2016; pp. 209–215. [Google Scholar]
- Porwal, P.; Pachade, S.; Kokare, M.; Deshmukh, G.; Son, J.; Bae, W.; Liu, L.; Wang, J.; Liu, X.; Gao, L.; et al. IDRiD: Diabetic Retinopathy–Segmentation and Grading Challenge. Med. Image Anal. 2020, 59, 101561. [Google Scholar] [CrossRef] [PubMed]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Sánchez, C.I.; Niemeijer, M.; Dumitrescu, A.V.; Suttorp-Schulten, M.S.; Abramoff, M.D.; van Ginneken, B. Evaluation of a computer-aided diagnosis system for diabetic retinopathy screening on public data. Invest. Ophthalmol. Visual Sci. 2011, 52, 4866–4871. [Google Scholar] [CrossRef] [PubMed]
- Chen, Q.; Peng, Y.; Keenan, T.; Dharssi, S.; Agro, E. A multi-task deep learning model for the classification of Age-related Macular Degeneration. arXiv 2018, arXiv:1812.00422. [Google Scholar]
- Liu, L.; Dou, Q.; Chen, H.; Qin, J.; Heng, P.A. Multi-task deep model with margin ranking loss for lung nodule analysis. IEEE Trans. Med. Imaging 2019, 39, 718–728. [Google Scholar] [CrossRef] [PubMed]
- Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
- Tu, Z.; Gao, S.; Zhou, K.; Chen, X.; Fu, H.; Gu, Z.; Chen, J.; Yu, Z.; Liu, J. SUNet: A Lesion Regularized Model for Simultaneous Diabetic Retinopathy and Diabetic Macular Edema Grading. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), Iowa City, IA, USA, 4–7 April 2020; pp. 1378–1382. [Google Scholar]
- Sokolova, M.; Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 2009, 45, 427–437. [Google Scholar] [CrossRef]
- Kendall, A.; Gal, Y.; Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 7482–7491. [Google Scholar]
- Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features from Tiny Images. Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf (accessed on 17 June 2020).
Method | AUC | Acc. (%) | Pre. (%) | Rec. (%) |
---|---|---|---|---|
Pires et al. [30] | 0.863 | - | - | - |
VNXK/LGI [32] | 0.887 | 89.3 | - | - |
CKML Net/LGI [32] | 0.891 | 89.7 | - | - |
CANet [12] | 0.895 | 81.0 | - | - |
Comprehensive CAD [35] | 0.910 | - | - | - |
DSF-RFcara [5] | 0.916 | - | - | - |
Expert [35] | 0.940 | - | - | - |
Multitask net [36] † | 0.948 | 89.9 | 89.7 | 85.7 |
MTMR-Net [37] | 0.949 | 90.3 | 90.0 | 86.7 |
Zoom-in-net [31] | 0.957 | 91.1 | - | - |
CANet + MultiTask [12] | 0.963 | 92.6 | 90.6 | 92.0 |
SKD w/o CAM-Attention (ours) | 0.959 | 91.7 | 89.3 | 87.5 |
SKD (ours) | 0.966 ± 0.02 | 92.9 ± 3.99 | 91.0 ± 1.72 | 91.2 ± 2.18 |
Rank | Method | Main Branch | Side Branch 3 | Side Branch 2 | Side Branch 1 |
---|---|---|---|---|---|
1 | LzyUNCC * | 74.76 | - | - | - |
2 | SKD(ours) | 67.96 | 66.99 | 60.19 | 67.96 |
3 | SUNet [39] | 65.06 | - | - | - |
4 | VRT * | 59.22 | - | - | - |
5 | Mammoth * | 55.34 | - | - | - |
5 | HarangiM1 * | 55.34 | - | - | - |
6 | AVSASVA * | 54.37 | - | - | - |
7 | HarangiM2 * | 47.57 | - | - | - |
Method | Main Branch | Side Branch 3 | Side Branch 2 | Side Branch 1 |
---|---|---|---|---|
baseline | 91.67 | - | - | - |
baseline + CA | 93.33 | - | - | - |
baseline + CA + MM | 94.17 | 92.50 | 93.33 | 93.33 |
baseline + CA + MM + SKD | 95.83 | 95.83 | 95.00 | 94.17 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Luo, L.; Xue, D.; Feng, X. Automatic Diabetic Retinopathy Grading via Self-Knowledge Distillation. Electronics 2020, 9, 1337. https://doi.org/10.3390/electronics9091337
Luo L, Xue D, Feng X. Automatic Diabetic Retinopathy Grading via Self-Knowledge Distillation. Electronics. 2020; 9(9):1337. https://doi.org/10.3390/electronics9091337
Chicago/Turabian StyleLuo, Ling, Dingyu Xue, and Xinglong Feng. 2020. "Automatic Diabetic Retinopathy Grading via Self-Knowledge Distillation" Electronics 9, no. 9: 1337. https://doi.org/10.3390/electronics9091337