Multi-Task Semi-Supervised Approach for Counting Cones in Adaptive Optics Images
Abstract
1. Introduction
2. Methodology
2.1. Inpainting SSL
2.2. Learning-to-Rank SSL
2.3. Downstream Cone Counting
2.4. Multi-Task Semi-Supervised Learning
2.5. Point Map Generation
3. Experiment Setup
- Single-Task Self-Supervised Learning (STSSL)—We first evaluated each pretext task—IP and L2R individually. Each model was pretrained using only the respective self-supervised objective on unlabeled images, and subsequently finetuned using the labeled images for the downstream cone counting task via supervised density-map regression.
- Multi-Task Self-Supervised Pretraining + FineTuning (MTSSP)—Next, we jointly pretrained the model using both IP and L2R pretext tasks on unlabeled data. The frozen shared encoder was then combined with a decoder finetuned on labeled data for cone counting.
- Proposed MTSSL—Finally, we evaluated our proposed approach, where IP, L2R, and density-map supervision were jointly optimized in a single training loop.
3.1. Dataset
3.2. STSSL—IP
3.3. STSSL—L2R
3.4. MTSSP
3.5. Downstream
3.6. MTSSL
Algorithm 1 Annotation | |
1: Input: Image to label | |
2: Output: Ground Truth Density Map | |
3: | |
4: | ▹ Density Maps |
Algorithm 2 MTSSL | |
Input: Unlabeled Images , labeled images , density maps , Model with encoder and decoders for IP, L2R, and density estimation, respectively | |
Output: Predicted Density Maps , hierarchical patch features , reconstructed masked region | |
1: Sample mask m on | ▹ Random Masking m |
2: , | ▹ Hierarchical patch extractor |
3: | ▹ Decoder , Encoder E, Inpainting |
4: | |
5: , |
Algorithm 3 MTSSL mixed loss | |
Input: Images | |
Output: Predicted Density Maps | |
1: | |
2: | |
3: | |
4: | ▹ Compute reconstruction loss |
5: | ▹ Ranking loss R |
6: | |
7: | |
8: Update the model parameters using |
4. Experiments and Results
4.1. Generalization to Different Scales
4.2. Counting Performance in Different Regions
4.3. Performance of IP-Related Self-Supervised Tasks vs. Counting Performance
4.4. Point Map Localization
4.5. Ablations
5. Discussion
6. Conclusions
Author Contributions
Funding
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Majumder, P.D.; Marchese, A.; Pichi, F.; Garg, I.; Agarwal, A. An update on autoimmune retinopathy. Indian J. Ophthalmol. 2020, 68, 1829. [Google Scholar] [CrossRef]
- Campochiaro, P.A.; Mir, T.A. The mechanism of cone cell death in Retinitis Pigmentosa. Prog. Retin. Eye Res. 2018, 62, 24–37. [Google Scholar] [CrossRef] [PubMed]
- Chen, Y.; Ratnam, K.; Sundquist, S.M.; Lujan, B.; Ayyagari, R.; Gudiseva, V.H.; Roorda, A.; Duncan, J.L. Cone photoreceptor abnormalities correlate with vision loss in patients with Stargardt disease. Investig. Ophthalmol. Vis. Sci. 2011, 52, 3281–3292. [Google Scholar] [CrossRef]
- Davidson, B.; Kalitzeos, A.; Carroll, J.; Dubra, A.; Ourselin, S.; Michaelides, M.; Bergeles, C. Automatic cone photoreceptor localisation in healthy and Stargardt afflicted retinas using deep learning. Sci. Rep. 2018, 8, 7911. [Google Scholar] [CrossRef]
- Tang, J.; Liu, H.; Mo, S.; Zhu, Z.; Huang, H.; Liu, X. Cone Density Distribution and Related Factors in Patients Receiving Hydroxychloroquine Treatment. Investig. Ophthalmol. Vis. Sci. 2023, 64, 29. [Google Scholar] [CrossRef]
- Toledo-Cortés, S.; Dubis, A.M.; González, F.A.; Müller, H. Deep Density Estimation for Cone Counting and Diagnosis of Genetic Eye Diseases From Adaptive Optics Scanning Light Ophthalmoscope Images. Transl. Vis. Sci. Technol. 2023, 12, 25. [Google Scholar] [CrossRef]
- Vera-Díaz, F.A.; Doble, N. The Human Eye and Adaptive Optics. In Topics in Adaptive Optics; Tyson, R.K., Ed.; IntechOpen: Rijeka, Croatia, 2012; Chapter 6. [Google Scholar] [CrossRef]
- Burns, S.A.; Elsner, A.E.; Sapoznik, K.A.; Warner, R.L.; Gast, T.J. Adaptive optics imaging of the human retina. Prog. Retin. Eye Res. 2019, 68, 1–30. [Google Scholar] [CrossRef]
- Samelska, K.; Szaflik, J.P.; Śmigielska, B.; Zaleska-Żmijewska, A. Progression of Rare Inherited Retinal Dystrophies May Be Monitored by Adaptive Optics Imaging. Life 2023, 13, 1871. [Google Scholar] [CrossRef] [PubMed]
- Nakanishi, A.; Ueno, S.; Kawano, K.; Ito, Y.; Kominami, T.; Yasuda, S.; Kondo, M.; Tsunoda, K.; Iwata, T.; Terasaki, H. Pathologic changes of cone photoreceptors in eyes with occult macular dystrophy. Investig. Ophthalmol. Vis. Sci. 2015, 56, 7243–7249. [Google Scholar] [CrossRef]
- Kupis, M.; Wawrzyniak, Z.M.; Szaflik, J.P.; Zaleska-Żmijewska, A. Retinal Photoreceptors and Microvascular Changes in the Assessment of Diabetic Retinopathy Progression: A Two-Year Follow-Up Study. Diagnostics 2023, 13, 2513. [Google Scholar] [CrossRef] [PubMed]
- Mc Grath, O.; Sarfraz, M.W.; Gupta, A.; Yang, Y.; Aslam, T. Clinical utility of artificial intelligence algorithms to enhance wide-field optical coherence tomography angiography images. J. Imaging 2021, 7, 32. [Google Scholar] [CrossRef]
- Legras, R.; Gaudric, A.; Woog, K. Distribution of cone density, spacing and arrangement in adult healthy retinas with adaptive optics flood illumination. PLoS ONE 2018, 13, e0191141. [Google Scholar] [CrossRef]
- Cunefare, D.; Huckenpahler, A.L.; Patterson, E.J.; Dubra, A.; Carroll, J.; Farsiu, S. RAC-CNN: Multimodal deep learning based automatic detection and classification of rod and cone photoreceptors in adaptive optics scanning light ophthalmoscope images. Biomed. Opt. Express 2019, 10, 3815–3832. [Google Scholar] [CrossRef]
- Cunefare, D.; Langlo, C.S.; Patterson, E.J.; Blau, S.; Dubra, A.; Carroll, J.; Farsiu, S. Deep learning based detection of cone photoreceptors with multimodal adaptive optics scanning light ophthalmoscope images of achromatopsia. Biomed. Opt. Express 2018, 9, 3740–3756. [Google Scholar] [CrossRef] [PubMed]
- Morgan, J.I.; Chen, M.; Huang, A.M.; Jiang, Y.Y.; Cooper, R.F. Cone identification in choroideremia: Repeatability, reliability, and automation through use of a convolutional neural network. Transl. Vis. Sci. Technol. 2020, 9, 40. [Google Scholar] [CrossRef] [PubMed]
- Smith, T.B.; Smith, N. Detection of cone photoreceptors in adaptive optics retinal images using topographical features and machine learning. In Proceedings of the Classical Optics 2014, Kohala Coast, HI, USA, 22–26 June 2014; Optica Publishing Group: Washington, DC, USA, 2014; p. JTu5A.41. [Google Scholar] [CrossRef]
- Hamwood, J.; Alonso-Caneiro, D.; Sampson, D.M.; Collins, M.J.; Chen, F.K. Automatic detection of cone photoreceptors with fully convolutional networks. Transl. Vis. Sci. Technol. 2019, 8, 10. [Google Scholar] [CrossRef]
- Garrioch, R.; Langlo, C.; Dubis, A.M.; Cooper, R.F.; Dubra, A.; Carroll, J. Repeatability of in vivo parafoveal cone density and spacing measurements. Optom. Vis. Sci. 2012, 89, 632–643. [Google Scholar] [CrossRef]
- Zaleska-Żmijewska, A.; Wawrzyniak, Z.M.; Dąbrowska, A.; Szaflik, J.P. Adaptive optics (rtx1) high-resolution imaging of photoreceptors and retinal arteries in patients with diabetic retinopathy. J. Diabetes Res. 2019, 2019, 9548324. [Google Scholar] [CrossRef]
- Morgan, J.I.; Vergilio, G.K.; Hsu, J.; Dubra, A.; Cooper, R.F. The reliability of cone density measurements in the presence of rods. Transl. Vis. Sci. Technol. 2018, 7, 21. [Google Scholar] [CrossRef] [PubMed]
- Akhavanrezayat, A.; Bommanapally, V.; Mudiyanselage, D.L.G.; Halim, M.S.; Or, C.; Karaca, I.; Uludag, G.; Yavari, N.; Bazojoo, V.; Mobasserian, A.; et al. A novel objective method to detect the foveal center point in the rtx1TM device using artificial intelligence. Investig. Ophthalmol. Vis. Sci. 2023, 64, 1068. [Google Scholar]
- Bommanapally, V.; Abeyrathna, D.; Chundi, P.; Subramaniam, M. Super resolution-based methodology for self-supervised segmentation of microscopy images. Front. Microbiol. 2024, 15, 1255850. [Google Scholar] [CrossRef]
- Abeyrathna, D.; Ashaduzzaman, M.; Malshe, M.; Kalimuthu, J.; Gadhamshetty, V.; Chundi, P.; Subramaniam, M. An AI-based approach for detecting cells and microbial byproducts in low volume scanning electron microscope images of biofilms. Front. Microbiol. 2022, 13, 996400. [Google Scholar] [CrossRef] [PubMed]
- Deng, L.; Zhou, Q.; Wang, S.; Górriz, J.M.; Zhang, Y. Deep learning in crowd counting: A survey. CAAI Trans. Intell. Technol. 2024, 9, 1043–1077. [Google Scholar] [CrossRef]
- Liu, X.; Van De Weijer, J.; Bagdanov, A.D. Leveraging unlabeled data for crowd counting by learning to rank. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7661–7669. [Google Scholar]
- Babu Sam, D.; Agarwalla, A.; Joseph, J.; Sindagi, V.A.; Babu, R.V.; Patel, V.M. Completely self-supervised crowd counting via distribution matching. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022; Springer: Cham, Switzerland, 2022; pp. 186–204. [Google Scholar]
- Bommanapally, V.; Akhavanrezayat, A.; Nguyen, Q.D.; Subramaniam, M. COINS: Counting Cones Using Inpainting Based Self-supervised Learning. In Proceedings of the 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 15–19 July 2024; pp. 1–4. [Google Scholar]
- Sun, G.; Liu, Y.; Probst, T.; Paudel, D.P.; Popovic, N.; Gool, L. Rethinking global context in crowd counting. Mach. Intell. Res. 2024, 21, 640–651. [Google Scholar] [CrossRef]
- Ma, Z.; Zhou, L.; Wu, D.; Zhang, X. A small object detection method with context information for high altitude images. Pattern Recognit. Lett. 2025, 188, 22–28. [Google Scholar] [CrossRef]
- Zhao, T.; Yue, Y.; Sun, H.; Li, J.; Wen, Y.; Yao, Y.; Qian, W.; Guan, Y.; Qi, S. MAEMC-NET: A hybrid self-supervised learning method for predicting the malignancy of solitary pulmonary nodules from CT images. Front. Med. 2025, 12, 1507258. [Google Scholar] [CrossRef]
- Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A.A. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 2536–2544. [Google Scholar]
- Ching, J.H.; See, J.; Wong, L.K. Learning image aesthetics by learning inpainting. In Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 25–28 October 2020; pp. 2246–2250. [Google Scholar]
- Heinrich, K.; Roth, A.; Zschech, P. Everything counts: A Taxonomy of Deep Learning Approaches for Object Counting. In Proceedings of the ECIS, Stockholm, Sweden, 8–14 June 2019. [Google Scholar]
- Liao, W.; Xiong, H.; Wang, Q.; Mo, Y.; Li, X.; Liu, Y.; Chen, Z.; Huang, S.; Dou, D. Muscle: Multi-task self-supervised continual learning to pre-train deep models for x-ray images of multiple body parts. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore, 18–22 September 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 151–161. [Google Scholar]
- Crawshaw, M. Multi-task learning with deep neural networks: A survey. arXiv 2020, arXiv:2009.09796. [Google Scholar] [CrossRef]
- Xie, Q.; Dai, Z.; Hovy, E.; Luong, T.; Le, Q. Unsupervised data augmentation for consistency training. Adv. Neural Inf. Process. Syst. 2020, 33, 6256–6268. [Google Scholar]
- Gu, S.; Lian, Z. A unified multi-task learning framework of real-time drone supervision for crowd counting. arXiv 2022, arXiv:2202.03843. [Google Scholar]
- Zhu, P.; Li, J.; Cao, B.; Hu, Q. Multi-task credible pseudo-label learning for semi-supervised crowd counting. IEEE Trans. Neural Netw. Learn. Syst. 2023, 35, 10394–10406. [Google Scholar] [CrossRef] [PubMed]
- Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; Raffel, C.A. Mixmatch: A holistic approach to semi-supervised learning. Adv. Neural Inf. Process. Syst. 2019, 32, 5050–5060. [Google Scholar]
- Wang, Y.; Song, D.; Wang, W.; Rao, S.; Wang, X.; Wang, M. Self-supervised learning and semi-supervised learning for multi-sequence medical image classification. Neurocomputing 2022, 513, 383–394. [Google Scholar] [CrossRef]
- Qiu, Z.; Zeng, W.; Liao, D.; Gui, N. A-SFS: Semi-supervised feature selection based on multi-task self-supervision. Knowl.-Based Syst. 2022, 252, 109449. [Google Scholar] [CrossRef]
- Pfister, J.; Kobs, K.; Hotho, A. Self-supervised multi-task pretraining improves image aesthetic assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 816–825. [Google Scholar]
- Fang, X.; Zhang, G.; Zhang, G.; Zhou, X.; Wu, J.; Zhao, L. A hybrid self-supervised learning framework for hyperspectral image classification. In Proceedings of the 2023 International Conference on Computer, Vision and Intelligent Technology, Chenzhou, China, 25–28 August 2023; pp. 1–7. [Google Scholar]
- Lin, W.; Chan, A.B. Optimal transport minimization: Crowd localization on density maps for semi-supervised counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 21663–21673. [Google Scholar]
- Bidaut Garnier, M.; Flores, M.; Debellemanière, G.; Puyraveau, M.; Tumahai, P.; Meillat, M.; Schwartz, C.; Montard, M.; Delbosc, B.; Saleh, M. Reliability of cone counts using an adaptive optics retinal camera. Clin. Exp. Ophthalmol. 2014, 42, 833–840. [Google Scholar] [CrossRef]
- Li, Y.; Zhang, X.; Chen, D. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 1091–1100. [Google Scholar]
- Shi, C.; Sun, C.; Wu, Y.; Jia, Y. Video anomaly detection via sequentially learning multiple pretext tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 1–6 October 2023; pp. 10330–10340. [Google Scholar]
RMSE | |||||
---|---|---|---|---|---|
M1 | ✓ | x | x | x | |
M2 | ✓ | ✓ | x | x | |
M3 | ✓ | ✓ | ✓ | x | |
M4 | ✓ | ✓ | x | ✓ | |
M5 | ✓ | x | ✓ | ✓ | |
MTSSL | ✓ | ✓ | ✓ | ✓ |
RMSE | R-Squared | RMAE | |
---|---|---|---|
IP | 22.33 ± 0.152 | 0.923 | 9.9 |
L2R | 20.53 ± 0.191 | 0.901 | 8.7 |
MTSSP | 16.26 ± 0.107 | 0.955 | 7.2 |
MTSSL | 10.52 ± 0.080 | 0.974 | 4.1 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bommanapally, V.; Akhavanrezayat, A.; Chundi, P.; Nguyen, Q.D.; Subramaniam, M. Multi-Task Semi-Supervised Approach for Counting Cones in Adaptive Optics Images. Algorithms 2025, 18, 552. https://doi.org/10.3390/a18090552
Bommanapally V, Akhavanrezayat A, Chundi P, Nguyen QD, Subramaniam M. Multi-Task Semi-Supervised Approach for Counting Cones in Adaptive Optics Images. Algorithms. 2025; 18(9):552. https://doi.org/10.3390/a18090552
Chicago/Turabian StyleBommanapally, Vidya, Amir Akhavanrezayat, Parvathi Chundi, Quan Dong Nguyen, and Mahadevan Subramaniam. 2025. "Multi-Task Semi-Supervised Approach for Counting Cones in Adaptive Optics Images" Algorithms 18, no. 9: 552. https://doi.org/10.3390/a18090552
APA StyleBommanapally, V., Akhavanrezayat, A., Chundi, P., Nguyen, Q. D., & Subramaniam, M. (2025). Multi-Task Semi-Supervised Approach for Counting Cones in Adaptive Optics Images. Algorithms, 18(9), 552. https://doi.org/10.3390/a18090552