Dual-Training-Based Semi-Supervised Learning with Few Labels
Abstract
:1. Introduction
2. Related Work
2.1. Semi-Supervised Learning
2.2. Self-Supervised Learning
3. Our Approach
Algorithm 1 Dual-Training-Based Semi-Supervised Learning. | |
1 | Input: Labeled dataset , Unlabeled dataset |
2 | Setting: Learning rate , Batch size , Ratio of unlabeled data , Loss weight , , , Threshold , SGD algorithm with momentum, maximum epochs |
3 | Initialization: Encoder , Projection , Classifier |
4 | for epoch < maximum epochs do |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | Update Encoder , Projection , Classifier |
17 | end for |
18 | Output: Encoder , Projection , Classifier |
3.1. Dual Training Strategy
3.2. The Generation of Sample Weight
3.3. Self-Supervised Learning Based on Cosine Distance
4. Experiments
4.1. Datasets
4.2. Implementation Details
4.3. Comparison with State-of-the-Art Methods
4.4. Ablation Study
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Arco, J.E.; Ortiz, A.; Ramírez, J.; Martínez-Murcia, F.J.; Zhang, Y.-D.; Górriz, J.M. Uncertainty-driven ensembles of multi-scale deep architectures for image classification. Inf. Fusion 2023, 89, 53–65. [Google Scholar] [CrossRef]
- Sun, L.; Zhao, G.; Zheng, Y.; Wu, Z. Spectral–spatial feature tokenization transformer for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–14. [Google Scholar] [CrossRef]
- Shi, S.; Jiang, L.; Deng, J.; Wang, Z.; Guo, C.; Shi, J.; Wang, X.; Li, H. PV-RCNN++: Point-voxel feature set abstraction with local vector representation for 3D object detection. Int. J. Comput. Vis. 2023, 131, 531–551. [Google Scholar] [CrossRef]
- Cheng, G.; Wang, J.; Li, K.; Xie, X.; Lang, C.; Yao, Y.; Han, J. Anchor-free oriented proposal generator for object detection. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–11. [Google Scholar] [CrossRef]
- Lin, A.; Chen, B.; Xu, J.; Zhang, Z.; Lu, G.; Zhang, D. Ds-transunet: Dual swin transformer u-net for medical image segmentation. IEEE Trans. Instrum. Meas. 2022, 71, 4005615. [Google Scholar] [CrossRef]
- Ma, F.; Zhang, F.; Xiang, D.; Yin, Q.; Zhou, Y. Fast task-specific region merging for SAR image segmentation. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–16. [Google Scholar] [CrossRef]
- Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Li, F.-F. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar] [CrossRef]
- Snell, J.; Swersky, K.; Zemel, R. Prototypical networks for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; Volume 30, pp. 4077–4087. Available online: https://proceedings.neurips.cc/paper_files/paper/2017/file/cb8da6767461f2812ae4290eac7cbc42-Paper.pdf (accessed on 22 May 2024).
- Wang, Y.; Yao, Q.; Kwok, J.T.; Ni, L.M. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv. (CSUR) 2020, 53, 1–34. [Google Scholar] [CrossRef]
- Hua, Z.; Yang, Y. Robust and sparse label propagation for graph-based semi-supervised classification. Appl. Intell. 2022, 52, 3337–3351. [Google Scholar] [CrossRef]
- Liu, P.; Qian, W.; Cao, J.; Xu, D. Semi-supervised medical image classification via increasing prediction diversity. Appl. Intell. 2022, 52, 10162–10175. [Google Scholar] [CrossRef]
- Chen, Y.; Mancini, M.; Zhu, X.; Akata, Z. Semi-supervised and unsupervised deep visual learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 46, 1327–1347. [Google Scholar] [CrossRef]
- Xu, H.; Xiao, H.; Hao, H.; Dong, L.; Qiu, X.; Peng, C. Semi-supervised learning with pseudo-negative labels for image classification. Knowl-Based Syst. 2023, 260, 110166. [Google Scholar] [CrossRef]
- Wang, J.; Lukasiewicz, T.; Massiceti, D.; Hu, X.; Pavlovic, V.; Neophytou, A. NP-Match: When Neural Processes meet Semi-Supervised Learning. arXiv 2022, arXiv:2207.01066. [Google Scholar]
- Yang, F.; Wu, K.; Zhang, S.; Jiang, G.; Liu, Y.; Zheng, F.; Zhang, W.; Wang, C.; Zeng, L. Class-Aware Contrastive Semi-Supervised Learning. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 14401–14410. [Google Scholar] [CrossRef]
- Sohn, K.; Berthelot, D.; Carlini, N.; Zhang, Z.; Zhang, H.; Raffel, C.A.; Cubuk, E.D.; Kurakin, A.; Li, C.L. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–12 December 2020; Curran Associates, Inc.: New York, NY, USA, 2020; pp. 596–608. Available online: https://proceedings.neurips.cc/paper_files/paper/2020/file/06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf (accessed on 10 January 2024).
- Lee, D.-H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the Workshop on Challenges in Representation Learning (ICML), Atlanta, GA, USA, 16–21 June 2013; p. 896. Available online: https://ui.adsabs.harvard.edu/abs/2022arXiv220110836G (accessed on 10 January 2024).
- Tarvainen, A.; Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv 2017, arXiv:1703.01780. [Google Scholar]
- Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; Raffel, C.A. MixMatch: A Holistic Approach to Semi-Supervised Learning. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; Curran Associates, Inc.: New York, NY, USA, 2019. Available online: https://proceedings.neurips.cc/paper/2019/file/1cd138d0499a68f4bb72bee04bbec2d7-Paper.pdf (accessed on 10 January 2024).
- Cubuk, E.D.; Zoph, B.; Shlens, J.; Le, Q.V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Virtual, 6–12 December 2020; pp. 702–703. [Google Scholar] [CrossRef]
- Berthelot, D.; Carlini, N.; Cubuk, E.D.; Kurakin, A.; Sohn, K.; Zhang, H.; Raffel, C. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv 2019, arXiv:1911.09785. [Google Scholar]
- Li, J.; Xiong, C.; Hoi, S.C.H. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 11–17 October 2021; pp. 9475–9484. [Google Scholar] [CrossRef]
- Zheng, M.; You, S.; Huang, L.; Wang, F.; Qian, C.; Xu, C. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 14471–14481. [Google Scholar] [CrossRef]
- Yang, X.; Hu, X.; Zhou, S.; Liu, X.; Zhu, E. Interpolation-based contrastive learning for few-label semi-supervised learning. IEEE Trans. Neural Netw. Learn. Syst. 2022, 35, 2054–2065. [Google Scholar] [CrossRef] [PubMed]
- Hu, X.; Xu, X.; Zeng, Y.; Yang, X. Patch-Mixing Contrastive Regularization for Few-Label Semi-Supervised Learning. IEEE Trans. Artif. Intell. 2023, 5, 384–397. [Google Scholar] [CrossRef]
- Chen, X.; He, K. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 15750–15758. [Google Scholar] [CrossRef]
- Rizve, M.N.; Duarte, K.; Rawat, Y.S.; Shah, M. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv 2021, arXiv:2101.06329. [Google Scholar]
- Iscen, A.; Tolias, G.; Avrithis, Y.; Chum, O. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 5070–5079. [Google Scholar] [CrossRef]
- Cascante-Bonilla, P.; Tan, F.; Qi, Y.; Ordonez, V. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 2–9 February 2021; Volume 35, pp. 6912–6920. [Google Scholar] [CrossRef]
- Miyato, T.; Maeda, S.; Koyama, M.; Ishii, S. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 41, 1979–1993. [Google Scholar] [CrossRef] [PubMed]
- Zhang, H.; Cisse, M.; Dauphin, Y.N.; Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv 2017, arXiv:1710.09412. [Google Scholar]
- Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (PMLR), Virtual, 13–18 July 2020; pp. 1597–1607. Available online: http://proceedings.mlr.press/v119/chen20j/chen20j.pdf (accessed on 10 January 2024).
- Grill, J.B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M.; et al. Bootstrap your own latent-a new approach to self-supervised learning. In Proceedings of the Advances in neural Information Processing Systems, Virtual, 6–12 December 2020; Volume 33, pp. 21271–21284. Available online: https://proceedings.neurips.cc/paper_files/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf (accessed on 10 January 2024).
- Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–12 December 2020; Volume 33, pp. 9912–9924. Available online: https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf (accessed on 10 January 2024).
- Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–10 December 2013; Volume 26. Available online: https://proceedings.neurips.cc/paper/2013/file/af21d0c97db2e27e13572cbf59eb343d-Paper.pdf (accessed on 10 January 2024).
- Wang, Y.; Chen, H.; Fan, Y.; Sun, W.; Tao, R.; Hou, W.; Wang, R.; Yang, L.; Zhou, Z.; Guo, L.Z.; et al. Usb: A unified semi-supervised learning benchmark for classification. In Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA, 28 November–9 December 2022; Volume 35, pp. 3938–3961. Available online: https://proceedings.neurips.cc/paper_files/paper/2022/file/190dd6a5735822f05646dc27decff19b-Paper-Datasets_and_Benchmarks.pdf (accessed on 10 January 2024).
- Jiang, L.; Zhou, Z.; Leung, T.; Li, J.; Li, F.-F. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Proceedings of the International Conference on Machine Learning (PMLR), Stockholm, Sweden, 10–15 July 2018; pp. 2304–2313. Available online: https://proceedings.mlr.press/v80/jiang18c.html (accessed on 10 January 2024).
- Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar] [CrossRef]
- Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto. 2009. Available online: http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf (accessed on 10 January 2024).
- Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; Ng, A.Y. Reading Digits in Natural Images with Unsupervised Feature Learning. 2011. Available online: http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf (accessed on 10 January 2024).
- Zagoruyko, S.; Komodakis, N. Wide residual networks. arXiv 2016, arXiv:1605.07146. [Google Scholar]
- Laine, S.; Aila, T. Temporal ensembling for semi-supervised learning. arXiv 2016, arXiv:1610.02242. [Google Scholar]
Method | FixMatch [16] | CoMatch [22] | SimMatch [23] | Ours |
Parameters(M) | 1.4676 | 1.4924 | 3.0013 | 1.4773 |
Dataset | Image Size | Training Set | Test Set | Categories |
---|---|---|---|---|
Cifar10 | 32 × 32 | 50,000 | 10,000 | 10 |
Cifar100 | 32 × 32 | 50,000 | 10,000 | 100 |
SVHN | 32 × 32 | 73,257 | 26,032 | 10 |
Method | Cifar10 | Cifar100 | ||||
---|---|---|---|---|---|---|
20 Labels | 30 Labels | 40 Labels | 200 Labels | 400 Labels | 800 Labels | |
-Model [42] | - | - | - | 8.53 ± 0.25 | 11.67 ± 0.37 | 17.64 ± 1.0 |
MeanTeacher [18] | 21.79 ± 0.57 | 24.51 ± 0.35 | 24.93 ± 0.62 | 7.11 ± 0.06 | 11.54 ± 0.28 | 17.82 ± 0.09 |
MixMatch [19] | 38.51 ± 8.48 | 50.10 ± 5.81 | 59.08 ± 3.04 | 4.55 ± 0.45 | 17.68 ± 0.07 | 26.75 ± 1.13 |
FixMatch [16] | 72.63 ± 5.37 | 86.65 ± 3.56 | 89.69 ± 4.58 | 9.31 ± 0.08 | 24.44 ± 0.35 | 28.12 ± 0.30 |
CoMatch [22] | 83.43 ± 9.20 | 88.68 ± 3.79 | 90.14 ± 2.86 | 22.39 ± 1.35 | 29.60 ± 0.88 | 37.00 ± 0.59 |
SimMatch [23] | 78.13 ± 6.12 | 90.01 ± 4.15 | 91.05 ± 3.11 | 25.43 ± 1.98 | 38.66 ± 1.61 | 52.41 ± 0.76 |
ICL-SSL [24] | 88.73 ± 5.69 | 90.30 ± 3.10 | 91.78 ± 2.23 | 14.06 ± 0.52 | 26.52 ± 1.20 | 33.81 ± 0.63 |
-Mix [25] | 91.95 ± 5.95 | 92.64 ± 1.69 | 93.65 ± 0.10 | 25.25 ± 3.28 | 38.01 ± 2.96 | 48.24 ± 3.91 |
Ours | 92.67 ± 2.59 | 93.21 ± 2.45 | 94.71 ± 0.26 | 29.43 ± 2.02 | 43.90 ± 1.49 | 53.52 ± 0.54 |
Method | SVHN | ||
---|---|---|---|
250 Labels | 500 Labels | 1000 Labels | |
-Model [42] | 42.66 ± 0.91 | 53.33 ± 1.39 | 65.90 ± 0.03 |
Mean-Teacher [18] | 42.70 ± 1.79 | 55.71 ± 0.53 | 67.71 ± 1.22 |
MixMatch [19] | 92.12 ± 0.06 | 94.53 ± 0.43 | 95.13 ± 0.04 |
FixMatch [16] | 95.45 ± 0.07 | 95.73 ± 0.15 | 95.94 ± 0.10 |
ICL-SSL [24] | 95.58 ± 0.14 | 95.80 ± 0.12 | 96.05 ± 0.14 |
-Mix [25] | 97.41 ± 0.02 | 97.39 ± 0.01 | 97.45 ± 0.01 |
Ours | 97.42 ± 0.06 | 97.62 ± 0.06 | 97.71 ± 0.08 |
Method | Test Accuracy (%) |
---|---|
Single hard labels | 50.96 |
Single soft labels | 51.98 |
Dual training with hard labels | 53.79 |
Dual training with soft labels | 52.37 |
Dual training strategy | 54.34 |
Method | Test Accuracy (%) |
---|---|
Ours w/o DA | 51.79 |
Ours with DA | 54.34 |
Method | Test Accuracy (%) |
---|---|
Ours w/o SW | 52.30 |
Ours with SW | 54.34 |
Method | Test Accuracy (%) |
---|---|
Ours w/o CDL | 54.04 |
Ours with CDL | 54.34 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wu, H.; Sun, J.; Chen, Q. Dual-Training-Based Semi-Supervised Learning with Few Labels. Appl. Sci. 2024, 14, 4993. https://doi.org/10.3390/app14124993
Wu H, Sun J, Chen Q. Dual-Training-Based Semi-Supervised Learning with Few Labels. Applied Sciences. 2024; 14(12):4993. https://doi.org/10.3390/app14124993
Chicago/Turabian StyleWu, Hao, Jun Sun, and Qidong Chen. 2024. "Dual-Training-Based Semi-Supervised Learning with Few Labels" Applied Sciences 14, no. 12: 4993. https://doi.org/10.3390/app14124993
APA StyleWu, H., Sun, J., & Chen, Q. (2024). Dual-Training-Based Semi-Supervised Learning with Few Labels. Applied Sciences, 14(12), 4993. https://doi.org/10.3390/app14124993