# Semi-Supervised Semantic Segmentation-Based Remote Sensing Identification Method for Winter Wheat Planting Area Extraction

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Study Area

#### 2.2. Data Preprocessing and Label Generation

#### 2.3. Data Augmentation

#### 2.4. Experimental Environment and Parameter Setting

#### 2.5. Methodology

#### 2.5.1. Entropy Minimization

_{θ}(x)

_{k}is the model’s confidence in predicting whether x belongs to class k. The sum of confidence for all classes is 1, meaning that when the prediction value for one class is close to 1, the prediction values for other classes are close to 0, resulting in minimized entropy.

#### 2.5.2. Consistency Regularization

#### 2.5.3. Self-Training Algorithm

- (1)
- Supervised learning: Train a teacher model T using the cross-entropy loss on a labeled dataset of ${D}^{l}$.
- (2)
- Pseudo-labeling: Use the trained teacher model T to predict one-hot pseudo-labels on an unlabeled dataset of ${D}^{u}$, resulting in ${\widehat{D}}^{u}={\{({u}_{i},T({u}_{i}))\}}_{i=1}^{N}$.
- (3)
- Retraining: Combine the labeled and pseudo-labeled data of ${D}^{l}\cup {\widehat{D}}^{u}$ and retrain a student model S for final testing.

^{w}applies random, weak data augmentation to the original image. H minimizes the entropy between the student and teacher.

^{s}denotes the application of strong data augmentation to the unlabeled data.

Algorithm 1 Self-training pseudocode |

Input: Labeled training set ${D}^{l}={\{({x}_{i},{y}_{i})\}}_{i=1}^{M}$Unlabeled training set ${D}^{u}={\left\{{u}_{i}\right\}}_{i=1}^{N}$ Weak/Strong data augmentations A ^{w}/A^{s}Teacher/Student model T/S Output: Student model STrain T on ${D}^{l}$ with cross-entropy loss ${L}_{ce}$ Obtain pseudo labeled ${\widehat{D}}^{u}={\{({u}_{i},T({u}_{i}))\}}_{i=1}^{N}$ Over-sample ${D}^{l}$ to around the sized of ${\widehat{D}}^{u}$ for minbatch ${\{({x}_{k},{y}_{k})\}}_{k=1}^{B}\subset ({D}^{l}\cup {\widehat{D}}^{u})$ dofor $k\in \{1,\cdot \cdot \cdot ,B\}$ doif $k\in \{1,\cdot \cdot \cdot ,B\}$ then${x}_{k},{y}_{k}\leftarrow {A}^{s}({A}^{w}({x}_{k},{y}_{k}))$ else${x}_{k},{y}_{k}\leftarrow {A}^{w}({x}_{k},{y}_{k})$ ${\widehat{y}}_{k}=S({x}_{k})$ Update S to minimize ${L}_{ce}$ of ${\{({\widehat{y}}_{k},{y}_{k})\}}_{k=1}^{B}$ return S |

#### 2.6. Evaluation Metrics and Comparative Methods

_{ij}denote the number of pixels belonging to category i that are incorrectly classified as category j, p

_{ii}denote the number of pixels correctly classified as category i, and p

_{jj}denote the number of pixels correctly classified as category j.

## 3. Results

#### 3.1. Comparison of Segmentation Accuracies

#### 3.2. Visualization Effects Using Different Models

#### 3.3. Mapping Wheat Planting Areas Using the Semi-Supervised U-Net

## 4. Discussion

#### 4.1. Analysis of Training Ratios of Labeled Data for Different Models

#### 4.2. Influence of Spatial Resolution on Remote Imagery

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Mo, Y.; Wu, Y.; Yang, X.; Liu, F.; Liao, Y. Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing
**2022**, 493, 626–646. [Google Scholar] [CrossRef] - Yuan, X.; Shi, J.; Gu, L. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Syst. Appl.
**2021**, 169, 114417. [Google Scholar] [CrossRef] - Yang, E.; Zhou, W.; Qian, X.; Lei, J.; Yu, L. DRNet: Dual-stage refinement network with boundary inference for RGB-D semantic segmentation of indoor scenes. Eng. Appl. Artif. Intell.
**2023**, 125, 106729. [Google Scholar] [CrossRef] - Casado-García, A.; Heras, J.; Milella, A.; Marani, R. Semi-supervised deep learning and low-cost cameras for the semantic segmentation of natural images in viticulture. Precis. Agric.
**2022**, 23, 2001–2026. [Google Scholar] [CrossRef] - Zheng, Y.; Yang, M.; Wang, M.; Qian, X.; Yang, R.; Zhang, X.; Dong, W. Semi-Supervised Adversarial Semantic Segmentation Network Using Transformer and Multiscale Convolution for High-Resolution Remote Sensing Imagery. Remote Sens.
**2022**, 14, 1786. [Google Scholar] [CrossRef] - Mukhtar, H.; Khan, M.U.G.; Saba, T.; Latif, R. Wheat Plant Counting Using UAV Images Based on Semi-supervised Semantic Segmentation. In Proceedings of the 2021 1st International Conference on Artificial Intelligence and Data Analytics, Riyadh, Saudi Arabia, 6–7 April 2021; pp. 257–261. [Google Scholar]
- Oliver, A.; Odena, A.; Raffel, C.A.; Cubuk, E.D.; Goodfellow, I. Realistic evaluation of deep semi-supervised learning algorithms. In Proceedings of the 32nd annual Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, QC, Canada, 3–8 December 2018; pp. 3235–3246. [Google Scholar]
- Xie, Q.; Luong, M.T.; Hovy, E.; Le, Q.V. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 10687–10698. [Google Scholar]
- Wei, C.; Sohn, K.; Mellina, C.; Yuille, A.; Yang, F. CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 10857–10866. [Google Scholar]
- Zhu, Y.; Zhang, Z.; Wu, C.; Zhang, Z.; He, T.; Zhang, H.; Manmatha, R.; Li, M.; Smola, A.J. Improving Semantic Segmentation via Efficient Self-Training. arXiv
**2020**, arXiv:2004.14960. [Google Scholar] [CrossRef] [PubMed] - Feng, Z.; Zhou, Q.; Cheng, G.; Tan, X.; Shi, J.; Ma, L. Semi-supervised semantic segmentation via dynamic self-training and class-balanced curriculum. arXiv
**2020**, arXiv:2004.08514. [Google Scholar] - Zoph, B.; Ghiasi, G.; Lin, T.Y.; Cui, Y.; Liu, H.; Cubuk, E.D.; Le, Q. Rethinking pre-training and self-training. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada, 6–12 December 2020; pp. 3833–3845. [Google Scholar]
- Luo, Y.; Zhu, J.; Li, M.; Ren, Y.; Zhang, B. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–21 June 2018; pp. 8896–8905. [Google Scholar]
- Jeong, J.; Shin, J. Consistency regularization for certified robustness of smoothed classifiers. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada, 6–12 December 2020; pp. 10558–10570. [Google Scholar]
- French, G.; Laine, S.; Aila, T.; Mackiewicz, M.; Finlayson, G. Semi-supervised semantic segmentation needs strong, varied perturbations. arXiv
**2019**, arXiv:1906.01916. [Google Scholar] - Kim, J.; Jang, J.; Park, H.; Jeong, S. Structured consistency loss for semi-supervised semantic segmentation. arXiv
**2020**, arXiv:2001.04647. [Google Scholar] - Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, 12–18 July 2020; pp. 1597–1607. [Google Scholar]
- Yun, S.; Han, D.; Oh, S.J.; Chun, S.; Choe, J.; Yoo, Y. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6023–6032. [Google Scholar]
- Olsson, V.; Tranheden, W.; Pinto, J.; Svensson, L. Classmix: Segmentation-based data augmentation for semi-supervised learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 3–8 January 2021; pp. 1369–1378. [Google Scholar]
- Laine, S.; Aila, T. Temporal ensembling for semi-supervised learning. arXiv
**2016**, arXiv:1610.02242. [Google Scholar] - Tarvainen, A.; Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–8 December 2017; pp. 1195–1204. [Google Scholar]
- Ouali, Y.; Hudelot, C.; Tami, M. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 12674–12684. [Google Scholar]
- Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; Raffel, C.A. Mixmatch: A holistic approach to semi-supervised learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS 2019), Vancouver, BC, Canada, 8–14 December 2019; pp. 5049–5059. [Google Scholar]
- Mittal, S.; Tatarchenko, M.; Brox, T. Semi-Supervised Semantic Segmentation with High- and Low-Level Consistency. IEEE Trans. Pattern Anal. Mach. Intell.
**2019**, 43, 1369–1379. [Google Scholar] [CrossRef] [PubMed] - Lee, D.H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the Workshop on Challenges in Representation Learning, ICML (2013), Atlanta, GA, USA, 16–21 June 2013; pp. 1–6. [Google Scholar]
- Sohn, K.; Berthelot, D.; Carlini, N.; Zhang, Z.; Zhang, H.; Raffel, C.A.; Cubuk, E.D.; Kurakin, A.; Li, C.L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada, 6–12 December 2020; pp. 596–608. [Google Scholar]
- Zou, Y.; Zhang, Z.; Zhang, H.; Li, C.L.; Bian, X.; Huang, J.B.; Pfister, T. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv
**2020**, arXiv:2010.09713. [Google Scholar] - Santos, C.F.G.D.; Papa, J.P. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Comput. Surv.
**2022**, 54, 1–25. [Google Scholar] [CrossRef] - Grandvalet, Y.; Bengio, Y. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 13–18 December 2004; pp. 529–536. [Google Scholar]
- Wang, J.; Ding, C.H.Q.; Chen, S.; He, C.; Luo, B. Semi-Supervised Remote Sensing Image Semantic Segmentation via Consistency Regularization and Average Update of Pseudo-Label. Remote Sens.
**2020**, 12, 3603. [Google Scholar] [CrossRef] - Yang, L.; Zhuo, W.; Qi, L.; Shi, Y.; Gao, Y. St++: Make self-training work better for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 4268–4277. [Google Scholar]
- Alberto, G.G.; Sergio, O.E.; Sergiu, O.; Victor, V.M.; Jose, G.R. A review on deep learning techniques applied to semantic segmentation. arXiv
**2017**, arXiv:1704.06857. [Google Scholar] - Badrinarayanan, V.; Kendall, A.; Cipolla, R. SegNet: A deep convolutional encoder-decoder architecture for image segmen-tation. IEEE Trans. Pattern Anal. Mach. Intell.
**2017**, 39, 2481–2495. [Google Scholar] [CrossRef] [PubMed] - Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
- Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 801–818. [Google Scholar]
- Song, E.; Huang, D.; Ma, G.; Hung, C.-C. Semi-supervised multi-class Adaboost by exploiting unlabeled data. Expert Syst. Appl.
**2011**, 38, 6720–6726. [Google Scholar] [CrossRef] - Wang, M.; Dong, Z.; Cheng, Y.; Li, D. Optimal Segmentation of High-Resolution Remote Sensing Image by Combining Superpixels with the Minimum Spanning Tree. IEEE Trans. Geosci. Remote Sens.
**2018**, 56, 228–238. [Google Scholar] [CrossRef] - Yang, R.; Xu, Q.; Long, H. Spatial distribution characteristics and optimized reconstruction analysis of China’s rural settlements during the process of rapid urbanization. J. Rural Stud.
**2016**, 47, 413–424. [Google Scholar] [CrossRef]

**Figure 2.**Comparison of different data augmentation methods: (

**a**) Original image; (

**b**) Grayscale; (

**c**) Colorjitter; (

**d**) Blur; (

**e**) RandomInvert; and (

**f**) Cutout.

**Figure 4.**Comparison of extraction effects for the fully supervised SegNet under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 5.**Comparison of extraction effects for the semi-supervised SegNet under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 6.**Comparison of extraction effects for the fully supervised DeepLabv3+ under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 7.**Comparison of extraction effects for the semi-supervised DeepLabv3+ under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 8.**Comparison of extraction effects for the fully supervised U-Net under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 9.**Comparison of extraction effects for the semi-supervised U-Net under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16; (

**d**) 1/8; (

**e**) 1/4; and (

**f**) 1/2.

**Figure 10.**Comparison of extraction effects for the semi-supervised U-Net under different ratios of labeled data: (

**a**) Original image; (

**b**) labeled data; (

**c**) 1/16 fully supervised; (

**d**) 1/8 fully supervised; (

**e**) 1/4 fully supervised; (

**f**) 1/2 fully supervised; (

**g**) 1/16 semi-supervised; (

**h**) 1/8 semi-supervised; (

**i**) 1/4 semi-supervised; and (

**j**) 1/2 semi-supervised.

Method | Indicator | 1/16(312) | 1/8(625) | 1/4(1250) | 1/2(2500) |
---|---|---|---|---|---|

Fully supervised SegNet with only the labeled data | MPA/% | 61.82 | 68.87 | 76.95 | 78.17 |

MIoU/% | 50.26 | 62.52 | 68.19 | 70.18 | |

Semi-supervised SegNet | MPA/% | 66.01 | 73.77 | 79.66 | 80.51 |

MIoU/% | 54.81 | 66.46 | 72.28 | 73.48 | |

Fully supervised DeepLabv3+ with only the labeled data | MPA/% | 78.29 | 79.33 | 81.14 | 82.27 |

MIoU/% | 71.82 | 72.48 | 73.39 | 74.49 | |

Semi-supervised DeepLabv3+ | MPA/% | 79.25 | 80.60 | 82.13 | 82.50 |

MIoU/% | 73.45 | 74.29 | 75.30 | 75.74 | |

Fully supervised U-Net with only the labeled data | MPA/% | 81.63 | 82.88 | 83.99 | 84.62 |

MIoU/% | 73.31 | 74.11 | 75.73 | 76.45 | |

Semi-supervised U-Net | MPA/% | 82.50 | 84.60 | 84.27 | 85.52 |

MIoU/% | 76.01 | 76.84 | 77.52 | 77.83 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zhang, M.; Xue, Y.; Zhan, Y.; Zhao, J.
Semi-Supervised Semantic Segmentation-Based Remote Sensing Identification Method for Winter Wheat Planting Area Extraction. *Agronomy* **2023**, *13*, 2868.
https://doi.org/10.3390/agronomy13122868

**AMA Style**

Zhang M, Xue Y, Zhan Y, Zhao J.
Semi-Supervised Semantic Segmentation-Based Remote Sensing Identification Method for Winter Wheat Planting Area Extraction. *Agronomy*. 2023; 13(12):2868.
https://doi.org/10.3390/agronomy13122868

**Chicago/Turabian Style**

Zhang, Mingmei, Yongan Xue, Yuanyuan Zhan, and Jinling Zhao.
2023. "Semi-Supervised Semantic Segmentation-Based Remote Sensing Identification Method for Winter Wheat Planting Area Extraction" *Agronomy* 13, no. 12: 2868.
https://doi.org/10.3390/agronomy13122868