Semantic Segmentation of Cabbage in the South Korea Highlands with Images by Unmanned Aerial Vehicles
Abstract
:1. Introduction
- To measure how well our framework generalizes well despite differences for highlands and shooting dates, we operated UAVs equipped with a multispectral sensor and collected multispectral images under different spatiotemporal conditions.
- Our proposed framework shows exceptional detection performance for test images collected from the areas included in the training data but on different dates. Moreover, our method generalizes well to unseen areas not used during training.
- To analyze which wavelength in multispectral images has a positive effect on detection performance, we experimented with four different combinations of input wavelengths and compared their detection performances. Based on the results, we demonstrate that the semantic segmentation model trained with blue, green, red, and red edge wavelengths is the most suitable for automating the process of identifying cabbage cultivation fields.
2. Related Work
2.1. Semantic Segmentation Models
2.2. Land Cover Classification
2.3. Applications of Semantic Segmentation for Agriculture
3. Materials and Methods
3.1. Target Crop and Regions
3.2. Image Preprocessing
3.3. Semantic Segmentation Models
4. Experiments
4.1. Dataset
4.2. Hyperparameters
4.3. Model Performance
5. Discussion
5.1. Comparisons of Models
5.2. Difference in Input Wavelengths
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Gebrehiwot, A.; Hashemi-Beni, L.; Thompson, G.; Kordjamshidi, P.; Langan, T.E. Deep convolutional neural network for flood extent mapping using unmanned aerial vehicles data. Sensors 2019, 19, 1486. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Scott, G.J.; England, M.R.; Starms, W.A.; Marcum, R.A.; Davis, C.H. Training deep convolutional neural networks for land—Cover classification of high-resolution imagery. IEEE Geosci. Remote Sens. Lett. 2017, 14, 549–553. [Google Scholar] [CrossRef]
- Rustowicz, R.M.; Cheong, R.; Wang, L.; Ermon, S.; Burke, M.; Lobell, D. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 16–20 June 2019; pp. 75–82. [Google Scholar]
- Kwak, G.H.; Park, N.W. Impact of texture information on crop classification with machine learning and UAV images. Appl. Sci. 2019, 9, 643. [Google Scholar] [CrossRef] [Green Version]
- Vali, A.; Comai, S.; Matteucci, M. Deep learning for land use and land cover classification based on hyperspectral and multispectral earth observation data: A review. Remote Sens. 2020, 12, 2495. [Google Scholar] [CrossRef]
- Kussul, N.; Lavreniuk, M.; Skakun, S.; Shelestov, A. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geosci. Remote Sens. Lett. 2017, 14, 778–782. [Google Scholar] [CrossRef]
- Maes, W.H.; Steppe, K. Perspectives for remote sensing with unmanned aerial vehicles in precision agriculture. Trends Plant Sci. 2019, 24, 152–164. [Google Scholar] [CrossRef]
- Sarkar, T.K.; Ryu, C.S.; Kang, Y.S.; Kim, S.H.; Jeon, S.R.; Jang, S.H.; Park, J.W.; Kim, S.G.; Kim, H.J. Integrating UAV remote sensing with GIS for predicting rice grain protein. J. Biosyst. Eng. 2018, 43, 148–159. [Google Scholar]
- Zhou, X.; Zheng, H.; Xu, X.; He, J.; Ge, X.; Yao, X.; Cheng, T.; Zhu, Y.; Cao, W.; Tian, Y. Predicting grain yield in rice using multi-temporal vegetation indices from UAV-based multispectral and digital imagery. ISPRS J. Photogramm. Remote Sens. 2017, 130, 246–255. [Google Scholar] [CrossRef]
- Li, Z.; Hu, H.M.; Zhang, W.; Pu, S.; Li, B. Spectrum characteristics preserved visible and near-infrared image fusion algorithm. IEEE Trans. Multimed. 2020, 23, 306–319. [Google Scholar] [CrossRef]
- Pôças, I.; Calera, A.; Campos, I.; Cunha, M. Remote sensing for estimating and mapping single and basal crop coefficientes: A review on spectral vegetation indices approaches. Agric. Water Manag. 2020, 233, 106081. [Google Scholar] [CrossRef]
- Rawat, W.; Wang, Z. Deep convolutional neural networks for image classification: A comprehensive review. Neural Comput. 2017, 29, 2352–2449. [Google Scholar] [CrossRef]
- Zhao, Z.Q.; Zheng, P.; Xu, S.T.; Wu, X. Object detection with deep learning: A review. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3212–3232. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Guo, Y.; Liu, Y.; Georgiou, T.; Lew, M.S. A review of semantic segmentation using deep neural networks. Int. J. Multimed. Inf. Retr. 2018, 7, 87–93. [Google Scholar] [CrossRef] [Green Version]
- Li, Y.; Peng, B.; He, L.; Fan, K.; Tong, L. Road segmentation of unmanned aerial vehicle remote sensing images using adversarial network with multiscale context aggregation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2019, 12, 2279–2287. [Google Scholar] [CrossRef]
- Jiang, L.; Xie, Y.; Ren, T. A deep neural networks approach for pixel-level runway pavement crack segmentation using drone-captured images. arXiv 2020, arXiv:2001.03257. [Google Scholar]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- Everingham, M.; Eslami, S.A.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vis. 2015, 111, 98–136. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
- Badrinarayanan, V.; Kendall, A.; Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 2481–2495. [Google Scholar] [CrossRef] [PubMed]
- Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2881–2890. [Google Scholar]
- Yu, F.; Koltun, V. Multi-scale context aggregation by dilated convolutions. arXiv 2015, arXiv:1511.07122. [Google Scholar]
- Rembold, F.; Atzberger, C.; Savin, I.; Rojas, O. Using low resolution satellite imagery for yield prediction and yield anomaly detection. Remote Sens. 2013, 5, 1704–1733. [Google Scholar] [CrossRef] [Green Version]
- Alam, M.; Wang, J.F.; Guangpei, C.; Yunrong, L.; Chen, Y. Convolutional Neural Network for the Semantic Segmentation of Remote Sensing Images. Mob. Netw. Appl. 2021, 26, 200–215. [Google Scholar] [CrossRef]
- Kerkech, M.; Hafiane, A.; Canals, R. Vine disease detection in UAV multispectral images using optimized image registration and deep learning segmentation approach. Comput. Electron. Agric. 2020, 174, 105446. [Google Scholar] [CrossRef]
- Wang, T.; Thomasson, J.A.; Yang, C.; Isakeit, T.; Nichols, R.L. Automatic classification of cotton root rot disease based on UAV remote sensing. Remote Sens. 2020, 12, 1310. [Google Scholar] [CrossRef] [Green Version]
- Yang, M.D.; Tseng, H.H.; Hsu, Y.C.; Tsai, H.P. Semantic segmentation using deep learning with vegetation indices for rice lodging identification in multi-date UAV visible images. Remote Sens. 2020, 12, 633. [Google Scholar] [CrossRef] [Green Version]
- Sa, I.; Popović, M.; Khanna, R.; Chen, Z.; Lottes, P.; Liebisch, F.; Nieto, J.; Stachniss, C.; Walter, A.; Siegwart, R. WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming. Remote Sens. 2018, 10, 1423. [Google Scholar] [CrossRef] [Green Version]
- Seferbekov, S.; Iglovikov, V.; Buslaev, A.; Shvets, A. Feature pyramid network for multi-class land segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018; pp. 272–275. [Google Scholar]
- Revanasiddappa, B.; Arvind, C.; Swamy, S. Real-time early detection of weed plants in pulse crop field using drone with IoT. Technology 2020, 16, 1227–1242. [Google Scholar]
- Akiva, P.; Dana, K.; Oudemans, P.; Mars, M. Finding berries: Segmentation and counting of cranberries using point supervision and shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA, 14–19 June 2020; pp. 50–51. [Google Scholar]
- Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Seattle, WA, USA, 14–19 June 2018; pp. 801–818. [Google Scholar]
- National Geographic Information Institute. Available online: http://map.ngii.go.kr/ (accessed on 15 March 2020).
- Evangelidis, G.D.; Psarakis, E.Z. Parametric image alignment using enhanced correlation coefficient maximization. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 1858–1865. [Google Scholar] [CrossRef] [Green Version]
- MicaSense Imageprocessing. Available online: https://github.com/micasense/imageprocessing (accessed on 10 August 2019).
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Schroff, F.; Adam, H. Rethinking atrous convolution for semantic image segmentation. arXiv 2017, arXiv:1706.05587. [Google Scholar]
- Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1251–1258. [Google Scholar]
- Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv 2017, arXiv:1704.04861. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, Sardinia, Italy, 13–15 May 2010; pp. 249–256. [Google Scholar]
- Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv 2017, arXiv:1711.05101. [Google Scholar]
- Qin, J.; Wang, B.; Wu, Y.; Lu, Q.; Zhu, H. Identifying Pine Wood Nematode Disease Using UAV Images and Deep Learning Algorithms. Remote Sens. 2021, 13, 162. [Google Scholar] [CrossRef]
- You, J.; Liu, W.; Lee, J. A DNN-based semantic segmentation for detecting weed and crop. Comput. Electron. Agric. 2020, 178, 105750. [Google Scholar] [CrossRef]
- Kemker, R.; Salvaggio, C.; Kanan, C. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning. ISPRS J. Photogramm. Remote Sens. 2018, 145, 60–77. [Google Scholar] [CrossRef] [Green Version]
- Jiang, J.; Liu, F.; Xu, Y.; Huang, H. Multi-spectral RGB-NIR image classification using double-channel CNN. IEEE Access 2019, 7, 20607–20613. [Google Scholar] [CrossRef]
Model | Input Wavelengths | The Number of Parameters | Validation MIoU | MBS Dataset MIoU |
---|---|---|---|---|
DeepLab V3+ | RGB | 54,700,434 | 0.9021 (0.0028) | 0.8997 (0.0183) |
U-Net | RGB | 40,446,786 | 0.8979 (0.0023) | 0.8455 (0.0187) |
SegNet | RGB | 29,444,166 | 0.8506 (0.0198) | 0.4851 (0.1298) |
DeepLab V3+ | RGB + RE | 54,700,722 | 0.8974 (0.0025) | 0.8483 (0.0466) |
U-Net | RGB + RE | 40,447,362 | 0.9108 (0.0028) | 0.8042 (0.0215) |
SegNet | RGB + RE | 29,444,742 | 0.8676 (0.0051) | 0.6604 (0.0482) |
DeepLab V3+ | RGB + NIR | 54,700,722 | 0.8768 (0.0097) | 0.8741 (0.0396) |
U-Net | RGB + NIR | 40,447,362 | 0.9085 (0.0029) | 0.7613 (0.0742) |
SegNet | RGB + NIR | 29,444,742 | 0.7833 (0.0155) | 0.1856 (0.0537) |
DeepLab V3+ | RGB+RE+NIR | 54,701,010 | 0.8923 (0.0045) | 0.5700 (0.1291) |
U-Net | RGB+RE+NIR | 40,447,938 | 0.8878 (0.0120) | 0.5312 (0.0864) |
SegNet | RGB+RE+NIR | 29,445,318 | 0.8136 (0.0301) | 0.4829 (0.1887) |
Input Wavelengths | Model | GNM Dataset MIoU | ABD Dataset MIoU |
---|---|---|---|
DeepLab V3+ | RGB | 0.9072 (0.0045) | 0.7294 (0.0764) |
U-Net | RGB | 0.8999 (0.0076) | 0.4734 (0.0689) |
SegNet | RGB | 0.8191 (0.0210) | 0.4873 (0.2066) |
textbfDeepLab V3+ | RGB + RE | 0.9097 (0.0030) | 0.8223 (0.0483) |
U-Net | RGB +RE | 0.8983 (0.0030) | 0.7459 (0.0605) |
SegNet | RGB + RE | 0.7921 (0.0288) | 0.6435 (0.0611) |
DeepLab V3+ | RGB + NIR | 0.8605 (0.0221) | 0.7812 (0.0340) |
U-Net | RGB +NIR | 0.8912 (0.0174) | 0.5933 (0.0976) |
SegNet | RGB + NIR | 0.7054 (0.0214) | 0.1440 (0.0873) |
DeepLab V3+ | RGB + RE + NIR | 0.8525 (0.0246) | 0.3084 (0.1622) |
U-Net | RGB + RE + NIR | 0.8665 (0.0326) | 0.3121 (0.1901) |
SegNet | RGB + RE + NIR | 0.7048 (0.0443) | 0.5444 (0.2246) |
Dataset | MBS | GNM | ABD | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
B | G | R | RE | NIR | B | G | R | RE | NIR | B | G | R | RE | NIR | |
T-statistic | 989.70 | 602.73 | 858.70 | 3540.27 | 6767.05 | −280.60 | −298.04 | 419.88 | 155.37 | 2394.65 | 1309.56 | 2190.63 | 1251.48 | 4305.60 | 7144.72 |
p-value | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jo, Y.; Lee, S.; Lee, Y.; Kahng, H.; Park, S.; Bae, S.; Kim, M.; Han, S.; Kim, S. Semantic Segmentation of Cabbage in the South Korea Highlands with Images by Unmanned Aerial Vehicles. Appl. Sci. 2021, 11, 4493. https://doi.org/10.3390/app11104493
Jo Y, Lee S, Lee Y, Kahng H, Park S, Bae S, Kim M, Han S, Kim S. Semantic Segmentation of Cabbage in the South Korea Highlands with Images by Unmanned Aerial Vehicles. Applied Sciences. 2021; 11(10):4493. https://doi.org/10.3390/app11104493
Chicago/Turabian StyleJo, Yongwon, Soobin Lee, Youngjae Lee, Hyungu Kahng, Seonghun Park, Seounghun Bae, Minkwan Kim, Sungwon Han, and Seoungbum Kim. 2021. "Semantic Segmentation of Cabbage in the South Korea Highlands with Images by Unmanned Aerial Vehicles" Applied Sciences 11, no. 10: 4493. https://doi.org/10.3390/app11104493
APA StyleJo, Y., Lee, S., Lee, Y., Kahng, H., Park, S., Bae, S., Kim, M., Han, S., & Kim, S. (2021). Semantic Segmentation of Cabbage in the South Korea Highlands with Images by Unmanned Aerial Vehicles. Applied Sciences, 11(10), 4493. https://doi.org/10.3390/app11104493