E-MPSPNet: Ice–Water SAR Scene Segmentation Based on Multi-Scale Semantic Features and Edge Supervision
Abstract
:1. Introduction
- We propose an ice–water scene segmentation network, E-MPSPNet. It fuses the multi-scale features with scale-wise attention to produce an ice–water segmentation feature map and combines the segmentation feature map with an edge feature map to achieve better segmentation accuracy. The proposed E-MPSPNet performs well with a relatively higher efficiency compared to mainstream segmentation networks, U-Net, PSPNet, DeepLabV3, and HED-UNet.
- To eliminate the uncertainty of ice–water segmentation edges, we design an edge supervision module based on the idea of deep supervision. It plays a two-fold role: directly predicting the ice–water edge feature map and providing additional edge constraints to feature extraction. This module helps capture the edge characteristics of ice and water more effectively.
- We design a joint loss function that combines the edge loss and the semantic loss for the network optimization and take into account the problem of class imbalance between edge pixels and non-edge pixels.
2. Study Area and Data
2.1. Data Source
2.2. Dataset Processing
- Identify polygons for ice–water segments. The AI4Arctic/ASIP Sea Ice Dataset contains a DMI ice chart for the area corresponding to each SAR image. Each polygon in the ice chart is recorded in a table with its unique ID and the code of ice concentration in SIGRID3. To generate ice and water polygons for this study, we simplify the ice concentration SIGRID3 codes into two categories, as shown in Table 1. Label “0” defines pixels with ice concentrations less than 1/10 as sea water (according to the WMO’s definition), and label “1” defines pixels with ice concentrations in the range 1-10/10 as sea ice.
- Identify land masks. According to the distance information between pixels and the land zones provided in the netCDF files, the pixels containing land are used as masks. The parts of the SAR image outside the ice chart area are also considered as masks. The pixels being masked are not used for the training of the model.
- Generate ground truth labels. After completing the above two steps, the ground truth maps for ice–water segmentation can be generated. Then, a Sobel operator is run on the ice–water segmentation maps to produce ice–water boundaries. The produced edge ground truth map has the value “zero” for the ice–water boundaries, and it will be used for edge supervision in this paper.
Definition, Concentration | Sigrid3 Code (CT, CA, CB, and CC) | Category | Label |
---|---|---|---|
Ice Free | 00 | Sea corresponds to the concentration of codes < 1/10 | 0 |
Less than 1/10 | 01 | ||
Bergy water | 02 * | ||
1/10 | 10 | Ice corresponds to the concentration of codes 1–10/10 | 1 |
2/10 | 20 | ||
3/10 | 30 | ||
4/10 | 40 | ||
5/10 | 50 | ||
6/10 | 60 | ||
7/10 | 70 | ||
8/10 | 80 | ||
9/10 | 90 | ||
9+/10 (95%) | 91 ** | ||
10/10 | 92 |
3. Methodology
3.1. Overview of the Network Structure
3.2. Backbone Network
3.3. Edge Supervision Module
3.4. Multi-Scale Feature Fusion Module
3.5. The Joint Loss Function
4. Experiment and Analysis
4.1. Experimental Environment and Settings
4.2. Evaluation Indicators
4.3. Edge Supervision Module
4.3.1. Comparison with Different Segmentation Models
4.3.2. Influence of MFFM and EEM on Network Segmentation Performance
4.3.3. Influence of Different Loss Functions on Network Segmentation Performance
5. Discussion
5.1. The Application of Ice–Water Segmentation in a SAR Scene
5.2. The Impact of the Incident Angle
5.3. The Impact of Characteristics of Ice–Water Boundaries
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Carter, N.; Dawson, J.; Joyce, J.; Ogilvie, A. Arctic Corridors and Northern Voices: Governing Marine Transportation in the Canadian Arctic (Arviat, Nunavut Community Report); Arctic Corridors: Ottawa, ON, Canada, 2017. [Google Scholar]
- Kang, M.-S.; Kim, K.-T. Automatic SAR Image Registration via Tsallis Entropy and Iterative Search Process. IEEE Sens. J. 2020, 20, 7711–7720. [Google Scholar] [CrossRef]
- Gong, M.; Cao, Y.; Wu, Q. A Neighborhood-Based Ratio Approach for Change Detection in SAR Images. IEEE Geosci. Remote Sens. Lett. 2012, 9, 307–311. [Google Scholar] [CrossRef]
- Hakim, W.L.; Achmad, A.R.; Eom, J.; Lee, C.-W. Land Subsidence Measurement of Jakarta Coastal Area Using Time Series Interferometry with Sentinel-1 SAR Data. J. Coast. Res. 2020, 102, 75–81. [Google Scholar] [CrossRef]
- Partington, K.C.; Flach, J.D.; Barber, D.; Isleifson, D.; Meadows, P.J.; Verlaan, P. Dual-Polarization C-Band Radar Observations of Sea Ice in the Amundsen Gulf. IEEE Trans. Geosci. Remote Sens. 2010, 48, 2685–2691. [Google Scholar] [CrossRef]
- Makynen, M.; Manninen, A.; Simila, M.; Karvonen, J.; Hallikainen, M. Incidence angle dependence of the statistical properties of C-band HH-polarization backscattering signatures of the Baltic Sea ice. IEEE Trans. Geosci. Remote Sens. 2002, 40, 2593–2605. [Google Scholar] [CrossRef]
- Nghiem, S.; Bertoïa, C. Study of Multi-Polarization C-Band Backscatter Signatures for Arctic Sea Ice Mapping with Future Satellite SAR. Can. J. Remote Sens. 2014, 27, 387–402. [Google Scholar] [CrossRef]
- Holmes, Q.A.; Nuesch, D.R.; Shuchman, R.A. Textural Analysis and Real-Time Classification of Sea-Ice Types Using Digital SAR Data. IEEE Trans. Geosci. Remote Sens. 2007, 2, 113–120. [Google Scholar] [CrossRef]
- Soh, L.-K.; Tsatsoulis, C. Texture analysis of SAR sea ice imagery using gray level co-occurrence matrices. IEEE Trans. Geosci. Remote Sens. 1999, 37, 780–795. [Google Scholar] [CrossRef] [Green Version]
- Clausi, D.A. An analysis of co-occurrence texture statistics as a function of grey level quantization. Can. J. Remote Sens. 2002, 28, 45–62. [Google Scholar] [CrossRef]
- Clausi, D.A.; Yue, B. Comparing Cooccurrence Probabilities and Markov Random Fields for Texture Analysis of SAR Sea Ice Imagery. IEEE Trans. Geosci. Remote Sens. 2004, 42, 215–228. [Google Scholar] [CrossRef]
- Ochilov, S.; Clausi, D.A. Operational SAR Sea-Ice Image Classification. IEEE Trans. Geosci. Remote Sens. 2012, 50, 4397–4408. [Google Scholar] [CrossRef]
- Korosov, A.; Zakhvatkina, N.; Muckenhuber, S. Ice/Water Classification of Sentinel-1 Images. In EGU General Assembly Conference Abstracts; EGU: Munich, Germany, 2015. [Google Scholar]
- Liu, H.; Guo, H.; Zhang, L. SVM-Based Sea Ice Classification Using Textural Features and Concentration From RADARSAT-2 Dual-Pol ScanSAR Data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015, 8, 1601–1613. [Google Scholar] [CrossRef]
- Zakhvatkina, N.; Korosov, A.; Muckenhuber, S.; Sandven, S.; Babiker, M. Operational algorithm for ice–water classification on dual-polarized RADARSAT-2 images. Cryosphere 2017, 11, 33–46. [Google Scholar] [CrossRef] [Green Version]
- Kim, M.; Im, J.; Han, H.; Kim, J.; Lee, S.; Shin, M.; Kim, H.-C. Landfast sea ice monitoring using multisensor fusion in the Antarctic. Mapp. Sci. Remote Sens. 2015, 52, 239–256. [Google Scholar] [CrossRef]
- Wiebke, A.; Céline, H.; Eriksson, L. Comparison of ice/water classification in Fram Strait from C- and L-band SAR imagery. Ann. Glaciol. 2018, 59, 112–123. [Google Scholar]
- Karvonen, J.; Simila, M.; Mäkynen, M. Open Water Detection from Baltic Sea Ice Radarsat-1 SAR Imagery. IEEE Geosci. Remote Sens. Lett. 2005, 2, 275–279. [Google Scholar] [CrossRef]
- Mohammadimanesh, F.; Salehi, B.; Mahdianpari, M.; Gill, E.; Molinier, M. A new fully convolutional neural network for semantic segmentation of polarimetric SAR imagery in complex land cover ecosystem. ISPRS J. Photogramm. Remote Sens. 2019, 151, 223–236. [Google Scholar] [CrossRef]
- Huang, L.; Liu, B.; Li, B.; Guo, W.; Yu, W.; Zhang, Z.; Yu, W. OpenSARShip: A dataset dedicated to Sentinel-1 ship interpretation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2017, 11, 195–208. [Google Scholar] [CrossRef]
- Xia, G.-S.; Bai, X.; Ding, J.; Zhu, Z.; Belongie, S.; Luo, J.; Datcu, M.; Pelillo, M.; Zhang, L. DOTA: A Large-Scale Dataset for Object Detection in Aerial Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Heidler, K.; Mou, L.; Baumhoer, C.; Dietz, A.; Zhu, X.X. HED-UNet: Combined segmentation and edge detection for monitoring the Antarctic coastline. IEEE Trans. Geosci. Remote Sens. 2021, 60, 4300514. [Google Scholar] [CrossRef]
- Wang, Z.; Bai, L.; Song, G.; Zhang, J.; Tao, J.; Mulvenna, M.; Bond, R.; Chen, L. An Oil Well Dataset Derived from Satellite-Based Remote Sensing. Remote Sens. 2021, 13, 1132. [Google Scholar] [CrossRef]
- Khaleghian, S.; Ullah, H.; Kræmer, T.; Hughes, N.; Eltoft, T.; Marinoni, A. Sea Ice Classification of SAR Imagery Based on Convolution Neural Networks. Remote Sens. 2021, 13, 1734. [Google Scholar] [CrossRef]
- Song, W.; Gao, W.; He, Q.; Liotta, A.; Guo, W. SI-STSAR-7: A Large SAR Images Dataset with Spatial and Temporal Information for Classification of Winter Sea Ice in Hudson Bay. Remote Sens. 2022, 14, 168. [Google Scholar] [CrossRef]
- Wulf, T.; Kreiner, M.B.; Buus-Hinkler, J.; Tonboe, R.T.; Høyer, J.L.; Saldo, R.; Pedersen, L.T.; Nielsen, A.A.; Skriver, H.; Malmgren-Hansen, D. Fusion of Satellite SAR and Passive Microwave Radiometer Data for Automated Sea Ice Mapping and the Expected Impact of CIMR Observations. In Proceedings of the From Science to Operations for the Copernicus Imaging Microwave Radiometer (CIMR) Mission, Noordwijk, The Netherlands, 15–17 April 2020; ESA: Paris, France, 2021. [Google Scholar]
- Wang, L.; Scott, K.A.; Clausi, D.A. Sea ice concentration estimation during freeze-up from SAR imagery using a convolutional neural network. Remote Sens. 2017, 9, 408. [Google Scholar] [CrossRef]
- Malmgren-Hansen, D.; Pedersen, L.T.; Nielsen, A.A.; Kreiner, M.B.; Saldo, R.; Skriver, H.; Lavelle, J.; Buus-Hinkler, J.; Krane, K.H. A Convolutional Neural Network Architecture for Sentinel-1 and AMSR2 Data Fusion. IEEE Trans. Geosci. Remote Sens. 2021, 59, 1890–1902. [Google Scholar] [CrossRef]
- Dirscherl, M.; Dietz, A.; Kneisel, C.; Kuenzer, C. A Novel Method for Automated Supraglacial Lake Mapping in Antarctica Using Sentinel-1 SAR Imagery and Deep Learning. Remote Sens. 2021, 13, 197. [Google Scholar] [CrossRef]
- Kruk, R.; Fuller, M.; Komarov, A.; Isleifson, D.; Jeffrey, I. Proof of Concept for Sea Ice Stage of Development Classification Using Deep Learning. Remote Sens. 2020, 12, 2486. [Google Scholar] [CrossRef]
- Han, Y.; Liu, Y.; Hong, Z.; Zhang, Y.; Yang, S.; Wang, J. Sea Ice Image Classification Based on Heterogeneous Data Fusion and Deep Learning. Remote Sens. 2021, 13, 592. [Google Scholar] [CrossRef]
- Zhang, T.; Yang, Y.; Shokr, M.; Mi, C.; Li, X.-M.; Cheng, X.; Hui, F. Deep Learning Based Sea Ice Classification with Gaofen-3 Fully Polarimetric SAR Data. Remote Sens. 2021, 13, 1452. [Google Scholar] [CrossRef]
- Boulze, H.; Korosov, A.A.; Brajard, J. Classification of Sea Ice Types in Sentinel-1 SAR Data Using Convolutional Neural Networks. Remote Sens. 2020, 12, 2165. [Google Scholar] [CrossRef]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. Comput. Sci. 2014, 4, 357–361. [Google Scholar]
- Chen, L.-C.; Barron, J.T.; Papandreou, G.; Murphy, K.; Yuille, A.L. Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; IEEE: Pistacaway, NJ, USA, 2016; pp. 4545–4554. [Google Scholar]
- Takikawa, T.; Acuna, D.; Jampani, V.; Fidler, S. Gated-SCNN: Gated Shape CNNs for Semantic Segmentation. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; IEEE: Pistacaway, NJ, USA, 2020. [Google Scholar]
- Sun, G.Y.; Huang, B.H.; Zhao, X.J. The Analysis of Edge Detection Uncertainty of Remote Sensing Images and its Processing Method. Remote Sens. Inf. 2010, 32, 110–114. [Google Scholar]
- Wu, T.; Jin, Y.-F.; Hou, R.; Yang, J.-J. Cognitive physics-based method for image edge representation and extraction with uncertainty. Acta Phys. Sin. 2013, 62, 675. [Google Scholar]
- Park, J.-W.; Korosov, A.A.; Babiker, M.; Sandven, S.; Won, J.-S. Efficient Thermal Noise Removal for Sentinel-1 TOPSAR Cross-Polarization Channel. IEEE Trans. Geosci. Remote Sens. 2018, 56, 1555–1565. [Google Scholar] [CrossRef]
- Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid Scene Parsing Network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; IEEE: Pistacaway, NJ, USA, 2017; pp. 6230–6239. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
- Lee, C.-Y.; Xie, S.; Gallagher, P.; Zhang, Z.; Tu, Z. Deeply-Supervised Nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, San Diego, CA, USA, 9–12 May 2015. [Google Scholar]
- Lin, T.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal Loss for Dense Object Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 99, 2999–3007. [Google Scholar]
- Kingma, D.P.; Adam, J.B. A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Yang, H.; Yu, B.; Luo, J.; Chen, F. Semantic segmentation of high spatial resolution images with deep neural networks. GISci. Remote Sens. 2019, 56, 749–768. [Google Scholar] [CrossRef]
- Stokholm, A.; Wulf, T.; Kucik, A.; Saldo, R.; Buus-Hinkler, J.; Hvidegaard, S.M. AI4SeaIce: Toward Solving Ambiguous SAR Textures in Convolutional Neural Networks for Automatic Sea Ice Concentration Charting. IEEE Trans. Geosci. Remote Sens. 2022, 60, 4304013. [Google Scholar] [CrossRef]
- Wang, Z.; Wang, J.; Yang, K.; Wang, L.; Su, F.; Chen, X. Semantic segmentation of high-resolution remote sensing images based on a class feature attention mechanism fused with Deeplabv3+. Comput. Geosci. 2022, 158, 104969. [Google Scholar] [CrossRef]
- Singha, S.; Johansson, A.M.; Doulgeris, A.P. Robustness of SAR Sea Ice Type Classification Across Incidence Angles and Seasons at L-Band. IEEE Trans. Geosci. Remote Sens. 2020, 59, 9941–9952. [Google Scholar] [CrossRef]
- Singha, S.; Johansson, A.M.; Doulgeris, A.P. Incident Angle Dependence of Sentinel-1 Texture Features for Sea Ice Classification. Remote Sens. 2021, 13, 552. [Google Scholar]
Layer Type | Kernel Size | Channels | Stride | Output Size |
---|---|---|---|---|
input | 3 × 3 | 64 | 1 | 800 × 800 × 64 |
conv1 | 7 × 7 | 128 | 2 | 400 × 400 × 128 |
conv2 | Max pooling | 128 | 2 | 200 × 200 × 128 |
256 | 1 | 200 × 200 × 256 | ||
conv3 | 512 | 1 | 100 × 100 × 512 | |
conv4 | 1024 | 1 | 100 × 100 × 1024 | |
conv5 | 2048 | 1 | 100 × 100 × 2048 |
Methodology | Accuracy | F-Score | MIoU |
---|---|---|---|
UNet | 0.903 | 0.881 | 0.822 |
DeepLabV3 | 0.928 | 0.919 | 0.864 |
HED-UNet | 0.932 | 0.910 | 0.870 |
PSPNet | 0.932 | 0.921 | 0.873 |
E-MPSPNet | 0.942 | 0.930 | 0.892 |
Network Structure | Accuracy | F-Score | MIoU |
---|---|---|---|
Backbone network | 0.934 | 0.919 | 0.875 |
Backbone + MFFM | 0.936 | 0.923 | 0.882 |
Backbone + MFFM + ESM | 0.942 | 0.930 | 0.892 |
Methodology | Accuracy | F-Score | MIoU |
---|---|---|---|
0.928 | 0.915 | 0.866 | |
0.936 | 0.924 | 0.879 | |
0.942 | 0.930 | 0.892 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Song, W.; Li, H.; He, Q.; Gao, G.; Liotta, A. E-MPSPNet: Ice–Water SAR Scene Segmentation Based on Multi-Scale Semantic Features and Edge Supervision. Remote Sens. 2022, 14, 5753. https://doi.org/10.3390/rs14225753
Song W, Li H, He Q, Gao G, Liotta A. E-MPSPNet: Ice–Water SAR Scene Segmentation Based on Multi-Scale Semantic Features and Edge Supervision. Remote Sensing. 2022; 14(22):5753. https://doi.org/10.3390/rs14225753
Chicago/Turabian StyleSong, Wei, Hongtao Li, Qi He, Guoping Gao, and Antonio Liotta. 2022. "E-MPSPNet: Ice–Water SAR Scene Segmentation Based on Multi-Scale Semantic Features and Edge Supervision" Remote Sensing 14, no. 22: 5753. https://doi.org/10.3390/rs14225753