Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
Abstract
:1. Introduction
2. Datasets and Methods
2.1. Datasets
2.1.1. Data Descriptions
2.1.2. Data Partition Method
2.2. Method
2.2.1. CTAFNet Architecture Overview
2.2.2. Convolution–Transformer Adaptive Fusion Kernel
- Conv Block
- 2.
- Trans Block
- 3.
- CTAFK workflow
2.2.3. Comparison Methods
3. Experiments and Analysis
3.1. Implementation Details and Metrics
3.2. Experiment Result
3.2.1. Experiment Results on AeroRIT Dataset
3.2.2. Experiment Results on DFC2018 Dataset
3.2.3. Experiment Results on Xiongan Dataset
4. Discussion
4.1. Ablation Study
4.2. Parameter Sensitivity Analysis
4.3. Generalization of CTAFNet on Small Dataset
4.4. Limitations and Future Works
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Li, X.; Li, Z.; Qiu, H.; Hou, G.; Fan, P. An Overview of Hyperspectral Image Feature Extraction, Classification Methods and The Methods Based on Small Samples. Appl. Spectrosc. Rev. 2021, 11, 1–34. [Google Scholar] [CrossRef]
- Khan, M.J.; Khan, H.S.; Yousaf, A.; Khurshid, K.; Abbas, A. Modern Trends in Hyperspectral Image Analysis: A Review. IEEE Access 2018, 6, 14118–14129. [Google Scholar] [CrossRef]
- Adão, T.; Hruška, J.; Pádua, L.; Bessa, J.; Peres, E.; Morais, R.; Sousa, J.J. Hyperspectral Imaging: A Review on UAV-Based Sensors, Data Processing and Applications for Agriculture and Forestry. Remote Sens. 2017, 9, 1110. [Google Scholar] [CrossRef] [Green Version]
- Krupnik, D.; Khan, S. Close-Range, Ground-Based Hyperspectral Imaging for Mining Applications at Various Scales: Review and Case Studies. Earth-Sci. Rev. 2019, 198, 102952. [Google Scholar] [CrossRef]
- Liu, B.; Liu, Z.; Men, S.; Li, Y.; Ding, Z.; He, J.; Zhao, Z. Underwater Hyperspectral Imaging Technology and Its Applications for Detecting and Mapping the Seafloor: A Review. Sensors 2020, 20, 4962. [Google Scholar] [CrossRef]
- Chen, Y.; Xing, Z.; Jia, X. Spectral-Spatial Classification of Hyperspectral Data Based on Deep Belief Network. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 2015, 8, 2381–2392. [Google Scholar] [CrossRef]
- Li, S.; Song, W.; Fang, L.; Chen, Y.; Ghamisi, P.; Benediktsson, J.A. Deep Learning for Hyperspectral Image Classification: An Overview. IEEE Trans. Geosci. Remote Sens. 2019, 57, 6690–6709. [Google Scholar] [CrossRef] [Green Version]
- Prasad, S.; Bruce, L.M. Limitations of Principal Components Analysis for Hyperspectral Target Recognition. IEEE Geosci. Remote Sens. Lett. 2008, 5, 625–629. [Google Scholar] [CrossRef]
- Li, W.; Prasad, S.; Fowler, J.E.; Bruce, L.M. Locality-Preserving Dimensionality Reduction and Classification for Hyperspectral Image Analysis. IEEE Trans. Geosci. Remote Sens. 2012, 50, 1185–1198. [Google Scholar] [CrossRef] [Green Version]
- Liao, W.; Pizurica, A.; Scheunders, P.; Philips, W.; Pi, Y. Semisupervised Local Discriminant Analysis for Feature Extraction in Hyperspectral Images. IEEE Trans. Geosci. Remote Sens. 2013, 51, 184–198. [Google Scholar] [CrossRef]
- Al-khafaji, S.L.; Zhou, J.; Zia, A.; Liew, A.W. Spectral-Spatial Scale Invariant Feature Transform for Hyperspectral Images. IEEE Trans. Image Process. 2018, 27, 837–850. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, W.; Chen, C.; Su, H.; Du, Q. Local Binary Patterns and Extreme Learning Machine for Hyperspectral Imagery Classification. IEEE Trans. Geosci. Remote Sens. 2015, 53, 3681–3693. [Google Scholar] [CrossRef]
- Gu, Y.; Liu, H. Sample-Screening MKL Method via Boosting Strategy for Hyperspectral Image Classification. Neurocomputing 2016, 173, 1630–1639. [Google Scholar] [CrossRef]
- Melgani, F.; Bruzzone, L. Classification of Hyperspectral Remote Sensing Images with Support Vector Machines. IEEE Trans. Geosci. Remote Sens. 2004, 42, 1778–1790. [Google Scholar] [CrossRef] [Green Version]
- Ham, J.; Chen, Y.; Crawford, M.M.; Ghosh, J. Investigation of the Random Forest Framework for Classification of Hyperspectral Data. IEEE Trans. Geosci. Remote Sens. 2005, 43, 492–501. [Google Scholar] [CrossRef] [Green Version]
- Zhou, Y.; Peng, J.; Chen, C.L.P. Extreme Learning Machine with Composite Kernels for Hyperspectral Image Classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015, 8, 2351–2360. [Google Scholar] [CrossRef]
- Hu, W.; Huang, Y.; Wei, L.; Zhang, F.; Li, H. Deep Convolutional Neural Networks for Hyperspectral Image Classification. J. Sens. 2015, 2015, 258619. [Google Scholar] [CrossRef] [Green Version]
- Zhao, W.; Du, S. Spectral-Spatial Feature Extraction for Hyperspectral Image Classification: A Dimension Reduction and Deep Learning Approach. IEEE Trans. Geosci. Remote Sens. 2016, 54, 4544–4554. [Google Scholar] [CrossRef]
- Chen, Y.; Jiang, H.; Li, C.; Jia, X.; Ghamisi, P. Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks. IEEE Trans. Geosci. Remote Sens. 2016, 54, 6232–6251. [Google Scholar] [CrossRef] [Green Version]
- Zhang, H.; Li, Y.; Zhang, Y.; Shen, Q. Spectral-Spatial Classification of Hyperspectral Imagery Using a Dual-Channel Convolutional Neural Network. Remote Sens. Lett. 2017, 8, 438–447. [Google Scholar] [CrossRef]
- Li, J.S.; Xia, X.; Li, W. Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios. arXiv 2022, arXiv:2207.05501v2. [Google Scholar]
- Wang, H.; Wu, X.; Huang, Z.; Xing, E.P. High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 8681–8691. [Google Scholar] [CrossRef]
- Zhao, F.; Zhang, J.; Meng, Z.; Liu, H. Densely Connected Pyramidal Dilated Convolutional Network for Hyperspectral Image Classification. Remote Sens. 2021, 13, 3396. [Google Scholar] [CrossRef]
- Liu, D.; Han, G.; Liu, P.; Yang, H.; Sun, X.; Li, Q.; Wu, J. A Novel 2D-3D CNN with Spectral-Spatial Multi-Scale Feature Fusion for Hyperspectral Image Classification. Remote Sens. 2021, 13, 4621. [Google Scholar] [CrossRef]
- Fang, B.; Li, Y.; Zhang, H.; Chan, J.C.-W. Hyperspectral Images Classification Based on Dense Convolutional Networks with Spectral-Wise Attention Mechanism. Remote Sens. 2019, 11, 159. [Google Scholar] [CrossRef] [Green Version]
- Shen, Y.; Zhu, S.; Chen, C.; Du, Q.; Xiao, L.; Chen, J.; Pan, D. Efficient Deep Learning of Nonlocal Features for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2021, 59, 6029–6043. [Google Scholar] [CrossRef]
- Wang, L.; Peng, J.; Sun, W. Spatial-Spectral Squeeze-and-Excitation Residual Network for Hyperspectral Image Classification. Remote Sens. 2019, 11, 884. [Google Scholar] [CrossRef] [Green Version]
- Xue, Z.H.; Xu, Q.; Zhang, M.X. Local Transformer with Spatial Partition Restore for Hyperspectral Image Classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022, 15, 4307–4325. [Google Scholar] [CrossRef]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is All You Need. arXiv 2017, arXiv:1706.03762. [Google Scholar]
- Qing, Y.; Liu, W.; Feng, L.; Gao, W. Improved Transformer Net for Hyperspectral Image Classification. Remote Sens. 2021, 13, 2216. [Google Scholar] [CrossRef]
- Sun, J.; Zhang, J.; Gao, X.; Wang, M.; Ou, D.; Wu, X.; Zhang, D. Fusing Spatial Attention with Spectral-Channel Attention Mechanism for Hyperspectral Image Classification via Encoder-Decoder Networks. Remote Sens. 2022, 14, 1968. [Google Scholar] [CrossRef]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual, 12 October 2021; pp. 10012–10022. [Google Scholar] [CrossRef]
- D’Ascoli, S.; Touvron, H.; Leavitt, M.; Morcos, A.; Biroli, G.; Sagun, L. ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. arXiv 2021, arXiv:2103.10697. [Google Scholar] [CrossRef]
- Han, K.; Xiao, A.; Wu, E.; Guo, J.; Xu, C.; Wang, Y. Transformer in Transformer. arXiv 2021, arXiv:2103.00112. [Google Scholar]
- Yang, L.; Yang, Y.; Yang, J.; Zhao, N.; Wu, L.; Wang, L.; Wang, T. FusionNet: A Convolution-Transformer Fusion Network for Hyperspectral Image Classification. Remote Sens. 2022, 14, 4066. [Google Scholar] [CrossRef]
- Sun, L.; Zhao, G.; Zheng, Y.; Wu, Z. Spectral-Spatial Feature Tokenization Transformer for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2022, 60, 5522214. [Google Scholar] [CrossRef]
- Song, R.; Feng, Y.; Cheng, W.; Mu, Z.; Wang, X. BS2T: Bottleneck Spatial-Spectral Transformer for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2022, 60, 5532117. [Google Scholar] [CrossRef]
- Park, N.; Kim, S. How Do Vision Transformers Work? arXiv 2022, arXiv:2202.06709. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
- Nalepa, J.; Myller, M.; Kawulok, M. Validating Hyperspectral Image Segmentation. IEEE Geosci. Remote Sens. Lett. 2019, 16, 1264–1268. [Google Scholar] [CrossRef] [Green Version]
- Liang, J.; Zhou, J.; Qian, Y.; Wen, L.; Bai, X.; Gao, Y. On the Sampling Strategy for Evaluation of Spectral-Spatial Methods in Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2017, 55, 862–880. [Google Scholar] [CrossRef] [Green Version]
- Zou, L.; Zhu, X.; Wu, C.; Liu, Y.; Qu, L. Spectral-Spatial Exploration for Hyperspectral Image Classification via the Fusion of Fully Convolutional Networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 659–674. [Google Scholar] [CrossRef]
- Rangnekar, A.; Mokashi, N.; Ientilucci, E.J.; Kanan, C.; Hoffman, M.J. AeroRIT: A New Scene for Hyperspectral Image Analysis. IEEE Trans. Geosci. Remote Sens. 2020, 58, 8116–8124. [Google Scholar] [CrossRef]
- Cen, Y.; Zhang, L.; Zhang, X.; Wang, Y.; Qi, W.; Tang, S.; Zhang, P. Aerial Hyperspectral Remote Sensing Classification Dataset of Xiongan New Aera (Matiwan Village). J. Remote Sens. 2020, 24, 10–17. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; Lecture Notes in Computer Science in Medical Image Computing and Computer-Assisted Intervention; Springer: Cham, Switzerland, 2015; Volume 9351, pp. 234–241. [Google Scholar] [CrossRef] [Green Version]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar] [CrossRef] [Green Version]
- Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the International Conference on Machine Learning (ICML), Lille, France, 6–11 July 2015; pp. 448–456. [Google Scholar] [CrossRef]
- Glorot, X.; Bordes, A.; Bengio, Y.S. Deep Sparse Rectifier Neural Networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA, 11–13 April 2011; pp. 315–323. Available online: http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf (accessed on 25 October 2022).
- Ba, J.L.; Kiros, J.R.; Hinton, G.E. Layer Normalization. arXiv 2016, arXiv:1607.06450. [Google Scholar]
- Hendrycks, D.; Gimpel, K. Gaussian Error Linear Units (GELUs). arXiv 2016, arXiv:1606.08415. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 7132–7141. [Google Scholar] [CrossRef] [Green Version]
- Chen, L.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv 2018, arXiv:1802.02611v3. [Google Scholar]
- Xie, E.; Wang, W.; Yu, Z.; Anandkumar, A.; Alvarez, J.M.; Luo, P. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. arXiv 2021, arXiv:2105.15203v2. [Google Scholar]
- Cao, H.; Wang, Y.; Chen, J.; Jiang, D.; Zhang, X.; Tian, Q.; Wang, M. Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. arXiv 2021, arXiv:2105.05537. [Google Scholar]
- Zhu, Q.; Deng, W.; Zheng, Z.; Zhong, Y.; Guan, Q.; Lin, W.; Zhang, L.; Li, D. A Spectral-Spatial-Dependent Global Learning Framework for Insufficient and Imbalanced Hyperspectral Image Classification. IEEE Trans. Cybern. 2021, 52, 11709–11723. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar] [CrossRef] [Green Version]
- Loshchilov, I.; Hutter, F. Decoupled Weight Decay Regularization. arXiv 2017, arXiv:1711.05101. [Google Scholar]
- Yu, C.; Wang, J.; Gao, C.; Yu, G.; Shen, C.; Sang, N. Context Prior for Scene Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 12413–12422. [Google Scholar] [CrossRef]
- Sun, Z.; Liu, M.; Liu, P.; Li, J.; Yu, T.; Gu, X.; Yang, J.; Mi, X.; Cao, W.; Zhang, Z. SAR Image Classification Using Fully Connected Conditional Random Fields Combined with Deep Learning and Superpixel Boundary Constraint. Remote Sens. 2021, 13, 271. [Google Scholar] [CrossRef]
- Karimi, D.; Salcudean, S.E. Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networks. IEEE Trans. Med. Imaging 2020, 39, 499–513. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Sinha, S.; Ohashi, H.; Nakamura, K. Class-Wise Difficulty-Balanced Loss for Solving Class-Imbalance. In Proceedings of the Asian Conference on Computer Vision (ACCV), Kyoto, Japan, 30 November–4 December 2020; pp. 549–565. [Google Scholar] [CrossRef]
- Wambugua, N.; Chen, Y.; Xiao, Z.; Tan, K.; Wei, M.; Liu, X.; Li, J. Hyperspectral Image Classification on Insufficient-Sample and Feature Learning Using Deep Neural Networks: A Review. Int. J. Appl. Earth Obs. Geoinf. 2021, 105, 102603. [Google Scholar] [CrossRef]
- Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; Schiele, B. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 3213–3223. [Google Scholar] [CrossRef] [Green Version]
- Mottaghi, R.; Chen, X.; Liu, X.; Cho, N.; Lee, S.; Fidler, S.; Urtasun, R.; Yuille, A.L. The Role of Context for Object Detection and Semantic Segmentation in the Wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 24–27 June 2014; pp. 891–898. [Google Scholar] [CrossRef]
- Xu, Y.; Gong, J.; Huang, X.; Hu, X.; Li, J.; Li, Q.; Peng, M. Luojia-HSSR: A High Spatial-Spectral Resolution Remote Sensing Dataset for Land-Cover Classification with a New 3D-HRNet. Geo-Spat. Inf. Sci. 2022. [Google Scholar] [CrossRef]
Class | Train | Val | Test |
---|---|---|---|
Buildings | 423,605 | 141,424 | 352,788 |
Vegetation | 1,277,105 | 349,211 | 1,551,317 |
Roads | 843,770 | 319,228 | 781,508 |
Water | 112,946 | 0 | 5718 |
Cars | 70,313 | 19,537 | 42,243 |
Total | 2,727,739 | 829,400 | 2,733,574 |
Class | Train | Test |
---|---|---|
Healthy grass | 24,546 | 14,650 |
Stressed grass | 95,176 | 34,832 |
Evergreen trees | 42,446 | 11,876 |
Deciduous trees | 13,973 | 6199 |
Bare earth | 11,844 | 6220 |
Residential buildings | 119,456 | 39,539 |
Non-residential buildings | 616,125 | 278,644 |
Roads | 129,820 | 53,463 |
Sidewalks | 94,431 | 41,604 |
Crosswalks | 3875 | 2184 |
Major thoroughfares | 134,167 | 51,271 |
Highways | 29,175 | 10,263 |
Railways | 18,308 | 9440 |
Paved parking lots | 31,833 | 14,099 |
Cars | 19,875 | 6414 |
Trains | 14,524 | 6955 |
Stadium seats | 17,074 | 10,222 |
Total | 1,416,648 | 597,875 |
Class | Train | Test |
---|---|---|
Acer negundo | 140,953 | 84,694 |
Willow | 119,010 | 61,756 |
Elm | 12,523 | 2830 |
Paddy | 342,682 | 109,462 |
Sophora japonica | 305,611 | 169,980 |
Fraxinus chinensis | 123,224 | 46,118 |
Goldenrain tree | 19,126 | 4178 |
Waters | 122,790 | 42,857 |
Bare ground | 27,526 | 10,883 |
Stubble | 134,439 | 59,391 |
Corn | 43,796 | 15,369 |
Pyrus | 764,745 | 261,768 |
Soybean | 6682 | 469 |
Poplar | 62,489 | 28,583 |
Vegetable field | 20,822 | 8326 |
Grass | 315,130 | 106,660 |
Peach | 43,220 | 22,294 |
Building | 18,452 | 11,164 |
Total | 2623,220 | 1,046,782 |
Stage | Encoder (H × W × C) | Decoder (H × W × C) |
---|---|---|
1 | 64 × 64 × 96 | 16 × 16 × 96 |
2 | 32 × 32 × 96 | 32 × 32 × 96 |
3 | 16 × 16 × 96 | 64 × 64 ×1 |
4 | 8 × 8 × 96 |
Class | UNet | DeepLab v3+ | SegFormer | Swin-UNet | UNet-m | SS3FCN | ENLFCN | SSDGL | FSSANet | CTAFNet |
---|---|---|---|---|---|---|---|---|---|---|
Buildings | 85.47 | 85.53 | 63.47 | 82.89 | 82.64 | 83.86 | 85.17 | 81.66 | 84.93 | 86.63 |
Vegetation | 94.87 | 94.99 | 93.09 | 95.94 | 95.45 | 95.71 | 95.25 | 95.72 | 95.8 | 95.72 |
Roads | 81.17 | 82.09 | 66.95 | 81.07 | 80.04 | 83.93 | 80.77 | 82.01 | 81.62 | 84.64 |
Water | 72.11 | 68.78 | 30.75 | 75.52 | 76.74 | 74.65 | 76.89 | 75.23 | 75.06 | 78.42 |
Cars | 48.94 | 47.74 | 32.34 | 38.4 | 47.17 | 60.44 | 47.56 | 57.38 | 45.84 | 61.64 |
pixel acc. | 93.86 | 94.08 | 88.51 | 93.72 | 93.61 | 94.71 | 93.88 | 94.2 | 94.09 | 95.07 |
mIoU | 76.52 | 75.83 | 57.32 | 74.76 | 76.41 | 79.72 | 77.13 | 78.4 | 76.65 | 81.41 |
Class | UNet | DeepLab v3+ | SegFormer | Swin-UNet | UNet-m | SS3FCN | ENLFCN | SSDGL | FSSANet | CTAFNet |
---|---|---|---|---|---|---|---|---|---|---|
Healthy grass | 78.88 | 74.67 | 64.39 | 87.59 | 82.68 | 87.01 | 80.77 | 84.56 | 85.36 | 77.79 |
Stressed grass | 86.36 | 83.94 | 69.95 | 88.68 | 88.92 | 90.34 | 86.86 | 85.30 | 87.41 | 88.08 |
Evergreen trees | 81.53 | 84.70 | 78.05 | 81.47 | 83.75 | 87.67 | 82.83 | 85.48 | 80.27 | 89.47 |
Deciduous trees | 61.14 | 68.57 | 36.77 | 68.18 | 65.83 | 75.80 | 60.05 | 51.59 | 61.20 | 77.39 |
Bare earth | 84.67 | 93.53 | 71.37 | 82.90 | 86.68 | 85.84 | 88.95 | 70.52 | 89.12 | 93.53 |
Residential buildings | 66.30 | 83.75 | 64.26 | 70.33 | 70.05 | 81.89 | 79.40 | 78.61 | 72.18 | 91.21 |
Non-residential buildings | 86.99 | 89.68 | 69.82 | 86.89 | 86.82 | 93.09 | 88.03 | 85.80 | 89.92 | 93.69 |
Roads | 55.15 | 57.47 | 32.47 | 50.39 | 55.79 | 65.72 | 51.22 | 48.82 | 53.57 | 69.71 |
Sidewalks | 50.69 | 55.47 | 34.60 | 47.74 | 56.89 | 64.23 | 53.96 | 43.93 | 53.19 | 62.98 |
Crosswalks | 2.61 | 12.13 | 4.20 | 14.15 | 23.34 | 29.23 | 17.57 | 8.84 | 11.31 | 14.00 |
Major thoroughfares | 38.41 | 72.58 | 48.41 | 47.83 | 69.96 | 72.84 | 66.66 | 61.08 | 53.62 | 83.77 |
Highways | 76.58 | 80.54 | 75.60 | 38.04 | 86.48 | 76.20 | 74.53 | 82.80 | 53.67 | 90.06 |
Railways | 79.06 | 90.99 | 59.26 | 94.60 | 92.41 | 96.99 | 92.36 | 98.96 | 93.89 | 97.17 |
Paved parking lots | 67.52 | 88.01 | 69.21 | 81.18 | 92.97 | 92.85 | 92.46 | 84.20 | 91.47 | 95.96 |
Cars | 69.57 | 79.15 | 54.47 | 54.41 | 69.04 | 63.23 | 84.98 | 81.62 | 65.22 | 93.56 |
Trains | 96.19 | 94.57 | 90.42 | 72.24 | 96.13 | 94.80 | 95.83 | 98.43 | 88.52 | 99.16 |
Stadium seats | 57.83 | 85.46 | 64.68 | 82.01 | 93.90 | 95.40 | 88.87 | 62.24 | 81.17 | 96.52 |
pixel acc. | 84.84 | 88.66 | 73.44 | 83.56 | 87.56 | 91.12 | 87.24 | 84.72 | 85.97 | 92.74 |
mIoU | 68.29 | 76.19 | 58.11 | 67.57 | 76.57 | 79.60 | 75.61 | 71.34 | 71.24 | 83.18 |
Class | UNet | DeepLab v3+ | SegFormer | Swin-UNet | UNet-m | SS3FCN | ENLFCN | SSDGL | FSSANet | CTAFNet |
---|---|---|---|---|---|---|---|---|---|---|
Acer negundo | 68.54 | 52.18 | 14.18 | 63.24 | 72.43 | 85.71 | 88.25 | 20.27 | 49.43 | 89.29 |
Willow | 85.62 | 77.04 | 56.18 | 70.77 | 85.10 | 91.62 | 98.05 | 43.78 | 72.67 | 97.74 |
Elm | 79.88 | 57.30 | 0.00 | 62.33 | 59.82 | 82.54 | 86.08 | 16.71 | 45.42 | 92.53 |
Paddy | 94.82 | 97.62 | 89.89 | 93.76 | 97.05 | 98.68 | 98.95 | 95.53 | 95.76 | 99.20 |
Sophora japonica | 73.32 | 78.77 | 35.37 | 70.00 | 70.31 | 91.88 | 90.62 | 47.58 | 52.18 | 91.31 |
Fraxinus chinensis | 80.43 | 77.90 | 34.64 | 49.20 | 77.37 | 91.14 | 93.15 | 41.71 | 74.34 | 85.19 |
Goldenrain tree | 99.59 | 93.01 | 7.80 | 87.24 | 74.96 | 97.43 | 99.86 | 50.12 | 97.93 | 99.52 |
Waters | 96.95 | 94.97 | 89.31 | 88.36 | 95.54 | 94.83 | 94.14 | 87.16 | 91.59 | 95.11 |
Bare ground | 93.60 | 96.37 | 76.12 | 93.02 | 91.82 | 98.88 | 95.36 | 85.76 | 85.60 | 94.39 |
Stubble | 87.53 | 98.01 | 82.62 | 95.13 | 97.72 | 99.56 | 98.17 | 88.09 | 97.36 | 99.96 |
Corn | 58.02 | 56.89 | 21.06 | 34.57 | 56.34 | 65.95 | 77.61 | 8.97 | 61.65 | 80.21 |
Pyrus | 76.00 | 80.09 | 58.47 | 61.74 | 68.51 | 86.90 | 86.94 | 48.97 | 65.91 | 93.86 |
Soybean | 8.52 | 23.32 | 0.00 | 13.01 | 9.57 | 19.71 | 9.85 | 4.91 | 9.50 | 34.36 |
Poplar | 74.69 | 71.24 | 28.68 | 59.29 | 69.95 | 72.20 | 70.30 | 28.48 | 48.72 | 78.53 |
Vegetable field | 31.12 | 32.62 | 12.56 | 15.03 | 54.24 | 36.28 | 44.92 | 21.25 | 25.99 | 53.72 |
Grass | 73.27 | 68.16 | 40.42 | 61.54 | 67.21 | 84.69 | 88.61 | 39.88 | 67.36 | 92.90 |
Peach | 78.26 | 78.22 | 39.09 | 51.83 | 81.30 | 80.58 | 79.71 | 43.56 | 49.74 | 95.99 |
Building | 83.70 | 83.49 | 76.26 | 60.84 | 88.33 | 88.25 | 90.83 | 67.33 | 64.61 | 89.38 |
pixel acc. | 87.29 | 87.52 | 66.58 | 79.63 | 86.17 | 93.79 | 94.37 | 64.91 | 79.38 | 96.17 |
mIoU | 74.66 | 73.18 | 42.37 | 62.83 | 73.20 | 81.49 | 82.86 | 46.67 | 64.21 | 86.84 |
Conv | Trans | Hybrid Strategy | Aerial | DFC2018 | Xiongan | |||
---|---|---|---|---|---|---|---|---|
Pixel Acc. | mIoU | Pixel Acc. | mIoU | Pixel Acc. | mIoU | |||
√ | None | 94.90 | 76.54 | 92.31 | 81.40 | 87.86 | 76.66 | |
√ | None | 93.12 | 72.70 | 75.59 | 56.34 | 61.27 | 40.61 | |
√ | √ | CCTT | 94.02 | 75.97 | 88.46 | 76.41 | 90.30 | 76.75 |
√ | √ | Add | 94.25 | 76.15 | 89.45 | 76.46 | 88.62 | 71.32 |
√ | √ | Cat | 94.69 | 77.79 | 91.12 | 80.62 | 92.36 | 79.05 |
√ | √ | Adapt | 95.07 | 81.41 | 92.74 | 83.18 | 96.17 | 86.84 |
Number of Heads | Aerial | DFC2018 | Xiongan | |||
---|---|---|---|---|---|---|
Pixel Acc. | mIoU | Pixel Acc. | mIoU | Pixel Acc. | mIoU | |
1 | 94.51 | 77.98 | 92.08 | 82.37 | 94.05 | 81.79 |
2 | 95.07 | 81.41 | 92.74 | 83.18 | 96.17 | 86.84 |
4 | 94.83 | 79.25 | 91.76 | 82.00 | 94.49 | 81.35 |
8 | 94.94 | 80.22 | 91.96 | 81.83 | 93.95 | 80.20 |
Number of Channels | Aerial | DFC2018 | Xiongan | |||
---|---|---|---|---|---|---|
Pixel Acc. | mIoU | Pixel Acc. | mIoU | Pixel Acc. | mIoU | |
32 | 94.91 | 78.02 | 90.37 | 78.30 | 89.80 | 74.74 |
64 | 94.53 | 77.95 | 91.81 | 81.25 | 93.76 | 75.97 |
96 | 95.07 | 81.41 | 92.74 | 83.18 | 96.17 | 86.84 |
128 | 94.84 | 78.78 | 91.75 | 82.21 | 94.19 | 82.10 |
Class | UNet | DeepLab v3+ | SegFormer | Swin-UNet | UNet-m | SS3FCN | ENLFCN | SSDGL | FSSANet | CTAFNet |
---|---|---|---|---|---|---|---|---|---|---|
Broccoli green weeds 1 | 99.14 | 0.00 | 16.63 | 77.59 | 97.06 | 100.00 | 100.00 | 9.49 | 100.00 | 100.00 |
Broccoli green weeds 2 | 100.00 | 74.01 | 1.33 | 89.49 | 99.87 | 100.00 | 100.00 | 4.20 | 100.00 | 100.00 |
Fallow | 95.41 | 11.29 | 39.90 | 82.89 | 100.00 | 85.28 | 85.04 | 17.39 | 83.23 | 100.00 |
Fallow rough plow | 86.45 | 73.03 | 72.03 | 98.29 | 95.02 | 98.97 | 91.11 | 89.13 | 97.92 | 98.62 |
Fallow smooth | 95.22 | 95.72 | 94.34 | 94.18 | 99.48 | 99.27 | 98.33 | 99.58 | 98.35 | 99.69 |
Stubble | 100.00 | 67.84 | 64.73 | 94.63 | 99.56 | 99.85 | 100.00 | 94.87 | 96.76 | 100.00 |
Celery | 99.71 | 99.86 | 7.71 | 97.29 | 100.00 | 99.28 | 100.00 | 23.18 | 97.57 | 100.00 |
Grapes untrained | 38.36 | 37.28 | 40.66 | 61.13 | 65.59 | 72.46 | 75.94 | 18.15 | 66.20 | 75.06 |
Soil vineyard develop | 98.82 | 83.38 | 83.81 | 99.88 | 95.67 | 99.71 | 99.88 | 97.85 | 97.76 | 99.51 |
Corn senesced green weeds | 67.75 | 50.51 | 51.65 | 62.56 | 80.10 | 79.92 | 83.03 | 45.93 | 81.58 | 98.66 |
Lettuce romaine 4 wk | 87.41 | 24.68 | 38.46 | 69.05 | 92.38 | 86.62 | 85.62 | 52.24 | 84.62 | 100.00 |
Lettuce romaine 5 wk | 96.45 | 5.80 | 13.77 | 83.41 | 85.33 | 97.24 | 98.60 | 51.62 | 98.24 | 100.00 |
Lettuce romaine 6 wk | 93.64 | 0.00 | 4.37 | 80.88 | 93.16 | 89.52 | 96.09 | 87.70 | 77.00 | 100.00 |
Lettuce romaine 7 wk | 88.19 | 64.58 | 70.08 | 75.21 | 87.26 | 77.41 | 83.72 | 79.46 | 69.49 | 99.13 |
Vineyard untrained | 56.43 | 0.00 | 31.97 | 64.59 | 67.97 | 66.49 | 67.11 | 17.89 | 54.25 | 60.48 |
Vineyard vertical trellis | 18.59 | 0.00 | 0.00 | 82.10 | 0.00 | 91.86 | 88.28 | 0.00 | 78.49 | 100.00 |
pixel acc. | 85.70 | 63.44 | 61.24 | 88.72 | 90.99 | 92.95 | 93.55 | 59.84 | 90.44 | 94.62 |
mIoU | 82.60 | 43.00 | 39.47 | 82.07 | 84.90 | 90.24 | 90.80 | 49.29 | 86.34 | 95.70 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, J.; Xing, H.; Ao, Z.; Wang, H.; Liu, W.; Zhang, A. Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification. Appl. Sci. 2023, 13, 492. https://doi.org/10.3390/app13010492
Li J, Xing H, Ao Z, Wang H, Liu W, Zhang A. Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification. Applied Sciences. 2023; 13(1):492. https://doi.org/10.3390/app13010492
Chicago/Turabian StyleLi, Jiaju, Hanfa Xing, Zurui Ao, Hefeng Wang, Wenkai Liu, and Anbing Zhang. 2023. "Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification" Applied Sciences 13, no. 1: 492. https://doi.org/10.3390/app13010492
APA StyleLi, J., Xing, H., Ao, Z., Wang, H., Liu, W., & Zhang, A. (2023). Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification. Applied Sciences, 13(1), 492. https://doi.org/10.3390/app13010492