Figure 1.
The illustration of the proposed framework in training and testing phases.
Figure 1.
The illustration of the proposed framework in training and testing phases.
Figure 2.
The network architecture of full pre-activation RBs.
Figure 2.
The network architecture of full pre-activation RBs.
Figure 3.
Network design of the proposed coupled residual convolutional neural networks.
Figure 3.
Network design of the proposed coupled residual convolutional neural networks.
Figure 4.
Houston 2013: From top to bottom, the LiDAR-derived DSM image, the false color HSI image, the training samples, and the testing samples.
Figure 4.
Houston 2013: From top to bottom, the LiDAR-derived DSM image, the false color HSI image, the training samples, and the testing samples.
Figure 5.
Houston 2018: From top to bottom, the LiDAR-derived DSM image, the VHR RGB Image (downsampled), the training samples, and the testing samples.
Figure 5.
Houston 2018: From top to bottom, the LiDAR-derived DSM image, the VHR RGB Image (downsampled), the training samples, and the testing samples.
Figure 6.
Trento: From top to bottom, the LiDAR-derived DSM image, the false color HSI image, the training samples, and the testing samples.
Figure 6.
Trento: From top to bottom, the LiDAR-derived DSM image, the false color HSI image, the training samples, and the testing samples.
Figure 7.
The Houston 2013 dataset: Classifications generated from different features and models. (a) HSI-ResNet, (b) LiDAR-ResNet, (c) EPs-HSI-ResNet, (d) EPs-LiDAR-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 7.
The Houston 2013 dataset: Classifications generated from different features and models. (a) HSI-ResNet, (b) LiDAR-ResNet, (c) EPs-HSI-ResNet, (d) EPs-LiDAR-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 8.
The Trento dataset: Classifications generated from different features and models. (a) HSI-ResNet, (b) LiDAR-ResNet, (c) EPs-HSI-ResNet, (d) EPs-LiDAR-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 8.
The Trento dataset: Classifications generated from different features and models. (a) HSI-ResNet, (b) LiDAR-ResNet, (c) EPs-HSI-ResNet, (d) EPs-LiDAR-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 9.
The Houston 2018 dataset: (a) Ground-truth label map; (b–f) classification maps generated on different features and models. (b) HSI-ResNet, (c) LiDAR-ResNet, (d) RGB-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 9.
The Houston 2018 dataset: (a) Ground-truth label map; (b–f) classification maps generated on different features and models. (b) HSI-ResNet, (c) LiDAR-ResNet, (d) RGB-ResNet, (e) CResNet, and (f) CResNet-AUX.
Figure 10.
Analysis of the classification OA w.r.t the number of training samples on the Houston 2018 dataset. We select 10, 25, 50, or 100 training samples per each class.
Figure 10.
Analysis of the classification OA w.r.t the number of training samples on the Houston 2018 dataset. We select 10, 25, 50, or 100 training samples per each class.
Figure 11.
Analysis of classification OA w.r.t the weights of auxiliary losses on Houston 2018 dataset.
Figure 11.
Analysis of classification OA w.r.t the weights of auxiliary losses on Houston 2018 dataset.
Figure 12.
Comparison of classification accuracy with and without auxiliary loss functions for three datasets.
Figure 12.
Comparison of classification accuracy with and without auxiliary loss functions for three datasets.
Table 1.
Houston University 2013: The number of training samples, testing samples, and the total number of samples per class.
Table 1.
Houston University 2013: The number of training samples, testing samples, and the total number of samples per class.
Class No. | Class Name | Training | Testing | Samples |
---|
1 | Healthy grass | 198 | 1053 | 1251 |
2 | Stressed grass | 190 | 1064 | 1254 |
3 | Synthetic grass | 192 | 505 | 697 |
4 | Tree | 188 | 1056 | 1244 |
5 | Soil | 186 | 1056 | 1242 |
6 | Water | 182 | 143 | 325 |
7 | Residential | 196 | 1072 | 1268 |
8 | Commercial | 191 | 1053 | 1244 |
9 | Road | 193 | 1059 | 1252 |
10 | Highway | 191 | 1036 | 1227 |
11 | Railway | 181 | 1054 | 1235 |
12 | Parking Lot 1 | 192 | 1041 | 1233 |
13 | Parking Lot 2 | 184 | 285 | 469 |
14 | Tennis court | 181 | 247 | 428 |
15 | Running track | 187 | 473 | 660 |
| Total | 2832 | 12,197 | 15,029 |
Table 2.
Houston University 2018: The number of training samples, testing samples, and the total number of samples per class.
Table 2.
Houston University 2018: The number of training samples, testing samples, and the total number of samples per class.
Class No. | Class Name | Training | Testing | Samples |
---|
1 | Healthy grass | 1458 | 8341 | 9799 |
2 | Stressed grass | 4316 | 28,186 | 32,502 |
3 | Synthetic grass | 331 | 353 | 684 |
4 | Evergreen Trees | 2005 | 11,583 | 13,588 |
5 | Deciduous Trees | 676 | 4372 | 5048 |
6 | Soil | 1757 | 2759 | 4516 |
7 | Water | 147 | 119 | 266 |
8 | Residential | 3809 | 35,953 | 39,762 |
9 | Commercial | 2789 | 220,895 | 223,684 |
10 | Road | 3188 | 42,622 | 45,810 |
11 | Sidewalk | 2699 | 31,303 | 34,002 |
12 | Crosswalk | 225 | 1291 | 1516 |
13 | Major Thoroughfares | 5193 | 41,165 | 46,358 |
14 | Highway | 700 | 9149 | 9849 |
15 | Railway | 1224 | 5713 | 6937 |
16 | Paved Parking Lot | 1179 | 10,296 | 11,475 |
17 | Gravel Parking Lot | 127 | 22 | 149 |
18 | Cars | 848 | 5730 | 6578 |
19 | Trains | 493 | 4872 | 5365 |
20 | Seats | 1313 | 5511 | 6824 |
| Total | 34,477 | 470,235 | 504,712 |
Table 3.
Trento: The number of training samples, testing samples, and the total number of samples per class.
Table 3.
Trento: The number of training samples, testing samples, and the total number of samples per class.
Class No. | Class Name | Training | Testing | Samples |
---|
1 | Apple trees | 129 | 3905 | 4034 |
2 | Buildings | 125 | 2778 | 2903 |
3 | Ground | 105 | 374 | 479 |
4 | Wood | 154 | 8969 | 9123 |
5 | Vineyard | 184 | 10,317 | 10,501 |
6 | Roads | 122 | 3052 | 3174 |
| Total | 819 | 29,395 | 30,214 |
Table 4.
Houston 2013: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
Table 4.
Houston 2013: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
# | Class | HSI-ResNet | LiDAR-ResNet | EPs-HSI-ResNet | EPs-LiDAR-ResNet | CResNet | CResNet-AUX |
---|
| Number of features | (144) | (1) | (225) | (71) | (144+225+71) | (144+225+71) |
---|
1 | Healthy grass | 77.68 | 51.76 | 74.83 | 54.13 | 83.00 | 86.51 |
2 | Stressed grass | 98.59 | 47.09 | 76.60 | 56.77 | 99.81 | 98.01 |
3 | Synthetic grass | 86.53 | 87.33 | 87.33 | 94.06 | 84.36 | 87.87 |
4 | Tree | 86.46 | 51.52 | 51.89 | 68.09 | 96.69 | 85.52 |
5 | Soil | 89.11 | 43.56 | 93.94 | 52.37 | 99.91 | 87.02 |
6 | Water | 81.12 | 78.32 | 91.61 | 79.02 | 95.80 | 99.81 |
7 | Residential | 93.75 | 67.07 | 74.07 | 75.93 | 90.11 | 100.00 |
8 | Commercial | 81.86 | 75.12 | 80.53 | 83.57 | 95.73 | 95.72 |
9 | Road | 88.67 | 58.55 | 55.71 | 59.87 | 90.65 | 96.68 |
10 | Highway | 74.52 | 73.84 | 54.05 | 72.78 | 70.46 | 100.00 |
11 | Railway | 95.64 | 90.32 | 68.98 | 98.29 | 94.68 | 85.54 |
12 | Parking Lot 1 | 85.78 | 68.20 | 73.20 | 78.10 | 97.50 | 95.80 |
13 | Parking Lot 2 | 82.81 | 75.44 | 68.07 | 72.28 | 79.30 | 94.05 |
14 | Tennis court | 100.00 | 90.28 | 93.12 | 88.66 | 100.00 | 95.10 |
15 | Running track | 68.92 | 39.32 | 41.23 | 15.43 | 89.85 | 93.87 |
| OA(%) | 86.60 | 63.82 | 70.63 | 69.39 | 91.42 | 93.57 |
| AA(%) | 86.10 | 66.51 | 72.34 | 69.96 | 91.19 | 93.44 |
| Kappa | 0.8545 | 0.6074 | 0.6809 | 0.6676 | 0.9068 | 0.9302 |
Table 5.
Trento: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
Table 5.
Trento: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
# | Class | HSI-ResNet | LiDAR-ResNet | EPs-HSI-ResNet | EPs-LiDAR-ResNet | CResNet | CResNet-AUX |
---|
| Number of Features | (63) | (1) | (225) | (71) | (63+225+71) | (63+225+71) |
---|
1 | Apple trees | 98.21 | 0.00 | 96.67 | 98.39 | 98.10 | 99.74 |
2 | Buildings | 93.12 | 15.77 | 87.83 | 97.52 | 97.77 | 99.60 |
3 | Ground | 77.54 | 39.84 | 77.01 | 64.71 | 77.01 | 75.40 |
4 | Wood | 98.99 | 98.27 | 99.74 | 100.00 | 99.90 | 100.00 |
5 | Vineyard | 99.96 | 97.00 | 94.92 | 97.77 | 100.00 | 100.00 |
6 | Roads | 60.52 | 2.62 | 75.75 | 83.65 | 92.46 | 92.27 |
| OA (%) | 94.40 | 66.30 | 93.74 | 96.62 | 98.43 | 98.81 |
| AA (%) | 88.06 | 42.25 | 88.65 | 90.34 | 94.21 | 94.50 |
| Kappa | 0.9250 | 0.5178 | 0.9166 | 0.9548 | 0.9790 | 0.9841 |
Table 6.
Houston 2018: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
Table 6.
Houston 2018: Classification accuracies for per class, OA, AA (in %), kappa coefficient (is of no unit). The bold refers to the best OA, AA, and Kappa performance.
# | Class | HSI-ResNet | LiDAR-ResNet | RGB-ResNet | CResNet | CResNet-AUX |
---|
| Number of features | (48) | (7) | (3) | (48+7+3) | (48+7+3) |
---|
1 | Healthy grass | 46.35 | 24.25 | 41.54 | 18.77 | 75.90 |
2 | Stressed grass | 79.64 | 74.80 | 79.37 | 90.43 | 67.79 |
3 | Synthetic grass | 82.72 | 100.00 | 100.00 | 100.00 | 100.00 |
4 | Evergreen Trees | 93.59 | 90.02 | 93.05 | 94.74 | 95.24 |
5 | Deciduous Trees | 46.27 | 43.62 | 44.26 | 59.54 | 59.47 |
6 | Soil | 36.17 | 31.39 | 86.48 | 43.82 | 36.82 |
7 | Water | 42.02 | 0.00 | 22.69 | 30.25 | 1.68 |
8 | Residential | 89.86 | 87.51 | 91.08 | 90.79 | 88.00 |
9 | Commercial | 71.24 | 70.89 | 66.35 | 92.71 | 92.75 |
10 | Road | 54.44 | 61.35 | 65.97 | 64.14 | 72.77 |
11 | Sidewalk | 63.14 | 73.80 | 75.18 | 62.26 | 71.27 |
12 | Crosswalk | 3.95 | 2.40 | 2.87 | 3.02 | 3.95 |
13 | Major Thoroughfares | 47.50 | 62.67 | 56.97 | 65.15 | 57.62 |
14 | Highway | 31.82 | 34.97 | 37.22 | 42.34 | 44.82 |
15 | Railway | 77.58 | 84.75 | 84.74 | 63.77 | 63.96 |
16 | Paved parking Lot | 85.60 | 97.31 | 94.80 | 83.64 | 89.48 |
17 | Gravel parking Lot | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
18 | Cars | 32.24 | 37.24 | 50.89 | 29.91 | 34.57 |
19 | Trains | 93.49 | 99.36 | 98.75 | 92.44 | 97.74 |
20 | Seats | 63.49 | 99.84 | 98.42 | 61.13 | 73.42 |
| OA (%) | 67.83 | 70.26 | 69.53 | 80.62 | 81.20 |
| AA (%) | 62.16 | 63.81 | 69.53 | 64.47 | 66.36 |
| Kappa | 0.5944 | 0.6287 | 0.6253 | 0.7416 | 0.7506 |
Table 7.
Houston 2013: Performance comparison with the state-of-the-art models. The bold refers to the best OA, AA, and Kappa performance.
Table 7.
Houston 2013: Performance comparison with the state-of-the-art models. The bold refers to the best OA, AA, and Kappa performance.
Methods | MLRsub [10] | OTVCA [13] | SLRCA [15] | DeepFusion [23] | EPs-CNN [8] | CK-CNN [24] | CResNet | CResNet-AUX |
---|
OA (%) | 92.05 | 92.45 | 91.30 | 91.32 | 91.02 | 92.57 | 91.42 | 93.57 |
AA (%) | 92.85 | 92.68 | 91.95 | 91.96 | 91.82 | 92.48 | 91.19 | 93.44 |
Kappa | 0.9137 | 0.9181 | 0.9056 | 0.9057 | 0.9033 | 0.9193 | 0.9068 | 0.9302 |
Table 8.
Computational time for three multi-sensor datasets. The bold refers to the best OA, AA, and Kappa performance.
Table 8.
Computational time for three multi-sensor datasets. The bold refers to the best OA, AA, and Kappa performance.
Houston 2013 | HSI-ResNet | LiDAR-ResNet | EPs-HSI-ResNet | EPs-LiDAR-ResNet | CResNet | CResNet-AUX |
Train (min) | 8.84 | 5.837 | 8.61 | 5.67 | 16.11 | 16.61 |
Test (s) | 4.38 | 3.04 | 5.53 | 3.61 | 8.15 | 16.25 |
Trento | HSI-ResNet | LiDAR-ResNet | EPs-HSI-ResNet | EPs-LiDAR-ResNet | CResNet | CResNet-AUX |
Train (min) | 5.69 | 5.04 | 6.88 | 5.79 | 11.73 | 13.66 |
Test (s) | 6.28 | 5.62 | 9.15 | 7.13 | 13.57 | 14.06 |
Houston 2018 | HSI-ResNet | LiDAR-ResNet | RGB-ResNet | CResNet | CResNet-AUX | |
Train (min) | 82.50 | 63.11 | 58.13 | 159.9 | 168.33 | |
Test (s) | 53.64 | 35.84 | 38.38 | 102.91 | 107.79 | |