In this section we describe the experimental work and analyze the results obtained by our proposal when applied to crop classification task in satellite images.
4.4. Comparison Measures
The comparison measures, for validation purposes, are based on the confusion matrix
C which is used in multi-class classification problems [
42,
43] to assess the performance of algorithms. Here, we assume that the rows of the confusion matrix correspond to the actual classes (the “ground truth”) and the columns correspond to the predicted classes. Then, the entry
of the confusion matrix
C represents the number of data of the class
i that are predicted in class
j. The performance measure we used for comparing the algorithms are: the overall accuracy, the precision for each class
j and the Cohen’s kappa, i.e.,
where
The overall accuracy (
16) is the number of correct classifications divided by the total number of classified data. The precision (
17) is a measure of the accuracy of a specific class
j. Thus, the precision is the ratio between the number of correctly predicted data of the class
j divided by the total number of classified data in the class
j. Cohen’s kappa (
18) is a statistical measure of inter-rater agreement or inter-annotator agreement for 2 raters [
41]. Cohen’s kappa measures the agreement between two raters who each classify
N items into
K mutually exclusive categories, see
Table 4 for the interpretation of the values of Kappa index [
53]. In Ref. [
53] the reader can find more details about the measure of agreement Kappa.
4.5. Results and Discussion
We carried out several experiments where we combined the feature spaces described in the
Section 4.3.
Table 5 gives a brief description about all possible feature space combinations. Column 1 indicates the number of combination and column 2 specifies the feature spaces included in the combination.
We conducted 11 experiments with real data described in
Section 4.2. The first three experiments considered only one feature space with the standard GMMF algorithm [
29]. The next four experiments were performed using the lowest entropy model [
28], see Equations (
4) and (
5) and the space combinations 4–7 given in
Table 5. The last four experiments also used the combinations 4–7, and the segmentation process was conducted using the fusion sources model proposed in this work, see Equations (
10)–(
12).
Table 6,
Table 7 and
Table 8 summarize the numerical results of all 11 experiments. These tables present the numerical information about the classification of 5 different crops, see
Table 1, the precision of classification for each class, the overall accuracy and Kappa coefficient. We note that the results of the first row of
Table 6 and
Table 7 differ from those presented in Ref. [
28,
29]. This is due to, in this results, we only consider sites in the image that correspond to crops in the five categories of interest, i.e., only pixels in the region of interest.
Table 6 shows the results obtained using the standard GMMF algorithm with different features spaces [
29,
54]. Experiments 1–3 consider the information from Space 1, Space 2 and Space 3 respectively, see
Table 5. Note that best result is obtained when using the first three principal components based on all spectral bands, except the TM6 band (Experiment 3). Note also that, the first three principal components using the 10 spectral indexes (Experiment 2) achieved a similar performance to the one obtained with the typical combination of TM2, TM3 and TM4 bands (Experiment 1). This means that for the crop classification using GMMF, the amount of information is relevant.
Table 7 summarizes numerical results for the experiments in which we considered the lowest entropy model [
28], see Equations (
4) and (
5). Experiment 4 combines the Space 1 and the Space 2, Experiment 5 takes into account the Space 1 and the Space 3, Experiment 6 includes Space 2 and Space 3 and Experiment 7 uses all studied spaces. Observe that the combination of different feature spaces together with the lowest entropy model allowed to improve the precision of crop recognition in general, and hence the overall accuracy and Kappa index increased.
Table 7 also confirms that not only the information from TM2, TM3 and TM4 (Space 1) is relevant from crop recognition, but also the information derived from other bands.
Table 8 shows the numerical results of the experiments 8–11. These four experiments considered the same feature space combination as in the experiments 4–7 in
Table 7, but using the fusion sources model (
10)–(
12). Note that the Experiment 9 achieved the best performance. This experiment considered the combination of Space 1 and Space 3.
Although the results of the experiments in
Table 8 are similar in comparison with the results obtained by means of lowest entropy model, see
Table 7, they have a better performance than those obtained by lowest entropy model. In general, the new proposal obtained the best performance compared with our previous conference papers. Furthermore, the study of feature spaces performed in this work also allowed us to improve the performance of our previous work.
Our experimental work corroborated that the vegetation indices enhance the vegetation information, and they are able to distinguish between vegetation and non-vegetation. However, for the classification task, they are not sufficient as an information source to discriminate different types of crops.
Though more information sources we have a greater computation time, for this task we suggest using more feature spaces in order to improve the accuracy of the classification process. For that reason the proposal described in
Section 3.2 considers a fusion of multiples feature spaces.
In this work we used feature spaces generated from the pixel information and different spectral bands, however the proposal is more general and it allows to include other feature spaces and not only punctual information, but also local information derived from different image modalities. The study of local information and the fusion of different image modalities is a part of our future work.
On the other hand,
Table 9 shows a comparative analysis of different classification methods that have been widely used in the context of crop classification problems. In the experiment we use the real image described in
Section 4.2. Here we compare different algorithms implemented in the software MultiSpec, funded by NASA, that is available online [
55]. MultiSpec is a freeware multispectral image analysis software developed at Purdue University and the latest release date is on March 2017. In this experiment, we include the following methods: Minimum Euclidean Distance (MED) [
56], Maximum Likelihood Classifier (ML) [
57], Fisher Linear Likelihood (FLL) [
58] and (ESS) [
59,
60] because they reached the best performance results. We also include two versions of the one vs all multiclass SVM method, the linear and the non-linear Radial Basis Function alternatives, using the winner-takes-all strategy. In this case, we use the matlab built-in function for SVM. In the comparison study we also consider the original version of the probabilistic segmentation approach described in [
28,
29]. Additionally, for a fair comparison, we include the performance analysis of these algorithms using the best results reached in the feature space study in this work, see
Table 6 and
Table 7, denoted as MICAI
and MICAI
in
Table 9. Finally, we include the results achieved by our proposal based on the fusion of information sources. Note that for our proposal, and in general for probabilistic segmentation approaches, the natural feature space is the space of probabilistic distributions or the likelihood space, unlike others methods under study where the natural space is the original image. A detailed comparison study of feature spaces and classification methods for crop classification is a very interesting task, however, this is out of the scope of the present research and we leave it for future work.
Based on the comparison results in
Table 9, we note that our proposal, in general, outperforms the remainder analyzed methods. The precision, overall accuracy and the Kappa index attained a better performance with respect to our previous conference papers [
28,
29]. Observe, that the non-linear SVM version with Radial Basis Function leaded to the best classification for class C4, however the best classification for the rest classes was reached by our proposal. On the the hand, the analysis of the feature spaces developed in this work, also allowed to improve the results of the probabilistic methods in [
28,
29]. The method MICAI
corresponds to the GMMF algorithm [
29] using the Space 3 studied here. Note that the overall accuracy and Kappa coefficient increased in comparison to the Space 1 used in the original paper in Ref. [
29]. Similarly, MICAI
refers to the proposal in [
28] using the feature space combination Space 1 and Space 3. Note that performance measures of classification also increased.
For illustrative purposes, the classification maps of the three most accurate methods given in
Table 9 are shown in
Figure 4.