An Integrated Change Detection Method Based on Spectral Unmixing and the CNN for Hyperspectral Imagery

Abstract

Hyperspectral remote sensing image (HSI) include rich spectral information that can be very beneficial for change detection (CD) technology. Due to the existence of many mixed pixels, pixel-wise approaches can lead to considerable errors in the resulting CD map. The spectral unmixing (SU) method is a potential solution to this problem, as it decomposes mixed pixels into a set of fractions of land cover. Subsequently, the CD map is created by comparing the abundance images. However, based only on the abundance images created through the SU method, they are unable to effectively provide detailed change information. Meanwhile, the features of change information cannot be sufficiently extracted by the traditional sub-pixel CD framework, which leads to a poor CD result. To address these problems, this paper presents an integrated CD method based on multi-endmember spectral unmixing, joint matrix and CNN (MSUJMC) for HSI. Three main steps are considered to accomplish this task. First, considering the endmember spectral variability, more reliable endmember abundance information is obtained by multi-endmember spectral unmixing (MSU). Second, the original image features are incorporated with the abundance images using a joint matrix (JM) algorithm to provide more temporal and spatial land cover change information characteristics. Third, to efficiently extract the change features and to better handle the fused multi-source information, the convolutional neural network (CNN) is introduced to realize a high-accuracy CD result. The proposed method has been verified on simulated and real multitemporal HSI datasets, which provide multiple changes. Experimental results verify the effectiveness of the proposed approach.

1. Introduction

Change detection (CD) [1,2] using remote sensing images is essential for protecting the ecological environment, managing natural resources and studying social development, etc. [3,4,5,6]. The traditional CD technique is mainly applied in multispectral images (MSI) with the aim of detecting land cover changes in multi-temporal remote sensing images [7,8]. However, due to the limited spectral resolution of the MSI, only significant changes can be detected. In contrast, hyperspectral images (HSIs) have more abundant spectral information of ground objects, which can reflect the subtle spectral properties of the measured objects in detail [9]. Many researchers have been concerned about the study of CD in HSI. For example, sparse unmixing with dictionary pruning for HSI-CD is proposed to reduce the computation time of the CD process and improve the CD performance by using the high spectral resolution property of HSI [10]. A novel hierarchical CD method that considers both change magnitude and spectral change information is proposed, aiming to identify the change classes with discriminable spectral behaviour and thereby improving the accuracy of CD results [11]. A subspace-based HSI-CD method is proposed, which makes full use of the rich spectral information in the HSI and obtains more reliable CD results by measuring the spectral changes [12], etc.

Even so, many mixed pixels occur widely in HSI due to the complexity of the ground cover and the restriction of the sensor resolution in HSI, which cause difficulties for CD [13]. The interior of the mixed pixel is not a single element but rather a combination of several elements [14,15]. If these mixed pixels are employed in CD on a pixel level, some significant information will be lost [16]. To address the effect of mixed pixels, many researchers have attempted to explore efficient and robust CD methods. Spectral unmixing (SU) is a representative method that solves the problem of mixed pixels to a certain extent [17]. The whole procedure is described by the following steps. First, the endmember spectra of each feature type are extracted from the HSI. Second, the abundance of each feature type is estimated. Finally, the CD map is created by comparing the corresponding abundance results. The CD method at sub-pixel level has been shown to be superior to the traditional CD method at the pixel level [18,19,20]. During the process, obtaining abundance results with high accuracy is very important. As the imaging environment of each region on an image is likely to be different, the same ground element may not have the same endmember spectra or may have significant differences. The illumination variations or shadows degrade the performance of the spectral unmixing due to the spectral variability within each endmember class. If this issue is not addressed, the validity of endmember abundance information will be adversely impacted, leading to a poor CD result [21]. Many existing unmixing models that include spectral variability or nonlinearity aim to address these problems. For example, an unsupervised multitemporal SU method that considers endmember spectral variability is proposed for detecting multiple changes in HSI [22]. A novel CD approach is based on SU from stacked multitemporal remote sensing images with a variability of endmembers [23]. A new spectral mixing model, called the augmented linear mixing model (ALMM), is proposed to solve the spectral variability problem by applying a data-driven learning strategy in inverse problems of HSI unmixing [24], etc. These methods have achieved better results in solving the problem of endmember spectral variability.

However, previous SU methods were mainly focused on obtaining the abundance images, and afterward, the traditional comparison methods were used to obtain the CD results. In such a framework, the CD methods at the “sub-pixel level” are often unable to provide detailed change information effectively based only on the limited resources. The original image feature information at the pixel level is not utilized, which can be seen as a significant complement. This means that the original image feature information should be sufficiently considered and extracted for the HSI-CD task. Thus, to improve the accuracy of the HSI-CD results, the original image feature information is incorporated with the endmember abundance information to fully exploit the information contained in the HSI. Subsequently, a suitable method is adopted to efficiently extract change features from the fused information. In the early stages, some machine learning-based methods, such as the k-means clustering algorithm (k-means) [25], change vector analysis (CVA) [26], random forest (RF) [27], support vector machine (SVM) [28], Markov random field (MRF) [29], decision trees (DT) [30], neural network (NN) [31], logistic regression (LR) [32], etc., were widely used. However, these methods only extract shallow features based on the spectral information of the HSI, using individual pixels and all their bands as input. As a result, these linear and nonlinear classifiers do not extract subtle features of HSI well, limiting their application. In recent years, deep learning has become increasingly popular in the field of remote sensing, as it is capable of dealing with certain sophisticated and abstract problems. Deep learning can utilize training samples to train the neural network, which allows the neural network to recognize subtle and abstract features and eventually, extract these features efficiently [33,34,35].

From the above discussion, an integrated CD method based on a multi-endmember spectral unmixing joint matrix and CNN (MSUJMC) is proposed in this paper. It incorporates the original image feature information on sub-pixel level endmember abundance information. In addition, CNN is used to extract multiple types of the change features for detecting multiple changes in the HSI. The overall flow chart of the MSUJMC approach is described in the following steps: (1) Considering the variability of the endmember spectra, this approach utilizes MSU to generate the abundance maps for the whole image; (2) To make full use of the valid information contained in the HSI data, the JM algorithm is used to fuse the multi-source information; (3) The CNN is used to extract complex fine-grained change features and improve the CD performance of the method. This method showed excellent performance on both simulated and real HSI datasets. The experimental results validate the effectiveness of the method in detecting multiple changes in HSI. The remainder of this work is structured in the following format. The second section reviews the related research on remote sensing image CD. The third section thoroughly describes the MSUJMC presented in this article. The fourth section provides the related datasets, experimental results, and analysis. Finally, the fifth section is the conclusion.

2. Related Works

As mentioned in Section 1, the CD is a common application in the field of remote sensing. The aim is to extract pixels from multitemporal remote sensing images where ground types have changed. The relevant CD methods are summarized below.

2.1. Spectral Unmixing

With the increasing availability of HSIs, the problem of mixed pixels included in HSIs is becoming increasingly apparent, limiting the accuracy of HSI-CD results. In order to solve mixed pixels in HSIs, SU has been applied in HSI-CD and achieved good results. For example, a framework based on multi-level SU is proposed for use in HSI-CD tasks [13]. An unsupervised method based on SU and a new formulation for CD is proposed to detect binary changes and multiple changes [18]. In reference [20], HSI-CD by SU is investigated, and systematically present the advantages that can be gained by using such an approach. A sub-pixel CD algorithm using variability in endmembers is proposed, through a simple but effective model that takes into account the real change in endmember combination, the performance of SU is enhanced and the accuracy of the CD results is improved [21], etc.

2.2. Machine Learning

Machine learning was widely used in the field of HSI-CD when deep learning was not widespread. These methods use a number of pixels and all their bands as input to extract features from the spectral information of the HSI and achieve good results. For example, through a formal definition and theoretical study of the CVA technique, a suitable framework is proposed to solve the unsupervised CD problem [26]. An improved RF algorithm is proposed for the classification step in the feature selection process to improve the accuracy of CD [27]. A framework based on image calibration, SVM training and tuning, statistical evaluation of model accuracy, and temporal pixel-based image differencing was used for the coral reef change detection task [28]. Furthermore, a method based on spatial domain analysis and MRF was used to detect building changes in remotely sensed images with good results [29], etc.

2.3. Deep Learning

Deep learning is an important approach for processing high-dimensional data. Different from traditional machine learning algorithms, these deep learning-based methods can automatically extract high-level semantic information from HSIs with no handcrafted feature extraction. Numerous studies have shown that deep learning has outstanding capabilities in feature extraction. When deep learning was introduced into HSI-CD, it achieved remarkable performance. For example, a three-dimensional spectral spatial CNN was proposed to extract the spectral and spatial features of HSIs, which improved the accuracy of the HSI-CD results [33]. A method of the HSI-CD based on tensor and deep learning was proposed, which improves the accuracy of the results by extracting the change information of the underlying features [34]. A CNN-based CD framework was proposed to detect the subtle change features as a way to improve the accuracy of the binary HSI-CD results [35], etc.

3. Methodology

In the proposed method, we first consider the variability of the endmember spectra, using the MSU method to obtain more reliable endmember abundance information. Then, to exploit the rich information contained in the HSI, the JM algorithm is used to fuse the sub-pixel level endmember abundance information and the original image feature information. This procedure converts the corresponding two one-dimensional pixel vectors into a two-dimensional matrix to provide richer cross-channel gradient information. Finally, to perform efficient feature extraction, the CNN is introduced to detect various change categories and generate a CD result map. Figure 1 shows the general architecture of the proposed HSI-CD approach. According to the above theory, this approach consists of the following three steps: (1) The abundance maps were obtained based on the MSU method; (2) Multi-source information was combined based on the JM algorithm; (3) multiple changes were detected based on CNN.

3.1. MSU Method for Acquiring Abundance Images

We used the MSU technique to obtain reliable sub-pixel level endmember abundance information, which improves the performance of the HSI-CD method. Figure 2 shows the flow chart of the MSU method. Considering the large range of the HSI, the same land cover category is in different imaging environments or has different existence states, which often results in the endmember spectra of the same land cover category not being the same or even having large differences. Therefore, the image is divided into several patch images (P₁–P_n) based on the size and complexity of the HSI. Thus, subtle endmember features can be highlighted. In this situation, the endmember spectral signatures can be sufficiently analysed in each patch image. The divided patch image is considerably smaller in size than the entire image. The patch scheme simultaneously handles both issues of a possible large number of endmembers and the effects of local spectral variability. Although the number and type of endmembers are increased, the redundant endmembers that are generated can be merged or eliminated later. In this case, the relative change in the deformation can be fully considered, which effectively reduces the errors of the endmember extraction.

Afterward, the endmember spectra of each patch image are identified by the vertex component analysis (VCA) [36] algorithm, since it is a robust and universal tool. This technique builds the observation vector into a convex cone, and the vertices of the convex cone are considered endmembers. Compared with other techniques, the VCA method is computationally inexpensive and efficient. In addition, the VCA method has a relatively good extraction accuracy and noise immunity. Following the endmember extraction, each patch image has its own endmember pool (U₁–U_n). Then, the spectra in each endmember pool are compared for classification. The defined rule is obviously the key factor in the procedure. Due to the explicit physical meaning, the spectral angle mapper (SAM) technique [37] is used to distinguish the similarity by calculating the angle through the following formula:

$$\alpha \left({\epsilon }_i,{\epsilon }_j\right)=\arccos (\frac{{\epsilon }_i^T{\epsilon }_j}{\sqrt{{\epsilon }_i^T{\epsilon }_i}\sqrt{{\epsilon }_j^T{\epsilon }_j}})$$

(1)

where ${\epsilon }_i$ and ${\epsilon }_j$ are the spectra of the two specified endmembers. The more similar the two spectra are, the smaller the angle is. If the angle of the two endmembers is smaller than the empirical threshold, they belong to the same type. Following the classification of the endmember spectra, a ground cover category corresponds to numerous endmember spectra. However, a large amount of redundant computation arises when there are many candidate endmember spectra. In particular, blocking easily caused many similar spectra to occur according to the same land cover type. In this case, the endmember average root mean square error (EAR) indicator is proposed to optimize the selection of the endmember spectra [38]. The average error in the endmember spectra of a ground cover type can be determined by EAR. The endmember spectra with a smaller EAR are the more representative endmember spectra of the ground cover type among several comparable spectra. Assuming a land cover type with m candidate endmember spectra $\left\{{\mathrm{E}}_1,{\mathrm{E}}_2,\dots ,{\mathrm{E}}_m\right\}$, the EAR of the ith endmember spectrum can be stated as follows:

$${\mathrm{EAR}}_i=\frac{1}{m}{\displaystyle \sum _{j=1}^m\mathrm{RMSE}({\mathrm{E}}_i,{\mathrm{E}}_j)}$$

(2)

where ${\mathrm{EAR}}_i$ represents the EAR of the ith endmember spectrum, and $\mathrm{RMSE}({\mathrm{E}}_i,{\mathrm{E}}_j)$ represents the average value of the root mean square error between ${\mathrm{E}}_i$ and ${\mathrm{E}}_j$. The lower the EAR value is, the more representative the spectra are. According to the calculated EAR value, some of the more representative spectra were selected to be the endmember spectra of the ground cover type. Through this method, the endmember spectra of the various ground cover categories are optimized and selected. Finally, the endmember pools of all the patch images are combined to obtain the final endmember pool (U). In this endmember pool, a land cover type has several representative endmember spectra.

The LMM is widely used to identify and quantify pure components in remotely sensed images due to its simple physical interpretation and trackable estimation process [39]. This model assumes that the reflectance measured within each pixel is a unique linear combination of the reflectance of each sub-pixel endmember, weighted by its abundance and some noise [40]. The spectral properties of mixed pixels are as follows:

$$x=\mathbf{E}\alpha +n$$

(3)

where x is the spectral vector value of the mixed pixel, E is the endmember matrix, $\alpha$ is the abundance column vector of each endmember, and n is the noise. Finally, the least squares approach is used to obtain the abundance $\alpha$ that is most appropriate. There are two crucial limitations in computing the abundance; the abundance must be nonnegative, and the sum must be one [41]. If the number of endmembers in the image is N, Formula (3) can be represented as:

$$x={\displaystyle \sum _{i=1}^N{s}_i{\alpha }_i}+n$$

(4)

where $s_i$ and ${\alpha }_i$ are the ith endmember spectrum and the corresponding abundance values, respectively. The mixed pixels are decomposed according to a specific endmember pool (U) obtained previously. The MSU model is revised based on Equation (4) and shown below:

$$x={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^{M_i}q{s}_{ij}{\alpha }_{ij}}+n}$$

(5)

where N is the number of the endmembers and $M_i$ is the number of spectra according to the ith land cover type. The values of $s_{ij}$ and ${\alpha }_{ij}$ are the jth endmember spectrum in the ith class type and the corresponding abundance value, respectively, q is the label (zero or one) representing whether the relevant endmember spectrum is used, and n is the noise. Like the LMM, the MSU has the following two constraints: $0\le {\alpha }_{ij}\le 1$ and ${\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^{M_i}{\alpha }_{ij}}=1}$. Finally, more reliable endmember abundance maps are obtained.

3.2. JM Algorithm for Information Fusion

After obtaining the abundance images, it can be seen that the traditional direct comparison approaches are not effective in extracting the change information. In addition, the detailed change information is not effectively provided if it was only based on the abundance images that were obtained through the MSU method. Therefore, we consider fusing the abundance image information with the original image feature information to provide more information on the land cover change. In this process, principal component analysis (PCA) [42] is used to extract the feature information of the original image. It can map the high-dimensional data to the low-dimensional space. In other words, the original features are projected onto the selected feature vector to obtain new low-dimensional feature information. This process removes redundant information and noise to obtain high-quality original image feature information and then normalizes it to limit the value to the range of zero to one. Subsequently, the JM algorithm is used to efficiently process the multi-source information simultaneously, fusing the sub-pixel level endmember abundance information with the original image feature information to fully explore the rich information contained in the HSI. By converting the corresponding two one-dimensional pixel vectors into a two-dimensional matrix, the change patterns between two corresponding spectral vectors on the spatial pixels are better explored. The flow chart is shown in Figure 3.

Assuming that the original image size is $I\times J\times k$, m bands are retained after the PCA processing. Suppose that the abundance image (F₁, F₂) has n bands. The abundance image information and its corresponding original image feature information are stacked in the spectral dimension to obtain two stacked images of size $I\times J\times (m+n)$. To properly characterize the local spatial patterns, a neighbourhood is used. For each stacked image, a neighbourhood of size $I_n\times J_n$ is divided with each pixel as the centre. Then, the JM algorithm is used to reshape and stack the corresponding neighbourhood image information to obtain tensor K. The tensor K contains the context information of each pixel, and the noise and misalignment artefacts can be suppressed or removed. This can be expressed with the following formula:

$$K_{ijpq}=(r_{ijp}^1-r_{ijq}^2)/\left|r_{ijq}^2\right|\hspace{1em}\hspace{1em}i=1,2,\cdots ,I_n;j=1,2,\cdots J_n;p,q=1,2,\cdots ,m+n$$

(6)

where $K_{ijpq}$ represents the value of row p and column q of the joint matrix generated by the pixel vector of row i and column j of the corresponding neighbourhood image; $r_{ijp}^1$ is the p-band value of the pixel vector of row i and column j in the T₁ image; and $r_{ijq}^2$ is the q-band value of the pixel vector of row i and column j in the T₂ image.

According to Equation (6), each pair of the corresponding one-dimensional pixel vectors is converted into a two-dimensional joint matrix $K_{ij}$ of size $(m+n)\times (m+n)$. In addition, each pair of corresponding neighbourhood images is converted into a joint matrix cluster K of size $(m+n)\times (m+n)\times (I_n\times J_n)$. As shown in Figure 3, each $K_{ij}$ contains three parts of information. In Part A, the abundance image of a certain endmember for one pixel in the T₁ image subtracts the abundance image of the n endmember corresponding to the pixel of the T₂ image. This part of the information represents the difference information at the sub-pixel level of the HSI, which is distributed in the upper left corner of the two-dimensional matrix. In Part B, a certain band of a pixel in the T₁ image subtracts m bands of pixels corresponding to the T₂ image. This part of the information represents the pixel level difference information of the HSI, which is distributed in the lower right corner of the two-dimensional matrix. Since the affinity between the endmember abundance information and original image feature information is meaningless, the remaining $K_{ij}$ is set to zero. The smaller the value of $K_{ij}$ is, the more similar it is between the corresponding pixels. In contrast, the greater the value of $K_{ij}$ is, the greater the likelihood for pixel change. Following the calculation of the joint matrix, K is obtained with a quantity of $I\times J$.

This whole process combines the endmember abundance information with the original HSI feature information. Furthermore, the JM algorithm maps the differences between the corresponding one-dimensional pixel vectors in the neighbourhood to a two-dimensional matrix, which can provide richer cross-channel gradient information and maximize the utilization of multi-source information. This is an efficient way to simultaneously process multi-source information.

3.3. CNN for Detecting Multiple Changes

To accurately extract the change features from the fused information, the CNN is introduced to improve the accuracy of the HSI-CD results [43]. The CNN is significant in computer vision. It maintains spatial information by using convolution, pooling, and activation functions and learns locally invariant spatial characteristics as well as nonlinear features. It is commonly utilized for image classification, target detection, change detection, and other image processing tasks. Furthermore, the CNN convolution layer with an activation function can be described through the following calculation:

$$\begin{array}{l}{F}_{i,j}^{l+1,k}=f({(W^{l+1,k}\ast F^l)}_{i,j}+b^{l+1,k})\\ =f({\displaystyle \sum _{m=1}^{s_{l+1}}{\displaystyle \sum _{n=1}^{s_{l+1}}{\displaystyle \sum _{p=1}^{s_{l+1}}{W}_{m,n,p}^{l+1,k}}}}\cdot F_{i+m-1,j+n-1,p}^l+b^{l+1,k})\end{array}$$

(7)

where $F_{i,j,p}^l$ is the characteristic graph p output by the pixel (i, j) of the lth layer, $W^{l+1,k}$ and $b^{l+1,k}$ are the kth convolution kernel and deviation of layer l + 1, respectively, $s_{l+1}$ is the size of the convolution kernel of layer l + 1, k is the number of characteristic graphs of this layer, and $f(\cdot )$ is the activation function, such as the sigmoid function (sigma) [44], hyperbolic tangent function (tanh) [45], and corrected linear function (relu) [46,47].

The overall network architecture is shown in Figure 4, and the specific parameters are shown in

Table 1. Specific details of network architecture.

Layers	Type	Channels	Kernel Size
Conv1	Convolution + BN Activation (relu)	16	3 × 3
Pool1	MaxPooling	-	2 × 2
Conv2	Convolution + BN Activation (relu)	32	3 × 3
Pool2	MaxPooling	-	2 × 2
Conv3	Convolution + BN Activation (relu)	64	3 × 3
Pool3	MaxPooling	-	2 × 2
Conv4	Convolution + BN Activation (relu)	128	3 × 3
Pool4	MaxPooling	-	2 × 2
FC1	Fully Connected + BN Activation (relu)	128	-
FC2	Fully Connected + BN Activation (softmax)	nchange + 1	-

. After convolution, a batch normalization (BN) is used to accelerate the training process, and the relu function is used for nonlinear activation. The previously obtained joint matrix clusters are fed into this neural network to integrate this rich multi-source information. Finally, the predicted type labels of the pixels are output to obtain the HSI multicategory CD result map.

To optimize the network model, the cross-entropy loss function is used as the loss function of the neural network to detect multicategory changes in the HSI. The cross-entropy loss function is as follows:

$$\mathrm{Loss}=-\frac{1}{M}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{c=1}^C{y}_c^m\log (p_c^m)}}$$

(8)

where y and p are real labels and predicted output labels, respectively, C is the number of categories, and M is the number of samples. The parameters in the model are updated by backpropagation and random gradient descent.

4. Experiment

In this section, we will first introduce the HSI datasets used in the experiment, then introduce the evaluation measures of the selected method, followed by the discussion of the results, and finally, give the computational cost analysis.

4.1. Dataset Description

To make the experiment more representative, we chose three datasets with large differences in style for the experiment. The simulated experimental dataset had a large number of change types but a strong similarity between the images, which was relatively simple. The first real experimental data had fewer types of changes and a more concentrated range of changes, and the main difficulty was reflected in the complex farm paths. The second real experimental dataset had poor image similarity, and the images had a complex environment with multiple scattered change types.

The first dataset is made up of an HSI acquired by the AVIRIS sensor in 1998 on Salinas Valley, California. The original image has 224 contiguous spectral bands with wavelengths from 400 to 2500 nm, which are characterized by a spatial resolution of 3.7 m and a spectral resolution of 10 nm. Ground truth data that contain 16 material classes (vegetation, bare soil, vineyard, etc.) are available. A subset of the whole image was selected, having a size of 100 × 217 pixels. In pre-processing, 20 water absorption bands (bands 108–112, 154–167, and 224) were discarded, resulting in 204 bands for the experiments. Taking this image as the T₁ image, the T₂ image was simulated based on the T₁ image. To obtain realistic simulation data, six different regions were extracted from the T₁ image and inserted back into different spatial positions on the T₁ image by replacing the whole spectral vector. Thus, the T₂ image was generated with six simulated change classes. To simulate the radiation measurement difference, Gaussian white noise (mean value = 0, variance = 0.001) was added. A pair of simulated images constructed from the original image was obtained. Figure 5a,b show the original T₁ image and simulated T₂ image with noise. The reference land cover change map in Figure 5c was generated from artificial exchange areas. Change 1 is shown in red, change 2 is shown in green, change 3 is shown in blue, change 4 is shown in yellow, change 5 is shown in magenta, change 6 is shown in cyan, and the unchanged is shown in black.

The second dataset, “farmland”, is from the Earth Observation-1 (EO-1) Hyperion images, as shown in Figure 6a,b. The dataset covers farmland near Yancheng City, Jiangsu Province, China, with a size of 450 × 140 pixels. The two images were acquired on 3 May 2006, and 23 April 2007, respectively. The spectral resolution is 10 nm, and the spatial resolution is 30 m. The original image has 242 bands. Some bands with a low signal-to-noise ratio (uncalibrated, overlapping, highly influenced by water vapour and noisy bands) are eliminated in the pre-processing process, reserving 155 bands for the experiments. Visually, the main change in the dataset is the size of farmland. The change reference image is shown in Figure 6c, change 1 is shown in red, change 2 is shown in green, and the unchanged is shown in black.

The third dataset is a pair of real bitemporal images acquired by a Hyperion sensor mounted on the EO-1 satellite on 1 May 2004, and 8 May 2007. The study area is an agricultural irrigated land of Umatilla County, Oregon, USA, which has a size of 390 × 200 pixels. The wavelength range of the image is 350~2580 nm, the spectral resolution is 10 nm, and the spatial resolution is 30 m. After pre-processing (repairing bad stripes, removing uncalibrated and noisy bands, atmospheric correction and registration), 159 bands of the original 242 bands (8–57, 82–119, 131–164, 182–184, and 187–220) were used for the CD experiment. Changes in this scenario mainly include the land-cover transitions between the crops, bare soil, and variations in soil moisture and water content of the vegetation. Figure 6d,e show the false-colour composite of the images. The change reference image is shown in Figure 6f. Change 1 is shown in red, change 2 is shown in green, change 3 is shown in blue, change 4 is shown in yellow, change 5 is shown in magenta, and the unchanged is shown in black.

4.2. Evaluation Measures

To assess the performance of the CD method, the CD map was compared to the real reference map. The precision of the computation results can directly represent a CD method’s dependability and practicability. In this research, the following four assessment indices were used: overall accuracy (OA); Kappa coefficient; precision and recall for each class.

When computing the assessment statistics, each category was considered “positive”, while the remaining categories were considered “negative”. The number of pixels that correctly predicted a positive class is the TP, while the number of pixels that correctly predicted a negative class is the TN. The FP represents the number of pixels that incorrectly predicted the negative class as positive, and the FN represents the number of pixels that incorrectly predicted the positive class as negative. The proportion of correctly classified pixels is calculated using the OA. The formula is as follows:

$$\mathrm{OA}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(9)

The Kappa coefficient measures the consistency of the CD result map and the actual surface map. In comparison to the OA, the kappa coefficient more objectively reflects the correctness of the CD result. The kappa coefficient is calculated as follows:

$$\mathrm{Kappa}=\frac{\mathrm{OA}-P}{1-P}$$

(10)

where the formula to calculate P is the following:

$$P=\frac{(\mathrm{TP}+\mathrm{FP})(\mathrm{TP}+\mathrm{FN})}{{(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})}^2}+\frac{(\mathrm{TN}+\mathrm{FN})(\mathrm{FP}+\mathrm{TN})}{{(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})}^2}$$

(11)

Precision is the proportion of the pixels in the predicted positive class that truly belong to the positive class, which demonstrates the prediction ability of the algorithm. The formula is as follows:

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(12)

Recall refers to the proportion of the positive class pixels correctly predicted in the total positive class pixels, which reflects the differentiation ability of the algorithm. The following formula calculates the recall:

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(13)

4.3. Results and Discussion

To prove the feasibility of this method (MSUJMC), experiments were performed on three different HSI datasets. Comparisons with seven methods, which are the pixel-level CD method based on machine learning (K-means, RF, SVM), sub-pixel level CD method based on spectral unmixing (SU, MSU), CNN comparison method with abundance maps as input (MSUC), and JM algorithm based on traditional spectral unmixing and CNN method (SUJMC), were made to evaluate the performance of the proposed methods. During the experiment, the neighbourhood size was set to 3 × 3 ($I_n\times J_n$). The batch size for the training was set to 64, the optimizer was Adam, the learning rate was 0.0005, and the decay was 10⁻³. Experiment was optimized for 300 epochs for the training samples, and the early stop method was used to avoid overfitting. The number of training samples was 10% of the total pixels. Each experiment is repeated five times and the average accuracy are reported.

4.3.1. Simulation Dataset

The MSUJMC method was used to perform the CD on the simulated images. In acquiring the endmember abundance information, the number of rows and columns was set to two, and the whole image was segmented into four patch images. In acquiring the original image feature information, 10 bands were retained after the PCA processing. Figure 7b–i show the CD results obtained by the different CD methods: K-means, RF, SVM, SU, MSU, MSUC, SUIMC and MSUJMC, respectively. The evaluation statistics of the simulated data are shown in

Table 2. Accuracy statistics of simulation dataset.

		K-means	RF	SVM	SU	MSU	MSUC	SUJMC	MSUJMC
	OA (%)	96.07	97.06	98.41	97.81	99.01	99.37	99.80	99.95
	Kappa	0.74	0.80	0.88	0.84	0.92	0.95	0.98	0.996
unchanged	Precision	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	Recall	0.96	0.97	0.98	0.98	0.99	0.99	1.00	1.00
change 1	Precision	0.80	1.00	1.00	0.86	1.00	0.95	1.00	1.00
	Recall	0.99	1.00	1.00	0.99	1.00	1.00	1.00	1.00
change 2	Precision	0.48	0.80	0.91	0.68	0.80	1.00	0.99	1.00
	Recall	0.91	1.00	1.00	0.98	0.95	1.00	1.00	1.00
change 3	Precision	0.54	0.80	0.88	0.66	0.72	0.95	1.00	1.00
	Recall	0.96	1.00	1.00	0.82	0.97	0.97	1.00	1.00
change 4	Precision	0.75	1.00	1.00	0.82	1.00	0.95	1.00	1.00
	Recall	0.99	1.00	1.00	1.00	0.99	1.00	1.00	1.00
change 5	Precision	0.21	0.10	0.17	0.23	0.41	0.76	1.00	1.00
	Recall	0.72	1.00	1.00	0.40	0.84	0.95	0.16	0.84
change 6	Precision	0.09	0.07	0.13	0.32	0.36	0.72	1.00	1.00
	Recall	0.44	1.00	1.00	0.72	1.00	0.75	0.12	0.76

.

Through the analysis of the evaluation statistics and visual comparison with the reference map, the result obtained by the K-means method is shown to be the worst. There are many misclassified isolated pixels caused by the wrong CD, and some of the change 5 and change 6 pixels were not detected. The RF and SVM methods are much better than K-means. The six change classes were almost completely detected, but a large number of unchanged pixels were detected as change 5 and change 6. Therefore, in the two experimental results, the precision statistics of these two change classes are very low, barely exceeding 0.1, and the performance of the SVM was better than that of the RF. From the OA and Kappa, we found that the SU method outperforms RF but is weaker than SVM. The MSU method is a significant improvement over the SU method but still not as good as the methods based on the CNN. The SUJMC and MSUJMC, which fuse the sub-pixel endmember abundance information and original image feature information, have better performance than the MSUC, which does not consider the original image feature information. In particular, since the MSUJMC method considers the endmember spectral variability, this method significantly improves the CD accuracy and has fewer false pixels. Although some pixels of change 5 and change 6 were not detected, the MSUJMC method has the best CD performance overall.

To explore the stability of the method under the influence of different levels of noise, we added two simulations for comparison. By increasing the value of the noise variance (variance = 0.003 and variance = 0.005) in the simulated experiments, the difference in the performance between the methods is more significantly represented. Figure 8a–d show the spectral curves at different noise levels. It can be seen that the higher the noise level, the greater the effect on the spectral curve, which greatly increases the difficulty of CD. By analysing the statistics of these three simulated experiments, we found that the CD results of the various methods were affected by interference to different degrees as the noise level increases. As shown in Figure 8e,f, the MSUJMC method is the least affected by the noise and the most resistant to interference. When the noise level increases, the OA decreases by only 0.18% and 0.25%, and the Kappa decreases by only 0.016 and 0.02.

4.3.2. Real HSI Dataset-1

In this experiment, when acquiring the endmember abundance information, the number of rows and columns was set to two, and the whole image was segmented into four patch images. When acquiring the original image feature information, 10 bands were retained after the PCA processing. The MSUJMC method result are shown in Figure 9i, and the results of the comparison methods (K-means, RF, SVM, SU, MSU, MSUC, SUIMC) are shown in Figure 9b–h, respectively. The evaluation statistics are shown in

Table 3. Accuracy statistics of the real HSI dataset-1.

		K-means	RF	SVM	SU	MSU	MSUC	SUJMC	MSUJMC
	OA (%)	89.33	85.99	88.84	88.31	94.60	96.73	97.36	98.63
	Kappa	0.74	0.73	0.78	0.76	0.89	0.93	0.95	0.97
unchanged	Precision	0.89	1.00	1.00	0.97	0.99	0.98	0.98	0.99
	Recall	0.98	0.86	0.87	0.86	0.93	0.97	0.98	0.99
change 1	Precision	0.86	0.52	0.60	0.79	0.77	0.97	0.95	0.99
	Recall	0.97	0.94	0.94	0.96	0.98	0.91	0.91	0.95
change 2	Precision	0.99	0.78	0.79	0.70	0.91	0.92	0.97	0.98
	Recall	0.53	0.83	0.92	0.94	0.99	0.99	0.97	0.98

.

By comprehensively analysing the CD result maps of the various methods and their corresponding evaluation statistics, we found that the K-means method generated the least accurate result, with a large number of undetected change 2 pixels and a recall statistic of 0.53. In the resulting maps obtained by the RF, SVM and SU methods, a large number of roads are misidentified as change regions. The performance of the SVM and SU methods was similar, and both outperformed the RF method. Compared with previous methods, the result obtained by the MSU method show much visual and statistical improvement. Due to the strong learning capability of the CNN, the methods based on the CNN are far superior than the traditional methods. Among them, the SUJMC method and the MSUJMC method fused multi-source information and outperformed the MSUC method, which only considers the sub-pixel level endmember abundance information. However, the endmember abundance information used by the SUJMC method does not consider the variability of the endmember spectra, only one candidate spectrum for each endmember type is considered. This leads to the SUJMC method being less reliable than the MSUJMC method. It is also clear from the resulting maps and statistics that the MSUJMC is a very effective way to handle the HSI-CD tasks. All the evaluation statistics are above 0.95, and most of them are above 0.98, indicating that the resulting maps obtained by the MSUJMC method are generally consistent with the ground truth maps.

4.3.3. Real HSI Dataset-2

This dataset is complex compared to the previous datasets. Experiments were carried out by using the proposed MSUJMC method. In acquiring the endmember abundance information, the numbers of rows and columns are set to three and two, respectively, and the whole image is divided into six patch images. When acquiring the original image feature information, 15 bands were retained after the PCA processing. The MSUJMC method results are shown in Figure 10i. Figure 10b–h show the CD results obtained by the comparison of the following CD methods: K-means, RF, SVM, SU, MSU, MSUC, SUIMC, respectively. The evaluation statistics are shown in

Table 4. Accuracy statistics of real HSI dataset-2.

		K-means	RF	SVM	SU	MSU	MSUC	SUJMC	MSUJMC
	OA (%)	94.16	94.94	96.89	95.26	96.69	97.47	98.45	98.89
	Kappa	0.75	0.80	0.86	0.80	0.86	0.88	0.93	0.95
unchanged	Precision	0.99	0.99	0.99	0.99	0.99	0.99	0.99	1.00
	Recall	0.99	0.97	0.99	0.99	0.99	0.99	0.99	0.99
change 1	Precision	0.90	0.96	0.89	0.98	0.95	0.90	0.96	0.97
	Recall	0.65	0.92	0.92	0.71	0.86	0.93	0.96	0.96
change 2	Precision	0.31	0.39	0.84	0.48	0.66	0.92	0.93	0.93
	Recall	0.66	0.89	0.88	0.92	0.86	0.60	0.95	0.95
change 3	Precision	0.00	0.98	1.00	0.16	0.18	0.86	0.70	0.84
	Recall	0.00	0.59	0.58	0.82	0.81	0.85	0.90	0.85
change 4	Precision	0.54	0.53	0.60	0.58	0.72	0.91	0.89	0.90
	Recall	0.97	0.98	0.98	0.71	0.70	0.77	0.89	0.95
change 5	Precision	0.39	0.95	0.96	0.65	0.69	0.76	0.80	0.90
	Recall	0.23	0.36	0.36	0.51	0.73	0.90	0.86	0.91

.

We found that the result of the K-means method was significantly different from the real surface of the reference map. Although most of the change areas were detected, the various change classes were mixed together, resulting in many misjudgements. In particular, change 3 was not detected, leading to the precision and recall statistics of zero for this change type. Compared with the K-means method, the RF and SVM methods can provide more reliable and clearer information between change classes. The discernibility between the change classes is strong, but there are many unchanged pixels that are detected as changed pixels. The overall performance of the SU and MSU methods is similar to that of the RF and SVM. In the obtained results, the unchanged pixels that are detected as change pixels are less than those detected in the RF and SVM. However, some change 4 pixels were mistaken for change 5, resulting in lower precision and recall statistics for both change types. Moreover, in the result of the SU method, the pixels of change 1 and change 2 are mixed and difficult to distinguish. The MSUC, SUJMC and MSUJMC methods based on the CNN show higher performance than the traditional CD methods. Only a small number of unchanged pixels are detected as change pixels in the results of these three methods. The detection ability of the MSUC method for change 4 is slightly inferior, and the SUJMC method is less capable of detecting change 3. The MSUJMC method has the highest comprehensive performance both from the resulting map and from the evaluation statistics. The result of the MSUJMC method is the most consistent with the real reference map, and only very few pixels are incorrectly detected. The precision and recall statistics for each change type are very high. The OA and Kappa are also the highest among all experimental methods.

4.4. Computational Cost Analysis

The hardware and software parameters were as follows: Lenovo Desktops, AMD Ryzen 7 5800H with Radeon Graphics 3.20 GHz, 16.0 GB RAM, NVIDIA GeForce RTX 3050Ti 4 GB GPU, Windows 10, python3.8. The average computational costs of the three datasets are shown in

Table 5. Computational cost of the three datasets.

Time (s)		K-means	RF	SVM	SU	MSU	MSUC	SUJMC	MSUJMC
simulation dataset	v = 0.001	7.46	4.37	5.68	8.06	12.75	20.95	37.12	41.98
	v = 0.003	8.47	5.16	6.23	8.84	13.64	22.15	38.96	43.78
	v = 0.005	9.85	6.53	7.42	9.86	14.83	23.95	40.31	45.26
real dataset-1		9.83	6.94	7.85	10.21	15.36	24.66	41.08	46.24
real dataset-2		15.23	11.02	12.82	18.32	26.38	46.76	67.25	74.86

. We can see that among these methods, machine learning based methods (K-means, RF, SVM) are less time consuming. The reason for this is due to the fact that these methods only extract shallow features during training. The K-means method has the highest time cost of the three methods, while RF and SVM perform much better in comparison, and RF has the lowest computational time cost. Spectral unmixing based methods (SU, MSU) are somewhat more costly in terms of computational time than machine learning based methods. The reason for the higher computation time cost of the MSU method, compared to the SU method, is analysed mainly as the MSU method adopts a patching strategy, dividing the original image into multiple patch images to expand and optimize the endmember pool, which therefore takes more time to complete. The time costs of deep learning-based methods (MSUC, SUJMC, MSUJMC) increases abruptly, mainly as neural networks need more training time to extract deep features. Among these three deep learning-based methods, the MSUC method has the lowest time cost, mainly as the amount of data input to its network is much smaller. The difference in computational time costs between the SUJMC method and the MSUJMC method depends primarily on the time difference in data processing prior to network input. In the simulation experiments, the time consumption increases with increasing noise levels, but not significantly. In summary, the MSUJMC method requires the highest cost in terms of computational time, but is still acceptable.

5. Conclusions

In this paper, the MSUJMC method is proposed to solve the challenging problem of detecting multiple changes in multitemporal HSIs. The main contributions of the method are reflected in the following three aspects: (1) The MSU method provides sub-pixel endmember abundance information for the CD. In this process, the variability of the endmember spectra is fully considered so that the obtained endmember abundance information is more accurate and reliable; (2) The JM algorithm fuses the sub-pixel level endmember abundance information and original image feature information into lightweight matrices to make full use of the valid information in the HSI to determine the type of pixel change; (3) The CNN are used to fully exploit a series of change features in the joint matrix clusters to better accomplish the task of detecting multiple changes in HSI.

Although the proposed method consumes more time than the comparative methods in this paper, the time costs were still acceptable. Experimental results on the simulated and real HSI datasets verify the effectiveness of the method. The following conclusions can be drawn from the analysis of the theoretical and experimental results:

(1) A consideration of the variability in the endmember spectra should be made when obtaining image endmember abundance information. Dividing the original image into multiple patch images according to the complexity of the image enables the selection of a large number of possible candidate endmember spectra. The redundant candidate endmember spectra can be removed by the EAR index optimization. Finally, the mixed pixels of the whole image can be unmixed using the MSU method and the final endmember pool to obtain reliable sub-pixel level endmember abundance information.

(2) In the process of fusing the multi-source information, the JM algorithm also considers the neighbouring pixels around each pixel that jointly participate in the decision on the type of change of the central pixel. The one-dimensional pixel information is converted into two-dimensional matrix information, providing richer cross-channel gradient information. Then, a maximization of the use of the valid information in the HSI can be performed.

(3) The powerful learning ability of the CNN can work seamlessly with the joint matrix clusters. It can identify the change features and integrate the multi-source information. Therefore, the task of detecting multiple changes in the HSIs can be performed efficiently.

Considering that the HSI dataset with the real change reference map is relatively scarce, the real change reference maps have a high production cost. In order to achieve high accuracy CD on the HSI dataset without a real change reference map, in future work we will explore the application of the transfer learning and remote sensing images from different data sources on the HSI-CD.