Self-Paced Multi-Scale Joint Feature Mapper for Multi-Objective Change Detection in Heterogeneous Images

Abstract

Heterogeneous image change detection is a very practical and challenging task because the data in the original image have a large distribution difference and the labeled samples of the remote sensing image are usually very few. In this study, we focus on solving the issue of comparing heterogeneous images without supervision. This paper first designs a self-paced multi-scale joint feature mapper (SMJFM) for the mapping of heterogeneous data to similar feature spaces for comparison and incorporates a self-paced learning strategy to weaken the mapper’s capture of non-consistent information. Then, the difference information in the output of the mapper is evaluated from two perspectives, namely noise robustness and detail preservation effectiveness; then, the change detection problem is modeled as a multi-objective optimization problem. We decompose this multi-objective optimization problem into several scalar optimization subproblems with different weights, and use particle swarm optimization to optimize these subproblems. Finally, the robust evaluation strategy is used to fuse the multi-scale change information to obtain a high-precision binary change map. Compared with previous methods, the proposed SMJFM framework has the following three main advantages: First, the unsupervised design alleviates the dilemma of few labels in remote sensing images. Secondly, the introduction of self-paced learning enhances SMJFM’s capture of the unchanged region mapping relationship between heterogeneous images. Finally, the multi-scale change information fusion strategy enhances the robustness of the framework to outliers in the original data.

1. Introduction

The process of determining a difference in an object or phenomenon state via observation at various times is known as remote sensing image change detection [1,2,3,4]. Depending on whether or not the image acquisition sources are the same, it is separated into the following two categories: homogeneous image change detection and heterogeneous image change detection. For the study of urban growth, agriculture, forest resources, and disaster assistance, remote sensing image change detection has a strong practical significance [5,6,7,8,9,10,11]. The remote sensing images captured by multiple sensors have huge distribution differences. Therefore, heterogeneous remote sensing images will differ greatly, even within the same scene. For homogeneous images, numerous change detection techniques have been presented under this paradigm [12,13,14,15,16,17,18]. However, because heterogeneous images have different features, applying the homogeneous change detection approach to them directly will not produce adequate results.

Optical and SAR images are the two common types of remote sensing images that we concentrate on in our work. An optical sensor converts the optical signal into an electrical signal, thereby generating an optical image. Compared to optical systems, SAR systems operate on distinct principles. In order to detect objects on the ground, an SAR sensor actively emits radio waves, hears echoes, and uses the echoes it receives to create SAR images. While the forms of the objects and the lighting conditions affect how they seem in optical images, the key determinant of object appearance in SAR images is the item’s reflection qualities [19,20,21]. Optical and SAR images typically exhibit distinct visual properties, making heterogeneous remote sensing image change detection highly valuable in practice. However, the significant differences in data distribution between these two types of remote sensing images pose considerable challenges to this endeavor.

This paper introduces an unsupervised deep learning model, SMJFM, for change detection. SMJFM is based on a multi-scale joint feature mapper (MJFM) with an autoencoder architecture and uses a self-paced learning strategy to assist in optimization, then uses a multi-objective evolutionary algorithm (MOEA) to post-process the model output results. Initially, the approach treats each pixel, along with its surrounding area, as a patch. We define two sizes for the neighborhood—one large and one small—resulting in image patches of varying scales. Correspondingly, two joint feature mappers (JFMs) are constructed to map heterogeneous image patches to a more consistent feature space for feature-level comparison, and the difference information between heterogeneous images is obtained. Usually, the difference information obtained using smaller image patches retains more original image details but introduces more noise information, and the difference information obtained using larger image patches has less noise, but the image is smoother and has less detail information. To balance between noise robustness and detail retention effectiveness, this study incorporates multi-objective optimization to address the challenge. Multi-objective evolutionary algorithms offer the advantage of producing a collection of non-dominated Pareto solutions in a single iteration [22,23]. Finally, the robust evaluation strategy is used to fuse the change information corresponding to multiple Pareto solutions to obtain a high-precision binary change map. Compared with previous threshold methods or clustering methods to deal with difference maps, the robust evaluation strategy fuses the difference map information under different conditions, which improves the robustness of the framework to noise or abnormal data in the original data space.

Considering a pair of heterogeneous images, in order to make the difference map obtained by the joint feature mapper better reflect the difference information of the heterogeneous images, we hope that the JFM focuses on the mapping relationships in the data of the unchanged region and weakly captures the mapping relationships in the data of the changed region. Therefore, in this paper, we introduce a self-paced learning (SPL) strategy to train the JFM. SPL learns training samples from easy to difficult [24]. In heterogeneous remote sensing images, there are usually more unchanged regions than changing regions, and the JFM can usually capture the mapping relationship between unchanged regions more easily, while SPL automatically assigns larger weights to these samples. As for the data of the changed region, its mapping relationship usually contradicts the data of the unchanged region, and self-paced learning assigns smaller weights to these samples, which makes the JFM learn the data mapping relationship between unchanged regions more adequately.

2. Background and Motivation

In this section, we begin with the definition of a remote sensing change detection problem and introduce related work in the field. Then, we introduce the basics of evolutionary multi-objective optimization and some basics of self-paced learning.

2.1. Problem Definition

The aim of heterogeneous image change detection is to detect the change in the same geographical area in two remote sensing images registered at different times. Note that our approach uses two coregistered images, and modest coregistration mistakes would have less of an effect on our approach’s speed because we employ a patch to represent a rather error-tolerant pixel.

2.2. Related Work

Three categories can be distinguished by the numerous research topics and approaches that have been expressed for heterogeneous remote sensing image change detection [22,23,24]. First, several classification-based techniques, including those found in [25,26,27,28,29], such as post-classification comparison (PCC) [30] and classified adversarial networks (CANs) [31], can be used to handle heterogeneous change detection. PCC is a workable method for detecting changes in heterogeneous images. Two heterogeneous photos are segmented according to the different classes of ground objects, and the segmentation result is then used to obtain the change detection result. Such methods are usually simple and practical, but their accuracy is limited by the performance of the classification algorithm and will be affected by the accumulation of classification errors. A CAN employs adversarial training to explore the relationship between heterogeneous data and their corresponding labels. Following training, the heterogeneous data are fed into the generator to generate the final change map. The change detection accuracy of this method is also affected by the classification accuracy, and the training of the generator is a tedious process. In [32], Lv et al. identified potential samples by evaluating the similarity between the initial sample and its neighboring blocks. These potential samples were then used to train the neural network and predict the intermediate results. These operations were integrated through an iterative algorithm to facilitate the training and prediction of the neural network. However, this method involves a training sample enhancement strategy, which is highly dependent on the quality and representativness of the initial samples. If the initial sample is improperly selected, it may affect the entire iterative process. In [33], Li et al. proposed the SSPCN framework by combining adaptive step learning and convolutional networks; the method dynamically selects reliable samples and learns the relationship between two heterogeneous images. Pseudo-labels are initialized using a classification-based approach, and weights are assigned to each sample to represent its tractability. Subsequently, self-paced learning (SPL) prioritizes the learning of easier samples initially, gradually incorporating more complex ones over time. The introduction of the SPL strategy makes this method able to alleviate the phenomenon whereby the accuracy of change detection decreases due to the insufficient performance of classification algorithm. In [34], in order to solve the problem of false positives in heterogeneous image change detection, Xu et al. proposed a change detection method (CGCD) based on a convolutional neural network (CNN) and graph convolutional network (GCN). This method uses a CNN to learn the feature maps of multi-temporal images and calculate the difference maps of different scales so that low-level and high-level features contribute equally to change detection, then uses a GCN to classify the difference maps to obtain the change maps. However, this method uses pseudo-label samples for unsupervised training, and its performance may depend on the quality and accuracy of the pseudo-label samples. Second, some techniques, such those found in [35,36,37,38,39,40] and a symmetric convolutional coupling network (SCCN) [11], combine the two heterogeneous pictures into a similar feature space. Two heterogeneous images are mapped to a feature space using an SCCN, which then compares the images and creates a difference map. The accuracy of the results is dependent on the percentage of unchanged regions in the photos, and the process is unsupervised. An SCCN largely depends on the definition and optimization of the coupling function. If the coupling function does not capture the changes between images well, it may lead to performance degradation. In order to learn local and high-level features from a given pixel’s local neighborhood in an unsupervised manner, SMC+FCA [40] employs a denoising autoencoder. Afterward, the combination of feature similarity analysis and difference map building allows SMC+FCA to determine the change detection result by segmenting the final change map. This method relies on reliable training examples selected from the rough initial change map (ICM). If the quality of the ICM is not high, it may affect the accuracy of the mapping function and the final change detection results. In [41], Wu et al. proposed CACD, which integrates heterogeneous information into feature vectors for comparison and has a certain degree of robustness to noise in remote sensing data. The deep network structure in CACD is simple and easy to train, and the whole process does not contain any special noise reduction algorithm, which simplifies the optimization process of the algorithm. In [42], Xing et al. performed progressive modal alignment (PMA) on heterogeneous remote sensing images to improve the detection accuracy. A pseudo-label self-learning strategy was designed, and the pseudo-label generated in the modal alignment process was used to guide the detection of the change area. The detection results, in turn, refine the pseudo-label. However, this method relies on initialization. In the preheating training phase, the generation of the initial change map has an important impact on the final performance of the model. If the quality of the initial change map is not high, it may affect the convergence of the model and the final detection results. Third, some approaches model the interdependence between two heterogeneous images or the combined statistics using parametric models such as Markov random field (MRF) [43]. MRF relies on an initial iterative estimation method to estimate the parameters of the Markovian mixture model while considering the diversity of laws in the distribution mixture. Subsequently, the change detection map’s maximum a posteriori solution is calculated based on the determined parameters. However, this method is based on some assumptions, such as conditional independent pixel pair data and a specific likelihood distribution. These assumptions may not always hold in practical applications. PCC, SCCN, CAN, CACD, MRF, CGCD, and PMA will be compared with our approach.

2.3. Evolutionary Multi-Objective Optimization

A multi-objective optimization problem (MOP) can be expressed as follows:

$$\begin{array}{cc}\hfill \min \hfill & F(x)={(f_1(x),\dots ,f_n(x))}^T\hfill \\ & \mathrm{subject}\mathrm{to}x\in \Omega \hfill \end{array}$$

(1)

where x is the solution of the MOP, $R^m$ is the target space, $F:\Omega \to R^m$ maps the solution x to the target space, and $\Omega$ is the feasible space. The objectives in an MOP are usually conflicting, which means that it is impossible for any point in the feasible space to minimize all objectives at the same time. For minimization, if and only if one solution ($x_1$) is better than another solution ($x_2$), the following holds:

$$\begin{array}{c}\hfill \forall i=1,2,\dots ,n{f}_i(x_1)\le f_i(x_2)\wedge \exists j=1,2,\dots ,n{f}_j(x_1)<f_j(x_2)\end{array}$$

(2)

When there is no solution dominating $x^{*}$ in the feasible space ($\Omega$), $x^{*}$ is called the Pareto-optimal solution. We make a series of $F(x^{*})$ composed of $x^{*}$ become the Pareto-optimal vector. The objectives within the Pareto-optimal vector exhibit the following trade-off relationship: decreasing one objective will result in an increase in another objective. All points in the Pareto-optimal set adhere to this principle, collectively forming what is known as the Pareto-optimal frontier (PF) [44].

Evolutionary algorithms have found successful applications in areas such as privacy protection [45], neural architecture search [46], block sparse recovery and reconstruction [47], remote sensing image processing [48], and self-paced learning [49]. Many studies have employed evolutionary algorithms (EAs) to tackle multi-objective problems due to their ability to handle multiple potential solutions simultaneously.

2.4. Self-Paced Learning

Taking inspiration from the human learning process, Bengio et al. introduced the concept of curriculum learning [24]. The intuition comes from the human learning process, and the data are sent into the model in order of difficulty according to prior knowledge. As an improvement of course learning, SPL updates the course through the maturity of the model. To date, SPL has been maturely used in many fields, such as matrix decomposition and consistency detection [50,51,52,53,54,55,56,57,58]. To mitigate the issue of encountering undesirable local minima in matrix decomposition amidst noise and missing data, Zhao et al. tackled the problem by utilizing SPL [57]. Meanwhile, Zhang et al. integrated the multi-instance learning (MIL) problem into an SPL-based approach termed SP-MIL, demonstrating successful application in cosaliency detection [58].

In this article, the training set is $\mathcal{D}=\{({\mathit{x}}_{sar}^1,{\mathit{x}}_{opt}^1),\dots ,({\mathit{x}}_{sar}^N,{\mathit{x}}_{opt}^N)\}$, where ${\mathit{x}}_{sar}^n,{\mathit{x}}_{opt}^n\in {\mathbb{R}}^m$ denotes the nth sample pair. $\mathit{g}({\mathit{x}}_{sar}^n,{\mathit{x}}_{opt}^n;\mathit{w})$ is the JFM, where $\mathit{w}$ is the model parameter. Let $L_n$ reflect the loss of the nth sample pair. In SPL, $\mathit{v}={[v_1,v_2,v_3,\dots ,v_N]}^T$ is used to represent the difficulty of the sample, and the model parameters and weight vectors are optimized by Equation (3).

$$\underset{\mathit{w},\mathit{v}}{\min }\mathbb{E}(\mathit{w},\mathit{v};\lambda )=\sum _{n=1}^N{v}_n·L_{JFM}(\mathit{g}({\mathit{x}}_{sar}^n,{\mathit{x}}_{opt}^n;\mathit{w}))+f(\mathit{v};\lambda )$$

(3)

The speed parameter ($\lambda$) governs the pace of sample learning, and it is closely linked to the sample’s loss and weight. However, in the absence of any prior knowledge, the determination of the velocity parameter ($\lambda$) is a difficult problem. Usually, the aim of a robust parameter ($\lambda$) determination method is to sort the loss of the participating training samples in ascending order and determine $\lambda =L_{sort}^{N_s}(0<N_s\le N)$, where $N_s$ denotes the number of samples used in training and $\lambda$ can be conveniently controlled by adjusting the $N_s$ parameter. The self-paced function ($f(\mathit{v};\lambda )$) profoundly affects the sample weight, and researchers made a variety of variants of the original self-paced function to cope with diverse tasks [54,55].

In the original SPL hard weighting strategy [59], the weight can only be 0 or 1, and the optimal weight is calculated as follows:

$$v_n^{*}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{L}_n<\lambda \hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

According to (4), samples with a loss of less than $\lambda$ are considered simple samples for selection. Thus, the speed parameter ($\lambda$) governs the quantity of training samples. When $\lambda$ is small, the model is relatively “young”, so easy samples are selected for learning. When $\lambda$ is large, the model is relatively “mature”. At this time, complex samples are included to promote the attainment of more comprehensive knowledge by the model. It can be easily imagined that when the deep model is trained using a data set with a small number of false labels, when the value of $\lambda$ is small, the number of samples in the training queue is small and the sample loss is small, which means that the samples in the training queue are more reliable (i.e., the samples with correct labels). However, due to the small sample size, the model can easily overfit the data in the training queue to reduce the generalization performance. When the value of $\lambda$ is large, the data included in the training queue are more complex and diverse, and the mislabeled samples will be included, which will seriously endanger the performance of the deep model. In the SPL process, $\lambda$ changes from small to large, and the training queue expands accordingly, which ensures the good performance of the model on reliable samples while taking into account the generalization of the model and suppresses the negative impact of mislabeled samples on model performance.

3. Methodology

In this section, we begin by outlining the specifics of SMJFM and elaborating on the loss function of JFM. Then, we introduce the optimization details of JFM after introducing self-paced learning. Finally, we describe the robust change map evaluation strategy based on Pareto-optimal solutions. The overall algorithm framework is shown in Algorithm 1, and Figure 1 shows the overall framework of SMJFM.

Algorithm 1 Overall framework of SMJFM

Input:  $N_l$: large neighborhood size, $N_s$: small neighborhood size Output:  Binary change map.  1: Step (1) Difference map calculation  2: Construct multi-scale heterogeneous image patches train set $D_s=\{({\mathit{x}}_{sa{r}_i}^s,{\mathit{x}}_{op{t}_i}^s),\dots ,({\mathit{x}}_{sa{r}_N}^s,{\mathit{x}}_{op{t}_N}^s)\}$ and $D_l=\{({\mathit{x}}_{sa{r}_i}^l,{\mathit{x}}_{op{t}_i}^l),\dots ,({\mathit{x}}_{sa{r}_N}^l,{\mathit{x}}_{op{t}_N}^l)\}$ based on $N_s$ and $N_l$.  3: Initialization: Initialize two JFMs with the spatial $D_s$ and $D_t$  4: while  not converge  do  5:     while not converge do  6:         Calculate each heterogeneous patch pair loss $L_n$.  7:         Majorization Step:  8:         Calculate the weight v of each patch pair by Equation (13)  9:         Minimization Step: 10:         Optimize the parameters of MJFM Train the MJFM by Equation (16) 11:     end while 12:     Increase $\lambda$. 13: end while 14: Use the trained MJFM to calculate the difference map $D_1$ and $D_2$ with more complete details and the difference map with smoother details. 15: Step (2) Compute change map with MOPSO 16: Solve a set of Pareto solutions according to Algorithm 2. 17: Robustly evaluation the change map corresponding to each Pareto-optimal solution to obtain a more accurate change map by Equation (25). return the change map.

Algorithm 2 Algorithm for MOPSO in remote sensing image change detection

Input:  N: neighborhood size, m: the fuzzy exponent, maxgen: max generation number, pop: the swarm size. Output:  Pareto front solutions.  1: Stage (1) State Initialization  2: Stage (1.1) Position and Velocity initialization: $P={\{{\mathit{z}}_{\mathbf{1}},{\mathit{z}}_{\mathbf{2}},\dots ,{\mathit{z}}_{\mathbf{pop}}\}}^T$ and $O={\{{\mathit{o}}_{\mathbf{1}},{\mathit{o}}_{\mathbf{2}},\dots ,{\mathit{o}}_{\mathbf{pop}}\}}^T$  3: Stage (1.2) Generate evenly distributed weight vectors $\mathit{\mu }={\{{\mu }_1,{\mu }_2,\dots ,{\mu }_{pop}\}}^T$  4: Stage (1.3) Calculate Personal bests and globals bests: Set $pbes{t}_i=gbes{t}_i=z_i$ for each $i=1,2,\dots ,pop$.  5: Stage (1.4) Initialize neighborhood $N={\{n_1,n_2,\dots ,n_{pop}\}}^T$  6: Stage (1.5) Calculate the fuzzy membership matrix $U=\{u_1,u_2,\dots ,u_{pop}\}$ by Equation (24) with m, $\mu$ and P.  7: Stage (2) set $t=0$.  8: Stage (3) Iteration  9: for  $i=1,2,\dots ,pop$  do 10:     (3.1) Randomly choose one particle from the neighbors as the global best particle. 11:     (3.2) Compute the updated velocity ${\mathbf{o}}_i^{t+1}$ for the ith particle using Equation (19) and determine the new position ${\mathbf{z}}_i^{t+1}$ for the same particle using Equation (20). 12:     (3.3) Compute the updated fuzzy membership matrix $u_i^{t+1}$ using Equation (24) with the updated cluster centers ${\mathbf{z}}_i^{t+1}$ 13:     (3.4) Evaluation of ${\mathbf{z}}_i^{t+1}$ 14:     (3.5) Update neighborhood solutions: for each particle $j(j=1,2,\dots ,N)$ within the neighborhood of particle i particle, if $g^{ws}({\mathbf{z}}_i^{t+1}|w^j)<g^{ws}({\mathbf{z}}_i^t|w^j)$, then set ${\mathbf{z}}_i^t={\mathbf{z}}_i^{t+1}$, $F({\mathbf{z}}_i^t)=F({\mathbf{z}}_i^{t+1})$ and $u_i^t=u_i^{t+1}$. 15:     Stage (3.6) Update personal best position $pbes{t}_i$. if $g^{ws}({\mathbf{z}}_{i+1}^t|{\mu }^i)<g^{ws}(pbes{t}_i^t|{\mu }^i)$, then set $pbes{t}_i={\mathbf{z}}_{i+1}$. 16: end for 17: Stage (4) Stopping condition: If $t<maxgen$, increment t by 1 and proceed to step 3; otherwise, terminate the algorithm and generate the output.

3.1. Multi-Scale Joint Feature Mapper

The joint feature mapper (JFM) is designed based on the architecture of the autoencoder, and the multi-scale is reflected in the size of the input image patch. The specific details of MJFM are shown in Figure 2. MJFM consists of two JFMs. The upper JFM inputs a small-scale patch to compare the difference information of heterogeneous image blocks in more detail. The lower JFM inputs a larger-scale patch to generate a smoother difference map.

Each JFM consists of two autoencoders tasked with extracting features from SAR and optical image patches, respectively. The autoencoder maps each image patch to the high-level representation space, and the reconstruction error constraint of the high-level feature vector contains the texture information and spatial information of each image patch.

$$L_{rec}(\mathit{x},\widehat{\mathit{x}})=\left|\right|\mathit{x}-\widehat{\mathit{x}}{\left|\right|}_2^2=\sum _{m=1}^{H_p}\sum _{n=1}^{W_p}{(x^{m,n}-{\widehat{x}}^{m,n})}^2$$

(5)

where $\mathit{x}$ denotes the original image patch, and $\widehat{\mathit{x}}$ stands for the patch reconstructed by the autoencoder, with $H_p$ and $W_p$ representing the patch size.

Each side of the autoencoder compresses the image patch to a more compact feature vector, and the mapper maps the feature vector on one side to the feature space similar to the other side through feature-level reconstruction constraints. This study maps the features of the SAR image patch.

$$\begin{array}{c}\hfill L_{com}(M_{{\mathit{\theta }}_M}({\mathit{f}}_{sar}),{\mathit{f}}_{opt})=\sum _{i=1}^k{({\widehat{f}}_{sar}^i-f_{opt}^i)}^2\end{array}$$

(6)

where $M_{{\mathit{\theta }}_M}$ is the mapper, and ${\widehat{\mathit{f}}}_{sar}$ is the feature vector mapped by the mapper.

In summary, the objective function of each JFM is expressed as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{\theta }}{argmin}{L}_{JFM}& ={\mathbb{E}}_{({\mathit{x}}_{sar},{\mathit{x}}_{opt})\sim D}[L_{rec}^{sar}+L_{rec}^{opt}+L_{com}]\hfill \\ \hfill & =\frac{1}{H\times W}·\sum _{i=1}^H\sum _{j=1}^W(\sum _{m=1}^{H_p}\sum _{n=1}^{W_p}{(x_{sa{r}_{i,j}}^{m,n}-{\widehat{x}}_{sa{r}_{i,j}}^{m,n})}^2\hfill \\ \hfill & +\sum _{m=1}^{H_p}\sum _{n=1}^{W_p}{(x_{op{t}_{i,j}}^{m,n}-{\widehat{x}}_{op{t}_{i,j}}^{m,n})}^2+\sum _{k=1}^K{({\widehat{f}}_{sa{r}_{i,j}}^k-f_{op{t}_{i,j}}^k)}^2)\hfill \end{array}$$

(7)

where H and W represent the size of the original remote sensing image, while $H_p$ and $W_p$ denote the size of the image patch. ${\mathit{x}}_{sa{r}_{i,j}}$ denotes the patch consisting of pixels in the ith row and jth column of the SAR image and its neighborhood. Other variables can be derived similarly.

MJFM consists of two JFMs with the same optimization goal and different input patch sizes.

3.2. MJFM with Self-Paced Learning (SMJFM)

In remote sensing image change detection, the count of pixels in the unchanged region typically surpasses the count in the changed region by a significant margin. In addition, the mapping relationship between pixels in the unchanged area is more consistent, and the JFM can more easily capture the mapping relationship of the unchanged area. For the changed region, the mapping relationship between pixels is usually contradictory to the mapping relationship of pixels in the unchanged region and is more complex and diverse. Based on this assumption, this paper integrates self-paced learning into the optimization process of MJFM so that MJFM pays more attention to the mapping relationship of the unchanged region so as to obtain a more significant difference map.

SPL learns samples step by step in the order of learning difficulty (i.e., sample loss). Samples with smaller weights almost do not affect the performance of the network. After introducing SPL into MJFM, the optimization objective is updated as follows:

$$\begin{array}{cc}\hfill & \underset{\mathit{w},\mathit{v}}{\min }\mathbb{E}(\mathit{w},\mathit{v};\lambda )\hfill \\ \hfill & =\sum _{n=1}^N{v}_n·L_{JFM}(\mathit{g}({\mathit{x}}_{sa{r}_n},{\mathit{x}}_{op{t}_n};\mathit{w}))+f(\mathit{v};\lambda )\hfill \end{array}$$

(8)

where N represents the total pixel count, $\mathit{g}({\mathit{x}}_{sa{r}_n},{\mathit{x}}_{op{t}_n};\mathit{w}$) is the JFM, and $\mathit{w}$ represents the parameter of the JFM.

The SPL model is solved using a majorization–minimization (MM) algorithm. The goal of MM algorithms, which are extensively employed in machine learning, is to simplify complex optimization problems by repeatedly repeating the majorization and minimization steps. Here, ${\mathit{w}}^t$ refers to the model parameter in the tth iteration of the MM algorithm discussed in this paper.

(1) Majorization Step: $v_n^{*}(\lambda ;L_n({\mathit{w}}^t))$ is determined by solving the following problem:

$$v_n^{*}(\lambda ;L_n({\mathit{w}}^t))=\underset{v_n\in [0,1]}{\mathrm{argmin}}{v}_n{L}_n({\mathit{w}}^t)+f(v_n;\lambda )$$

(9)

The self-paced function ($f(v_n;\lambda )$) significantly influences the optimization procedure of the self-paced learning model. In [56], Jiang et al. delineated the self-paced function from the following three perspectives:

• $f(\mathit{v};\lambda )$ is a convex function over $v_n\in [0,1]$, ensuring the uniqueness of $f(v_n;\lambda )$ with respect to $v_n$;
• With all parameters held constant except for $v_n^{*}$ and $L_n$, $v_n^{*}$ exhibits a monotonically decreasing trend with $L_n$. Moreover, as $L_n$ approaches 0, ${lim}_{L_n\to 0}{v}_n^{*}=1$, and as $L_n$ tends to infinity, ${lim}_{L_n\to \infty }{v}_n^{*}=0$;
• $v_n^{*}$ increases monotonically with respect to $\lambda$, ${lim}_{\lambda \to 0}{v}_n^{*}=0$, and ${lim}_{\lambda \to \infty }{v}_n^{*}=1$.

The mixed-weight form in most SPL methods is relatively simple. We designed the self-paced function as follows to extend the mixed weight to a more general form:

$$\begin{array}{cc}\hfill & f(\mathit{v};\lambda )=\hfill \\ \hfill & \sum _{n=1}^N-\frac{\lambda }{a(1-e^a)}(((1-e^a)v_n+e^a)ln((1-e^a)v_n+e^a)\hfill \\ \hfill & -(1-e^a)v_n),\hfill \end{array}$$

(10)

where $\lambda$ is the speed parameter, and a is the control parameter in the form of soft weight. In the early stage of SPL, the samples entering the training queue are mostly simple samples, so it is necessary to set larger weights for the samples. At this time, a takes a smaller value. In the later stage of learning, the model has matured, and the training queue contains a large number of complex samples. Therefore, it is necessary to increase the value of a, which is always less than 0. The value of a is updated adaptively as follows:

$$a=1-\frac{1}{\frac{len(L_{\lambda })}{N}+△},$$

(11)

where $len(L_{\lambda })$ represents the size of the training queue, where each sample in the training queue has a loss of less than $\lambda$, and △ is an infinitesimal quantity.

Referring to Equation (8), the SPL objective function can be obtained as follows:

$$\begin{array}{cc}\hfill & \underset{\mathit{w},\mathit{v}}{\min }\mathbb{E}(\mathit{w},\mathit{v};\lambda )=\hfill \\ \hfill & \sum _{n=1}^N{v}_n·L_{JFM}(\mathit{g}({\mathit{x}}_{sa{r}_n},{\mathit{x}}_{op{t}_n};\mathit{w}))+\hfill \\ \hfill & \sum _{n=1}^N-\frac{\lambda }{a(1-e^a)}(((1-e^a)v_n+e^a)ln((1-e^a)v_n+e^a)\hfill \\ \hfill & -(1-e^a)v_n)\hfill \end{array}$$

(12)

In order to satisfy condition 1, we calculate $f(\mathit{v};\lambda )$ of the Hessian matrix as follows:

$$\begin{array}{cc}H=& \\ & \left[\begin{array}{cccc}-\frac{\lambda e^a}{a((1-e^a)v_1+e^a)}& 0& \dots & 0\\ 0& -\frac{\lambda e^a}{a((1-e^a)v_2+e^a)}& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & -\frac{\lambda e^a}{a((1-e^a)v_n+e^a)}\end{array}\right]\end{array}$$

(13)

where $\lambda >0$, $a<0$, and $v_n\in [0,1]$, so we have $-\frac{\lambda e^a}{a((1-e^a)v_n+e^a)}>0$. Obviously, all the principal subformulas of the objective function are always greater than zero, so H is a positive definite matrix, and the self-paced function $f(\mathit{v};\lambda )$ is a convex function.

To determine the optimal solution for $v_n$, we set the partial derivative of the objective function ($\mathbb{E}(\mathit{w},\mathit{v};\lambda )$) with respect to $v_n$ equal to 0. This yields the form of the optimal solution for v as follows:

$$v_n=\left\{\begin{array}{cc}\frac{e^{\frac{a{L}_n}{\lambda }}-e^a}{1-e^a},\hfill & L_n\le \lambda \hfill \\ 0,\hfill & L_n>\lambda \hfill \end{array}\right.$$

(14)

where $L_n=L_{JFM}(\mathit{g}({\mathit{x}}_{sa{r}_n},{\mathit{x}}_{op{t}_n};\mathit{w}))$ represents the loss of the nth heterogeneous patch pair. Clearly, $v_n$ decreases monotonically with $L_n$ and increases monotonically with $\lambda$, and for mixed weights, ${lim}_{L_n\to 0}{v}_n^{*}=1$ and ${lim}_{L_n\to \infty }{v}_n^{*}=0$, so condition 2 is satisfied. This relationship between optimal weight and loss can cause the model to pay more attention to simple samples and learn sample knowledge in progression from easy to difficult. For the speed parameter ($\lambda$), when $\lambda$ tends toward 0, ${lim}_{\lambda \to 0}{v}_n^{*}=1$, indicating that during the initial training phase, the speed parameter is low, and there are fewer samples included in the training. When $\lambda$ tends toward ∞, ${lim}_{\lambda \to \infty }{v}_n^{*}=1$, indicating that in the later stage of training, the model reaches maturity, and more and more complex samples are taken into account.

(2) Minimization Step: In this stage, we are required to compute

$${\mathit{w}}^{t+1}=\underset{\mathit{w}}{\mathrm{argmin}}\sum _{n=1}^N{v}_n{L}_n({\mathit{w}}^t)$$

(15)

After incorporating self-paced learning, the updated optimization objective for JFM is expressed as follows:

$$\begin{array}{c}\hfill L_{JFM}=\frac{1}{N}·\sum _{i=1}^N{v}_i·(\sum _{m=1}^{H_p}\sum _{n=1}^{W_p}{(x_{sa{r}_i}^{m,n}-{\widehat{x}}_{sa{r}_i}^{m,n})}^2+\sum _{m=1}^{H_p}\sum _{n=1}^{W_p}{(x_{op{t}_i}^{m,n}-{\widehat{x}}_{op{t}_i}^{m,n})}^2+\sum _{k=1}^K{({\widehat{f}}_{sa{r}_i}^k-f_{op{t}_i}^k)}^2)\end{array}$$

(16)

We introduce a self-paced learning optimization strategy for two JFMs in MJFM to force JFM to focus on the mapping relationship of unchanged regions.

3.3. Change Detection with Multi-Objective PSO

All multi-scale image patches are sent to the trained MJFM, and the similarity of the features of each patch is computed according to $\left|\right|{\mathit{f}}_{sar}-{\mathit{f}}_{opt}{\left|\right|}_2$. After normalization, two difference images ($D_1$ and $D_2$) are obtained. $D_1$ is obtained by a smaller patch, so it has more detailed texture information. $D_2$ is obtained by a larger patch, with less noise and a smoother texture.

To ensure an accurate change map, this investigation assigns weights to the two objectives of preserving image details and suppressing noise. It formulates the binarization of the difference map as a multi-objective optimization problem and utilizes the multi-objective particle swarm optimization algorithm to address the challenge. Algorithm 2 shows the specific calculation process of MOPSO for change detection. $D_1$ and $D_2$ are used as the input of a standard FCM measure to preserve details and suppress noise, respectively. Two conflicting objective functions ($f_{PD}$ and $f_{RN}$) can be described as follows:

$$\begin{array}{c}\hfill minF(\mathit{z})=\left\{\begin{array}{ccc}\hfill f_{PD}& =\hfill & \hfill \sum _{i=1}^N\sum _{k=1}^c{u}_{ki}^m\left|\right|p_i^1-z_k{\left|\right|}^2\\ \hfill f_{RN}& =\hfill & \hfill \sum _{i=1}^N\sum _{k=1}^c{u}_{ki}^m\left|\right|p_i^2-z_k{\left|\right|}^2\end{array}\right.\end{array}$$

(17)

where $p_i^1$ is the ith pixel in difference image $D_1$, and $p_i^2$ is the ith pixel in difference image $D_2$. This study adopts a decomposition strategy to transform the multi-objective optimization problem (MOP) into a collection of separate scalar aggregation problems. The commonly used weighted sum approach is the decomposition method employed in the proposed algorithm and is expressed as follows:

$$\begin{array}{cc}\hfill g^{ws}(\mathit{z}|\mathit{\mu })& =\mu ·f_{PD}(\mathit{z})+(1-\mu )·f_{RN}(\mathit{z})\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\mathit{z}\in \mathit{Z}\hfill \end{array}$$

(18)

where $\mathit{\mu }={(\mu ,(1-\mu ))}^T$ is the weight vector with $0\le \mu \le 1$.

The position vector in PSO stands for an answer to the problem that is optimized. The decision vector depicts the two cluster centers for the picture change detection problem. The new velocity is calculated by

$$\begin{array}{c}\hfill o_k^i=w_0{o}_k^i+c_1{r}_1(pbes{t}_k^i-z_k^i)+c_2{r}_2(gbes{t}_k^i-z_k^i)\end{array}$$

(19)

where $c_1$ and $c_2$ are two coefficients, while $w_0$ is the inertia factor. $r_1$ and $r_2$ are two random numbers between 0 and 1, $pbes{t}_K^i$ represents the ith particle’s best individual performance, and $gbes{t}_k^i$ represents the leader’s best position globally.

Following the velocity update, each particle utilizes the new velocity to determine its updated position.

$$\begin{array}{c}\hfill z_k^{i+1}=z_k^i+o_k^i\end{array}$$

(20)

Additionally, this study incorporates the concept of decomposition when updating the membership value. This decomposition breaks down the multi-objective problem into various scalar aggregation problems. The function of each subproblem is defined as follows:

$$\begin{array}{c}\hfill F_{\mu }=\mu \sum _{i=1}^N\sum _{k=1}^c[u_{ki}^m\left|\right|p_i^1-z_k{\left|\right|}^2]+\\ \hfill (1-\mu )\sum _{i=1}^N\sum _{k=1}^c[u_{ki}^m\left|\right|p_i^2-z_k{\left|\right|}^2]\end{array}$$

(21)

with limitations of ${\sum }_{k=1}^c{u}_{ki}=1$.

By minimizing Equation (14) while adhering to the restrictions outlined in Equation (15), the fuzzy membership functions are produced iteratively. As a result, the energy function is altered as follows using the Lagrange multipliers:

$$\begin{array}{cc}\hfill & F_{\mu }=\mu \sum _{i=1}^N\sum _{k=1}^c[u_{ki}^m\left|\right|p_i^1-z_k{\left|\right|}^2]+\hfill \\ \hfill & (1-\mu )\sum _{i=1}^N\sum _{k=1}^c[u_{ki}^m\left|\right|p_i^2-z_k{\left|\right|}^2]+{\alpha }_i(1-\sum _{k=1}^c{u}_{ki})\hfill \end{array}$$

(22)

where ${\alpha }_i$ is a Lagrange multiplier.

The local minimizer ($z_k^{*}$) is derived from

$$\begin{array}{c}\hfill \frac{\partial F_{\mu }}{\partial z_k}{|}_{z_k=z_k^{*}}=0\end{array}$$

(23)

With various weight values, we obtain the new membership-updating formula, which is represented as follows:

$$\begin{array}{c}\hfill u_{ki}=\frac{1}{{\sum }_{j=1}^c(\frac{\mu \left|\right|p_i^1-z_k{\left|\right|}^2+(1-\mu )\left|\right|p_i^2-z_k{\left|\right|}^2}{\mu \left|\right|p_i^1-z_j{\left|\right|}^2+(1-\mu )\left|\right|p_i^2-z_j{\left|\right|}^2})\frac{1}{m-1}}\end{array}$$

(24)

Through MOPSO, a set of Pareto-optimal solutions can be obtained to balance detail preservation and noise suppression. According to the membership matrix, the binary change map corresponding to each Pareto-optimal solution can be obtained. In addition, this study uses all the change maps of the threshold method for robust evaluation to obtain the final high-precision change map. Specifically, for the change pixel on each change map, if the number of change maps that divide the pixel into the changed class is greater than the set threshold ($ϵ$), the pixel is divided into the change class. Otherwise, the pixel is considered to be an unchanged region. The formulaic expression is expressed as follows:

$$label(x_i)=\left\{\begin{array}{cc}\hfill 1,& if\sum _{j=1}^m\mathbb{I}(l(x_i;ma{p}_j)=1)>ϵ\hfill \\ \hfill 0,& otherwise\hfill \end{array}\right.$$

(25)

where $label(x_i)$ represents the label of pixel $x_i$ in the final change map, $\mathbb{I}(·)$ is the indicator function, and $l(x_i;ma{p}_j)$ represents the label of pixel $x_i$ determined by the jth change map.

4. Experiment

4.1. Experimental Settings

(1) Evaluation Criteria: Our framework produces a binary change image, where 0 denotes the unchanged area, and 1 signifies the changed region. We employ five widely recognized quantitative metrics in our experiments—false positive (FP), false negative (FN), overall error (OE), overall accuracy (OA), Kappa coefficient (Kappa), and intersection-over-union ratio (IoU)—to assess the effectiveness of various algorithms. The specific calculation process of the index can be consulted in [33,41].

(2) Data Sets: The initial data set shown in Figure 3 consists of two uniform images measuring $349\times 252$ with an 8 m spatial resolution. These images were captured before and after a typhoon struck the Stone-Gate Reservoir in Taiwan. It is worth noting that our proposed framework demonstrates extensive adaptability and is equally suitable for homogeneous remote sensing images.

The second data set, known as the Yellow River data set, comprises a SAR image and an optical image, as shown in Figure 4. Both images have dimensions of $291\times 343$ with a spatial resolution of 8 m. This data set primarily highlights areas affected by floods.

The third data set, known as the Sardinian data set, comprises a Landsat-5 TM image and an optical image, as shown in Figure 5. Both images have dimensions of $412\times 300$ and a resolution of 30 m. This data set documents the expansion of Lake Mulagea in Sardinia, Italy.

The fourth data set, named the Shuguang data set, comprises one SAR image and one optical image, as shown in Figure 6. Both images have dimensions of $921\times 593$ with a resolution of 8 m. This data set primarily illustrates instances where local residents have converted farmland to create additional space for buildings or other purposes.

(3) Parameter Setting Details for Experiments: The architecture of MJFM, as illustrated in Figure 2, represents individual pixels using small patches centered around them. These patches are input into each JFM. For small-scale patches, the patch size is set to $3\times 3$, while for large-scale patches, it is $9\times 9$. Each JFM’s encoder comprises four convolutional layers. The sizes of the first three convolutional layers are $3\times 3$, with a stride of 1; a padding size of 1; and 32, 64, and 96 convolution kernels, respectively. The last convolutional layer’s kernel size is $patch\_size\times patch\_size$, with a stride of $patch\_size$, no padding, and 128 convolution kernels. The output passes through a fully connected layer to yield the patch’s 16-lengthfeature vector. The decoder is symmetrical to the encoder structure and consists of four deconvolution layers and one fully connected layer. First, the 16-dimensional image patch features are input into a fully connected layer with a size of $16\times 128$ and activated by the LeakyReLU nonlinear function. Then, the 128-dimensional vector is input into the deconvolution layer for upsampling. The first deconvolution layer consists of 96 convolution kernels. The size of the convolution kernels is $patch\_size\times patch\_size$, the stride size is $patch\_size$, and zero padding is not performed. The remaining three deconvolution layers are composed of 64, 32, and 3 convolution kernels, respectively. The convolution kernels’ size is $3\times 3$, the stride size is 1, and the padding size is 1. Note that for each convolution layer and deconvolution layer, after convolution processing, the BatchNorm layer is used for normalization; then, the LeakyReLU function is used for nonlinear activation. The mapper consists of two fully connected layers with sizes of $16\times 128$ and $128\times 16$, respectively. After each fully connected layer, a LeakyReLU nonlinear activation layer is added. MJFM’s learnable parameters are optimized using the backpropagation algorithm. Each JFM is trained using the Adam optimizer with a batch size of 128 examples, a learning rate of 0.0001, and exponential decay rates of 0.9 for the first moment estimate and 0.999 for the second moment estimate. The LeakyReLU activation function is utilized in each JFM layer. The feature vector’s Euclidean distance is computed and normalized to a range between 0 and 1, representing the normalized difference map.

In self-paced learning, selecting the appropriate step-speed parameter ($\lambda$) can be challenging. As mentioned in [60], SPL computation can be streamlined by seeking a solution based on the number of training samples involved. We organize the losses of all training samples in ascending order to acquire $L_{sort}$, then designate the $N_s$th loss value of $L_{sort}$ as $\lambda$, represented as $\lambda =L_{sor{t}_{N_s}}$. After the self-paced learning phase, $N_{add}$ new samples are introduced, prompting a recalculation of $\lambda$. In subsequent experiments, we delve deeper into the influence of $\lambda$ and $N_{add}$ on the model’s efficacy.

4.2. Test of Self-Paced Learning

In order to verify the effect of self-paced learning on MJFM, we compare the difference map obtained by using the self-paced learning strategy to train MJFM with that obtained using an ordinary mode to train MJFM.

Figure 7, Figure 8, Figure 9 and Figure 10 record the difference maps generated by a JFM trained by a self-paced learning strategy and a JFM trained in normal mode on four data sets, where (a) and (c) are the difference maps obtained by small-scale JFMs, and (e) and (g) are the difference maps obtained by large-scale JFMs; (b), (d), (f), and (h) show the details of the difference maps. As previously analyzed, large-scale JFMs usually result in smoother difference maps but less detail retention, while small-scale JFMs result in higher detail retention but more abnormal noise points. In addition, it can be seen that after introducing the self-paced learning strategy into the JFM training process, the difference information of the generated difference map is more obvious. This is because the self-paced learning strategy focuses on low-loss samples, which are mainly distributed in the unchanged area in the remote sensing image. In the JFM training process, the high-loss samples in the changing area are usually weighted with a smaller weights relative to the loss, thereby inhibiting the model from learning the heterogeneous mapping relationship of this part of the sample. In the detailed display image, we can see that the JFM trained by the ordinary mode has more noise in the difference image. This is because the data of the changing region induce the non-uniform mapping on the JFM learning image, and this non-uniform mapping relationship causes the JFM to map the data of the unchanged region to the feature space with large differences, resulting in more noise in the unchanged region.

4.3. Test of Self-Paced Learning Parameters

Selecting appropriate parameters has a significant impact on self-paced learning. In this paper, the SPL calculation is converted into a simpler calculation by searching for the solution of the number of samples involved in training. We conducted experiments on the initial $N_s$ and the sample increment parameter ($N_{add}$).

In each iteration, the loss (L) of training-set samples is sorted in ascending order to obtain $L_{sort}$, and the $N_s$th loss value of $L_{sort}$ is chosen as the value of $\lambda$ to create a training queue with non-zero weight values. After completing the inner loop of self-paced learning, $N_{add}$ training samples are appended to the training queue to increase the $\lambda$ value (thus, in the subsequent iteration, the number of samples ($N_s^{i+1}$) in the self-paced learning training queue becomes $N_s^{i+1}=N_s^i+N_{add}$).

In the experiment, we initialize the training queue by selecting 10%, 20%, 30%, 40%, and 50% of the samples from the training set, then determine $\lambda$ as $\lambda =L_{sor{t}_{N_s}}$. Subsequently, we gradually expand the training queue to sizes of 1000, 5000, 10,000, 20,000, and 30,000 until all training samples are included. Figure 11 illustrates the impact of $N_s$ and $N_{add}$ on the final change detection performance. It is observed that the framework demonstrates robustness to variations in $N_s$ and $N_{add}$ to a certain extent. Furthermore, it is noted that superior change detection outcomes are commonly attained when the initial training-queue size constitutes roughly 30% of the total training-set size, with around 10,000 samples added in each iteration. In addition, we do not recommend selecting a smaller initial training queue, which may lead to self-paced learning overfitting a single category of sample data, resulting in misclassification.

4.4. Test of Multi-Scale Learning

In this section, we test the impact of multi-scale learning on change detection performance. Specifically, for the multi-scale case, we use a pair of patches with sizes of $3\times 3$ and $9\times 9$ as input to MJFMand use MOPSO to post-process the difference map to obtain the change map. For the case of a single scale, we use $3\times 3$ and $9\times 9$ image patches as input to the JFM because in the case of a single scale, it is impossible to establish a multi-objective optimization problem for the difference map. Therefore, we use FCM to cluster the two difference maps to obtain the change map and record the relevant indicators of the change map obtained in these three cases. JFM-S means training the JFM with $3\times 3$ patches, and JFM-L means training the JFM with $9\times 9$ patches. Figure 12, Figure 13, Figure 14 and Figure 15 and

Table 1. Evaluation of multi-scale learning on four data sets.

Data Set	Method	FP	FN	OE	OA	Kappa	IoU
Stone-Gate	SMJFM	216	780	996	98.87	0.88	80.27
	JFM-S	445	1009	1454	98.34	0.83	72.45
	JFM-L	281	2785	3066	96.51	0.56	40.04
Yellow River	SMJFM	536	1465	2001	98.00	0.62	46.31
	JFM-S	895	1514	2409	97.59	0.57	41.04
	JFM-L	589	2031	2620	97.38	0.58	30.69
Sardinia	SMJFM	700	2129	2829	97.79	0.78	66.02
	JFM-S	791	2170	2961	97.60	0.77	64.82
	JFM-L	1065	3307	4372	96.46	0.64	49.70
Shuguang	SMJFM	1768	7358	9126	98.33	0.76	61.98
	JFM-S	4050	13,010	17,060	96.88	0.50	35.09
	JFM-L	2880	7886	10,766	98.03	0.72	57.13

record the relevant results. It can be seen that in the case of a single scale, when the patch size is small, JFM-S detects more unchanged class pixels as changed classes. When the patch size is large, JFM-L detects more changed class pixels as unchanged classes. Multi-scale learning is complementary and greatly reduces the FP and FN indicators of change detection results. This is because multi-scale learning can make up for the lack of information in the case of a single scale and integrate the advantages of each scale, thereby providing richer information for change detection tasks and improving change detection performance.

4.5. Test of MOPSO

In order to evaluate the impact of MOPSO on change detection performance, we compare the change detection indices with and without MOPSO. In the first case, the robust evaluation threshold is set to 0.5. For the case without MOPSO, we perform an average weighted fusion of the two difference maps obtained by MJFM (i.e., $D_f=0.5D_1+0.5D_2$), then use FCM to classify the difference map to obtain the change map. The experimental results are shown in

Table 2. Evaluation of MOPSO on four data sets.

Data Set	Method	FP	FN	OE	OA	Kappa	IoU
Stone-Gate	SMJFM	216	780	996	98.87	0.88	80.27
	CMJFM	356	1101	1457	98.34	0.79	71.92
Yellow River	SMJFM	536	1465	2001	98.00	0.62	46.31
	CMJFM	733	1528	2261	97.73	0.59	42.38
Sardinia	SMJFM	700	2129	2829	97.79	0.78	66.02
	CMJFM	1328	2428	3756	96.96	0.71	58.05
Shuguang	SMJFM	1768	7358	9126	98.33	0.76	61.98
	CMJFM	3828	8119	11,947	97.81	0.69	54.16

, where SMJFM represents our method, and CMJFM represents the method after removing MOPSO. It can be seen that in the absence of MOPSO, the FP index increases significantly, which means that there are more cases in which the unchanged class is misclassified into the changed class in the change map. This is because in SMJFM, we robustly evaluate the changed class pixels of the change map corresponding to multiple Pareto-optimal solutions, thus alleviating this situation. In addition, it can be seen that the use of MOPSO to post-process the difference map improves the OA and Kappa indices, which indicates that the introduction of MOPSO improves the processing performance from the difference map to the change map and can obtain a more accurate prediction change map closer to the original label map.

4.6. Test of Robust Evaluation of Change Map

In contrast to previous approaches such as thresholding or clustering methods for binarizing the difference map to obtain the change map, this paper leverages the information from the difference map at various scales. Initially, we present the Pareto frontier achieved by employing the multi-objective PSO algorithm, as depicted in Figure 16a–d. Subsequently, we assess the accuracy of the change-class labels on the four data sets using different thresholds and provide recommended threshold settings.

On the Pareto fronts of the four data sets, we show the change maps obtained by only considering noise suppression (i.e., clustering the difference maps obtained by the large-scale JFM) and only considering detail retention (i.e., clustering the difference maps obtained by the small-scale JFM). It can be clearly seen that in the case of only considering noise suppression, there are few isolated outliers on the generated change map, but the change map loses a lot of details, and the FN index on this change map will increase. If only the details are retained, there are many isolated outliers on the generated change map, and the FP index on this change map will increase. In order to make full use of the change information provided by the Pareto-optimal solutions, we use the change maps corresponding to all the Pareto-optimal solutions to robustly evaluate the change samples and set the threshold sequence to [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. In Figure 17, the change map’s OA index corresponding to different thresholds on four data sets is shown. It can be seen that when the robust evaluation threshold is near 0.5, better change detection results can usually be obtained. If the robust evaluation threshold is too large or too small, it will have a negative impact on the final detection results. This is because a smaller evaluation threshold cannot suppress the isolated abnormal samples on the change map, and a larger evaluation threshold will over-threshold the correct change samples on the change map, thereby reducing the overall accuracy.

4.7. Test of Patch Size

In order to compare the effects of different patch sizes on the change detection results, we change the patch size and record the final results while keeping other experimental settings unchanged. Specifically, we choose the Yellow River data set and the Sardinia data set, set different patch sizes, and calculate the OA index and Kappa index of the change map. Figure 18 is the experimental result. It can be seen that when the small-scale patch size is set to $3\times 3$ and the large-scale patch size is set to $9\times 9$, the change detection result is the best. This may be because when the size of the small-scale patch is too large, fewer details of the change map are output by the JFM, and more errors are caused when change detection is performed on the ground–object boundary. When the large-scale patch is small, the model has difficulty in suppressing noise data and has low robustness to noise data. At the same time, it is not advisable to blindly increase the size of large-scale patches, which will make the difference map of the JFM output too vague, thus reducing the accuracy of change detection. Therefore, we recommend that the small-scale patch size be $3\times 3$ and the large-scale patch size be $9\times 9$.

4.8. Comparison of SMJFM against Other Methods

We compare our proposed framework with several existing methods, including PCC, MRF, CAN, SCCN, CACD, CDCG, and PMA.

4.8.1. Experiments on the Stone-Gate Data Set

This data set includes two panchromatic images. Despite the distinct characteristics of rivers and mountains, the presence of rainwater on mountaintops may lead to incorrect detection of unchanged areas as changed areas.

PCC utilizes FLICO to classify ground objects and subsequently detect change areas. Figure 19a,b display the classification results of PCC. Figure 19c–h depicts the difference maps obtained by SCCN, CACD, CGCD, PMA, the small-scale JFM (JFM-S), and the large-scale JFM (JFM-L), respectively. Notably, PCC, MRF, and CAN do not generate difference maps. Figure 20 displays the change maps obtained by all methods, while

Table 3. Evaluation on the Stone-Gate data set.

Method	FP	FN	OE	OA	Kappa	IoU
PCC	5772	1467	7239	91.77	0.44	31.73
SCCN	1399	312	1711	98.05	0.82	72.54
CACD	451	1237	1688	98.08	0.80	68.05
MRF	2664	2904	5568	93.67	0.38	25.72
CAN	165	2544	2709	96.92	0.61	45.79
CDCG	725	1608	2333	97.35	0.72	58.02
PMA	277	1505	1782	97.97	0.78	65.12
SMJFM	216	780	996	98.87	0.88	80.27

presents the evaluation metrics for the change map acquired by each method. Compared to other methods, SMJFM does not incorporate any denoising algorithms, indicating that the complementary multi-scale information mechanism introduced by SMJFM inherently offers a denoising effect and demonstrates more promising detection performance.

4.8.2. Experiments on the Yellow River Data Set

The features of the two images are simple; they just show land and river. The problem is in the significant speckle noise present in the SAR image. Figure 21a,b show the classification results of PCC. Figure 21c–h show the difference maps obtained by SCCN, CACD, CGCD, PMA, the small-scale JFM (JFM-S), and the large-scale JFM (JFM-L), respectively. Figure 22 is the change map obtained by all methods, and

Table 4. Evaluation on the Yellow River data set.

Method	FP	FN	OE	OA	Kappa	IoU
PCC	6488	1396	7884	92.10	0.27	18.55
SCCN	2825	481	3306	96.69	0.60	45.05
CACD	573	2302	2875	97.12	0.37	23.59
MRF	24,760	714	25,474	74.48	0.1120	8.86
CAN	20,958	2346	23,304	76.65	0.0125	3.50
CGCD	728	2192	2920	97.07	0.39	25.49
PMA	524	2328	2852	97.14	0.36	23.23
SMJFM	536	1465	2001	98.00	0.62	46.31

is the index of the change map obtained by all methods. It can be seen that the change map obtained by SMJFM is the closest to the real label visually, the number of errors detected is smaller, and the overall accuracy is higher. In addition, it can be seen from Figure 21 that the comparison between the changed region and the unchanged region of the difference map obtained by SMJFM is more significant, which is due to the introduction of the self-paced learning training mechanism, while CACD only introduces common information into the feature vector, which leads to the autoencoder capturing the inconsistent mapping in the changed region during the learning process.

4.8.3. Experiments on the Sardinia Data Set

The heterogeneous images in this data set are mainly composed of mountains and water. The intricate texture features of the mountains pose the primary challenge in aligning the intricate elements of unchanged mountain features between the two images, while the second challenge involves identifying the characteristics of the water boundary. Figure 23a,b display the classification results of PCC, while Figure 23c–h depict the difference maps generated by SCCN, CACD, CGCD, PMA, the small-scale JFM (JFM-S), and the large-scale JFM (JFM-L), respectively. Figure 24 presents the change maps obtained by all methods, and

Table 5. Evaluation on the Sardinia data set.

Method	FP	FN	OE	OA	Kappa	IoU
PCC	27,200	3962	31,162	74.79	0.10	10.52
SCCN	3015	1406	4421	96.42	0.72	58.45
CACD	1284	2533	3817	96.91	0.71	57.16
MRF	3246	2587	5833	95.28	0.61	46.35
CAN	43,197	2616	45,813	62.93	0.08	9.86
CGCD	1561	2574	4135	96.65	0.69	54.99
PMA	1072	3408	4480	96.38	0.63	48.49
SMJFM	700	2129	2829	97.79	0.78	66.02

provides the evaluation indices of the change maps obtained by all methods. Due to the complex texture structure of the two images, the classification accuracy achieved by PCC is notably low. In contrast, SCCN, CACD, CGCD, and SMJFM map the two images to a more consistent potential space, thereby suppressing false positives in the unchanged region. Moreover, due to the substantial variation in the distribution of complex texture SAR image data and optical image data, the majority of pixels in the CAN image are misclassified.

4.8.4. Experiments on the Shuguang Data Set

In comparison to the first three data sets, the fourth data set encompasses a broader array of ground items, including fields, buildings, and water areas, indicating the presence of numerous feature categories in these images. Consequently, the change detection task becomes increasingly challenging. In this data set, some attributes of the changed region closely resemble those of the unchanged region, leading to incorrect detection of these parts as unchanged. Distinguishing between these similar attributes poses a formidable challenge. Figure 25a,b illustrate the classification results of PCC, while Figure 25c–h display the difference diagrams generated by SCCN, CACD, CGCD, PMA, the small-scale JFM (JFM-S), and the large-scale JFM (JFM-L), respectively. Figure 26 presents the change maps obtained by all methods, and

Table 6. Evaluation on the Shuguang data set.

Method	FP	FN	OE	OA	Kappa	IoU
PCC	158,414	723	159,137	70.86	0.15	11.91
SCCN	600	18,513	19,113	96.50	0.27	16.29
CACD	920	17,388	18,308	96.64	0.33	20.92
MRF	12,702	5174	17,876	96.73	0.64	48.83
CAN	14,173	14,242	28,415	94.80	0.33	21.95
CGCD	1499	15,664	17,163	96.86	0.42	27.68
PMA	911	18,287	19,198	96.48	0.28	17.05
SMJFM	1768	7358	9126	98.33	0.76	61.98

provides the evaluation indices of the change maps obtained by all methods. From Figure 25, it can be observed that the difference maps generated by the CACD, SCCN, CGCD, and PMA methods are not satisfactory, as the feature difference in the change area is minimal, leading to an increase in the FN index in the change map. Meanwhile, although the difference map generated by the small-scale JFM is subpar, the difference map obtained by the large-scale JFM effectively highlights the change area. Thanks to the robust evaluation mechanism of Pareto-optimal solutions, an excellent change map is ultimately achieved.

5. Conclusions

This paper proposes an unsupervised method for detecting changes in heterogeneous images. The method consists of several key components. First, it introduces a multi-scale joint feature mapper (MJFM) based on an autoencoder architecture to effectively map heterogeneous data into comparable feature spaces. Secondly, it incorporates a self-paced learning training strategy to guide the MJFM in capturing the mapping relationship of unchanged regions while avoiding overfitting to changed regions. Finally, it balances noise suppression and detail retention objectives and utilizes a robust evaluation method to obtain a high-precision binary change image. The proposed method demonstrates resilience to noise in remote sensing images and effectively mitigates change detection errors stemming from complex ground-object information, ensuring its applicability across diverse scenarios. In addition, since the proposed framework does not require supervised information, this framework can handle more diverse remote sensing image (underwater remote sensing images, hyperspectral remote sensing images, etc.) change detection problems.