A Framelet-Based Iterative Pan-Sharpening Approach

Abstract

Pan-sharpening is used to fuse multispectral images and panchromatic images to produce a multispectral image with high spatial resolution. In this paper, we design a new iterative method based on framelet for pan-sharpening. The proposed model takes advantage of the upsampled multispectral image and a linear relation between the panchromatic image and the latent high-resolution multispectral image. Since the sparsity of the pan-sharpened image under a B-spline framelet transform is assumed, we regularize the model by penalizing $l_1$ norm of a framelet based term. The model is solved by a designed algorithm based on alternating direction method of multipliers (ADMM). For better performance, we propose an iterative strategy to pick up more spectral and spatial details. Experiments on four datasets demonstrate that the proposed method outperforms several existing pan-sharpening methods.

1. Introduction

An optical satellite usually acquires two images describing the same scene almost simultaneously, which are called multispectral (MS) image and panchromatic (PAN) image respectively. The former is a multichannel image with low spatial resolution, while the latter is a single channel image with rich spatial details. Despite the fact that MS image can be of over eight bands and the resolution of PAN image can be less than half a meter, their superiority cannot be synthesized in one image due to physical and technological constraints. However, pan-sharpening techniques are capable of creating a multichannel image with high spatial resolution out of these two images, which is of great importance for remote sensing. More specifically, pan-sharpening plays an important role in the interpretation of remote sensing scenes and can be used as a preliminary step of various remote sensing tasks such as object recognition [1], change detection [2] and so on. Therefore, these techniques attract much attention of scientific community.

Among various pan-sharpening methods having been proposed in the literature, many of them can be put into two main categories: component substitution (CS) and multiresolution analysis (MRA). CS mainly involves three steps: firstly a spectral transformation of MS image, then a replacement of its spatial component with PAN image, and finally an inverse transformation. This class includes classical methods such as intensity-hue-saturation (IHS) [3], principal component analysis (PCA) [4], Gram-Schmidt spectral sharpening (GS) [5] and recent methods based on mean information [6] and image matting model [7]. As for MRA, it focuses on an injection process in which spatial details extracted from PAN image are added to upsampled MS image. Examples of this class include wavelet transform based methods [8,9,10], Laplacian Pyramid based methods [11], and methods based on other transforms [12,13].

Different from the two categories mentioned above, there are also other types of pan-sharpening methods. This family includes those based on Bayesian paradigm [14], total variation [15,16], gradient operator [17,18], sparse representation [19], super-resolution techniques [20], convolution neural network [21] and so on. Recently, Deng et al. [22] propose a novel variational model for pan-sharpening in which intensity function of the unknown image is considered from a continuous point of view. The related continuous function is made up of two components. Assumption is made that the former lies in a Reproducing Kernel Hilbert Space (RKHS) while the latter can be approximated by linear combination of approximated Heaviside functions (AHF). This model outperforms several state-of-the-art pan-sharpening methods according to experiments on two datasets. However, its good performance relies on a large amount of computation, which results in rather long running time.

In this paper, a new iterative algorithm for pan-sharpening is proposed as an attempt to simplify RKHS method. We make use of the information from MS image by generating its upsampled form. The linear relation [22,23] between PAN image and bands of the image to be estimated is also considered in the model. A framelet based term is introduced as regularization. Besides, we adopt an iterative strategy similar to [22] to improve performance of the algorithm. The framework of the proposed approach can be seen in Figure 1. By utilizing real data from Pléiades, Quickbird, WorldView-2 and SPOT-6, we compare several existing pan-sharpening methods with the proposed method. These results show spatial and spectral fidelity of the proposed method. Meanwhile, they confirm a much less running time than RKHS method, suggesting that framelet based regularization is more mature and easier to compute.

The rest of this paper is arranged as follows. Section 2 reviews the related work [22]. Then in Section 3 the new iterative algorithm is presented. Section 4 is a display of visual and numerical experimental results together with some discussions about the proposed method. Finally, conclusion is drawn in Section 5.

2. Related Work

To begin with, we introduce several notations to be used throughout this paper. Let $\mathrm{MS}\in {\mathbb{R}}^{m_1\times n_1\times T}$ be the original multispectral image with T bands, each band denoted as ${\mathrm{MS}}_i\in {\mathbb{R}}^{m_1\times n_1}$. Let $\widetilde{\mathrm{MS}}\in {\mathbb{R}}^{m_2\times n_2\times T}$ be the upsampled multispectral image, each band denoted as $\widetilde{{\mathrm{MS}}_i}\in {\mathbb{R}}^{m_2\times n_2}$. $\widehat{\mathrm{MS}}$ represents the high-resolution multispectral image to be estimated, with $\widehat{{\mathrm{MS}}_i}\in {\mathbb{R}}^{m_2\times n_2}$ being the ith band of it. Moreover, the original panchromatic image is $\mathrm{P}\in {\mathbb{R}}^{m_2\times n_2}$.

Deng et al. propose a new iterative pan-sharpening algorithm [22], which is an extension of their previous work on super-resolution [24]. They view the pan-sharpening problem as an intensity estimation process for the unknown image $\widehat{\mathrm{MS}}$. The intensity of $\widehat{\mathrm{MS}}$ is modelled as a hidden continuous function. It consists of two different components, a smooth one and a non-smooth one. The former is assumed to be an element of a special function space called Reproducible Kernel Hilbert Space (RKHS), while the latter is formulated as a linear combination of approximated Heaviside functions (AHF).

Specifically, let $f_i,i=1,2,\dots ,T$, be the underlying continuous intensity function corresponding to the ith band of $\widehat{\mathrm{MS}}$. Without loss of generation, the domain is restricted such that $\mathbf{z}=(x,y)\in [0,1]\times [0,1]$. The smooth component is expressed as ${\sum }_{\nu =1}^M{d}_{\nu ,i}{\varphi }_{\nu ,i}(\mathbf{z})+{\sum }_{s=1}^n{c}_{s,i}{\xi }_{s,i}(\mathbf{z})$. These two series lie in two different RKHS, with ${\varphi }_{\nu ,i}(\mathbf{z}),\nu =1,2,\dots ,M$, and ${\xi }_{s,i}(z),s=1,2,\dots ,n$, being basis functions in each RKHS respectively. $d_{\nu ,i},\nu =1,2,\dots ,M$, together with $c_{s,i},s=1,2,\dots ,n$, are the corresponding coefficients. As for the non-smooth component, a family of Heaviside functions is considered to implement the approximation. However, since Heaviside functions are singular at the origin, which makes differentiation impossible to be taken there, they are approximated by a smooth form, i.e., approximated Heaviside functions (AHF). In 2D case, this smooth alternative takes the form of $\psi ((cos{\theta }_{j,i},sin{\theta }_{j,i})·\mathbf{z}+c_{\rho ,i})$, actually representing an edge with $\theta$ elevation at a location specified by $c_{\rho ,i}$. It is a generalization of the 1D form $\psi (x)=\frac{1}{2}+\frac{1}{\pi }arctan(\frac{x}{\xi })$, where $\xi$ is a positive parameter to control smoothness. Therefore, $f_i,i=1,2,\dots ,T$, is finally modelled as:

$$f_i=\sum _{\nu =1}^M{d}_{\nu ,i}{\varphi }_{\nu ,i}(\mathbf{z})+\sum _{s=1}^n{c}_{s,i}{\xi }_{s,i}(\mathbf{z})+\sum _{j=1}^k\sum _{\rho =1}^n{\omega }_{j,i}\psi ((cos{\theta }_{j,i},sin{\theta }_{j,i})·\mathbf{z}+c_{\rho ,i}).$$

(1)

For more details, see [22,24].

By evaluating on a fine grid, (1) can be discretized so that it becomes a form involving simple matrix multiplication. Let ${\mathrm{T}}^h\in {\mathbb{R}}^{m_2{n}_2\times M},{\mathrm{K}}^h\in {\mathbb{R}}^{m_2{n}_2\times m_2{n}_2}$ and ${\Psi }^h\in {\mathbb{R}}^{m_2{n}_2\times m}(m=k·n)$ denote matrices whose entries are generated by evaluating ${\varphi }_{\nu ,i}(\mathbf{z})$, ${\xi }_{s,i}(\mathbf{z})$, and $\psi ((cos{\theta }_{j,i},sin{\theta }_{j,i})·\mathbf{z}+c_{\rho ,i})$ on a fine grid. Then each band of the desired multichannel image with high spatial resolution can be computed as follows:

$$\widehat{{\mathrm{MS}}_i}={\mathrm{T}}^h{\mathrm{d}}_i+{\mathrm{K}}^h{\mathrm{c}}_i+{\Psi }^h{\beta }_i,i=1,2,\dots ,T.$$

(2)

Hence the pan-sharpening problem is converted into a problem of coefficient estimation. In [22], the coefficients are computed by minimizing the following model:

$$\begin{array}{c}\hfill \underset{{\mathrm{d}}_i,{\mathrm{c}}_i,{\beta }_i}{\min }\frac{1}{N}\sum _{i=1}^T\parallel {\mathrm{T}}^h{\mathrm{d}}_i+{\mathrm{K}}^h{\mathrm{c}}_i+{\Psi }^h{\beta }_i-\widetilde{{\mathrm{MS}}_i}{\parallel }_2^2+\frac{\mu }{2}\sum _{i=1}^T{\mathrm{c}}_i^T{\mathrm{K}}^l{\mathrm{c}}_i+\frac{{\lambda }_1}{2}\sum _{i=1}^T{\parallel {\beta }_i\parallel }_1\\ \hfill +\frac{{\lambda }_2}{2}{∥\sum _{i=1}^T{\omega }_i({\mathrm{T}}^h{\mathrm{d}}_i+{\mathrm{K}}^h{\mathrm{c}}_i+{\Psi }^h{\beta }_i)-\mathrm{P}∥}_2^2,\end{array}$$

(3)

where N is the number of pixels that $\widehat{\mathrm{MS}}$ contains. $\mu ,{\lambda }_1$, and ${\lambda }_2$ are positive parameters. $\widetilde{{\mathrm{MS}}_i}$ is generated by an upsampling process via GS method [5]. ${\mathrm{K}}^l\in {\mathbb{R}}^{m_1{n}_1\times m_1{n}_1}$ is a coarser version of ${\mathrm{K}}^h$, i.e., the discretization process is done on a coarser grid. ${\omega }_i$ are weights reflecting the contribution of each band of $\widehat{\mathrm{MS}}$ to the linear combination which approximates the panchromatic image P. This linear approximation and its variants are assumed by many methods, e.g., [23,25].

After the coefficients ${\mathrm{d}}_i,{\mathrm{c}}_i,{\beta }_i$ are obtained, the high-resolution multispectral image can be computed by (2). Furthermore, model (3) is combined with an iterative strategy. It will be detailed in Section 3 since it is used in the method proposed in this paper as well.

3. The Proposed Method

Empirical results in [22] show that RKHS method outperforms several state-of-the-art pan-sharpening methods both in the perspective of spatial and spectral fidelity. However, the RKHS based model of $\widehat{\mathrm{MS}}$ is quite complicated, since it is not easy to implement in actual calculation. In addition, the regularization terms of (3) involve not only $l_1$ norm of ${\beta }_i$ but also quadratic function of ${\mathrm{c}}_i$ (both summed by band), which adds to computational burden. As a result, the algorithm is rather time consuming. Therefore, it is natural to raise a question. Can we simplify model (3) without loss of its advantage, i.e., high-level spatial and spectral performance? A possible way is not to make efforts to build a complicated model of $\widehat{\mathrm{MS}}$, but seek for another technique of regularization instead. This is exactly why we turn to framelet.

Piecewise smooth functions, for instance images, can be sparsely approximated by framelet system efficiently [26]. As a result, framelet techniques are used in literature to address the problems like image restoration (e.g., [26,27,28]). Recently, it is also applied to pan-sharpening. For instance, a framelet based MRA scheme is considered in [29]. A variational model [30] based on assumptions related to framelet coefficients, geometry keeping, spectral preserving and the sparsity of the image in the framelet domain is also proposed. They are able to obtain good results. Nevertheless, in this section, we consider a combination of variational model and iterative strategy instead of building a complicated model, i.e., we try to build a simple framelet based variational model and then combine it with an iterative strategy to yield a novel and effective approach for pan-sharpening.

In discrete case, let W and $W^T$ denote fast framelet decomposition and construction respectively. They are constructed by unitary extension principle (UEP) [31], and satisfy the relation $W^{T}W=I$. In this paper, we use the piecewise linear B-spline framelets [26]. An L-level framelet transform of an image u [26] can be denoted as:

$$Wu=\{W_{l,j}u,0\le l\le L-1,j\in \mathcal{I}\},$$

(4)

where $W_{l,j}u$ denotes the coefficients of u in framelet band j at level l under the framelet transform, and $\mathcal{I}$ is the index set of all framelet bands. Throughout this paper, we empirically set $L=1$. For more theoretical details of framelet, see e.g., [32].

Consequently, by employing framelet based regularization, (3) can be modified. Each band of $\widehat{\mathrm{MS}}$ is computed by minimizing the following function:

$$\underset{\widehat{{\mathrm{MS}}_i}}{\min }\frac{1}{2}\sum _{i=1}^T\parallel \widehat{{\mathrm{MS}}_i}-\widetilde{{\mathrm{MS}}_i}{\parallel }_2^2+\frac{\alpha }{2}{∥\sum _{i=1}^T{\omega }_i\widehat{{\mathrm{MS}}_i}-\mathrm{P}∥}_2^2+\sum _{i=1}^T{\parallel {\lambda }_i·W(\widehat{{\mathrm{MS}}_i})\parallel }_1,$$

(5)

where $\alpha$ and ${\lambda }_i$ are parameters. Dot product is used in the third term since there are more than one framelet band, as (4) suggested. Coefficients for bands of $\widehat{\mathrm{MS}}$, i.e., ${\omega }_i$, are estimated automatically by a linear regression [23] between the original multispectral image and downsampled panchromatic image. The upsampled multispectral image $\widetilde{{\mathrm{MS}}_i}$ is generated via GS method. Note that this model can also be viewed as an extension of the so-called analysis based model for image restoration (e.g., [33,34]). Each $l_1$ term of (5) is in accordance with that defined in the analysis based model. Concretely, it can be expressed as

$$\parallel {\lambda }_i·W(\widehat{{\mathrm{MS}}_i}){\parallel }_1={∥\sum _{j\in \mathcal{I}}{\lambda }_{i,j}\left|W_{i,j}(\widehat{{\mathrm{MS}}_i})\right|∥}_1.$$

(6)

This expression is the case where 1-level framelet transform is imposed on $\widehat{{\mathrm{MS}}_i}$, as we emphasized above.

Model (5) can be solved by methods such as primal-dual method [35] and ADMM [36] efficiently. We choose ADMM here, whose application covers a wide range of image processing, such as image denoising [37], image super-resolution [38], tensor completion [39], image destriping [40,41] and so on. Its convergence is guaranteed by many works such as [42,43]. Due to non-smoothness caused by $l_1$ term, the first step is to rewrite (5) as an equivalent form through substitution of variables:

$$\begin{array}{c}\hfill \underset{u_i,V_i,\widehat{{\mathrm{MS}}_i}}{\min }\frac{1}{2}\sum _{i=1}^T\parallel \widehat{{\mathrm{MS}}_i}-\widetilde{{\mathrm{MS}}_i}{\parallel }_2^2+\frac{\alpha }{2}{∥\sum _{i=1}^T{\omega }_i{V}_i-\mathrm{P}∥}_2^2+\sum _{i=1}^T{\parallel {\lambda }_i·u_i\parallel }_1\\ \hfill s.t.,u_i=W(\widehat{{\mathrm{MS}}_i}),V_i=\widehat{{\mathrm{MS}}_i},i=1,\dots ,T.\end{array}$$

(7)

Then we can obtain the augmented Lagrangian of (7), i.e.,

$$\begin{array}{c}\hfill \mathcal{L}(\widehat{{\mathrm{MS}}_i},u_i,V_i,D_i,E_i)=\frac{1}{2}\sum _{i=1}^T\parallel \widehat{{\mathrm{MS}}_i}-\widetilde{{\mathrm{MS}}_i}{\parallel }_2^2+\frac{\alpha }{2}{∥\sum _{i=1}^T{\omega }_i{V}_i-\mathrm{P}∥}_2^2+\sum _{i=1}^T{\parallel {\lambda }_i·u_i\parallel }_1\\ \hfill +\frac{{\beta }_2}{2}\sum _{i=1}^T{\parallel u_i-W(\widehat{{\mathrm{MS}}_i})\parallel }_2^2+\sum _{i=1}^T{E}_i^T(u_i-W(\widehat{{\mathrm{MS}}_i}))\\ \hfill +\frac{{\beta }_1}{2}\sum _{i=1}^T{\parallel V_i-\widehat{{\mathrm{MS}}_i}\parallel }_2^2+\sum _{i=1}^T{D}_i^T(V_i-\widehat{{\mathrm{MS}}_i}),\end{array}$$

(8)

where $D_i$ and $E_i$ are Lagrangian multipliers, ${\beta }_1$ and ${\beta }_2$ are two positive parameters.

Now we denote $F_i=D_i/{\beta }_1$ and $G_i=E_i/{\beta }_2$. According to ADMM, problem (7) can be solved by implementing the following iterative scheme:

(1)

For $i=1,\dots ,T$, update each $u_i^{(k)}$ by solving:

$$u_i^{(k+1)}=\arg \underset{u_i}{\min }\parallel {\lambda }_i·u_i{\parallel }_1+\frac{{\beta }_2}{2}{\parallel u_i-W({\widehat{{\mathrm{MS}}_i}}^{(k)})+G_i^{(k)}\parallel }_2^2.$$

(9)

(2)

For $i=1,\dots ,T$, update each $V_i^{(k)}$ by solving:

$$V_i^{(k+1)}=\arg \underset{V_i}{\min }\frac{\alpha }{2}{∥\sum _{1\le j<i}{\omega }_j{V}_j^{(k+1)}+{\omega }_i{V}_i+\sum _{i<j\le T}{\omega }_j{V}_j^{(k)}-\mathrm{P}∥}_2^2+\frac{{\beta }_1}{2}{\parallel V_i-{\widehat{{\mathrm{MS}}_i}}^{(k)}+F_i^{(k)}\parallel }_2^2.$$

(10)

(3)

For $i=1,\dots ,T$, update each ${\widehat{{\mathrm{MS}}_i}}^{(k)}$ by solving:

$$\begin{array}{c}\hfill {\widehat{{\mathrm{MS}}_i}}^{(k+1)}=\arg \underset{\widehat{{\mathrm{MS}}_i}}{\min }\frac{1}{2}\parallel \widehat{{\mathrm{MS}}_i}-\widetilde{{\mathrm{MS}}_i}{\parallel }_2^2+\frac{{\beta }_1}{2}{\parallel V_i^{(k+1)}-\widehat{{\mathrm{MS}}_i}+F_i^{(k)}\parallel }_2^2\\ \hfill +\frac{{\beta }_2}{2}{\parallel u_i^{(k+1)}-W(\widehat{{\mathrm{MS}}_i})+G_i^{(k)}\parallel }_2^2.\end{array}$$

(11)

(4)

For $i=1,\dots ,T$, update each $F_i^{(k)}$ by $F_i^{(k+1)}=F_i^{(k)}+(V_i^{(k+1)}-{\widehat{{\mathrm{MS}}_i}}^{(k+1)})$.

(5)

For $i=1,\dots ,T$, update each $G_i^{(k)}$ by $G_i^{(k+1)}=G_i^{(k)}+(u_i^{(k+1)}-W({\widehat{{\mathrm{MS}}_i}}^{(k+1)})$.

Note that (9)–(11) have closed-form solutions. Using soft-thresholding operator (e.g., [44]) ${\mathcal{T}}_{\tau }$, $u_i^{(k+1)}$ can be rewritten as:

$$u_i^{(k+1)}={\mathcal{T}}_{{\lambda }_i/{\mathbf{\beta }}_2}(W({\widehat{{\mathrm{MS}}_i}}^{(k)})-G_i^{(k)}),$$

(12)

where ${\mathcal{T}}_{\tau }(\nu )$ is defined entry-wise by

$${\mathcal{T}}_{\tau }(\nu )=\frac{\nu }{\left|\nu \right|}\max \left\{\right|\nu |-\tau ,0\},$$

(13)

As for (10) and (11), they can be solved easily as follows:

$$V_i^{(k+1)}=\frac{\alpha {\omega }_i(\mathrm{P}-{\sum }_{1\le j<i}{\omega }_j{V}_j^{(k+1)}-{\sum }_{i<j\le T}{\omega }_j{V}_j^{(k)})+{\beta }_1({\widehat{{\mathrm{MS}}_i}}^{(k)}-F_i^{(k)})}{\alpha {\omega }_i^2+{\beta }_1},$$

(14)

$${\widehat{{\mathrm{MS}}_i}}^{(k+1)}=\frac{\widetilde{{\mathrm{MS}}_i}+{\beta }_1(V_i^{(k+1)}+F_i^{(k)})+{\beta }_2{W}^T(u_i^{(k+1)}+G_i^{(k)})}{{\beta }_1+{\beta }_2+1}.$$

(15)

In order to facilitate illustration of the proposed algorithm, we summarize these steps of ADMM as Algorithm 1. Note that for simplicity of form, we write each iteration of ADMM in Algorithm 1 in an order slightly different from what we mentioned above, but it is easy to validate that they give the same results.

Algorithm 1 ADMM scheme for the proposed model

Input: panchromatic image $\mathrm{P}$, upsampled multispectral image $\widetilde{\mathrm{MS}}$, ${\omega }_i$ $\alpha ,{\lambda }_i,{\mathbf{\beta }}_1,{\mathbf{\beta }}_2$.  Output: high-resolution multispectral image $\widehat{\mathrm{MS}}$.   while not converged do    for $i=1:T$ do     (1) Solve $u_i^{(k+1)}$ by (12).     (2) Solve $V_i^{(k+1)}$ by (14).     (3) Solve ${\widehat{{\mathrm{MS}}_i}}^{(k+1)}$ by (15).     (4) Update $F_i^{(k)}$ by $F_i^{(k+1)}=F_i^{(k)}+(V_i^{(k+1)}-{\widehat{{\mathrm{MS}}_i}}^{(k+1)})$.     (5) Update $G_i^{(k)}$ by $G_i^{(k+1)}=G_i^{(k)}+(u_i^{(k+1)}-W({\widehat{{\mathrm{MS}}_i}}^{(k+1)}))$.    end for   end while

Although Algorithm 1 itself is a complete algorithm, there is still room for improvement. Thus an iterative stategy is considered. For accordance of notations, let ${\mathrm{P}}^{(1)}=\mathrm{P}$ and ${\mathrm{MS}}^{(1)}=\mathrm{MS}$. Similarly, denote the first output of Algorithm 1 as $I^{(1)}$. After obtaining $\widehat{\mathrm{MS}}$, we compute ${\mathrm{P}}^{(2)}=\mathrm{P}$−${\sum }_{i=1}^T{\omega }_i\widehat{{\mathrm{MS}}_i}$. In addition, let ${\widetilde{\mathrm{MS}}}^{(2)}=U(\mathrm{MS}$-$D(\widehat{\mathrm{MS}}))$. D represents a downsampling operator, and U represents the upsampling process by which $\widetilde{\mathrm{MS}}$ is generated. Now we view ${\mathrm{P}}^{(2)}$ and ${\widetilde{\mathrm{MS}}}^{(2)}$ as new inputs for Algorithm 1 instead of the original $\mathrm{P}$ and $\widetilde{\mathrm{MS}}$. The resulting new output can be denoted as $I^{(2)}$. By repeating this strategy, we obtain a series of $I^{(j)},j=2,3,\dots ,\gamma$. Then the sum of all $I^{(j)},j=1,2,\dots ,\gamma$, is taken as the final high-resolution multichannel image. This iterative strategy is adopted in not only pan-sharpening [22] but also image super-resolution [24].

Now we can summarize the procedures above as Algorithm 2:

Algorithm 2 The proposed iterative pan-sharpening algorithm

Input: panchromatic image $\mathrm{P}$, multispectral image $\mathrm{MS}$, ${\omega }_i$ $\alpha ,{\lambda }_i,{\mathbf{\beta }}_1,{\mathbf{\beta }}_2$.  Output: high-resolution multispectral image $\widehat{\mathrm{MS}}$.   1. Initialization: ${\mathrm{MS}}^{(1)}=\mathrm{MS},{\mathrm{P}}^{(1)}=\mathrm{P}$.   for $j=1:\gamma$ do     (1) Upsample ${\mathrm{MS}}^{(j)}$ to obtain ${\widetilde{\mathrm{MS}}}^{(j)}$.     (2) Compute $I^{(j)}$ by implementing Algorithm 1 (${\widetilde{\mathrm{MS}}}^{(j)},{\mathrm{P}}^{(j)}$ instead of $\widetilde{\mathrm{MS}},\mathrm{P}$ as input).     (3) Update ${\mathrm{P}}^{(j)}$ by ${\mathrm{P}}^{(j+1)}={\mathrm{P}}^{(j)}-{\sum }_{i=1}^T{\omega }_i{I}_i^{(j)}$.     (4) Update ${\mathrm{MS}}^{(j)}$ by ${\mathrm{MS}}^{(j+1)}={\mathrm{MS}}^{(j)}-D(I^{(j)})$.   end for   2. Compute the final output: $\widehat{\mathrm{MS}}={\sum }_{j=1}^{\gamma }{I}^{(j)}$.

Note that $\gamma$ is the number of outer iterations. The downsampling process D in each iteration is completed through a combination of two steps: compute the modulation transfer function of $I^{(j)}$ with Gaussian filter and then interpolate it to the size of $\mathrm{MS}$ in a “nearest” way [22]. And the upsampling process is done by GS method.

4. Results and Discussion

In this section, we firstly utilize four datasets to compare the proposed method with several pan-sharpening methods. After that, discussions related to the number of outer iterations of Algorithm 2 are discussed. Results on time cost are presented as well.

The tested datasets are acquired by Quickbird (4 bands, $512\times 512$), Pléiades (4 bands, $1024\times 1024$), WorldView-2 (4 bands, $800\times 800$), and SPOT-6 (4 bands, $1024\times 1024$). The dataset of Quickbird can be downloaded from http://glcf.umd.edu/data/quickbird/chilika.shtml. And the dataset of Pléiades is downloaded together with the source codes of [45] from http://openremotesensing.net/knowledgebase/quality-assessment-of-pan-sharpening-methods-in-high-resolution-satellite-images-using-radiometric-and-geometric-index/. As for datasets of WorldView-2 and SPOT-6, we downloaded them from http://cms.mapmart.com/Samples.aspx.

Since high resolution multispectral images are not available in the datasets, we follow Wald’s protocol [46]. Therefore, the original multispectral images in the datasets are treated as ground truth. The scale ratio is 4, thus the simulated low-resolution multispectral images (4 bands) are of the size $128\times 128,256\times 256,200\times 200$, and $256\times 256$ respectively. Each of them is downsampled from the corresponding ground truth in the same way as that in Algorithm 2, i.e., filter the ground truth by a Gaussian filter matched with the modulation transfer function (MTF) and then downscale it by “nearest” interpolation. As for each $\mathrm{P}$, it is generated by combining bands of the ground truth linearly.

Parameters of the proposed algorithm are empirically set as follows. We use 1-level piecewise linear B-spline framelet. For each band, ${\lambda }_i$ is equally set as:

$${\lambda }_i={10}^{-4}\left(\begin{array}{ccc}0& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right).$$

(16)

Other model parameters in (8) are set as $\alpha =1.5$, ${\beta }_1=0.5$, and ${\beta }_2=0.5$, with coefficients ${\omega }_i$ estimated by linear regression. In addition, the number of the external iterations in Algorithm 2, i.e., $\gamma$, is set as 5. For different datasets, these settings may not always be the best choice, but we unify them to display stability of the proposed method and also to save efforts of tuning.

Methods compared with the proposed method comprise some classical pan-sharpening methods (PCA [4], GS [5], high-pass filtering (HPF) [47], and modulation transfer function-generalized Laplacian Pyramid (MTFGLP) [48,49]) and different kinds of recent state-of-the-art methods (pan-sharpening with hyper-Laplacian prior (PHLP) [50], nonlinear intensity-hue-saturation (NIHS) [51], and RKHS [22]). All the experiments are conducted in MATLAB on a laptap with 4GB RAM and 1.70 GHz Intel(R) Core(TM) i5-4210U CPU.

4.1. Visual Comparison

Figure 2, Figure 3, Figure 4 and Figure 5 show visual results obtained by conducting experiments on four datasets aforementioned. Each set of figures contains output images produced by eight different methods. Each ground truth is also presented as reference. For better visualization, we show local enlarged images at the bottom-left corner of each output image.

It is obvious that all of the other methods perform better than PCA method both in terms of spatial quality and spectral fidelity according to Figure 2, Figure 3 and Figure 4. GS method outperforms PCA method significantly but generally fails to avoid great spectral distortion, which is most visible on Quickbird dataset and Pléiades dataset. NIHS method and PHLP method preserve spectral characteristics quite well. However, from the perspective of spatial details, they tend to generate excessive smooth results, thus the pan-sharpened images provided by them lack much sharp spatial information compared with the reference images.

HPF method and MTFGLP method are visually without much spectral distortion and are able to keep more spatial details. However, a closer look at their resulting images, e.g., in Figure 3, suggests that they are not able to provide as many spatial details as the proposed method does. Finally, it is noticeable that RKHS method and the proposed method achieve the best visual performance on the last three experimented datasets. Actually, visual comparisons show little disparity for these four methods. However, we will demonstrate that the proposed method gives the best quantitative results in Section 4.2.

4.2. Quantitative Comparison

Several quantitative indices are employed to report the performance of different pan-sharpening methods. To evaluate spectral distortion, we use spectral angle mapper (SAM) [52], erreur relative globale adimensionnelle de synthése (ERGAS) [52], universal image quality index (Q) [53] together with its vector extension Q4 [54], and relative average spectral error (RASE) [55] (the larger Q and Q4 and the smaller SAM and ERGAS, the better performance). Correlation coefficient (CC) [56], with 1 being its ideal value, acts as spatial quality metric. Meanwhile, we use peak signal-to-noise ratio (PSNR) and root mean square error (RMSE) as metrics of fusion accuracy. Generally speaking, better performance is achieved when PSNR is larger and RMSE is smaller.

In each experiment, most of the compared methods require an upsampled multispectral image as an input. Unless specially specified in the literature, we unifiedly generate them by interpolating via a kernel function which is a polynomial with 23 coefficients [57].

The quantitative results of four datasets with regard to eight metrics are reported in

Table 1. Quantitative results for Quickbird dataset.

Method	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
PCA	5.2812	0.7734	0.3895	21.5874	4.7453	0.7520	0.1246	18.0910
GS	2.3162	0.8510	0.8345	12.3841	2.9310	0.9433	0.0715	22.9177
HPF	2.1727	0.8561	0.8299	8.5465	2.0681	0.9420	0.0493	26.1392
MTFGLP	2.2767	0.8756	0.8399	6.1273	1.6287	0.9439	0.0354	29.0296
PHLP	5.0053	0.8184	0.7736	11.5526	2.9020	0.9077	0.0667	23.5214
NIHS	2.9060	0.7212	0.7521	14.4955	3.4247	0.9161	0.0837	21.5503
RKHS	3.0682	0.8725	0.8380	7.4955	1.9868	0.9413	0.0433	27.2789
Proposed	2.2422	0.8816	0.8525	5.1265	1.4605	0.9484	0.0296	30.5785

,

Table 2. Quantitative results for Pléiades dataset.

Method	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
PCA	9.5457	0.7829	0.8387	32.8033	8.0507	0.9276	0.0637	23.9147
GS	9.1222	0.8336	0.8900	28.8060	6.4917	0.9645	0.0560	25.0434
HPF	10.8694	0.8376	0.9243	27.8449	6.7583	0.9722	0.0541	25.3382
MTFGLP	4.7925	0.9063	0.9653	14.6515	3.2470	0.9801	0.0285	30.9155
PHLP	3.9558	0.7749	0.9186	23.6349	4.8308	0.9508	0.0459	26.7620
NIHS	5.8053	0.7807	0.8954	32.9408	6.6098	0.9500	0.0640	23.8784
RKHS	3.8294	0.9071	0.9710	12.0542	2.5088	0.9829	0.0234	32.6103
Proposed	3.2465	0.9278	0.9775	10.3886	2.1452	0.9857	0.0202	33.9019

,

Table 3. Quantitative results for WorldView-2 dataset.

Method	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
PCA	7.1137	0.9256	0.9137	17.4524	3.9960	0.9608	0.0778	22.1796
GS	5.2254	0.9312	0.9438	13.1991	3.2049	0.9776	0.0588	24.6058
HPF	4.1352	0.9185	0.9487	9.9563	2.4925	0.9832	0.0444	27.0547
MTFGLP	4.3926	0.9372	0.9531	9.1386	2.3852	0.9857	0.0407	27.7991
PHLP	4.0716	0.7879	0.9058	13.5540	3.2978	0.9720	0.0604	24.3753
NIHS	1.7128	0.7751	0.8781	29.6650	5.6749	0.9645	0.1323	17.5718
RKHS	5.3524	0.9340	0.9546	11.7716	2.9588	0.9864	0.0525	25.6000
Proposed	3.5176	0.9441	0.9647	6.6818	1.7829	0.9879	0.0298	30.5188

and

Table 4. Quantitative results for SPOT-6 dataset.

Method	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
PCA	8.4179	0.8594	0.9002	23.3863	6.1677	0.9653	0.0565	24.9547
GS	7.0558	0.8976	0.9248	25.6653	5.2751	0.9804	0.0620	24.1470
HPF	7.7539	0.8707	0.9172	26.0012	5.5211	0.9727	0.0628	24.0340
MTFGLP	3.8118	0.9162	0.9362	20.5174	4.1687	0.9795	0.0496	26.0915
PHLP	5.8827	0.7768	0.9044	21.1517	4.8940	0.9659	0.0511	25.8270
NIHS	3.8008	0.7848	0.8770	34.3983	6.4343	0.9588	0.0831	21.6032
RKHS	2.5938	0.9266	0.9530	11.8152	2.8106	0.9842	0.0286	30.8851
Proposed	3.2818	0.9300	0.9566	10.8640	2.6335	0.9843	0.0263	31.6141

. They clearly validate that the rest of the methods outperform PCA method as visual comparison in Section 4.1 preliminarily confirmed. GS method maintain spectral fidelity well on Quickbird dataset and WorldView-2 dataset, while on the rest two datasets it performs not so well. Compared with methods such as MTFGLP method and HPF method, PHLP method and NIHS method give comparable results with respect to SAM. However, there is still a gap when it comes to metrics reflecting spatial quality and fusion accuracy. Similar to GS method, HPF method is unable to preserve enough characteristics on Pléiades dasaset and SPOT-6 dataset from a spectral point of view. If not taking the proposed method into consideration, RKHS method or MTFGLP method performs the best. However, the proposed method consistently achieves better quantitative performance than them in terms of all metrics (except for SAM) on the four tested datasets. These observations are sufficient to demonstrate that the proposed method preserves spectral information and sharp spatial details accurately.

4.3. Discussion on the Number of Outer Iterations

In Algorithm 2, we use an iterative strategy to improve the performance of Algorithm 1. The number of this outer iterations, i.e., $\gamma$, is set to 5. It is meaningful to inspect how the performance of Algorithm 2 changes as $\gamma$ increases.

In Figure 6, we present $I^{(j)}$, $j=1,2,3,4,5$, each being the result of the jth outer iteration of Algorithm 2 with respect to Pléiades dataset. To focus on spatial details, only the first channel of each image is shown. For each of the last four images, we subtract its intensities by the corresponding smallest value in the channel before normalization, since these images contain negative intensities which cannot be plotted by MATLAB directly. For better visualization, the normalization of these four images are implemented by dividing the largest intensity of $I^{(2)}$ since their absolute values are rather small compared with $I^{(1)}$. From these visual results, we know that the outer iteration in Algorithm 2 is able to pick up more image details.

From

Table 5. Quantitative results for Quickbird dataset with different numbers of outer iterations.

Case	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
$\gamma =1$	3.6036	0.8633	0.8274	10.0852	2.5146	0.9393	0.0582	24.7013
$\gamma =2$	2.6197	0.8796	0.8468	6.3718	1.6839	0.9471	0.0368	28.6898
$\gamma =3$	2.2813	0.8816	0.8499	5.4827	1.5165	0.9476	0.0316	29.9951
$\gamma =4$	2.3563	0.8828	0.8496	5.2938	1.5354	0.9465	0.0306	30.2996
$\gamma =5$	2.2422	0.8816	0.8525	5.1265	1.4605	0.9484	0.0296	30.5785
$\gamma =6$	2.3382	0.8822	0.8500	5.1863	1.5145	0.9465	0.0299	30.4778

,

Table 6. Quantitative results for Pléiades dataset with different numbers of outer iterations.

Case	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
$\gamma =1$	5.8470	0.8758	0.9383	19.2451	4.1779	0.9754	0.0374	28.5467
$\gamma =2$	3.5671	0.9187	0.9725	11.8189	2.4454	0.9830	0.0230	32.7816
$\gamma =3$	3.4947	0.9244	0.9746	11.5754	2.3527	0.9839	0.0225	32.9624
$\gamma =4$	4.1877	0.9247	0.9731	12.5506	2.5343	0.9843	0.0244	32.2598
$\gamma =5$	3.2465	0.9278	0.9775	10.3886	2.1452	0.9857	0.0202	33.9019
$\gamma =6$	3.4306	0.9275	0.9761	10.9675	2.2676	0.9854	0.0213	33.4309

,

Table 7. Quantitative results for WorldView-2 dataset with different numbers of outer iterations.

Case	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
$\gamma =1$	5.2761	0.9338	0.9484	12.1816	3.0319	0.9816	0.0543	25.3026
$\gamma =2$	4.2752	0.9422	0.9608	8.2856	2.1955	0.9872	0.0369	28.6502
$\gamma =3$	3.7884	0.9436	0.9637	7.1468	1.8866	0.9877	0.0319	29.9344
$\gamma =4$	3.6676	0.9439	0.9640	7.3470	1.9681	0.9878	0.0328	29.6944
$\gamma =5$	3.5176	0.9441	0.9647	6.6818	1.7829	0.9879	0.0298	30.5188
$\gamma =6$	3.2613	0.9433	0.9652	6.7456	1.8407	0.9876	0.0301	30.4363

and

Table 8. Quantitative results for SPOT-6 dataset with different numbers of outer iterations.

Case	SAM	Q4	Q	RASE	ERGAS	CC	RMSE	PSNR
$\gamma =1$	6.8677	0.9109	0.9284	23.4889	4.8319	0.9803	0.0568	24.9167
$\gamma =2$	3.6602	0.9270	0.9504	13.6097	3.1098	0.9824	0.0329	29.6569
$\gamma =3$	2.8394	0.9287	0.9535	10.9922	2.6297	0.9824	0.0266	31.5123
$\gamma =4$	2.6245	0.9268	0.9518	10.1182	2.5081	0.9819	0.0245	32.2318
$\gamma =5$	3.2818	0.9300	0.9566	10.8640	2.6335	0.9843	0.0263	31.6141
$\gamma =6$	3.8526	0.9295	0.9558	10.8789	2.6852	0.9841	0.0263	31.6022

, quantitative results of the proposed method with $\gamma$ changing from 1 to 6 are listed. When $\gamma =1$, the proposed method essentially becomes Algorithm 1, which makes it convenient to compare it with the performance of Algorithm 2 directly. As expected, after combining Algorithm 1 with the iterative strategy, better quantitative performance is achieved, which can also be inferred from Figure 6.

We observe from Table 5, Table 6, Table 7 and Table 8 that the best quantitative performance is achieved at $\gamma =5$ when $\gamma$ is not too large, i.e., smaller than 6. An exception can be noticed in the case of SPOT-6 dataset, where the best performance is observed at $\gamma =4$. However, for the rest of the datasets, the case of $\gamma =5$ outperforms other cases.

When we let $\gamma$ goes even larger, things become different. This can be seen in Figure 7, which shows how indices vary when $\gamma$ goes from 1 to 20. Since situation for all the tested datasets and all the indices is similar, we only present two indices related to Pléiades dataset here. From Figure 7, it is obvious that the curves of indices fluctuate when $\gamma$ is larger than 3. However, it is also noticeable that there is not much improvement for the performance of Algorithm 2 corresponds to larger $\gamma$ compared with the case of $\gamma =5$.

To explain this phenomenon, an iterative regularization algorithm [58] may be helpful, whose output signals become noisy if the outer iterations are not stopped properly, i.e., smaller than a threshold. And the authors explain that the phenomenon of noisiness is due to the influence of noisy input image. Despite the fact that the output images of Algorithm 2 do not necessarily degrade all the way when $\gamma$ becomes larger than 5, the explanation of the similar phenomenon in [58] is enlightening. Therefore, we suggest that the output of Algorithm 2 is influenced significantly by the blurred upsampling multispectral image when $\gamma$ is large enough, which leads to the fluctuation of numerical indices. Since the success of our iterative strategy has been preliminarily demonstrated by numerical results, an analysis of this conjecture can be explored in our future research. We simply set $\gamma$ as 5 as a tradeoff of computational burden and the performance of the proposed algorithm, and also as a strategy to save the efforts of tuning.

4.4. Time Comparison with RKHS Method

Since we mentioned in Section 3 that the proposed model is supposed to simplify model (3), it is reasonable to expect a reduction in running time when comparing these two methods. Time cost of both methods for the four datasets can be examined in

Table 9. Running time(s) of RKHS and the proposed method on four datasets.

Method	Quickbird	Pléiades	WorldView-2	SPOT-6
RKHS	7263.22	27882.87	16559.93	31576.65
Proposed	207.71	851.11	512.47	863.59

, whose values are presented in the unit of second. We point out that these results are obtained by running on the same laptap with 4GB RAM and 1.70GHz Intel(R) Core(TM) i5-4210U CPU as mentioned in the beginning of Section 4.

From the table, it can be validated that the proposed method is much less time consuming than RKHS method, which means the goal of reduction in computational burden and running time is achieved.

5. Conclusions

In this paper, a new iterative pan-sharpening approach is proposed for the fusion of panchromatic image and multispectral image. The proposed variational model inherits the framework of RKHS method but is essentially different. Instead of paying attention to modelling the unknown high resolution multispectral image in a complicated way, we utilize framelet technique in image restoration for more effectiveness of regularization. An iterative scheme similar to [22] is also employed to improve performance of the proposed algorithm. Experiments on data from Quickbird, Pléiades, WorldView-2, and SPOT are conducted for visual and numerical assessment. The results demonstrate that the proposed method outperforms several state-of-the-art pan-sharpening methods both visually and quantitatively. Meanwhile, it succeeds in reducing time cost compared with RKHS method.