Scaling-up Transformation of Multisensor Images with Multiple Resolutions

For scaling up low resolution multispectral images (LRMIs) with high resolution panchromatic image (HRPI), intensity-hue-saturation (IHS) can produce satisfactory spatial enhancement but usually introduces spectral distortion in the fused high resolution multispectral images (HRMIs). In this paper, to minimize this problem, we present a generalized intensity modulation (GIM) by extending the IHS transform to an arbitrary number of LRMIs, which uses the information of the spectral response functions (SRFs) of the multispectral and panchromatic sensors. Before modulation, the generalized intensity is enhanced by injecting details extracted from the HRPI by means of empirical mode decomposition. After the enhanced generalized intensity is substituted for the old one, the HRMIs are obtained through the GIM. Quickbird images are used to illustrate the superiority of this proposed method. Extensive comparison results based on visual analysis and Wald’s protocol demonstrate that the proposed method is more encouraging for scaling up the LRMIs with the HRPI spectrally and spatially than the tested fusion methods.


Introduction
In many remote sensing applications that require both high spatial and high spectral resolution, such as urban mapping, vegetation identification and land use classification, high resolution panchromatic images (HRPIs) and low resolution multispectral images (LRMIs) are fused using fusion methods to produce high resolution multispectral images (HRMIs), not only to increase the ability of humans to interpret the image dataset, but also for improving the accuracy of the classification [1].
Many image fusion methods have been proposed [1][2][3]. Initial methods mainly focused on intensity modulation for sharpening the LRMI by means of an HRPI. These methods provide good visual HRMIs, while overlooking the requirement of the high quality synthesis of spectral content which is very important for most remote sensing applications based on spectral signatures, such as soil and lithology [4]. Another family of methods, such as high pass filtering (HPF) [5] and gradient pyramid [6], yields HRMIs with much less spectral distortion by injecting high frequency information from the HRPI into the LRMI. However, it is not until the introduction of methods based on multiresolution analysis that HRMI achieved artistic results [7]. Conventional image fusion approaches based on à trous wavelet transform (AWT) [8] implement multiresoltuion decomposition on the HRPI, and then the HRMI can be recovered by performing the inverse AWT (IAWT) from the LRMI and the wavelet planes of the HRPI. However, wavelet based fusion methods do not consider the differences in high frequency information between the HRPI and the LRMIs [9].
The Intensity Hue Saturation (IHS) method can quickly merge massive volumes of data by requiring only resampled LRMIs aside from its high spatial enhancement capability [10]. Its concept is based on the representation of the LRMIs in the IHS system, and then substituting the low resolution intensity component (LRIC) with the HRPI. The inverse IHS transformation allows one to produce the HRMIs. However, the use of such a method for multisensor image fusion often leads to important modifications of the spectral properties of the LRMIs. This is due to the fact that all details contained in the HRPI are directly substituted to the LRIC [10].
A more appropriate use of the IHS method should rather consist of fusing the LRIC with the HRPI through image processing techniques to produce one high resolution intensity component (HRIC). For this purpose, empirical mode decomposition (EMD) is introduced into the fusion of the LRIC with the HRPI. The EMD is a recent method for analyzing nonlinear and nonstationary data, developed by Huang et al. [11]. The final representations of the signal are finite intrinsic mode functions (IMFs) that give not only sharp identifications of salient information but also smooth part of the signal. By manipulating the IMFs, the EMD is very suitable for image fusion [12]. This paper presents a novel scaling up multisensor image fusion method, based on the joint use of generalized intensity modulation (GIM) and the EMD. The GIM is the generalization of the IHS transform, and it incorporates information from the spectral response functions (SRFs) of the LRMI and the HRPI sensors to estimate the LRIC. The EMD is used to extract the spatial details of the HPRI to be injected into the LRIC. As a result, one texture modulated HRIC is produced. Experimental results based on Quickbird images are presented and discussed. Visual analysis and quantitative comparison demonstrate that the new approach provides a satisfactory result, both visually and quantitatively.

GIM based fusion method
The main advantage of the IHS method lies in the separation of spatial information such as an intensity (I) component from the spectral information represented by the hue (H) and saturation (S) components. One can independently manipulate the I component while maintaining the overall color balance of the original images. Traditionally, the IHS method comprises four steps: 1) transform three LRMIs to IHS components; 2) match the histogram of the HRPI with that of the LRIC; 3) replace the LRIC with the stretched HRPI; and 4) inverse-transform IHS channels to three HRMIs.
Forward transform: Backward transform: Backward transform: In (3), α n is the weight coefficient of the LRMI n , which is related with the SRFs of the nth multispectral and panchromatic sensors, and is discussed in the following section.

Production of the LRIC based on SRF
The SRF of a sensor defines the probability that the radiation is detected by this sensor. For producing the LRIC from the {LRMI n } 1≤n≤N and the HRPI, the SRF of the panchromatic sensor (φ(υ)) and the SRFs of the N multispectral sensors ({ψ n (υ)} 1≤n≤N ) are involved. Let the events m n and t be the detection of the radiation by the nth multispectral sensor and the HRPI sensor, respectively. The probabilities of the events m n and t are [7]: The probability of the radiation detected by both sensors (event m n ∩t) is: In geometrical terms, P(m n ∩t) can be understood as the area below φ(υ) and ψ n (υ) (Figure 1, http://www.spaceimaging.com/producs/QuickBird/QuickBird Relative Spectral Response.xls, accessed on July, 8,2005). Given the radiation detected by the nth multispectral sensor, the probability to be detected by the HRPI sensor is: From (8), we can obtain a new LRIC as: where α n is the spectral signature contribution factor of the LRMI n to the LRIC, and preserves the spectral properties of the scanned objects when producing the LRIC. That is, α n is the ratio of the spectral content identified by the HRPI sensor from what the LRMI n records to that identified by the HRPI sensor from all LRMI bands.

Introduction of EMD into the fusion of the LRIC and the HRPI
The IHS method for multisensor image fusion often causes significant spectral distortion in the HRMIs. This is due to the fact that all details contained in the HRPI are directly substituted to the LRIC [10]. A more suitable use of the IHS method should rather fuse the LRIC with the HRPI through an advanced image processing technique to produce a better HRIC. The EMD is a highly efficient and adaptive algorithm for analyzing nonlinear and nonstationary signal [11]. With the development of the EMD, one expects much room for improvement over the simple substitution scheme.
The EMD can decompose a signal into finite intrinsic mode functions (IMFs) and one residue component. Each IMF represents simple oscillatory mode imbedded in the signal [11]. Hence, the EMD offers higher frequency resolution and more accurate timing of nonlinear and nonstationary signal events than traditional integral transforms, and the sum of all IMFs match the original signal perfectly using the inverse EMD (IEMD). For the basic theory of the EMD, interested readers may consult [11] for more details.
For a two dimensional image, the sifting procedure of the EMD is summarized as follows: 1) Treating the original image I as the initial residue component I 0 .
2) Finding all the local extrema, then constructing two smooth cubic splines connecting all the local maxima and minima along rows to get upper envelope u r and lower envelope l r . Similarly, upper envelope u c and lower envelope l c along columns are also obtained. The mean plane ul is defined: Then, the difference between I 0 and ul is: This is one iteration of obtaining the IMF. Checking whether or not ω 1 is an IMF: if not, treating ω 1 as I 0 , and go to 2); if ω 1 is an IMF, and treating the following residue component as I 0 and go to 2): Because the value of ul decreases rapidly for the first several iterations and then decreases slowly, this suggests that the number of iterations can be used as the stopping criterion. Therefore, the appropriate number of iterations to obtain the IMF is used as the stopping criterion.
3) Treating the residue component as the new input. A series of {ω j } 1≤j≤J is obtained by repeating 2) until I J is a monotonic component (J denotes the decomposition level). I can be recovered using the IEMD: Figure 2 shows one example of the EMD. The original image was downloaded from http://www.inrialpes.fr /is2/people/pgoncalv (accessed in April 2007). Before and after the EMD, it is interesting to find that the original image contains three kinds of patterns, and the two modes and the residue component provide very useful information on a series of pattern structures which vary in scale from the smallest to the largest. Hence, the framework of the EMD is suitable for fusing multisensor images by managing the IMFs.

Combined GIM-EMD scaling-up transformation method
The fusion of the LRIC and the HRPI based on the EMD can be considered as constructing one HRIC with the same spectral response as the LRIC and the same spatial response as the HRPI. With the EMD, we expect much room for improvement over the traditional IHS fuser. The proposed procedure takes the following steps ( Figure 3

Experiments
The raw images were downloaded from http://studio.gge.unb.ca/UNB/images. These QuickBird images cover over the Pyramids area of Egypt and were taken in 2002. The test images of size 1024 by 1024 at the resolution of 0.7 m are cut from the raw images. The panchromatic band (450-900 nm) of 0.7 m resolution and blue (450-520 nm), green (520-600 nm), red (630-690 nm), near infrared (760-900 nm) bands of 2.8 m resolution are used as the HRPI and LRMIs, respectively. Figure 4(a) displays the LRMIs in color image by mapping the red, green, blue bands into the RGB color space. Figure 4 The qualities of the HRMIs are estimated both qualitatively and quantitatively. Visual inspection is used for qualitative estimation since visual inspection is an effective tool for analyzing local as well as global variations of spatial structures and spectral information of the HRMIs. Wald's protocol is used to assess the qualities of the HRMIs quantitatively.

Visual inspection
Visual inspection provides an overall impression of image clarity and the similarity of the original and fused images. Visual analysis shows that the spatial resolution of the HRMIs is much higher than that of the LRMIs. The HRMIs present more details without noticeable spectral distortion except that of the IHS method, such as edges and slopes. Many textures and details in the HRMIs, such as edges and lines, can be identified individually in each of the HRMIs. This means that all of the fusion methods can improve the spatial quality of the LRMIs via the fusion procedure.
From  Figure 4(i)] produced by the proposed method appear the best among the HRMIs, and the integration of spatial features and color is natural. This effect can be seen clearly in Figure 5 by enlarging a region of interest. For the IHS and BT methods, this is due to the fact that all details contained in the HRPI are directly injected into the LRMIs [10]. For additive methods, such as AWT, HPF, and HPM, this is probably due to over enhancement along the edge area because these methods have not considered the differences in high frequency information between the HRPI and the LRMIs [4]. For the DWT method, the critically sampled multiresolution analysis does not preserve the translation invariance [3].

Quantitative comparison
In addition to visual analysis, the performance of each method is further quantitatively analyzed by checking Wald's protocol [13] using the following quantitative indexes. 1) Correlation coefficient (CC) between each band of the original LRMIs and the HRMIs.
2) Root mean square error (RMSE) between the LRMI and the HRMI, computed using the following equation: where the bias is the difference between the mean values of the LRMI and the HRMI and SDD the standard deviation of the difference image. RMSE should be as close to 0 as possible.
3) Spectral angle mapper (SAM) is defined as: where M is the mean radiance of the N LRMI bands (B i ). RASE should be as close to 0 as possible. 5) Q 4 , defined as [14]: where x and y, which denote the four band LRMIs and the HRMIs, respectively, are both expressed as quaternions (e. g. x=x 1 +i·x 2 +j·x 3 +k·x 4 ). E[·] denotes the expected value,  x is the quaternion obtained by averaging the four LRMIs, and ||x|| is the magnitude of the quaternion. It should be as close to 1 as possible.
6) Erreur relative globale adimensionnelle de synthèse (ERGAS) [13] is given by: where h is the resolution of the HRPI, l the resolution of the LRMI, N the number of HRMIs, and M i the mean of the HRMI i . Bias is the difference between the mean of the LRMI and HRMI, and SDD the square root of the difference image between each band of the LRMIs and the HRMIs. Three criteria based on the Wald's protocol were employed to test the degree of spectral distortion caused by the fusion methods [14]: (1) In order to test the first property of Wald's protocol, the HRMIs are spatially degraded to the resolution level of the original LRMIs (2.8 m) by cubic interpolation. Then, the degraded HRMIs (DHRMIs) are compared with the original LRMIs. Table 1 shows the results. (2) In order to test the second and third properties of Wald's protocol, the fusion results (LHRMIs) of the degraded HRPI and LRMIs (4 times degraded in resolution by cubic convolution) are also compared with the LRMIs. Table 2 shows the results. In Tables 1 and 2, B 1 , B 2 , B 3 and B 4 denote the red, green, blue, and near infrared bands, respectively, and the last column reflects the ideal situation that should be reached after the fusion process.
It can be seen from Tables 1 and 2 that all fusion methods yield high scores for the DHRMIs and LHRMIs. In general, the proposed method produces less spectral distortion than other fusion methods. Hence, the proposed method allows a higher transformation of the texture information of the HRPI when preserving the spectral content of the LRMIs. The proposed method outperforms other fusion methods in fusing the LRMI with the HRPI, because the fusion model takes into account detail injection, as is the case of the EMD based fuser, and spectral signature, as is the case of the GIM based on the SRFs of the sensors. These aspects of the proposed method allow producing the HRMIs closer to the real HRMIs that the QuickBird multispectral sensor would take at the spatial resolution of the HRPI than other fusion methods. In order to estimate the spatial quality of the HRMIs, we follow the procedure proposed by Zhou [15]. First, the spatial detail information present in the two images to be compared is extracted using the following Laplacian filter. Second, spatial correlation coefficient (SCC) between these two filtered images is calculated. The SCC indicates that how much the detail information of one of the images is present in the other. A high SCC shows that most spatial information of the HPRI has been incorporated into the LRMI during the fusion process: Because fusion method injects different amount of details into different band of the LRMIs, for the purpose of evaluating roundly the detail injection performance of fusion method, the average SCC (SCC avg ) is used as a global spatial quality index for the HRMIs. A good fusion method must allow the injection into each band of the LRMIs of the details the multispectral sensor would capture if it worked at a spatial resolution similar to that of the panchromatic sensor. That means the higher the SCC avg value the higher the spatial quality of the HRMIs. Table 3 shows the results. The proposed method outperforms the AWT, BT, DWT, HPF, and HPM fusion methods in incorporating spatial details of the HRPI into the LRMIs by taking into account the separation of spatial information from the spectral information, as is the case of the EMD decomposition though the IHS method is the best. This injection model allows producing the HRMIs closer to the real HRMIs that the multispectral sensor would take at the spatial resolution of the HRPI. Visual inspection and quantitative comparison show that the proposed method gets the advantage of many traditional methods in fusing the LRMIs with the HRPI when the HRMIs are compared with the LRMIs.

Conclusions
In this paper, we wed the ideas of SRF based GIM and the EMD for fusing the LRMI with the HRPI of the same scene in order to obtain one HRMI. The LRIC used in the GIM is obtained from weighted averaging the LRMIs based on the SRFs of the multispectral and panchromatic sensors for separating the low spatial intensity from the spectral information while the EMD is introduced for alleviating the spectral distortion caused by the IHS approach. The LRIC is replaced with the produced HRIC. Finally, the HRMIs are produced by performing the GIM.
QuickBird LRMIs and HRPI are used to demonstrate the advantage of the proposed method over the traditional fusion approaches in terms of preserving the spectral properties of the LRMIs. The experimental results are compared with those of six fusion methods by visual inspection and quantitative comparison. The comparison results confirm the spectral preservation property of the proposed method. All these results are encouraging, and they show that the proposed method can achieve better spectral preservation together with spatial enhancement.