Multi-Band and Polarization SAR Images Colorization Fusion

Abstract

The image fusion of multi-band and multi-polarization synthetic aperture radar (SAR) images can improve the efficiency of band and polarization information processing. In this paper, we introduce a fusion method that simultaneously fuses multi-band and polarization SAR images. In the method, we first use non-subsampled shearlet transform (NSST) to fuse multi-band and polarization SAR images. The sub-band images decomposed from the NSST are fused by the coefficient of variation (CV) and phase consistency (PC) weighted fusion rules. Subsequently, we extract the band and polarization difference information from the multi-band and polarization SAR images. The fusion image is finally colorized according to the band and polarization differences. In the experiments, we used Ka and S-band multi-polarization SAR images to test the fusion performance. The experiment results prove that the proposed fused images not only preserve much valuable information but also can be interpreted easily.

1. Introduction

Synthetic aperture radar (SAR) is a high-resolution imaging technique that uses electromagnetic waves in the microwave spectrum to acquire electromagnetic scattering characteristics of the detection area [1,2,3]. Different with optical sensor, SAR is an active imaging technique, and its long-wavelength radiation can penetrate through most climatic conditions. Therefore, SAR imaging has the advantages of long imaging distance and all-day, all-weather operation [4]. However, SAR images are poor in object recognition and band information. This has motivated researchers to fuse SAR images with other remote sensing images to obtain better visual performance and additional valuable information.

SAR image fusion for enhancing visual performance has been widely studied by researchers in recent years. There are various ways to achieve the fusion [5,6,7,8,9,10,11,12,13,14]. Among them, the multi-scale decomposition methods, such as wavelet transform [5,6,7], non-subsampled contourlet transform (NSCT) [8], and non-subsampled shearlet transform (NSST) [9] have been frequently used in the fusion of SAR images because of their good fusion performance and fast implementation. There are various fields where the fused SAR image can be applied, such as searching ice cracks [5], improving the quality of urban remote sensing images [8,9], or revealing cloud-obscured areas in optical images [10]. After the fusion, the object recognition of the fused SAR image is significantly improved, thus, many researchers utilize the fused SAR image for target detection, which includes obtaining the distribution of buildings in cities [11], monitoring the damage of urban buildings after earthquakes [12], monitoring glaciers in the ocean [13], and the detection of ships in ports [14].

The fusion between SAR images has received attention from researchers in an effort to obtain more band and polarization information [15,16,17,18,19,20,21,22]. Due to the different penetration and electromagnetic characteristics of the different bands, the fusion of multi-band SAR images can be used to achieve the detection of special targets. Wu et al. used the fusion of X, C, and L-band SAR images to achieve bridge detection [17]. Guida et al. used the fusion of X and S-band SAR images to identify oil and gas [18]. Chanika et al. applied X and C-band SAR image fusion to enhance the classification accuracy of maize lands [19]. Moreover, the fusion of multi-polarization SAR images can be utilized to distinguish the regions with different surface structures. Ruan et al. fused horizontal–horizontal (HH), horizontal–vertical (HV) and vertical–vertical (VV) polarized SAR images to improve the classification accuracy of different areas [20]. Song et al. fused HH and HV polarized SAR images to reduce the false alarm rate of moving target detection [21]. Zhu et al. enhanced the performance of vessel detection by using the fusion of HH and VV polarized SAR images [22].

However, although the fused multi-band SAR image can be used to detect objects that vary in scattering characteristics with respect to the band, it is difficult to determine the kind of objects they are. In addition, although the fused multi-polarization SAR image can be used to distinguish the regions with different surface structures, it is difficult to find masked objects based on the information of a single band. Therefore, if we can fuse the information of multi-band and multi-polarization SAR images at the same time, we can detect the masked object and determine the kind of object it is simultaneously, which will greatly facilitate the application of the SAR images.

Unfortunately, there are few effective fusion methods for the multi-band and polarization SAR images. Traditional fusion methods compress input images into a grayscale image. This compression will result in the loss of critical band and polarization information. Furthermore, the grayscale image is hard to interpret. The objects with only a one-dimensional grayscale difference are difficult to detect and discriminate.

In this paper, we propose a colorization fusion method for multi-band and polarization SAR images. First, we use non-subsampled shearlet transform (NSST) to decompose and fuse multi-band and polarization SAR images. Among them, the low and high-frequency sub-band images acquired from NSST are fused using the fusion rules based on coefficient of variation (CV) and phase consistency (PC), respectively. Then, the band difference map and the polarization difference map are obtained from the multi-band and polarization SAR images through the calculation of intensity differences and color saturation, respectively. Finally, the fused image is colorized according to the difference maps. This proposed fusion method not only preserves the detail information between different bands of SAR images but also increases the fused image by the band and polarization differences information, which achieves a high visual performance.

The rest of this paper is divided into three parts. Section 2 introduces the detailed fusion rules of multi-band and polarization SAR images. Section 3 presents the results of the proposed fusion method as well as the comparison with other fusion methods. We conclude in Section 4.

2. Methodology

In this section, we first introduce the concepts of NSST. Then, we present the NSST-based image fusion rules, in which the low and high-frequency sub-band images are respectively fused according to the CV and PC. Subsequently, the coloring rules for the fused SAR image are proposed. At the end of this section, we introduce the dataset and evaluation indexes that will be used in the experiments.

2.1. Concepts of NSST

Non-subsampled shearlet transform (NSST) is a non-orthogonal transform derived from wavelet transform [23]. For a continuous wavelet, a two-dimensional affine system with composite dilations is defined as:

$$M_{AS}\left(\psi \right)=\left\{{\left(\psi \right)}_{j,l,k}\left(x\right)={\left|\det A\right|}^{j/2}\psi \left(S^l{A}^{j}x-k\right):l,j\in Z,k\in Z^2\right\}$$

(1)

where $\Psi \in L^2\left(R^2\right)$, $L^2\left(R^2\right)=\{f\left(x,y\right):{{\displaystyle \int }}_{-\infty }^{+\infty }{{\displaystyle \int }}_{-\infty }^{+\infty }{\left|f\left(x,y\right)\right|}^2<+\infty \}$. j, l, and k denote scale, direction, and shift parameter, respectively. $A$ is the dilation matrix, and $A^j$ is related to the scale decomposition. $S$ is the shear matrix, and $S^l$ is related to the direction decomposition. When $A=\left(\begin{array}{cc}4& 0\\ 0& 2\end{array}\right)$ and $S=\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)$, they are called the shearlet filter (SF), and this system is called the shearlet transform.

Compared with the shearlet transform, NSST utilizes the not sampling pyramid (NSP) to achieve the scale decomposition [23]. The process of NSST is described as follows: First, NSP scale decomposition is performed to obtain a low-frequency sub-band image and j-1 high-frequency sub-band images. Then, the high-frequency sub-band images are processed with SF to obtain the sub-band images in different directions. Figure 1 shows the three-level NSST image decomposition. As a multi-scale decomposition method, NSST can achieve image multi-scale and multi-directional decomposition with high speed.

2.2. Fusion Rules

After decomposing the multi-band and multi-polarization SAR images by NSST, we can acquire their low and high-frequency sub-band images. Then, we fuse the low and high-frequency sub-band images respectively, according to the individual fusion rule, as follows.

2.2.1. Low-Frequency Fusion Rule

The low-frequency sub-band image contains the approximate information as well as the majority of the energy of the original SAR image. We use the coefficient of variation (CV) [24] as the weight of the low-frequency sub-band images. The CV is calculated as follows:

$$CV\left(x,y\right)=\frac{\sqrt{\frac{1}{n}{\displaystyle {\sum }_{i=1}^n{\left(I\left(x_i,y_i\right)-\mu \right)}^2}}}{\mu }$$

(2)

where n represents the number of pixels in a window, $I\left(x_i,y_i\right)$ is the neighbor pixel of the center pixel $I\left(x,y\right)$, and $\mu$ represents the mean of the pixels in the window. The calculation window of the CV is $3\times 3$ in this paper.

CV can describe the degree of local variation of a pixel. If a pixel has larger CV, it will provide more useful information and should have larger weight than the corresponding pixel in the other low-frequency sub-band image. After calculating the CV for each pixel in the low-frequency sub-band image and making the CV as their weights, the fused low-frequency sub-band image is acquired as follows:

$$LF\left(x,y\right)=\frac{{\displaystyle {\sum }_k^{m}C{V}_k\left(x,y\right)L_k\left(x,y\right)}}{{\displaystyle {\sum }_k^{m}C{V}_k\left(x,y\right)}}$$

(3)

where m represents the number of the original images, and L is the low-frequency sub-band image decomposed from NSST.

2.2.2. High-Frequency Fusion Rule

The high-frequency sub-band image contains the detailed texture, contour features, as well as interfering noise. In order to preserve the detailed texture and contour features as much as possible and to minimize the effect of noise, we use the PC weighted fusion method to achieve the fusion of the high-frequency sub-band images.

Phase Congruency (PC) is an image contour extraction method. Because PC analyzes the phase information of image in the frequency domain, it is invariant to the illumination change [25] and thus, has been widely used in the SAR image registration [26,27,28].

For the phase analysis, we should first extract the multi-frequency and orientation phase information from the original SAR image. In practice, the image is convolved with multi-frequency and orientation band-pass filters to achieve the extraction as follows:

$$A_{\omega ,\theta }\left(x,y\right)=\mathrm{Re}\left(I\left(x,y\right)*f_{\omega ,\theta }\left(x,y\right)\right)$$

(4)

$${\phi }_{\omega ,\theta }\left(x,y\right)=\mathrm{Im}\left(I\left(x,y\right)*f_{\omega ,\theta }\left(x,y\right)\right)$$

(5)

where $I$ represents the original SAR image. $A_{\omega ,\theta }$ and ${\phi }_{\omega ,\theta }$ represent the amplitude and phase at the frequency ω and orientation $\theta$, respectively. $f_{\omega ,\theta }$ represents the band-pass filter in the time domain. The function $\ast$ denotes the convolution of two metrics. The oriented log-Gabor filter is suitable as the band-pass filter f. In the frequency domain, the oriented log-Gabor filter is defined as

$$f_{\log -Gabor}\left(\omega \right)=f_{\theta }\cdot \exp \left(\frac{-\log {\left(\omega /{\omega }_0\right)}^2}{2\left(\log \left(\kappa /{\omega }_0\right)\right)}\right)$$

(6)

where $f_{\theta }$ is direction filter. ${\omega }_0$ is the central frequency of the log-Gabor filter, and $\kappa$ is a parameter that controls the band width. Fourier transform can achieve the transformation of the oriented log-Gabor filter between the frequency domain and the time domain.

Considering the effect of image noise, the phase deviation $\Delta {\phi }_{s,\theta }$ as well as the PC in the orientation $\theta$ are defined as:

$$\Delta {\phi }_{s,\theta }\left(x,y\right)=\cos \left({\phi }_{s,\theta }\left(x,y\right)-{\overline{\phi }}_{\theta }\left(x,y\right)\right)-\left|\sin ({\phi }_{s,\theta }\left(x,y\right)-{\overline{\phi }}_{\theta }\left(x,y\right))\right|$$

(7)

$$P{C}_{\theta }\left(x,y\right)=\frac{{\displaystyle \sum _s{W}_{s,\theta }\left(x,y\right)⌊A_{s,\theta }\left(x,y\right)\Delta {\phi }_{s,\theta }\left(x,y\right)-T⌋}}{{\displaystyle \sum _s{A}_{s,\theta }\left(x,y\right)+\epsilon }}$$

(8)

where ${\overline{\phi }}_{\theta }$ is mean phase angle, $W_{s,\theta }$ is the weighting function related to the band-pass filter, T is the estimated noise threshold, and $\epsilon$ is a small constant that avoids division by zero. The function $\lfloor ·\rfloor$ denotes that the enclosed quantity is equal to itself when its value is positive, and zero otherwise. If the $P{C}_{\theta }$ value of the pixel is close to 1, this pixel has good phase congruency in the orientation $\theta$ and is likely an edge pixel of the SAR image.

Furthermore, taking the orientation into account, we can distinguish the kind of edge. The maximum moment PCM and the minimum moment PCm can be obtained as follows [29,30]:

$$PCM=\frac{1}{2}\left(c+a+\sqrt{b^2+{\left(a-c\right)}^2}\right)$$

(9)

$$PCm=\frac{1}{2}\left(c+a-\sqrt{b^2+{\left(a-c\right)}^2}\right)$$

(10)

in which the three intermediate quantities are calculated by:

$$\begin{array}{l}a={{\displaystyle \sum _o\left(P{C}_{{\theta }_o}\cos \left({\theta }_o\right)\right)}}^2\\ b=2{\displaystyle \sum _o\left(P{C}_{{\theta }_o}\cos \left({\theta }_o\right)\right)\cdot \left(P{C}_{{\theta }_o}\sin \left({\theta }_o\right)\right)}\\ c={{\displaystyle \sum _o\left(P{C}_{{\theta }_o}\sin \left({\theta }_o\right)\right)}}^2\end{array}$$

(11)

where $P{C}_{{\theta }_o}$ denotes the PC at the orientation of ${\theta }_o$. PCM and PCm represent the edge and corner maps of the original SAR image, respectively. In this paper, we select PCM as the output of the PC method. To make the contours more visible, the PCM is optimized as follows:

$$PCM\left(x,y\right)=\max \left(PCM\left(x+x^{\prime },y+y^{\prime }\right)\right),x^{\prime }\in \left[-1,1\right],y^{\prime }\in \left[-1,1\right]$$

(12)

Finally, the PCM is made as the weights of each high-frequency sub-band image, then, the fused high-frequency sub-band images can be obtained as follows:

$$H{F}_{j,l}\left(x,y\right)=\frac{{\displaystyle {\sum }_k^{m}PC{M}_k\left(x,y\right)H_{j,l,k}\left(x,y\right)}}{{\displaystyle {\sum }_k^{m}PC{M}_k\left(x,y\right)}}$$

(13)

where m denotes the number of the original images, and H is the high-frequency sub-band image decomposed from NSST. j and l represent the scale and direction in the NSST, respectively.

So far, we have obtained the fused low-frequency sub-band image and the fused high-frequency sub-band images. Then, the NSST inversion is applied to the sub-band images to realize the fusion of the multi-band and polarization SAR images. Subsequently, we extract the band and polarization difference information from the original multi-band and polarization SAR images to colorize the fused SAR image.

2.3. Coloring Rules

2.3.1. Band Difference Extraction

Since the penetration of the detection microwaves vary from different frequencies, the masked object will show different intensity in the multi-band SAR images. We can locate these objects by extracting the band difference map between the images. The extraction of band difference is conducted as follows:

$$D_{B1-2}\left(x,y\right)=\left|\max \left(I_{B1,P1}\left(x,y\right)\right)-\max \left(I_{B2,P2}\left(x,y\right)\right)\right|,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}P1,P2\in \left[HH,HV,VV\right]$$

(14)

in which the $I_{B1,P1}$ and $I_{B2,P2}$ represent the different band SAR images at P1 and P2 polarization, respectively. If there are more than two bands, the band differences between each band should be calculated. Then the maximum $D_B$ of pixels between each band is retained to obtain the final band difference map, as follows:

$$D_B\left(x,y\right)=\max \left(D_{B1-2}\left(x,y\right),D_{B2-3}\left(x,y\right),D_{B1-3}\left(x,y\right)\dots \right)$$

(15)

However, large amounts of noise in the SAR image is retained in the $D_B$, which will reduce the quality of the band difference map. To reduce the impact of noise, we calculate the mean of $D_B$ in the band difference map. The values less than mean of $D_B$ in the band difference map is set to zero, then the map is convolved with $3\times 3$ mean filter.

2.3.2. Polarization Difference Extraction

The regions with different surface structures will present different polarization information. Therefore, we can use the color saturation [31] to extract the polarization difference map to distinguish different regions easily.

The color saturation of each pixel in the polarization difference map is calculated as:

$$S_B\left(x,y\right)=\frac{1-3\min \left(I_{B,P}\left(x,y\right)\right)}{{\displaystyle {\sum }_P{I}_{B,P}\left(x,y\right)}},P\in \left[HH,HV,VV\right]$$

(16)

in which $I_{B,P}$ denotes a certain band SAR image at polarization P. If a pixel has high saturation, the intensity of the pixel varies sharply among the multi-polarization SAR images, indicating that the pixel belongs to a special region. According to the color saturation values of the pixels, we can obtain the mean of them in the polarization difference map. In order to reduce the effect of noise, the values less than mean of $S_B$ in the polarization difference map is set to zero, then the map is convolved with $3\times 3$ mean filter.

Finally, the maximum $S_B$ of pixels in each band is retained to obtain the final polarization difference map, as follows:

$$S\left(x,y\right)=\max \left(S_{B1}\left(x,y\right),S_{B2}\left(x,y\right),\dots \right)$$

(17)

2.3.3. Coloring Rules of Difference Information

So far, we have obtained the fused SAR image, the band difference map as well as the polarization difference map from the multi-band and polarization SAR images. Next, the band and polarization difference maps are expressed on the fused SAR image by different colors.

Since the blue objects cannot be distinguished well in the human eyes because of the low brightness, we use red and green to express the band and polarization difference map, respectively. The regions with both band and polarization differences will appear yellow.

As the polarization difference map is presented as saturation, it is first transformed into the intensity difference as follows:

$$D_P\left(x,y\right)=F\left(x,y\right)*\frac{2S\left(x,y\right)}{1+S\left(x,y\right)}$$

(18)

in which the F is the fused SAR image. The intensity of each color in the fused SAR image is assigned according to the following rules:

$$\begin{array}{l}R\left(x,y\right)=\left\{\begin{array}{c}F\left(x,y\right)+D_B\left(x,y\right)-D_P\left(x,y\right) , D_B\left(x,y\right)<D_P\left(x,y\right)\\ F\left(x,y\right) , D_B\left(x,y\right)\ge D_P\left(x,y\right)\end{array}\right.\\ G\left(x,y\right)=\left\{\begin{array}{c}F\left(x,y\right) , D_B\left(x,y\right)<D_P\left(x,y\right)\\ F\left(x,y\right)-D_B\left(x,y\right)+D_P\left(x,y\right) , D_B\left(x,y\right)\ge D_P\left(x,y\right)\end{array}\right.\\ B\left(x,y\right)=\left\{\begin{array}{l}F\left(x,y\right)-D_B\left(x,y\right) , D_B\left(x,y\right)\ge D_P\left(x,y\right)\\ F\left(x,y\right)-D_P\left(x,y\right) , D_B\left(x,y\right)<D_P\left(x,y\right)\end{array}\right.\end{array}$$

(19)

After that, we obtain the colorized fused SAR image that contains band and polarization differences information.

The specific flow chart of multi-band and polarization SAR images fusion in this paper is shown in Figure 2.

2.4. Datasets

Two pairs of air-borne SAR images were selected for the fusion quality analysis. Each image pair contains two band (Ka and S-band) and three polarization (HH, HV and VV) SAR images. The polarization information is painted in different colors, in which HH is red, HV is green, and VV is blue. The resizing and registration between the multi-band SAR images was accomplished. Each SAR image had the size of $800\times 800$ and the resolution of 1 m. We also provide the optical images of the same scenes as references. All images were taken during an airplane flight on 14 March 2021.

The two pairs of Ka and S-band multi-polarization SAR images are shown in the Figure 3a,b and Figure 4a,b, respectively. Since the frequency of Ka-band microwave is higher than S-band microwave, the Ka-band SAR image has a higher resolution [1,2,3]. After the resizing, the Ka-band multi-polarization SAR images appear sharper than the S-band. Moreover, because the electromagnetic scattering characteristics of objects varies with the frequencies of the detection microwaves, the detailed texture of the objects in the Ka and S-band multi-polarization SAR images is significantly different. Furthermore, the polarization information of the Ka and S-band SAR images is also inconsistent. Since different kinds of objects present different polarization information, we can easily distinguish these regions such as farmlands or buildings from the images according to their own colors.

2.5. Evaluation indexes

Several statistical indexes will be used to evaluate the fused images, which include average gradient (AG), information entropy (IE), standard deviation (STD), correlation coefficient (CC), and structural similarity index measure (SSIM) [4,9]. Their introduction and specific equations are shown as follows:

• Average Gradient

AG represents the detail-describing ability of the fused image. It can be obtained by calculating the mean of image gradients:

$$AG=\frac{1}{xy}{\displaystyle \sum _x{\displaystyle \sum _{y}G\left(x,y\right)}}$$

(20)

$$G\left(x,y\right)=\frac{1}{2}\left(\left|G_x\left(x,y\right)\right|+\left|G_y\left(x,y\right)\right|\right)$$

(21)

where $G_x$ and $G_y$ are the gradients of pixels in the vertical and horizontal directions, respectively. $G_x$ and $G_y$ can be acquired by convolving the image with ${\left[-\begin{array}{cc}1/2& 1/2\end{array}\right]}^T$ and $\left[-\begin{array}{cc}1/2& 1/2\end{array}\right]$, respectively.

• Information Entropy

IE is the most intuitive standard for reflecting the amount of image information. The following equation is used to calculate information entropy.

$$IE=-{\displaystyle \sum _{k=1}^{L-1}p\left(k\right)}{\log }_{2}p\left(k\right)$$

(22)

where L is the dynamic range of the image being analyzed, and $p\left(k\right)$ is the probability of occurrence of $k^{th}$ gray level. In the 8-bit image, L is 255.

• Standard Deviation

STD is a measure of contrast in the fused image. High contrast in the fused image indicates information richness. Standard deviation can be calculated as follows:

$$SD=\sqrt{\frac{1}{N-1}{\displaystyle \sum _{i=1}^N{\left|F_i-\mu \right|}^2}}$$

(23)

where μ is the mean value of the fused image, F, N is the number of the pixels in the fused image.

• Correlation Coefficient

CC is a measure of the correlation between the reference image and the fused image. It can be calculated as follows:

$$CC=\frac{{\displaystyle \sum _x{\displaystyle \sum _y\left(F\left(x,y\right)-\overline{F}\right)\left(R\left(x,y\right)-\overline{R}\right)}}}{\sqrt{\left({\displaystyle \sum _x{\displaystyle \sum _y{\left(F\left(x,y\right)-\overline{F}\right)}^2}}\right)\left({\displaystyle \sum _x{\displaystyle \sum _y{\left(R\left(x,y\right)-\overline{R}\right)}^2}}\right)}}$$

(24)

where F and R are fused and reference images, respectively. $\overline{F}$ and $\overline{R}$ are mean values of fused and reference images, respectively.

• Structural Similarity Index Measure

SSIM measures structural similarity between the reference image and the fused image. It is calculated using following equation:

$$SSIM=\frac{\left(2{\mu }_f{u}_r+C_1\right)\left(2{\sigma }_{fr}+C_2\right)}{\left({\mu }_f^2+{\mu }_r^2+C_1\right)\left({\sigma }_f^2+{\sigma }_r^2+C_2\right)}$$

(25)

where f and r represent the fused and reference images, respectively; ${\mu }_f$ and ${\mu }_r$ are their mean values; ${\sigma }_f^2$ and ${\sigma }_f^2$ are their variances; ${\sigma }_{fr}$ is the covariance between them; and $C_1$ and $C_2$ are small constants for stabilizing denominator with weak division.

It should be noted that if the input image is in color, the image needs to be converted to a grayscale image first.

3. Experiments

3.1. Comparison of Fusion Results

We first selected a special region in each scene to analyze the performance of the proposed fusion method. As mentioned in Section 2, these SAR images are firstly decomposed by three-level NSST, in which the number of decomposition directions is $\left[1,4,8\right]$. Subsequently, the low-frequency and high-frequency sub-band images obtained from the decomposition are respectively fused according to the CV and PC. Next, the inverse NSST transform of the fused sub-band images is conducted to obtain the fused SAR image. We also present the results of various image fusion methods as the comparison, which include principal component analysis (PCA) [17], wavelet transform [16], non-subsampled contourlet transform (NSCT) [32], and primary NSST [9].

Figure 5a–c show the special region in scene 1. In this region, the Ka-band SAR image possesses high sharpness and rich detail information, but it does not present complete building information. On the contrary, the S-band SAR image shows the masked building due to the high penetration of the S-band microwave, but it is filled with noise. Figure 5d–h present the results of various fusion methods of the representative region in scene 1. Among them, the wavelet-based fused image (Figure 5e) does not preserve detailed texture. The fused images of PCA and NSCT (Figure 5d,f) retain a part of detailed texture, but the masked building is not apparent enough. Both the NSST and proposed fusion images (Figure 5g,h) present clear buildings. However, as we enlarge the marked regions (Figure 6a,b), the contour of the building in the NSST fusion image is blurred by noise. In contrast, the building in the proposed fusion image is sharper.

Figure 7a–c show the special region in scene 2. The Ka-band SAR image has higher visual performance, but lacks underwater information. The S-band SAR image has underwater information, but its visual performance is poor. Figure 7d–h present the results of the fusion methods. Similar to the results of scene 1, the fused images of PCA, wavelet, and NSCT do not preserve much detailed texture. Additionally, both the NSST and the proposed method present clear underwater information. After enlarging the marked regions in the NSST and proposed fusion images (Figure 8a,b), we can observe that the texture in the NSST is covered by noise, but the texture in the proposed fusion image is presented in detail. These fusion results demonstrate that the proposed fusion method can remove much noise while retaining the detailed information of the original images, which outperforms other methods.

3.2. Comparison of Colorization Results

Although the fusion methods compress the information of the SAR images, it is difficult to distinguish different regions from the grayscale map. To facilitate the use of the fusion image, we extract the band and polarization difference information from the SAR images and then inserted the information into the fusion results according to the coloring rules.

Figure 9a,b show the band and polarization difference maps extracted from special region 1, respectively. From the difference maps we can see that all buildings in this region have unique polarization information, whereas the masked building have additional band difference. After the colorization, the visual performance of all fusion images is dramatically improved, as shown in Figure 9c–g. With the color distribution, we can easily locate the buildings. In particular, the exposed building is painted in green because of its special polarization information. In contrast, we marked the masked building in red due to its band difference. Benefiting from the fusion method, the colorized proposed fusion image reveals more completed building information and sharper contour than other colorized fusion images.

Difference extraction and colorization were also conducted on special region 2. According to the band and polarization difference maps (Figure 10a,b), the buildings are painted in green due to their special polarization information. The underwater information consists of both band and polarization differences and thus. is painted in yellow. In the colorized results (Figure 10c–g), the PCA, wavelet, and NSCT display little underwater information. Both the NSST and proposed fusion images highlight the underwater information, but the colorized proposed fusion image retains more detailed texture and less noise. These results demonstrate that the proposed fusion method is more suitable for colorization.

The complete difference maps and colorization fusion results of scenes 1 and 2 are shown in Figure 11 and Figure 12, respectively. We can see that the band and polarization difference maps mark the location of the special regions. Even at larger scales, we can still spot the buildings or farmland from the colorized fusion results at a glance. Furthermore, the special objects can be easily located according to their outstanding colors. These results prove that the colorization fusion method can not only preserve the valuable information from the multi-band and polarization SAR images, but also reduce the difficulty of interpreting the SAR images.

However, it is difficult for human eyes to compare these large-scale images. Therefore, we used various statistical evaluation indexes to complete the comparison.

3.3. Comparison of Evaluation Indexes

This section contains the analysis of various evaluation indexes for the colorized fusion images in Figure 11 and Figure 12. The evaluation indexes include average gradient (AG), information entropy (IE), standard deviation (STD), correlation coefficient (CC), and structural similarity index measure (SSIM). The Ka-band SAR image is used as the reference image in the CC and SSIM indexes.

AG, IE, and STD can reflect the abundance of the texture in the image. The more texture information in the image, the higher the value of these three indexes. As shown in

Table 1. Evaluation indexes of the fused images in scene 1.

	AG	IE	STD	CC	SSIM	Time(s)
PCA	10.373	7.270	38.721	0.929	0.753	8.031
wavelet	8.952	7.243	37.777	0.934	0.762	3.828
NSCT	10.333	7.233	37.668	0.920	0.752	103.903
NSST	10.820	7.282	39.062	0.917	0.735	16.625
proposed	12.137	7.314	40.075	0.903	0.716	25.233

and

Table 2. Evaluation indexes of the fused images in scene 2.

	AG	IE	STD	CC	SSIM	Time
PCA	10.748	7.271	40.922	0.926	0.723	8.013
wavelet	9.437	7.252	39.927	0.938	0.759	3.611
NSCT	10.601	7.235	39.716	0.932	0.757	103.419
NSST	11.161	7.257	40.807	0.925	0.734	16.599
proposed	11.956	7.287	41.702	0.920	0.723	25.165

, both the proposed methods present the highest values of the three indexes, which is consistent with the results of the visual-based analysis. The index results of the two scenes have a similar distribution. The three indexes of the PCA, wavelet, and NSCT fusion images are lower than the proposed fusion images, because these methods do not preserve much detail texture. The NSST-based fusion images retained both detail texture and noise, and thus their AG, IE and STD indexes is close to the proposed fusion images. Since the proposed fusion method preserves the detail texture while avoiding most of the noise, the AG, IE, and STD of the proposed fusion images are higher than the NSST.

CC and SSIM measure the similarity between the colorized fusion images and the Ka-band SAR images. The lower the CC and SSIM indexes are, the less information is retained from the Ka-band SAR images, i.e., the more information from the S-band SAR image. Although the S-band SAR images have less detail information than the Ka-band SAR images, they have some additional information, such as masked building and underwater information, that is worth preserving. Therefore, the indexes of CC and SSIM imply the ability of these fusion method to retain the additional information. In Table 1 and Table 2, the CC and SSIM of the wavelet fusion images are the highest, which indicates that wavelet fusion images preserve less information from the S-band SAR images. On the contrary, since the proposed fusion method makes full use of the information from the S-band SAR images, the CC and SSIM of the proposed fusion images are lowest.

We also recorded the running time of each fusion method. We can see that the computational speed of NSST is significantly faster than the NSCT. Compared with NSST, the proposed fusion method needs more time because of the additional sub-band fusion rules.

As mentioned above, compared with other fused images, the proposed fusion images have better performance. The AG, IE, and STD prove that the proposed fusion images retain more detail information, and the CC and SSIM prove that the proposed fusion images preserve more additional information from the S-band SAR images. Therefore, we can conclude that the proposed image fusion method for multi-band and polarization SAR images can achieve the SAR image information fusion with high quality.

4. Conclusions

In this paper, we proposed a multi-band and polarization SAR images colorization fusion method, in which the fusion image is improved by the band and polarization difference extracted from the SAR images. In the proposed method, the fused image is acquired from the NSST transform and the CV and PC weighted sub-band image fusion rules. Then, the fused image is colorized based on the intensity difference of the bands and the color saturation of the polarizations. We chose two representative multi-band and polarization SAR image pairs for the analysis of the fusion methods. Both the visual and quantitative evaluations prove that the proposed fusion images own more detail information and less noise. Moreover, the colorization process dramatically enhanced the visual performance of the fusion images. In the proposed fusion images, image user can easily locate the regions that are masked or have special surface structure.