Multiframe Super-Resolution of Color Images Based on Cross Channel Prior

: Color images have a wider range of applications than gray images. There are two ways to extend the traditional super-resolution reconstruction method to color images: Super resolution reconstructs each channel of the color image individually; Change the RGB color bands into YCrCb color bands, then super-resolution reconstructs the luminance component and interpolates the chrominance components.These algorithms cannot effectively utilize the property that the edges and textures are similar in the RGB channels, and the results of those methods may lead to color artifacts. Aiming to solve these problems, we propose a new super-resolution method based on cross channel prior. First, a cross channel prior is proposed to describe the similarity of gradient in RGB channels. Then, a new super-resolution method is proposed for color images via combination of the cross channel prior and the traditional super-resolution methods. Finally, the proposed method reconstructs the color channels alternately. The experimental results show that the proposed method could effectively suppress the generation of color artifacts and improve the quality of the reconstructed images. Author Contributions: Conceptualization, Z.Y.; methodology, S.S.; software, S.S.; validation, S.S.; formal analysis, S.S.; investigation, S.S.; resources, S.S.; data curation, S.S.; writing—original draft preparation, S.S.; writing—review and editing, S.S. and Z.Y.; visualization, S.S.; supervision, Z.Y.; project administration, B.X. and Z.Y.; funding acquisition, Z.Y. authors read of the manuscript.


Introduction
Image super-resolution (SR) is an effective image enhancement technology. It uses mathematical methods to increase image resolution without changing the imaging system hardware. It has great advantages in terms of technology and cost, and is widely used in scientific research and engineering [1,2]. Traditional SR methods are mainly based on gray images. Compared with gray images, color images could provide more information and are widely used in digital television, remote sensing, medical imaging, and cultural relic protection and display [3][4][5]. Color image SR is gradually becoming a new direction of SR research.
In 1984, Tsai and Huang [6] proposed the concept of multiframe super-resolution (MFSR) reconstruction, proved the theoretical feasibility of the SR algorithm and successfully applied it to the processing of Landsat satellite images. Kim, Bose and Valenzaule [7,8] improved Tsai's algorithm by considering image noise, image blurring, and image registration which extended the application of SR algorithm. Rhee and Kang [9] used discrete cosine transform (DCT) instead of discrete Fourier transform (DFT) in SR algorithm, which improved the computational efficiency of the algorithm. After more than 30 years of development, the current SR methods can be divided into two categories: reconstruction-based and learning-based methods.
The learning-based SR technology of a single image has received widespread attention in recent years [10]. By establishing a training set of high-resolution (HR) images corre-sponding to low-resolution (LR) images, a suitable data dictionary [11], neural network [12], or adversarial network [13] can be trained to achieve the effect of SR reconstruction. This type of methods has good SR effects and high efficiency, but the training process often requires a lot of computing resources, and the reconstruction results are limited to the training set. In contrast, MFSR methods use the sub-pixel displacement between the LR images to recovery the lost high-frequency information, which would improve the true (optical) resolution of the image [14].
The reconstruction-based methods can be divided into two categories: frequency domain methods and spatial domain methods. The frequency domain method uses the aliasing that exists in the LR images to reconstruct the HR image, the principle is straightforward, the computational complexity of the algorithm is low, and it is easy to implement in hardware. Therefore, the frequency methods can only handle the global translational motion and cannot utilize the image prior information. The spatial domain methods mainly include iterative back projection (IBP) [15][16][17], the convex set projection method (POCS) [18][19][20], the maximum a posteriori (MAP) probability estimation method [21][22][23] and other methods. These methods use the sub-pixel information existing between LR images to provide additional information to reconstruct images. The algorithms reconstruct well and are mainly used in scientific research, satellite remote sensing and other fields.
The above-mentioned SR algorithm research mainly concentrates on grayscale images. For color images with a wider range of applications, there is not much targeted research, mainly involving independently applying SR reconstruction technology to the three RGB channels of color images for reconstruction [24][25][26]. Since the independent reconstruction process ignores the correlation between the color channel components of the color image, the SR reconstruction of the color image is prone to artifacts. Some scholars have proposed a new method to extend the single-channel SR reconstruction method to color images based on the characteristics of human eyes being more sensitive to brightness information [27]. Yang et al. [11] converted the color image from the RGB color space to the YCrCb color space, performed SR reconstruction of luminance information and performed simple interpolation processing on chrominance components. Xu et al. [28] improved Yang's method with a new conversion method and an improved TV regularization method. The new conversion method allows the luminance channel to contain more texture information, and the improved TV regularization method can effectively suppress image artifacts. This type of method combines the intensity change information of each component of the image as a whole for processing, which effectively reduces the amount of calculation while ensuring the basic quality of the image, and at the same time overcomes the problem of ignoring channel correlation in independent calculations. However, separating image brightness and chrominance information will inevitably break the correlation between color channels. Therefore, the reconstruction results of this type of method still face the problem of color artifacts in the reconstruction results.
Starting from the analysis of the correlation between the channels of the color image, combining the imaging model and prior information in the traditional grayscale SR reconstruction algorithm, this paper proposes an effective color image SR reconstruction algorithm. The main contributions of this article are listed below, • Introduce a new image prior knowledge in multiframe SR algorithm for color images, which can describe the correlation between each channel of the image well. • Propose a new SR algorithm based on the introduced cross-channel prior, which effectively utilizes the correlation between color channels and improves the reconstructed image quality. • Design experiments to verify the proposed algorithm and compare it with other algorithms to show the effect.
Through simulation experiments and real data testing, our algorithm can effectively utilize color image channel correlation and the reconstruction results are better than the presented algorithms.

Observation Model
A set of LR images can be obtained from an HR image through a set of operations such as rotation, displacement, bluring, down-sampling, and mixed noisy. The mathematical expression of the observation model is where y ic and x c denote the cth band(R,G,or B) of the ith LR image and the cth band of the HR image. S represents the down-sampling matrix, K represents the blurring matrix, M i represents the warping matrix, and N i represents the additive noisy.

Multiframe Super-Resolution Methods for Grayscale Image
The purpose of the MFSR method is to reconstruct the HR image from a set of LR images. It is an ill-conditioned problem based on the observation model and cannot be solved directly. Therefore, a regularization method is used in this problem.
where y i − SKM i x 2 2 is the data fidelity terms, which measure the correlation between the solution and the data, and Γ(x) is the regularization cost function, which imposes a penalty on the estimated value x to obtain a stable solution. The parameter λ is a scalar value that controls the tradeoff between the data fidelity term and the regularization terms.
The selection of the regularization term is often related to the a priori information of the image, and different regularization terms will lead to different reconstruction results. Here, we use the total variation (TV) regularization [29] to test our algorithm. The TV regularization, which is based on the assumption that natural images have small TV norm, is widely employed in image SR methods. The expression of the TV regularization is as follows, where ∆ h and ∆ v respectively represents the horizontal and vertical gradient of the image. Given the current estimation of the blur kernel K and the motion matrix M i , we can estimate the HR image by Conjugate Gradient (CG) method.

Cross-Channel Prior
The current MFSR algorithm can reconstruct HR grayscale images; therefore, the main idea of the algorithm in this paper is to enhance the SR results of color images by using cross channel prior.
The color image consists of RGB color channels. The pixel values of the three color channels are different, but the shape information is similar. i.e., the edges of the object are presented in all color channels in the same location the same shape. This corss-channel prior can be mathematically approximated as, where x r , x g , x b represent the RGB channels of the image x.

Proposed Method
By adding the cross-channel prior, our algorithm can be formulated as the following optimization problem, where the first and second terms are the same as the terms of the SR methods applied to the grayscale images, and the third term represents the cross-channel prior. The parameters λ c , β cl with c, l ∈ r, g, b are weights for the image prior and the cross-channel prior, respectively.

Deconvolution Algorithm
The conjugate gradient method is used to solve the RGB channels of the image alternately, where each channel is solved while other two channels kept constant. The deconvolution algorithm for the RGB channels are similar, so we take the R-channel of the image as an example to illustrate the deconvolution algorithm. First, since x g and x b are fixed parameters, the cross-channel prior terms can be written as, where D l denotes the diagonal matrix with the diagonal taken from the parameter x l , and D hl and D vl are the diagonal matrices with ∆ h x l and ∆ v x l as the diagonal elements, respectively. Since the l 1 norm cannot be solved directly using the gradient method, the majorizationminimization (MM) method [30] is used in this paper to approximate the l 1 norm. By introducing an auxiliary vector w, the l 1 norm can be written as: where w > 0, and the equation is established when w = Ax. Therefore, the solution Equation (5) can be optimized in the following form, where x t+1 r is the (t + 1)th estimate to be computed, and the auxiliary matrix and are calculated by the tth estimate x. The termination condition of our iteration is set as:

Results
In this paper, we test our proposed algorithm by comparing it with various state-ofthe-art SR algorithms. The algorithm results are evaluated by the objective evaluation methods peak signal to noise ratio (PSNR) and structural similarity (SSIM) [31].

Simulation Images
Four HR images with the size 256 × 256 are used as original images for simulation experiments. The four original images are show in Figure 1. Each original image is used to create eight synthetic images through image degradation described in Formula (1). The degradation process is as follows. The HR images are first shifted with random sub-pixel displacement and rotated with random angle to creat the image sequence of eight images. Then the image sequence is blurred by a Gaussian kernel of 5 × 5 size and unit variance, and downsampled by a factor of 2 in both the vertical and horizontal directions. Finally, the LR image sequence is obtained by adding 20 dB Gaussian noise. In order to evaluate our proposed algorithm, four representative methods are used in our experiment: 1. Grayscale image SR performed on RGB color channels independently. 2. color image SR performed on YCrCb color space [27]. 3. color image SR performed on chrominance regularization [32]. 4. color image SR performed on cross-channel prior.
The method in [32] is a learning-based method with color constraint; here, we replace it with an MFSR method using the same color constraint. All methods use the same grayscale SR algorithm [26] to eliminate the error caused by different convolution methods. The PSNR is the most common and widely used objective evaluation index for images, which evaluates the image quality through the pixel error between images. Given a test imagex with size M × N and its original image x, the PSNR is defined as: where L is the peak value in the image data and if it a 8-bit image, the L is 255. The evaluated results of PSNR is shown in Table 1. It is obvious that our proposed method outperforms other SR methods. Smoothing the image will improve the image PSNR, but it will lead to the loss of image texture details. Therefore, it is not accurate to evaluate the algorithm results from the PSNR alone. SSIM [31] is another widely used objective evaluation standard. SSIM can measure image similarity in terms of brightness, contrast, and structure, which are more sensitive to edge information. Hence, the combined use of the SSIM and the PSNR can accurately evaluate the quality of the reconstructed image.
The SSIM is defined as follows: where µ x and µx are the means and σ x and σx are the standard deviations of the original image x and the test imagex, respectively. σ xx is the covariance between x andx, and C 1 and C 2 are constants. The SSIM is equal to 1 only if x =x. The comparison of the SSIM of different SR methods is show in Table 2. It appears that our algorithm provides better results than other SR methods.

Real Data
In the real data experiment, a dataset is used to verify our proposed algorithm. The dataset is a sequence of 40 images of size 115 × 138 obtained from the MDSP dataset [33]. The real data is SR with a magnification factor of 3, and the blur kernel is a Gaussian kernel with variance of 1 and size of 5 × 5. The SR results of the real data are show in Figure 10. The traditional bicubic interpolation method has little effect on image SR. The SR method with RGB channels reconstruct independently can restore the lost details by SR and correct color image chromatic aberration due to the robustness of the SR algorithm. The results of SR method 2 are seriously affected by chromatic aberration because that only the luminance channel is reconstructed when SR is performed in the YCbCr color space. When the chromatic aberration is not severe, the SR method 3 can use the chrominance regularization prior to enhance the SR effect, but when the chromatic aberration is severe, the chrominance regularization prior used in SR method 3 will cause the image resolution to be unclear. It is obvious from the figure that our proposed method with cross-channel prior has the best SR results.

Conclusions
In this paper, we propose a multiframe SR algorithm for color images based on a new cross-channel prior. The results of the simulation show that our proposed algorithm can effectively suppress the noise and preserve edge details with a good reconstruction effect. The real data experiments show that the proposed algorithm can also suppress chromatic aberration and achieve the state-of-the-art SR results.