Multi-Color Channels Based Group Sparse Model for Image Restoration

: The group sparse representation (GSR) model combines local sparsity and nonlocal similarity in image processing, and achieves excellent results. However, the traditional GSR model and all subsequent improved GSR models convert the RGB space of the image to YCbCr space, and only extract the Y (luminance) channel of YCbCr space to change the color image to a gray image for processing. As a result, the image processing process cannot be loyal to each color channel, so the repair effect is not ideal. A new group sparse representation model based on multi-color channels is proposed in this paper. The model processes R, G and B color channels simultaneously when processing color images rather than processing a single color channel and then combining the results of different channels. The proposed multi-color-channels-based GSR model is compared with state-of-the-art methods. The experimental contrast results show that the proposed model is an effective method and can obtain good results in terms of objective quantitative metrics and subjective visual effects.


Introduction
Image restoration is one of the fundamental tasks in image processing and computer vision, and it aims to recover the original image x from the observed degraded image y. Image restoration is an ill-posed inverse problem and can be chartered by the following Formula (1): Here, x and y represent the original image and the degraded image respectively, H denotes the degenerate operator, and n denotes additive Gaussian noise. H is modeled according to different image processing tasks. When H is a unit matrix, Formula (1) represents an image denoising task [1,2]. When H is a blurring operator, Formula (1) represents an image deblurring task [3]. When H is a mask, Formula (1) represents image restoration [4,5]. When H is a random projection matrix, Formula (1) represents image compressive sensing applications [6,7].
The image restoration task is studied in this paper. In order to overcome the ill-posed characteristic and recover a satisfied copy of the original image, image priors are used to regularize the minimal problem, argmin x 1 2 Hx − y 2 2 + λΨ(x) (2) where the first term 1 2 Hx − y 2 2 represent the data fidelity term, Ψ(x) represents the regularization term of the image a priori, and λ is a regularization parameter. Image priors

Image Group Construction
The construction of the image group in GSR model is illustrated in Figure 1. The image x with the size of √ N × √ N is divided into n overlapped patches with the size of The overlapped patches are denoted by x i where i = 1, 2, . . . , n. For each x i , its m similar matched patches are constructed using the Euclidean distance as the similarity metric in a training window with the size of W × W. The similar matched patches form a set denoted by S i . All the similar patches in S i form a matrix with the size of b × m, denoted by X i ∈ R b×m . Each element in S i is a column of the matrix X i , i.e., X i = {x i,1 , x i,2 , . . . , x i,m }. The column x i,j denotes the jth similar patch of the ith image group corresponding to x i . The matrix X i includes all the similar patches as columns, and hence is referred to as a group.
sparse representation (GSR) model uses the image group instead of a single image block as the basic unit of sparse representation, which produces satisfactory results in image processing tasks [13,28,29]. The model proposed in this paper is based on the traditional GSR model, which is briefly introduced here.

Image Group Construction
The construction of the image group in GSR model is illustrated in Figure 1 and hence is referred to as a group.

Sparse Representation Model
After constructing the image group, a sparse representation model is established in the image group. In order to obtain the sparse representation model of the image group, a dictionary, denoted by i D is given, which is usually learned from the image group, such as the graph-based dictionary [30] and principal component analysis (PCA) dictionary [31]. The aim of the sparse representation is to present the image group using as few atoms of the dictionary as possible. That is, the task is to code the dictionary sparsely using a sufficiently sparse vector. Each sparse representation, denoted by ˆi β , corresponding to the image group i X , is obtained via minimizing the 0 L norm using the following Formula (3).
Here, i β denotes the group sparse coefficient corresponding to the ith image group i X .

Sparse Representation Model
After constructing the image group, a sparse representation model is established in the image group. In order to obtain the sparse representation model of the image group, a dictionary, denoted by D i is given, which is usually learned from the image group, such as the graph-based dictionary [30] and principal component analysis (PCA) dictionary [31]. The aim of the sparse representation is to present the image group using as few atoms of the dictionary as possible. That is, the task is to code the dictionary sparsely using a sufficiently sparse vector. Each sparse representation, denoted byβ i , corresponding to the image group X i , is obtained via minimizing the L 0 norm using the following Formula (3).
Here, β i denotes the group sparse coefficient corresponding to the ith image group X i .
is the fitting term used to measure the similarity between the image to be repaired via the sparse representation model and the original image. λ is a regularization parameter, and 0 denotes the L 0 norm used to calculate the no-zero terms of each column in β i . Since L 0 norm minimization is NP-hard, L 1 norm minimization is used instead of L 0 norm minimization via the following Formula (4).
The original image x is not available in the image restoration task, and the image groups are from the degraded image y ∈ R N . The image y is segmented into n overlapped patches with the same size of i.e., a vector with the length of b. The m similar matched patches are found to generate the image groups corresponding to the n overlapped patches. Each image group denoted by Y i ∈ R b×m is a matrix with the size of b × m, which also can be written as Y i = {y i,1 , y i,2 , . . . , y i,m }. Now, the image restoration task is to restore X i from Y i using the GSR model via the following Formula (5).
After obtainingβ i , i = 1, 2, . . . , n, the restored whole imagex can be represented by a set of sparse codes β i n i=1 .

Learning a Dictionary
Sparse representation aims to obtain the sparsest representation of the original signal on the learned dictionary, and the signal is represented as a linear combination of a small number of dictionary atoms [32]. Therefore, in the framework of sparse representation, the construction of the dictionary is the key step of image restoration. However, in image restoration tasks, the original image x is unknown, and the original image group X i cannot be obtained directly. The dictionary is only constructed according to the degraded image y. The image y is first initialized using the bilinear interpolation algorithm. The following operations are similar to the procedures depicted in Section 2.1. Specifically, the estimated image group r i corresponding to the original image group X i is extracted from the initialized image y. Then the singular value decomposition is performed on the estimation group r i via the following Equation (6).
Here, γ i denotes [γ i,1 , γ i,2 , . . . , γ i,n1 ], ∑ i denotes a diagonal matrix, γ i,j denotes the main diagonal elements of ∑ i , d represents each atom of the dictionary D i , and n 1 is defined as the minimum of d and m. u i,j and v T i,j denotes the column vectors of the matrixes U i and V T i , respectively. Each atom d i⊗j ∈ R b×m of the adaptive dictionary D i is defined as follows, In this way, the adaptive dictionary D i , corresponding to each image group can be represented as follows: The singular value decomposition is only needed to be performed on each group once to learn the adaptive dictionary corresponding to the image group. After the dictionary is learned, the sparse representation model is updated and then the L 1 norm minimization is solved. Finally, the split Bregman iteration (SBI) algorithm [13] is used for iterative optimization to obtain the final restoration result.

Proposed Multi-Color Channels Based GSR Model
The image restoration algorithm based on the GSR model has achieved good results in applications. However, most image restoration algorithms based on the GSR model transform the image into YCbCr space when processing color images, and only process the Y channel. Finally, the processed Y channel image is combined with other unprocessed channels to obtain the final restoration image. Although the Y channel of the image is formed by the integration of the three channels of a RGB color image in a certain proportion, it cannot preserve the maximum fidelity of the original R, G and B channels. This paper proposes a new GSR model based on multi-color channels, which processes three color channels of a RGB color image at the same time.

Construction of Image Groups
For an input degraded image y, the image group Y i Ch , Ch ∈ {R, G, B}, is constructed based on the multi-color channel image, where R, G and B denote the corresponding image channels. The construction of each channel image group Y i Ch is the same as that of Y i in Section 2.2. Given a RGB image, in each channel, m similar patches of the repairing patch with the size of B denote the image groups corresponding to the R, G and B channels, respectively. The three image groups form the total image group Y i C as Formula (9).
The above Formula (10) denotes the image group constructed using the Y channel in a traditional GSR model. The traditional GSR model and its improved models transform the RGB space into YCbCr space and extract the Y component for processing. The advantage of traditional GSR model is that human vision is sensitive to the change of the Y component in YCbCr space, and the Y component requires fewer storage units than the three color channels in RGB space, which is conducive to transmission and processing. However, the Y component cannot be loyal to each color channel in image processing. In this paper, as shown in Equation (9), stacking the image groups of three different color channels R, G and B in a matrix make it possible to tackle the three channels simultaneously. The total image group Y i C standing for the three color channels can be directly decomposed by singular value decomposition to obtain an adaptive dictionary and the sparse representation model. In the proposed model, the total image group is processed at the same time to avoid the loss of information caused by the transformation from RGB space to YCbCr space. Moreover, the rich information between different color channels can make up for each other [33]. On the other hand, there is no need to set and adjust the parameters of each color channel respectively when processing three color channels simultaneously, compared with separately processing the image group of each color channel to fuse different results to form the final combination results. Therefore, the proposed construction of image groups is more accurate that of the traditional GSR model. The advantages of the proposed multi-color-channels-based GSR model are low computational cost, low time cost and high efficiency.

Proposed Sparse Representation Model
After constructing a multi-color channel image group Y i C , a sparse representation model can be established based on the image group. By taking the traditional model Formula (5) and Y i C in Formula (9) in the regularization framework of Formula (2), the group sparse representation model based on multi-color channels of the proposed model in this paper is obtained.β = argmin Here, D C denotes all the adaptive dictionaries obtained via performing singular value decomposition on Y C i n i=1 , i.e., D C = D C 1 , D C 2 , . . . , D C n . β C represents the sparse coefficient of all multi-color channel image groups Y C i n i=1 , i.e., β C = β C 1 , β C 2 , . . . , β C n . It can be seen from Formula (11) that the proposed model simultaneously processes the image groups of multi-color channels.

Adaptive Dictionaries
An important problem of the proposed multi-color-channels-based GSR model is the selection of the dictionary. The dictionary is usually learned from natural images. Compared with the dictionaries of traditional analysis and design [33], such as wavelet, curvelet curve and discrete cosine transform, the dictionary directly learned from images can better adapt to the local structure of images, enhance the sparsity, and hence obtain higher performance. For example, the well-known K-SVD over-complete dictionary not only preserves the local structure of images, but also provides excellent denoising results [20]. However, the sparse representation on the over-complete dictionary is very computationally expensive and easily leads to artifacts in the image restoration [34]. In order to better adapt to the local structure of images, an adaptive dictionary D i C is learned for each group Y i C . The learning procedures of the adaptive dictionary D i C are similar to those depicted in Section 2.3. Firstly, the bilinear interpolation algorithm is used to initialize the degraded image y, and the estimation group r C i as in Formula (12) is obtained from the initialized image. Then singular value decomposition is performed on the estimation group, and the adaptive dictionary D i C of each group is learned through the estimation group r C i .

Solution of the Coefficients
In this paper, an alternating minimization algorithm [15] is used to solve the minimization problem of Equation (11). Solving the GSR model based on multi-color channels is to solve the β i minimization problem in Equation (11). This subsection provides implementation details of solving the β i minimization problem. Before solving the model proposed in this paper, the following theorem [13] is introduced. Theorem 1. Let x, y ∈ R N , and X i , Y i ∈ R b×m . e ∈ R N is the error vector, i.e., e = x − y. e j is the jth element of the error vector. Suppose that e j is independent and comes from a Gaussian distribution with zero mean and variance σ 2 s . Then for any > 0, we have the following property to describe the relation between 1 N x − y 2 2 and 1 where P() represents the probability and S = b × m × n. (The proof is referred to in [13]).
Based on Theorem 1, Formula (11) is transformed into the following formula: where τ = λS N , and S = b × m × n. Formula (14) may be tackled as the solution to the traditional GSR model, where the problem is divided into n sub-problems corresponding to each Y i C . In order to adapt to the local structure of the images, each Y i C corresponds to a learning adaptive dictionary D i C . Formula (14) is transformed into where Y i C = Hr i C and r i C = D i C γ i C . The minimization problem of the Formula (14) is simplified to the solution to the problem of Formula (15). Different from the traditional GSR model, the proposed method obtains the coefficients via a soft threshold [35] as Formula (16).

Adaptive Parameters
In general, λ is set to a fixed value empirically. However, a fixed value cannot guarantee the stability of the whole algorithm. In order to overcome this challenge, an adaptive parameter-setting scheme is used to make the proposed multi-color-channels-based GSR model more stable and practical. Using spatially adaptive Laplacian priors, the regularization parameters adaptively update in each iteration [36]: Here σ i is the ith local estimated variance, and ε is a sufficiently small constant to guarantee a no-zero denominator. The likelihood estimation [36] of σ i can be represented aŝ whereγ i denotes the singular value of the estimation r C i , and m is the number of similar patches. The proposed method adopts an iterative regularization strategy [8] to alternatively update the estimation of the noise variance σ w and the signal variance σ i .
Here κ is the scale factor controlling the noise variance, and k is the number of iterations. The complete description of the proposed GSR model based on multi-color channels is shown in the following Algorithm 1.
For Each group Y i C do

Experimental Results
This section provides lots of experimental results to verify the effectiveness of the proposed GSR model based on multi-color channels. In the restoration experiments, four types of corrupted images corresponding to different pixel missing rates are used as examples. The selected four pixel missing rates are 80%, 70%, 60%, and 50%. The pixels are lost randomly under different missing rates. The size of image block, , is set to 5 × 5, the overlap width between adjacent blocks is set to 1 pixel, the size of search window, W × W, is set to 20 × 20, the number of similar blocks, m, is set to 75, and the constant, ε, is set to 0.2.
The quality of the restored images is evaluated according to objective quantitative metrics and subjective visual effects. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [37] are used as objective metrics. The PSNR is calculated as shown in Formulas (21) and (22), Here, X and Y denote the original image and the repaired image, respectively. H and W denote the height and the width of the of the image, respectively. Formula (21) is used to calculate the mean square variance (MSE) of the original image and the repaired image. Formula (22) is used for calculating PSNR, where n is the number of bits per pixel. The larger the value of PSNR, the smaller the distortion. The calculation of SSIM is shown in Formulas (23)-(27), SSI M(X, Y) = l(X, Y) · c(X, Y) · s(X, Y) Algorithms 2022, 15, 176 9 of 15 SSIM is measured according to brightness, l, contrast, c, and image structure, s, in Formula (23). H and W denote the height and the width of the original image X and the repaired image Y. u X and u Y denote the mean value of X and Y. σ X and σ Y denote the variance of X and Y, respectively, and σ XY denotes the covariance of X and Y. C 1 , C 2 and C 3 are constants to be introduced to avoid the denominator being zero. The range of SSIM is [0, 1]. The SSIM metric is closer to a human's subjective feelings. The larger SSIM value indicates that the two images are more similar and the restoration effect is better.
In this paper, 10 images as shown in Figure 2 below are selected as test images. The quantitative experimental results and visual restoration effects given in the subsection are obtained on these images. The names of the images are Bahoon, Butterfly, Cowboy, Flower, Girl, Lake, Lena, Mickey, Mural, and Nanna, respectively. (( ( , ) )( ( , ) )) 1 SSIM is measured according to brightness, l , contrast, c , and image structure, s , in Formula (23). H and W denote the height and the width of the original image X and the repaired image Y . X u and Y u denote the mean value of X and Y . X σ and Y σ denote the variance of X and Y , respectively, and XY σ denotes the covariance of X and Y . 1 C , 2 C and 3 C are constants to be introduced to avoid the denominator being zero. The range of SSIM is [ ] 0, 1 . The SSIM metric is closer to a human's subjective feelings. The larger SSIM value indicates that the two images are more similar and the restoration effect is better. In this paper, 10 images as shown in Figure 2 below are selected as test images. The quantitative experimental results and visual restoration effects given in the subsection are obtained on these images. The names of the images are Bahoon, Butterfly, Cowboy, Flower, Girl, Lake, Lena, Mickey, Mural, and Nanna, respectively.

Objective Assessment
The proposed model in this paper is compared with 11 state-of-the-art models. These models used for comparison include SALSA [38], BPFA [4], GSR [13], NGS [39], TSLRA [40], JPG-SR [14], GSRC-NLP [27], MAR-LOW [41], HSSE [42], IR-CNN [17], and IDBP [43]. Among the comparison models, GSR [13], JPG-SR [14], GSRC-NLP [27], and HSSE [42] are based on traditional GSR or improved GSR models, which are the same type as the proposed multi-color-channels-based GSR model in this paper. In order to comprehensively evaluate the performance of the proposed model for image restoration, the proposed model is also compared with non-GSR-based image restoration models. The BPFA [4] model uses the nonparametric Bayesian dictionary learning method for image sparse representation. The SALSA [38] model proposes an algorithm belonging to the extended

Objective Assessment
The proposed model in this paper is compared with 11 state-of-the-art models. These models used for comparison include SALSA [38], BPFA [4], GSR [13], NGS [39], TSLRA [40], JPG-SR [14], GSRC-NLP [27], MAR-LOW [41], HSSE [42], IR-CNN [17], and IDBP [43]. Among the comparison models, GSR [13], JPG-SR [14], GSRC-NLP [27], and HSSE [42] are based on traditional GSR or improved GSR models, which are the same type as the proposed multi-color-channels-based GSR model in this paper. In order to comprehensively evaluate the performance of the proposed model for image restoration, the proposed model is also compared with non-GSR-based image restoration models. The BPFA [4] model uses the nonparametric Bayesian dictionary learning method for image sparse representation. The SALSA [38] model proposes an algorithm belonging to the extended Lagrangian method family to deal with constrained problems. The NGS [39] model proposes a new gradient regularization framework using nonlocal similarity. The TSLRA [40] and MAR-LOW [41] are the type of low rank minimization methods. The MAR-LOW [41] combines spatial autoregression with low rank minimization methods and achieves excellent results. The IR-CNN [17] and IDBP [43] are deep-learning-based methods. Tables 1 and 2 show the PSNR values of the restoration results on different images under the four pixels missing rates of 80%, 70%, 60% and 50% of the proposed model and the comparison models. In the tables, the best results are highlighted in bold. Table 3 shows the SSIM values of the results on different images under the four pixel missing rates of the proposed model and the comparison models, and the best results are highlighted in bold. Tables 1 and 2 show the comparison results between the proposed model and the traditional models, and Table 3 shows the comparison results between the proposed model and the deep-learning-based models. In order to reduce the burden of experiments, the result data of NGS [39], TSLRA [40], and HSSE [42], provided in HSSE [42], are used here directly to avoid repetitive implementations. The experimental data of the deep learning methods IR-CNN [17] and IDBP [43] are directly from JPG-SR [14] and GSRC-NLP [27]. In addition to PSNR and SSIM values of each test image, the average PSNR and SSIM values of all test images under different pixel missing rates corresponding to different models are calculated for overall comparison. It can be seen from Tables 1-3 that the proposed multi-color-channels-based GSR model in this paper obtains better results than other competitive models. Firstly, the proposed model is compared with the non-GSRbased image restoration models. In terms of the average metric values, the proposed model is 7.20 dB higher than the SALSA model, 7.77 dB higher than the BPFA model, 6.00 dB higher than the NGS model, 4.94 dB higher than the TSLRA model, and 0.06 dB higher than the MAR-LOW model. Among the GSR based models, the JPG-SR method combines the PSR model with the GSR model, and it not only preserves the advantages of the two models, but also reduces their disadvantages, and therefore the sparse representation model is unified. The GSRC-NLP method introduces group sparse residual constraints to the traditional GSR model to approximate the group sparsity of the original group and improve the quality of image restoration. The HSSR model combines the external GSR model with the internal GSR model using the NSS prior knowledge of degraded images and external data sets to optimize the image. However, the above three GSR-based-models and many other improved models not compared in this paper transform the color image into YCbCr space, and only extract the Y channel for processing. In this way, image processing cannot be loyal to each color channel of the original color image, which makes the effect of image restoration poor. The proposed method utilizes three color channels in image restoration to obtain better results, and Tables 1 and 2 prove this conclusion. Compared with the aforementioned three GSR-based models in terms of average PSNR metric values, the proposed model is 4.15 dB higher than the traditional GSR model, 3.99 dB higher than the JPG-SR model, 3.77 dB higher than the GSRC-NLP model, and 3.70 dB higher than the HSSE model. It can be concluded from Tables 1 and 2 that the PSNR of the proposed model in this paper on the images with 80%, 70%, 60%, and 50% pixel missing rates are higher than the nine state-of-the-art comparison methods. Table 3 provides the comparison results of the proposed model with deep-learningbased IR-CNN and ISBP models. In terms of average metric values, the PSNR and SSIM values of the proposed model are 4.1 dB and 0.049 higher than those of the IR-CNN model, and 4.72 dB and 0.067 higher than those of the IDBP model, respectively. It can be seen from the PSNR and SSIM values in Table 3 that the proposed model based on multi-color channels is better than the IR-CNN and IDBP models. The experimental results in Tables 1-3 show that the model proposed in this paper is effective and can obtain good restoration results compared with the state-of-the-art models.

Subjective Assessment
In order to further demonstrate the effectiveness of the proposed multi-color-channelsbased GSR model, Figures 3 and 4 respectively show the visual butterfly and girl restoration results of the proposed model and some state-of-the-art models with an 80% pixel missing rate. As can be seen from Figures 3 and 4, the SALSA model cannot restore sharp edges and fine details. The BPFA model can have better visual effects than the salsa model, but it still has undesirable visual artifacts, such as ringing artifacts in Figure 4d. The GSR, JPG-SR and GSRC-NLP models produce over-smoothing effects. Compared with the model proposed in this paper, the MAR-LOW has a similar visual effect, which is not easy to distinguish with the naked eye. However, according to the experimental results in Tables 1-3, the proposed model has better objective evaluation results. In general, the visual effects of Figures 3 and 4 show that the GSR model based on multi-color channels proposed in this paper performs best in image restoration. It not only preserves fine details, but also effectively removes visual artifacts.
based IR-CNN and ISBP models. In terms of average metric values, the PSNR and SSIM values of the proposed model are 4.1 dB and 0.049 higher than those of the IR-CNN model, and 4.72 dB and 0.067 higher than those of the IDBP model, respectively. It can be seen from the PSNR and SSIM values in Table 3 that the proposed model based on multi-color channels is better than the IR-CNN and IDBP models. The experimental results in Tables  1-3 show that the model proposed in this paper is effective and can obtain good restoration results compared with the state-of-the-art models.

Subjective Assessment
In order to further demonstrate the effectiveness of the proposed multi-color-channels-based GSR model, Figures 3 and 4 respectively show the visual butterfly and girl restoration results of the proposed model and some state-of-the-art models with an 80% pixel missing rate. As can be seen from Figures 3 and 4, the SALSA model cannot restore sharp edges and fine details. The BPFA model can have better visual effects than the salsa model, but it still has undesirable visual artifacts, such as ringing artifacts in Figure 4d. The GSR, JPG-SR and GSRC-NLP models produce over-smoothing effects. Compared with the model proposed in this paper, the MAR-LOW has a similar visual effect, which is not easy to distinguish with the naked eye. However, according to the experimental results in Tables 1-3, the proposed model has better objective evaluation results. In general, the visual effects of Figures 3 and 4 show that the GSR model based on multi-color channels proposed in this paper performs best in image restoration. It not only preserves fine details, but also effectively removes visual artifacts.   (g) (h) (i)

Conclusions
In order to improve the performance of the traditional GSR model in image restoration, the paper extends the GSR model to a new multi-color-channels-based case, and process R, G, and B color channels of an image simultaneously. The problem of image restoration is transformed into how to use multi-color channels for sparse representation. The alternating minimization method with an adaptive parameter setting strategy is used to solve the proposed multi-color-channels-based model. The proposed method is compared with state-of-the-art methods. The experimental results show that the proposed model can obtain good results in terms of visual effects and quantitative metrics.