Single Image Super Resolution Technique: An Extension to True Color Images

: The super-resolution (SR) technique reconstructs a high-resolution image from single or multiple low-resolution images. SR has gained much attention over the past decade, as it has signiﬁcant applications in our daily life. This paper provides a new technique of a single image super-resolution on true colored images. The key idea is to obtain the super-resolved image from observed low-resolution images. A proposed technique is based on both the wavelet and spatial domain-based algorithms by exploiting the advantages of both of the algorithms. A back projection with an iterative method is implemented to minimize the reconstruction error and for noise removal wavelet-based de-noising method is used. Previously, this technique has been followed for the grayscale images. In this proposed algorithm, the colored images are taken into account for super-resolution. The results of the proposed method have been examined both subjectively by observation of the results visually and objectively by considering the peak signal-to-noise ratio (PSNR) and mean squared error (MSE), which gives signiﬁcant results and visually better in quality from the bi-cubic interpolation technique.


Introduction
There are numerous applications where high-resolution images are enviable like high-definition television broadcasting, surveillance system, sensor networks, and video conferencing, etc. It is not necessary that we can obtain the required high-resolution images due to certain limitations on resources, such as memory, power, bandwidth, and the cost of the camera. To mitigate this, several super-resolution techniques have been introduced and widely used to acquire the likely high-resolution images from the input, which will always be low-resolution images. The purpose of the image super-resolution technique is to use the low-resolution image to generate the high-resolution image, to compromise the speediness and complexity over the performance, because for the representation of Y one needs a linear combination of the sparse vector x and a given A, and in the field of image super-resolution one has to generate a dictionary D which acts like A for the approximation of Y = Dx. There are alternatives for achieving good outcomes in a faster way. The rest of the paper is organized as follows: In Section 2 different super-resolution techniques are discussed. Section 3 presents the implementation of the proposed algorithm. Experimental results are provided in Section 4. In Section 5 conclusions have been drawn.

Super-Resolution Techniques
Various interpolation techniques are present in the literature for the super-resolution of the images, nearest neighbor interpolation method is the easiest and the simple technique amongst them. The evaluation and computation of the neighbor's pixel value has been computed and duplicate the present values as a replacement. A distinct kind of tactic has been opted by the bi-linear interpolation, here the value of the unknown pixel is computed from a window of 2 × 2 neighbor's pixels. The computation of the unknown pixel's value is totally dependent on the weighted average of the defined block of 2 × 2 pixels of the neighbors and comparatively this gives good and smooth result from the nearest neighbor interpolation technique.
Among the interpolation techniques bi-cubic interpolation is one of the efficient and effective method, and due to this it is the most famous interpolation method. Normally every interpolation technique causes the generation of some artifacts, but the bi-cubic interpolation method results finer and smoother output with no artifacts. Bi-cubic is the advance version of the cubic interpolation. [32,33]. In last few years, super resolution using sparse representation have gained ample consideration. It was first used by yang et al. [34]. According to him the low-resolution patches in image can be used for the recovery of high-resolution images using sparse representation. Though their technique normally affected by irregularity between neighboring patches in an image. In [35], the authors scrutinized the non-local self-similarity across and within the spatial scales. The super-resolved image can be produced by using classical super resolution techniques if and only if the similar patches in an image are found in various different scaled versions [36].
Yang el al. [37] protracted the skeleton of [35]. In their algorithm, they have merged the image pairs both the low-and high-resolution from the image pyramid and sparse representation of the respective pairs has been jointly learned. Similar patches are examined in the image pyramid for the LR image patch and sparse representation that is jointly learned has been used for the estimating the final super-resolved result. There has been an inclination in super-resolution to combine reconstruction-based and example-based techniques for generating farther convincing outcomes [38,39].

Implementation
The proposed single image super-resolution technique is defined in this segment. The main skeleton of our proposed algorithm can be seen in Figure 1, where a true colored input has been taken into account and, furthermore, its channelization has been done into three channels. Image super resolution is technique that is used to enhance images in every aspect. Mostly blur images that are captured with the low-resolution cameras are clarified through averaging pixels. In image super-resolution, first image is converted into bands that are (HH HL LH LL) through wavelet transform, and then filters are used to maximize the details of the output image and to reduce the noise of the images. Algorithm 1 describes the implementation of proposed technique, and each step is briefly explained in the subsequent subsections.

Input RGB Image
A color input image has been considered for the image super resolution, previously the gray scaled image had been considered for the super resolution in spatial and wavelet domain [40]. For the pre-processing the next step is taken.

Channeling of Image
As the algorithm is to be applied on the separated channel of a true color image so for this purpose, the image is channelized into three planes that is red, green, and blue for further process.

Low-Resolution Image IL
The input image which is high resolution image is then converted to the low resolution shown in Figure 1, where the value of a single pixel of low-resolution image IL is computed from every four pixels of high-resolution input image. The method of getting the low-resolution image from the high-resolution image is shown in Figure 1.

Up-Sampling Image IH(n)
The technique proposed in [40] is used for the up sampling. Here, we applied the stationary wavelet transform (SWT) and discrete wavelet transform (DWT) on the acquired m × n dimensioned low-resolution image which produced four bands, i.e. HH, LH, HL, LL, with the similar m × n dimension as shown in Figure 2. The output of the SWT and DWT is different in terms of dimension of the low-resolution input image. The dimension of each of sub-bands is half of the dimension of the input image in DWT, however, in SWT each sub-band is generated exactly of the same dimension as the input image.
There are some other filter bank analysis techniques for the signal decomposition where at every single decomposition level it contains both the high and low pass filters and fragmented the signal into halves. The detail information corresponds to a differencing operation of a signal is fetched by the high pass filter while low pass filter gets the raw information corresponding to the averaging operator of the signal. Lastly, the resultant output of the filtering operation is divided by two [24,33,41]. For de-noising HAAR wavelet on HH band was applied because of it is computational fast instead of Cohen-Daubechies-Feauveas wavelet. In the final step the image is divided into four different bands indicated by HH, LH, HL, and LL subsequent to a single decomposition level [32,42]. The same practice is repeated for the LL band. This type of practice for image filtration is known as image pyramidal decomposition. This is illustrated in Figure 2. However, reconstruction of the image is done by the reversing the above process and it is reiterated till the fully reconstruction of image is not done [33].
The key details of the image lies in LL sub-band, therefore, the de-nosing algorithm is applied on HH sub-band only; however, the maximum noise is present in the remaining three sub-bands and HH sub-band contains the maximum high frequency noise. The explanation of the de-noising algorithm which is based on the algorithm defined in [33] is given as:

Algorithm 1 Single super Image resolution algorithm
Input: A true color image of 512 × 512 dimension Initialization: Channeling of input image to receive R, G, and B channels.
Repeat 1 to 6 for I times 1. Down-sampling of image in order to get the low-resolution image IL. 2. Achieve the up-sampled image by using the up-sampling algorithm shown in Figure 2.
3. Apply the Gaussian filter on the output came from step 2. 4. Generate the down-sampled image IH down 5. Reconstruction error: Use the formula described in Equation (7) for the computation of reconstruction error 6. Back Projection: Back projection is calculated using Equation (8). 7. Combination of the three channels back to get the desired output. Output: Super-resolved image.
Hard thresholding I(P, T): Soft thresholding: where T represents Threshold level, input sub-band is shown by P; however, D represents the de-noised band. For the computation of noise level median absolute deviation (MAD) is used: S i,j = HH, HL, LH, LL and the value of threshold is computed by: HM: Harmonic Symmetry 2019, 11, x FOR PEER REVIEW 5 of 13 Hard thresholding ( , ): Soft thresholding: where T represents Threshold level, input sub-band is shown by P; however, D represents the denoised band. For the computation of noise level median absolute deviation (MAD) is used: Si,j = HH, HL, LH, LL and the value of threshold is computed by: HM: Harmonic GM: Geometric The noise level (σ) is computed from the HH sub-band. After that, the value of the threshold (T) for that sub-band is evaluated and at last apply the soft thresholding formula to obtain the HH subband. In the final step all the four sub-bands are interpolated by using bicubic smoother along with the factor of K/2 and inverse DWT is applied to produce the sampled image with the dimension of Km x Kn.

Down-Sampling and Gaussain Filter
The output image after getting up sampling is looked blurred because of point spread function (PSF). For smoothing the up-sampled image a Gaussian filter worked very well. There are numerous other filters can be used such as Lucy-Richardson algorithm, winey filter or iterative blur DE convolution [43]. These down scaled images are again interpolated equal to dimension of the highresolution images to obtain the low-resolution images. The low-and high-resolution patches are required to pass through the Gaussian high pass filter and the discrete wavelet transform (DWT) for the corresponding low-and high-resolution images is computed The noise level (σ) is computed from the HH sub-band. After that, the value of the threshold (T) for that sub-band is evaluated and at last apply the soft thresholding formula to obtain the HH sub-band. In the final step all the four sub-bands are interpolated by using bicubic smoother along with the factor of K/2 and inverse DWT is applied to produce the sampled image with the dimension of Km × Kn.

Down-Sampling and Gaussain Filter
The output image after getting up sampling is looked blurred because of point spread function (PSF). For smoothing the up-sampled image a Gaussian filter worked very well. There are numerous other filters can be used such as Lucy-Richardson algorithm, winey filter or iterative blur DE convolution [43]. These down scaled images are again interpolated equal to dimension of the high-resolution images to obtain the low-resolution images. The low-and high-resolution patches are required to pass through the Gaussian high pass filter and the discrete wavelet transform (DWT) for the corresponding low-and high-resolution images is computed.

UpSampling of the Error and Reconstruction
In this phase of proposed technique, the error is computed between the down-sampled image acquired at step 3 and the original low-resolution image generated at step 1 defined in Equation (7). This is the vital step of this technique as the error that is computed in this step is taken as a factor of rectification for acquiring super resolution image and it is also helpful in the refinement of coefficients of the sub-bands.
For the reconstruction of super resolved image, the error ought to be back projected; the error matrix is up sampled to have the same dimension of the super resolved image.

UpSampling of the Error and Reconstruction
In this phase of proposed technique, the error is computed between the down-sampled image acquired at step 3 and the original low-resolution image generated at step 1 defined in Equation (7). This is the vital step of this technique as the error that is computed in this step is taken as a factor of rectification for acquiring super resolution image and it is also helpful in the refinement of coefficients of the sub-bands.
For the reconstruction of super resolved image, the error ought to be back projected; the error matrix is up sampled to have the same dimension of the super resolved image. Reconstruction error: E(n)=IL-IHdown (7)

Back Projection Error
Iterative back projection is a method with low computational complexity for super-resolution that can be useful for real-time application. This technique has been proposed by [44]. IBP is very useful iteration-based method to minimize the error energy. This method is very efficient to deal with constraints like smoothness. In IBP the high-resolution (HR) image is generated from the input lowresolution (LR) image by reducing the pixels. The observed LR image is generated from the downgraded initial HR image. The simulated LR image is then subtracted from the observed LR image. The high-pass filter is used for the estimation of HR image for edge projection and back projecting the error between the observed LR and simulated LR image [45].

Back Projection Error
Iterative back projection is a method with low computational complexity for super-resolution that can be useful for real-time application. This technique has been proposed by [44]. IBP is very useful iteration-based method to minimize the error energy. This method is very efficient to deal with constraints like smoothness. In IBP the high-resolution (HR) image is generated from the input low-resolution (LR) image by reducing the pixels. The observed LR image is generated from the downgraded initial HR image. The simulated LR image is then subtracted from the observed LR image. The high-pass filter is used for the estimation of HR image for edge projection and back projecting the error between the observed LR and simulated LR image [45].
The procedure is repeated iteratively in order to reduce the energy of the error. This procedure is continued for some pre-defined iteration. The block diagram of simple algorithm of IBP is shown in Figure 3. In the final stage, the high-resolution image which is acquired in step 3 is then added with the error computed in step 6 by using Equation (8). This procedure is repeated for four iterations to obtain the appropriate results. The quality of the images has been precisely measured, for the qualitative measurements being as a human viewer it is a very large challenge to observe an artifact as it typically depends on the prime contents of the image. Therefore, two important quality measures have been used to calculate the effect of the generated artifacts because these artifacts may have some influence in some applications, such as compression, steganography, and multimedia transmission over a communication channel. Peak signal to noise ratio (PSNR) and mean-squared error (MSE) are the two primary measures for comparison [23].
Back Projection: I H (n + 1) = E(n) + IH(n) (8) The procedure is repeated iteratively in order to reduce the energy of the error. This procedure is continued for some pre-defined iteration. The block diagram of simple algorithm of IBP is shown in Figure 3. In the final stage, the high-resolution image which is acquired in step 3 is then added with the error computed in step 6 by using Equation (8). This procedure is repeated for four iterations to obtain the appropriate results. The quality of the images has been precisely measured, for the qualitative measurements being as a human viewer it is a very large challenge to observe an artifact as it typically depends on the prime contents of the image. Therefore, two important quality measures have been used to calculate the effect of the generated artifacts because these artifacts may have some influence in some applications, such as compression, steganography, and multimedia transmission over a communication channel. Peak signal to noise ratio (PSNR) and mean-squared error (MSE) are the two primary measures for comparison [23].
Back Projection:

Combining of the RGB Channel
After applying the algorithm on the individual plane, it is then combined into one single RGB image.

Results
The above algorithm is applied on different test images and compared with the bi-cubic interpolation technique. Test images were of 512 × 512 dimensions. The performance metrics are PSNR and MSE and qualitative judgment has been done. PSNR is taken as an evaluation measure, it is expressed in dB, and it measures the closeness of the reconstructed image to the original image. Higher the PSNR means the reconstructed image is close enough to the original image. On the other side MSE computes the difference between the reconstructed and original image. Zero MSE suggests that both the images are perfectly matched. Table 1 shows the PSNR of bi-cubic and proposed algorithm, it is clearly observed that proposed algorithm has outperformed the bi-cubic except for the Brain, where bi-cubic shows better performance over proposed method. The MSE comparison is shown in Table 2, where it is self-explanatory that the proposed algorithm shows better performance

Combining of the RGB Channel
After applying the algorithm on the individual plane, it is then combined into one single RGB image.

Results
The above algorithm is applied on different test images and compared with the bi-cubic interpolation technique. Test images were of 512 × 512 dimensions. The performance metrics are PSNR and MSE and qualitative judgment has been done. PSNR is taken as an evaluation measure, it is expressed in dB, and it measures the closeness of the reconstructed image to the original image. Higher the PSNR means the reconstructed image is close enough to the original image. On the other side MSE computes the difference between the reconstructed and original image. Zero MSE suggests that both the images are perfectly matched. Table 1 shows the PSNR of bi-cubic and proposed algorithm, it is clearly observed that proposed algorithm has outperformed the bi-cubic except for the Brain, where bi-cubic shows better performance over proposed method. The MSE comparison is shown in Table 2, where it is self-explanatory that the proposed algorithm shows better performance over the Bi-cubic, however, the MSE value of bi-cubic for Brain gives better result. Figures 4 and 5 shows the quantitative comparison based on PSNR and MSE. However, qualitative results are shown in Figures 6-8. The coding of the proposed method is written and simulated in MATLAB. The comparison of Bicubic interpolation with the proposed method is measured with PSNR and is represented by Figure 4.  The comparison of Bicubic interpolation with the proposed method is measured with PSNR and is represented by Figure 4. The MSE comparison of the proposed algorithm with the existing bicubic interpolation is shown in Table 2.   The MSE comparison of the proposed algorithm with the existing bicubic interpolation is shown in Table 2.    Bi-Cubic Interpolation Proposed Method

Conclusions
A wavelet and spatial domain-based super-resolution algorithm has been developed that recuperates the best high-resolution image equivalent to the input image. Super-resolved images from low-resolution images are acquired by applying bicubic interpolation and the proposed method. The proposed technique gives the best results than the bi-cubic interpolation, since the super-resolved images acquired using bi-cubic interpolation still have the blurring artifacts. We can see the superior performance of the proposed method both visually as well as in terms of PSNR and MSE. On the other hand, the test images of high-resolution give the same results for both the bi-cubic interpolation and the proposed technique. The reason for this might be that the high-resolution images may contain the high-frequency components.
The proposed technique is faster as it takes advantages of both the spatial and wavelet domains. However, other possibilities can be exploited to find efficient methods for up sampling which will, in turn, make some improvements. Texture-based and wavelet domain-based algorithms can be used for up-sampling and some research should be carried out to observe which algorithm performs better for a class of images. In future work, high-resolution images will be targeted with smooth edges to improve the visual quality of the SR images. The technique can also be extended to the super-resolution of videos. This work can be extended by applying different filtering techniques (see [46] and references therein).