# Colorization-Based RGB-White Color Interpolation using Color Filter Array with Randomly Sampled Pattern

^{1}

^{2}

^{*}

Next Article in Journal

Department of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-ro, Seodaemun-Ku, Seoul 03722, Korea

Department of Software Engineering, Dongseo University, 47 Jurye-ro, Sasang-Ku, Busan 47011, Korea

Author to whom correspondence should be addressed.

Received: 26 April 2017 / Revised: 23 June 2017 / Accepted: 24 June 2017 / Published: 28 June 2017

(This article belongs to the Section Physical Sensors)

Recently, several RGB-White (RGBW) color filter arrays (CFAs) have been proposed, which have extra white (W) pixels in the filter array that are highly sensitive. Due to the high sensitivity, the W pixels have better SNR (Signal to Noise Ratio) characteristics than other color pixels in the filter array, especially, in low light conditions. However, most of the RGBW CFAs are designed so that the acquired RGBW pattern image can be converted into the conventional Bayer pattern image, which is then again converted into the final color image by using conventional demosaicing methods, i.e., color interpolation techniques. In this paper, we propose a new RGBW color filter array based on a totally different color interpolation technique, the colorization algorithm. The colorization algorithm was initially proposed for colorizing a gray image into a color image using a small number of color seeds. Here, we adopt this algorithm as a color interpolation technique, so that the RGBW color filter array can be designed with a very large number of W pixels to make the most of the highly sensitive characteristics of the W channel. The resulting RGBW color filter array has a pattern with a large proportion of W pixels, while the small-numbered RGB pixels are randomly distributed over the array. The colorization algorithm makes it possible to reconstruct the colors from such a small number of RGB values. Due to the large proportion of W pixels, the reconstructed color image has a high SNR value, especially higher than those of conventional CFAs in low light condition. Experimental results show that many important information which are not perceived in color images reconstructed with conventional CFAs are perceived in the images reconstructed with the proposed method.

Up to the present, most digital imaging systems obtain a full color image using a single sensor to reduce the cost and size of the system. The surface of the sensor is covered by a patterned color filter, called the color filter array (CFA), where each pixel in the pattern passes through only a certain color corresponding to a particular spectral band. The most widely used CFA pattern is the Bayer CFA pattern [1], where each pixel passes through only one of the primary colors (Red, Green and Blue), and the ratio of the numbers of the R, G, and B pixels is 1:2:1. Since each pixel captures only one of the primary colors, the other missing colors have to be obtained by interpolating neighboring colors. A lot of research has been done to find a good interpolation technique that reconstructs the missing colors as good as possible with respect to the given color filter array [2,3,4,5,6,7,8,9,10,11].

In order to use spectral information other than the RGB channel spectrum, a lot of study have been done on multispectral filter arrays [12,13,14,15,16]. By utilizing the additional spectral information obtained by the multispectral filter array, it is possible to reduce color artifacts [12,13], discriminate objects more precisely [14], and obtain invisible information such as that of NIR [15,16]. Simultaneously with these researches, demosaicing methods for various patterns have been also developed [17,18,19,20,21].

Recently, new CFA patterns [22,23,24,25] and demosaicing methods [18,19,20,21] have been proposed which also contain panchromatic “white” (W) pixels in the pattern. The W pixel has a much wider spectral band than the R, G, and B pixels as shown in Figure 1, and therefore can absorb more photons than other color pixels, making it more robust against the image noise. There are many causes of image noise, which is produced by the sensors and circuitry of a digital imager [26,27]. The fact that the W pixels are more robust against the noise is due to the fact that the signal to noise ratio (SNR) increases as the number of captured photons increases. This is especially true in low light condition, since in low light condition, the energies of the signal and the noise are similar and increasing the energy of the signal has a great effect in the SNR value. With conventional methods, the conversion from the acquired RGBW pattern to the color image is usually done in two steps: first, the conversion of the RGBW pattern to the widely used Bayer CFA pattern, and second, the conversion of the Bayer CFA pattern to the color image. The reason that the RGBW pattern is first converted to the Bayer CFA pattern is that this conversion is relatively easy and that numerous demosaicing and denoising algorithms [2,3,4,5,6,7,8,9,10,11,28,29] exist for the Bayer CFA pattern. Furthermore, conventional Bayer-oriented imaging signal processors(ISP) can be utilized with this approach. However, the reconstructed color image is degraded by this two-step conversion, since both steps introduce aliasing artifact and color distortion, which aggravates when combined together.

In this paper, we propose a new RGBW CFA pattern that has a large ratio of W pixels in the pattern and also propose the corresponding demosaicing method for this pattern. Due to the large ratio of W pixels in the CFA pattern, a large amount of light can be absorbed by the CFA pattern, resulting in a reconstructed color image with a high SNR value. Therefore, the proposed RGBW CFA pattern shows a great advantage over conventional ones in low light condition, since in low light condition, the quality of the reconstructed color image depends critically on the energy of the light absorbed by the CFA. The proposed demosaicing method for the proposed CFA pattern is based on the colorization technique which was initially proposed for automatic colorization of a gray image by the computer graphic society. The proposed demosaicing method is capable of recovering the RGB color channels from a small set of color pixels, which is the main reason that the proposed CFA pattern can have a relatively small set of RGB pixels while leaving a large space left for W pixels. Furthermore, the demosaicing method directly converts the proposed RGBW CFA pattern image to the color image, and therefore, the aliasing artifact introduced by the two step conversion in conventional RGBW CFA methods is reduced.

To understand the proposed method, the following preliminaries have to be understood.

A conventional Bayer CFA pattern contains twice as many G pixels as the R and B pixels as shown in Figure 2a. This is due to the fact that the spectral band of the G channel lies between the R and B channels, and therefore, has a relatively high spectral correlation with those channels. Furthermore, the G channel is highly related with the luminance channel of the image, and therefore, has a large influence on the perceptual quality of the color image, thus, it is advantageous that the G channel possesses a large portion of the CFA pattern.

However, as can be seen in Figure 1, compared to the G channel, the W channel has a much wider spectral band and also a larger overlapping region with the R and B spectral bands. Due to the wide spectral band, the W channel absorbs more light than other channels, and therefore, has a larger SNR value than other channels, which is especially true in low light condition. Furthermore, the W channel can be regarded as the luminance channel, and contains by itself most of the perceptual information in the image. Due to those facts, recently, it has been considered to use the W channel as the major channel in the CFA pattern, and many different RGBW CFAs have been proposed. For example, Figure 2b shows the RGBW CFA proposed in [22].

Compared with other RGBW CFAs [22,23,25] where the W pixels occupy about 50% of the CFA pattern, the proportion of the W channel in the proposed CFA pattern is much higher. This is possible because we use a color interpolation method based on the colorization technique, which can recover the colors in the large proportion of white pixels.

The color interpolation method for the proposed CFA is based on the colorization method [30]. In this method, a color image is estimated from a monochrome image with a small number of color seeds which contain the chrominance information as shown in Figure 3. In [30], Levin et al. propose how to formulate the colorization process as an optimization problem. Let M denote the number of pixels in the color image, r be the pixel position index in raster-scan order $(1\le r\le M)$, $\mathbf{y}$ be an $M\times 1$ vector denoting the luminance channel, $\mathbf{u}$, an $M\times 1$ vector denoting the chrominance channel (U or V) to be solved, and $\mathbf{x}$, an $M\times 1$ sparse vector containing the chrominance values only in certain positions (called as the representative pixels) and zeros in all other positions. Furthermore, let $\mathsf{\Psi}$ be the set of all r where $\mathbf{x}\left(r\right)\ne 0$, and $N\left(r\right)$ be the set of the 8-neighborhood pixels of the r-th. The colorization process is performed by minimizing the following energy function:
Here, ${\omega}_{rs}$ is a weighting value defined by
Define the weighting matrix $\mathbf{W}$ of size $M\times M$ as follows
and define $\mathbf{A}=\mathbf{I}-\mathbf{W}$, where $\mathbf{I}$ is an $M\times M$ identity matrix. Then, (1) can be expressed as

$$J\left(\mathbf{u}\right)=\sum _{p\in \mathsf{\Psi}}{\{\mathbf{x}\left(p\right)-\mathbf{u}\left(p\right)\}}^{2}+\sum _{r\notin \mathsf{\Psi}}{\{\mathbf{u}\left(r\right)-\sum _{s\in N\left(r\right)}{\omega}_{rs}\mathbf{u}\left(s\right)\}}^{2}.$$

$${\omega}_{rs}\propto {e}^{{(\mathbf{y}\left(r\right)-\mathbf{y}\left(s\right))}^{2}/2{\sigma}_{r}^{2}}.$$

$$\mathbf{W}(r,s)=\left\{\begin{array}{c}{\omega}_{rs}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\mathrm{if}\text{}r\notin \mathsf{\Psi}\mathrm{and}\text{}s\in N\left(r\right)\hfill \\ 0\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\mathrm{otherwise},\hfill \end{array}\right.$$

$$J\left(\mathbf{u}\right)={\parallel \mathbf{x}-\mathbf{A}\mathbf{u}\parallel}^{2}.$$

The minimizer $\mathbf{u}$ of $J\left(\mathbf{u}\right)$ is the color channel constructed from the small number of color seeds, i.e., from $\mathbf{x}$. The colorization technique brought about the idea that the R, G, and B channels can be reconstructed by a few number of sensed color pixels which contain the true color information, and the full-resolution white channel.

The proposed RGBW CFA pattern differs from other RGBW patterns in two aspects: it has a larger ratio of W pixels, and the positions of the R, G, and B pixels are random. Figure 4 shows the proposed randomly sampled RGBW pattern. pattern is 75%, and thus receives more light energy than other RGBW CFA patterns. This makes it stronger against the noise. Furthermore, since the W channel corresponds to the luminance channel which determines the resolution of the color image, a high resolution image can be obtained with the proposed CFA. The remaining 25% area of the CFA is composed of RGB pixels, which are randomly distributed over the pattern. The reason that we use a random RGB pixel pattern rather than a periodic one is that the aliasing artifact in the color channels can be reduced. In other RGBW CFAs, the periodic RGBW pattern is converted into the color image by directional interpolation. In this case, some pixels with certain directions cannot be included in the interpolation process, which results in the aliasing artifact.

Figure 5a shows the case of estimating the G channel value at an R pixel in a RGBW pattern which has same proportion of R, G, B, and W pixels, but whose R, G, and B pixels are periodically distributed over the pattern. Here, no G pixels are in the lower-left to upper-right diagonal direction. Therefore, the true G values in this direction cannot be referenced. As a result, aliasing artifact occurs if high frequency components of the G channel exist in this direction. Figure 5c shows the color interpolation result of the RGBW pattern in 5a, where the aliasing artifact can be observed in the upper right and lower left areas. In comparison, the proposed random RGBW pattern has G pixels in all directions, and therefore, the R pixel in Figure 5b can refer to those pixels in estimating the G channel value. Furthermore, the method in referring to those pixels is non-directional, i.e., based on a diffusion method as will be explained later. As a result, the aliasing artifact in the upper-right and lower-left areas is reduced as compared to the periodically sampled RGBW CFA, as can be seen in Figure 5d.

Most of the demosaicing methods for the Bayer CFA pattern first interpolate the G channel, since it contains most of the spatial information. The information of the interpolated G channel is then used in the subsequent R and B channel interpolation by referring to the inter-channel spectral correlations. Meanwhile, with conventional RGBW CFAs, the RGBW pattern is first converted to the RGB Bayer pattern, and then to the color image. Therefore, the demosaicing method is similar to the one with Bayer CFA pattern, including only an extra process of converting the RGBW pattern to the RGB pattern.

In comparison, with the proposed method, we reconstruct all the W pixels first, and then recover the colors by the colorization-based interpolation. This is due to the fact that the W channel has a higher spectral correlation with the R and B channels than the G channel, and therefore, is more suitable as a reference channel for recovering the color information. Furthermore, only 25% of the W channel has to be recovered with the proposed RGBW pattern, thus the recovery becomes highly reliable. Figure 6 shows a 3 × 3 local region of the RGBW CFA pattern where the W value of the central pixel has to be estimated. Here, w denotes the W channel, and c denotes one of the primary color channels, i.e., $c\in \{r,g,b\}$. With the Bayer CFA pattern, only four pixels, i.e., the pixels in the horizontal and vertical directions, can be referred to interpolate the missing G value. In comparison, the 8-neighborhood pixels can be used to estimate the missing W pixel values ($\widehat{w}$) with the proposed RGBW CFA pattern. Let $\{i,j\}$ denote the coordinates of the missing W pixel, $\mathcal{N}(i,j)$ represent the set of the 8 neighborhood of $\{i,j\}$, and $\{u,v\}$ be the coordinates of the pixels belonging to $\mathcal{N}(i,j)$. We define a weighting function ${\alpha}_{u,v}$ which determines how much the white pixel value in position $\{u,v\}$, i.e., ${w}_{u,v}$, should contribute to the reconstruction of the missing value at $\{i,j\}$, i.e., ${w}_{i,j}$ as follows:
Here, ${\alpha}_{u,v}$ consists of two terms, the inter-channel term ${\Delta}_{u,v}^{inter}$ and the intra-channel term ${\Delta}_{u,v}^{intra}$, where ${\beta}_{i,j}$ acts as a balance between these two terms. Then, reconstruction of $\widehat{w}$ is performed by the following equation:
Equation (6) is a weighted interpolation, where the weights are ${\alpha}_{u,v}$. As will be seen later, the term ${\beta}_{i,j}$ in (5) is designed to have a large value in achromatic regions, and a small value in colorful regions. If the difference between ${w}_{u,v}$ and ${w}_{i,j}$ is large, this means that the pixels $\{u,v\}$ and $\{i,j\}$ belong to different regions in the image, and therefore, the value ${w}_{u,v}$ should not contribute to the reconstruction of ${w}_{i,j}$. Therefore, the weight ${\alpha}_{u,v}$ should be inversely proportional to the difference between ${w}_{u,v}$ and ${w}_{i,j}$.

$${\alpha}_{u,v}=\frac{1}{{\beta}_{i,j}{\Delta}_{u,v}^{inter}+{\Delta}_{u,v}^{intra}},\phantom{\rule{0.166667em}{0ex}}\phantom{\rule{0.166667em}{0ex}}\forall u,v\in \mathcal{N}(i,j).$$

$${\widehat{w}}_{i,j}=\frac{{\sum}_{u,v\in \mathcal{N}(i,j)}{\alpha}_{u,v}{w}_{u,v}}{{\sum}_{u,v\in \mathcal{N}(i,j)}{\alpha}_{u,v}}.$$

In achromatic regions, the color value at $\{i,j\}$ itself is similar to the missing value ${w}_{i,j}$, and thus, the difference between ${w}_{i,j}$ and ${w}_{u,v}$ can be approximately measured by the difference between ${c}_{i,j}$ and ${w}_{u,v}$, where ${c}_{i,j}$ refers to one of the primary color values at $\{i,j\}$. Therefore, the inter-channel term is defined as

$${\Delta}_{u,v}^{inter}=|{c}_{i,j}-{w}_{u,v}|,\phantom{\rule{0.166667em}{0ex}}\phantom{\rule{0.166667em}{0ex}}\forall u,v\in \mathcal{N}(i,j).$$

On the contrary, in colorful regions, ${c}_{i,j}$ is not similar to ${w}_{i,j}$, and therefore, the difference between ${w}_{u,v}$ and ${w}_{i,j}$ should be measured by the differences of adjacent pixels. For example, if $\{u,v\}=\{i-1,j-1\}$, the difference between ${w}_{i,j}$ and ${w}_{u,v}$ is measured by the average of $|{w}_{i,j-1}-{w}_{i+1,j}|$ and $|{w}_{i-1,j}-{w}_{i,j+1}|$ as illustrated in the top left image in Figure 7.

The intra-channel terms ${\Delta}_{u,v}^{intra}$ for different positions of $\{u,v\}$ in $\mathcal{N}(i,j)$ are defined respectively as

$$\begin{array}{c}{\Delta}_{u=i-1,v=j-1}^{intra}=\left(\right|{w}_{i-1,j}-{w}_{i,j+1}|+|{w}_{i,j-1}-{w}_{i+1,j}|)/2,\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}{\Delta}_{u=i,v=j-1}^{intra}=\left(\right|{w}_{i-1,j-1}-{w}_{i-1,j}|+|{w}_{i+1,j-1}-{w}_{i+1,j}|)/2,\\ {\Delta}_{u=i-1,v=j+1}^{intra}=\left(\right|{w}_{i-1,j}-{w}_{i,j-1}|+|{w}_{i,j+1}-{w}_{i+1,j}|)/2,\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}{\Delta}_{u=i,v=j+1}^{intra}=\left(\right|{w}_{i-1,j+1}-{w}_{i-1,j}|+|{w}_{i+1,j+1}-{w}_{i+1,j}|)/2,\\ {\Delta}_{u=i+1,v=j-1}^{intra}=\left(\right|{w}_{i-1,j}-{w}_{i,j-1}|+|{w}_{i,j+1}-{w}_{i+1,j}|)/2,\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}{\Delta}_{u=i-1,v=j}^{intra}=\left(\right|{w}_{i-1,j-1}-{w}_{i,j-1}|+|{w}_{i-1,j+1}-{w}_{i,j+1}|)/2,\\ {\Delta}_{u=i+1,v=j+1}^{intra}=\left(\right|{w}_{i-1,j}-{w}_{i,j+1}|+|{w}_{i,j-1}-{w}_{i+1,j}|)/2,\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}\phantom{\rule{0.277778em}{0ex}}{\Delta}_{u=i+1,v=j}^{intra}=\left(\right|{w}_{i+1,j-1}-{w}_{i,j-1}|+|{w}_{i+1,j+1}-{w}_{i,j+1}|)/2.\end{array}$$

The value ${\beta}_{i,j}$ which determines whether the pixel $\{i,j\}$ belongs to an achromatic region or a colorful region is defined as
and has a large value in achromatic regions and a small value in colorful regions. Here, ${m}_{i,j}^{inter}$ and ${m}_{i,j}^{intra}$ are the minimum values related with ${\Delta}_{u,v}^{inter}$ and ${\Delta}_{u,v}^{intra}$, and are defined as

$${\beta}_{i,j}=\left\{\begin{array}{cc}0.5& \mathrm{if}{m}_{i,j}^{inter}\le {m}_{i,j}^{intra}\\ 2& \mathrm{if}{m}_{i,j}^{inter}\ge 2{m}_{i,j}^{intra}\\ \frac{1.5({m}_{i,j}^{inter}-{m}_{i,j}^{intra})}{{m}_{i,j}^{intra}}+0.5& \mathrm{otherwise},\end{array}\right.$$

$$\begin{array}{c}{m}_{i,j}^{inter}=\mathrm{min}({\Delta}_{i-1,j-1}^{inter}+{\Delta}_{i+1,j+1}^{inter},{\Delta}_{i-1,j+1}^{inter}+{\Delta}_{i+1,j-1}^{inter},{\Delta}_{i,j-1}^{inter}+{\Delta}_{i,j+1}^{inter},{\Delta}_{i-1,j}^{inter}+{\Delta}_{i+1,j}^{inter})\hfill \\ {m}_{i,j}^{intra}=\mathrm{min}({\Delta}_{i-1,j-1}^{intra}+{\Delta}_{i+1,j+1}^{intra},{\Delta}_{i-1,j+1}^{intra}+{\Delta}_{i+1,j-1}^{intra},{\Delta}_{i,j-1}^{intra}+{\Delta}_{i,j+1}^{intra},{\Delta}_{i-1,j}^{intra}+{\Delta}_{i+1,j}^{intra}).\hfill \end{array}$$

As described in Section 3.1, the number of color seeds, i.e., pixels which are sensing the primary colors, is very small, and their positions are randomly distributed. After the W channel is fully interpolated with the aforementioned method, the primary color channels can be obtained with a colorization scheme. However, in low light conditions, the primary color channels are more degraded by the noise than the W channel. Therefore, in this work, we design a colorization matrix which simultaneously diffuse the original color seeds to other pixels and remove the noise in the original color seeds.

Let $\mathbf{w}$ and $\mathbf{c}$ represent the lexicographically ordered vectors corresponding to the 2-dimensional images w and c, i.e.,
and
where ${N}_{r}$ and ${N}_{c}$ denote the height and the width of the image. Here, $\mathbf{c}$ represents the color channel which we want to recover from the sensed RGBW pattern image.

$$\mathbf{w}={[{w}_{1,1},{w}_{2,1},\dots ,{w}_{{N}_{r},1},{w}_{1,2},\dots ,{w}_{1,{N}_{c}},\dots ,{w}_{{N}_{r},{N}_{c}}]}^{T},$$

$$\mathbf{c}={[{c}_{1,1},{c}_{2,1},\dots ,{c}_{{N}_{r},1},{c}_{1,2},\dots ,{c}_{1,{N}_{c}},\dots ,{c}_{{N}_{r},{N}_{c}}]}^{T},$$

While other sophisticated residual images such as that proposed in [11] can also be used, we use a simple color difference channel. The color difference channel ${\mathbf{u}}_{c}$ between the primary color channel $\mathbf{c}\in \{\mathbf{r},\mathbf{g},\mathbf{b}\}$ and the W channel is defined as:

$${\mathbf{u}}_{c}=\mathbf{c}-\mathbf{w}.$$

Only 25% of the components in the ${\mathbf{u}}_{c}$ vector are sensed by the proposed RGBW CFA, and the problem of the primary color channel reconstruction, i.e., the color channel interpolation, is to recover the true ${\mathbf{u}}_{c}$ vector with proper color components at every position. We define by ${\mathbf{x}}_{c}$ the vector which contains the ${\mathbf{u}}_{c}$ values only at the positions where the corresponding $\mathbf{c}$ value is sensed by the proposed RGBW CFA and has zero values at all other positions, i.e.,
where $\widehat{\mathbf{w}}$ is the full W channel reconstructed by the method in Section 3.2, m is the position index of the pixel in the lexicographically ordered vector ${\mathbf{x}}_{c}$, and ${\mathsf{\Psi}}_{c}$ represents the set of the pixel positions where the channel c value is sensed by the RGBW CFA. In accordance with [30], we call the pixels in the set ${\mathsf{\Psi}}_{c}$ the representative pixels or the color seeds.

$${\mathbf{x}}_{c}\left(m\right)=\left\{\begin{array}{cc}\mathbf{c}\left(m\right)-\widehat{\mathbf{w}}\left(m\right)& \mathrm{if}m\in {\mathsf{\Psi}}_{c}\\ 0& \mathrm{otherwise},\end{array}\right.$$

The proposed colorization-based color interpolation solves the interpolation problem by minimizing the following functional with respect to ${\mathbf{u}}_{c}$ given ${\mathbf{x}}_{c}$ and $\widehat{\mathbf{w}}$:
The functional $J\left({\mathbf{u}}_{c}\right)$ consists of two energy terms: the diffusion term $D\left({\mathbf{u}}_{c}\right)$ and the noise-suppressing fidelity term $F\left({\mathbf{u}}_{c}\right)$. Let r denote the position index of the pixels not belonging to ${\mathsf{\Psi}}_{c}$, and $\mathcal{N}\left(r\right)$ the neighborhood pixels of r. The diffusion term is defined as
The minimization of $D\left({\mathbf{u}}_{c}\right)$ can be seen as weighted diffusion shown in Figure 8a, where the amount of diffusion between r and s is determined by the weight ${\u03f5}_{rs}$. The weight ${\u03f5}_{rs}$ is controlled by an edge directional ellipsoidal kernel as will be explained later. Let p denote the position index of the pixels belonging to ${\mathsf{\Psi}}_{c}$, and $\mathcal{N}\left(p\right)$ the neighborhood pixels of p. The noise-suppressing fidelity term $F\left({\mathbf{u}}_{c}\right)$ is defined as
It tries to not only make ${\mathbf{u}}_{c}$ similar to ${\mathbf{x}}_{c}$, but also subtract the weighted Laplacian of ${\mathbf{u}}_{c}$, i.e., $\gamma \{{\sum}_{q\in \mathcal{N}\left(p\right)}{\u03f5}_{pq}{\mathbf{u}}_{c}\left(q\right)-{\mathbf{u}}_{c}\left(p\right)\}$ from ${\mathbf{u}}_{c}$ at the seed pixel. It is different from the classical cost function defined as follow:
Minimizing (18), the neighborhood pixels tend to follow the noisy seed pixels and result in low frequency noise.

$$J\left({\mathbf{u}}_{c}\right)=D\left({\mathbf{u}}_{c}\right)+F\left({\mathbf{u}}_{c}\right).$$

$$D\left({\mathbf{u}}_{c}\right)=\sum _{r\notin {\mathsf{\Psi}}_{c}}{\{{\mathbf{u}}_{c}\left(r\right)-\sum _{s\in \mathcal{N}\left(r\right)}{\u03f5}_{rs}{\mathbf{u}}_{c}\left(s\right)\}}^{2}.$$

$$F\left({\mathbf{u}}_{c}\right)=\sum _{p\in {\mathsf{\Psi}}_{c}}{[{\mathbf{u}}_{c}\left(p\right)-\mathbf{x}\left(p\right)+\gamma \{{\mathbf{u}}_{c}\left(p\right)-\sum _{q\in \mathcal{N}\left(p\right)}{\u03f5}_{pq}{\mathbf{u}}_{c}\left(q\right)\}]}^{2}.$$

$${J}_{old}\left({\mathbf{u}}_{c}\right)=\sum _{p\in {\mathsf{\Psi}}_{c}}{[{\mathbf{u}}_{c}\left(p\right)-{\mathbf{x}}_{c}\left(p\right)]}^{2}+\gamma \sum _{p}{[\{{\mathbf{u}}_{c}\left(p\right)-\sum _{q\in \mathcal{N}\left(p\right)}{\u03f5}_{pq}{\mathbf{u}}_{c}\left(q\right)\}]}^{2}.$$

The first term in (17) preserves the fidelity of ${\mathbf{u}}_{c}\left(p\right)$ to the sensed value at p, i.e., ${\mathbf{x}}_{c}\left(p\right)$, while the second term tries to smooth it as a weighted average of the values of the neighborhood pixels as shown in Figure 8b, where the weights ${\u03f5}_{pq}$ are obtained in the same manner as ${\u03f5}_{rs}$ as will be explained later. The second term is necessary because, unlike the original colorization problem, the color seeds in the proposed method are degraded by the noise, especially in low light condition. This leads to spotted color noise after applying the colorization-based interpolation process. The value $\gamma $ controls the balance between the two terms, thus controlling the degree of diffusion. When $\gamma $ is large, the values of ${\mathbf{u}}_{c}$ at the position of the color seeds are influenced much by the nearby color values, whereas if $\gamma =0$, they are not influenced at all by the nearby values. Here, $\gamma $ is proportional to the noise variance. Let ${\sigma}_{n}^{2}$ denote the noise variance and $\tau $ represent a control constant, $\gamma $ is defined as
The noise variance ${\sigma}_{n}^{2}$ can be estimated by calculating the variance of the flat region in the image. In [30], the weight parameters were determined based on the differences of the luminance values between adjacent pixels, so that the color seeds were easily spread out in flat regions, while not in edge regions. In low light conditions, however, the difference between adjacent pixel values could not be estimated correctly due to the noise. As some edge regions were weakened by the noise, the color diffused across the edge region which results in the wash-out of the colors in edge regions. As the edge could not be determined exactly using the derivative at the current position only, the derivatives in a neighborhood region should be considered together to determine the edge direction.

$$\gamma =\tau {\sigma}_{n}^{2}.$$

The weight parameter is designed to take into account the directional tendency of adjacent pixels [31,32]. In order to estimate the tendency of the edge direction, the covariance matrix $\mathbf{C}$ is determined in a small region $R\left(m\right)$ centered at m
where ${\widehat{\mathbf{w}}}_{v}$ and ${\widehat{\mathbf{w}}}_{h}$ represent the derivatives of $\widehat{\mathbf{w}}$ in the vertical and the horizontal directions, respectively. Let ${\mathbf{d}}_{mn}={[{v}_{mn},{h}_{mn}]}^{T}$ be a pointing vector, where ${v}_{mn}$ and ${h}_{mn}$ are the vertical and the horizontal distances between m, the current pixel position, and n, a neighborhood pixel of m. The weight value at n is calculated as

$$\mathbf{C}=\left[\begin{array}{cc}\sum _{n\in R\left(m\right)}{\widehat{\mathbf{w}}}_{v}\left(n\right){\widehat{\mathbf{w}}}_{v}\left(n\right)& \sum _{n\in R\left(m\right)}{\widehat{\mathbf{w}}}_{v}\left(n\right){\widehat{\mathbf{w}}}_{h}\left(n\right)\\ \sum _{n\in R\left(m\right)}{\widehat{\mathbf{w}}}_{v}\left(n\right){\widehat{\mathbf{w}}}_{h}\left(n\right)& \sum _{n\in R\left(m\right)}{\widehat{\mathbf{w}}}_{h}\left(n\right){\widehat{\mathbf{w}}}_{h}\left(n\right)\end{array}\right],$$

$${\u03f5}_{mn}^{\prime}=\mathrm{exp}\{-\lambda {\mathbf{d}}_{mn}^{T}\mathbf{C}{\mathbf{d}}_{mn}\}.$$

It is derived from an ellipsoidal kernel function (Figure 9) which has a large value if n lies at the center of the kernel and a small value if n is far from the center. Here, $\lambda $ represents a parameter which controls the smoothness of the kernel. Furthermore, ${\u03f5}_{mn}^{\prime}$ is large if n lies in a direction orthogonal to the edge, and small if n lies in the direction of the edge. Therefore, ${\u03f5}_{mn}^{\prime}$ can be used as a measure whether a pixel n lies along or across an edge. For the weights in (16) and (17), we use a normalized version of ${\u03f5}_{mn}^{\prime}$, i.e., ${\u03f5}_{mn}=\frac{1}{k}{\u03f5}_{mn}^{\prime}$, where k is the normalizing factor.

The cost function defined in (15) can be re-written in matrix form as
where the diffusion matrix $\mathbf{D}$ which spreads the color seeds is defined as
The noise-suppressing fidelity matrix $\mathbf{F}$ is defined as
Solving (22), the color difference channel can be estimated as follows:
By using the estimated color difference channel ${\widehat{\mathbf{u}}}_{c}$, the primary color channel can be obtained as

$$J\left({\mathbf{u}}_{c}\right)={\parallel \mathbf{D}{\mathbf{u}}_{c}\parallel}^{2}+{\parallel \mathbf{F}{\mathbf{u}}_{c}-{\mathbf{x}}_{c}\parallel}^{2},$$

$$\mathbf{D}(r,s)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\text{}r\notin {\mathsf{\Psi}}_{c}\mathrm{and}\text{}s=r\hfill \\ -{\u03f5}_{rs}\hfill & \mathrm{else}\text{}\mathrm{if}\text{}r\notin {\mathsf{\Psi}}_{c}\mathrm{and}\text{}s\in \mathcal{N}\left(r\right)\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

$$\mathbf{F}(p,q)=\left\{\begin{array}{cc}1+\gamma \hfill & \mathrm{if}\text{}p\in {\mathsf{\Psi}}_{c}\mathrm{and}\text{}q=p\hfill \\ -\gamma {\u03f5}_{pq}\hfill & \mathrm{else}\text{}\mathrm{if}\text{}p\in {\mathsf{\Psi}}_{c}\mathrm{and}\text{}q\in \mathcal{N}\left(p\right)\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

$${\tilde{\mathbf{u}}}_{c}={({\mathbf{D}}^{T}\mathbf{D}+{\mathbf{F}}^{T}\mathbf{F})}^{-1}{\mathbf{F}}^{T}{\mathbf{x}}_{c}.$$

$$\tilde{\mathbf{c}}={\tilde{\mathbf{u}}}_{c}+\widehat{\mathbf{w}}.$$

The color channels $\tilde{\mathbf{c}}\in \{\tilde{\mathbf{r}},\tilde{\mathbf{g}},\tilde{\mathbf{b}}\}$ reconstructed by the method explained in Section 3.3 suffer from low frequency noise. This is due to the fact that the color seeds from which they are reconstructed contain noise themselves. In this section, we propose a post-process which subtracts the low frequency noise ${\mathbf{n}}_{LF}$ from $\tilde{\mathbf{c}}$ to obtain a noise removed color channel $\widehat{\mathbf{c}}$:
Here, the low frequency noise ${\mathbf{n}}_{LF}$ is estimated by subtracting $\widehat{\mathbf{w}}$ obtained in Section 3.2 from $\tilde{\mathbf{w}}$ which denotes a noisy W channel to be constructed as follow:
This is based on the assumptions that the low frequency noise contained in $\tilde{\mathbf{c}}$ is highly correlated with the noise in $\tilde{\mathbf{w}}$, and that $\widehat{\mathbf{w}}$ is free from this low frequency noise. The first assumption is satisfied since the noisy $\tilde{\mathbf{w}}$ is constructed by a linear combination of the noisy color channels:

$$\widehat{\mathbf{c}}=\tilde{\mathbf{c}}-{\mathbf{n}}_{LF}.$$

$${\mathbf{n}}_{LF}=\tilde{\mathbf{w}}-\widehat{\mathbf{w}}.$$

$$\tilde{\mathbf{w}}={\lambda}_{r}\tilde{\mathbf{r}}+{\lambda}_{g}\tilde{\mathbf{g}}+{\lambda}_{b}\tilde{\mathbf{b}}+{\lambda}_{1}.$$

Here, ${\lambda}_{r}$, ${\lambda}_{g}$, ${\lambda}_{b}$, and ${\lambda}_{1}$ are linear coefficients which are obtained by the following ${L}_{2}$ norm minimization with respect to multiple arguments ${\lambda}_{r}$, ${\lambda}_{g}$, ${\lambda}_{b}$, and ${\lambda}_{1}$:

$${\lambda}_{r},{\lambda}_{g},{\lambda}_{b},{\lambda}_{1}=\underset{{\lambda}_{r},{\lambda}_{g},{\lambda}_{b},{\lambda}_{1}}{\mathrm{argmin}}{\parallel \widehat{\mathbf{w}}-({\lambda}_{r}\tilde{\mathbf{r}}+{\lambda}_{g}\tilde{\mathbf{g}}+{\lambda}_{b}\tilde{\mathbf{b}}+{\lambda}_{1})\parallel}_{2}^{2}.$$

This is a multiple regression model [33] which makes the ${L}_{2}$ norm difference between $\widehat{w}$ and are $\tilde{w}$ as small as possible, thus resulting in the coefficients ${\lambda}_{r}$, ${\lambda}_{g}$, ${\lambda}_{b}$ and ${\lambda}_{1}$ that the overall brightness of $\tilde{w}$ becomes similar to $\widehat{w}$. This is similar to the guided filter approach proposed in [34], but instead of a single parameter we use multiple parameters, and unlike the guided filter which tries to obtain a noise-free channel, the purpose is to obtain a noisy white image $\tilde{w}$ which is used afterwards to eliminate the noise in the reconstructed noisy color channels. The constructed channel $\tilde{\mathbf{w}}$ contains the same low frequency noise as in $\tilde{\mathbf{c}}$ since it is constructed from it. The second assumption is satisfied since $\widehat{\mathbf{w}}$ is constructed from the sensed W channel, which has a larger SNR value than those of the color channels. Using the linear least square method, (30) can be solved as follow:
where $\mathbf{a}\in \{\tilde{\mathbf{r}},\tilde{\mathbf{g}},\tilde{\mathbf{b}},\widehat{\mathbf{w}}\}$, $\mu \left\{\mathbf{a}\right\}$ denotes the mean of a vector $\mathbf{a}$, and ∘ denotes the entrywise product operator. Figure 10a–d show $\tilde{\mathbf{w}}$, $\widehat{\mathbf{w}}$, $\tilde{\mathbf{c}}$, and $\widehat{\mathbf{c}}$, respectively. As can be seen in Figure 10d, the low frequency noise in $\widehat{\mathbf{c}}$ is quite reduced.

$$\left[\begin{array}{c}{\lambda}_{r}\\ {\lambda}_{g}\\ {\lambda}_{b}\\ {\lambda}_{1}\end{array}\right]={\left[\begin{array}{cccc}\mu \left\{\tilde{\mathbf{r}}\right\}& \mu \left\{\tilde{\mathbf{g}}\right\}& \mu \left\{\tilde{\mathbf{b}}\right\}& 1\\ \mu \{\tilde{\mathbf{r}}\circ \tilde{\mathbf{r}}\}& \mu \{\tilde{\mathbf{g}}\circ \tilde{\mathbf{r}}\}& \mu \{\tilde{\mathbf{b}}\circ \tilde{\mathbf{r}}\}& \mu \left\{\tilde{\mathbf{r}}\right\}\\ \mu \{\tilde{\mathbf{r}}\circ \tilde{\mathbf{g}}\}& \mu \{\tilde{\mathbf{g}}\circ \tilde{\mathbf{g}}\}& \mu \{\tilde{\mathbf{b}}\circ \tilde{\mathbf{g}}\}& \mu \left\{\tilde{\mathbf{g}}\right\}\\ \mu \{\tilde{\mathbf{r}}\circ \tilde{\mathbf{b}}\}& \mu \{\tilde{\mathbf{g}}\circ \tilde{\mathbf{b}}\}& \mu \{\tilde{\mathbf{b}}\circ \tilde{\mathbf{b}}\}& \mu \left\{\tilde{\mathbf{b}}\right\}\end{array}\right]}^{-1}\left[\begin{array}{c}\mu \left\{\widehat{\mathbf{w}}\right\}\\ \mu \{\widehat{\mathbf{w}}\circ \tilde{\mathbf{r}}\}\\ \mu \{\widehat{\mathbf{w}}\circ \tilde{\mathbf{g}}\}\\ \mu \{\widehat{\mathbf{w}}\circ \tilde{\mathbf{b}}\}\end{array}\right],$$

We compared the quality of the demosaicing result of the proposed method based on the proposed RGBW CFA pattern with the demosaicing results of other methods based on other CFA patterns. Experimental results show that in low light condition, spatial information which cannot be obtained by the Bayer CFA pattern using demosaicing method can be obtained by the proposed method. Furthermore, we also show that the proposed method produces results of good quality in sufficient light condition. In order to obtain the original full resolution R, G, B and W channel images, we used a filter-wheel-installed camera, where the wheel contains four different optical filters and is driven by a stepping motor. The four optical filters selectively filter the R, G, B, and W bands. We took four photographs of the same scene with the four different filters to obtain the R, G, B, and W channels. Figure 1 shows the spectral responsibilities of the R, G, B, and W filters. Here, the exposure time was set to 30 ms at 100 lux illumination. Using the full-resolution four channel images, we sampled the R, G, B, and W pixels corresponding to the different patterns, i.e., the Bayer CFA pattern, the Sony RGBW CFA pattern, and the proposed RGBW CFA pattern. For the proposed RGBW CFA pattern, the numbers of the R, G, and B pixels are 306,500, 306,697, and 306,489, respectively, which are not equal since there are randomly sampled, but together they cover 25% of the CFA pattern which has a total of 3,686,400 pixels since the size of the image is $1920\times 1920$. The control constant $\tau $ is set to a value around 100.

The images in the first column in Figure 11 show the demosaiced results of the Bayer CFA, and those in the second column are the demosaiced results of the Sony RGBW CFA. For the Bayer CFA results, the DLMMSE method [7] was used, while with the Sony RGBW CFA, the RGBW pattern was first converted into the Bayer CFA pattern using the method in [22], and then finally got demosaiced by the DLMMSE method. The images in the third column in Figure 11 show the results of the proposed RGBW CFA which are reconstructed by the proposed colorization-based interpolation method. As described in Section 3.1, the results using the Sony RGBW CFA shows the largest aliasing artifacts due to the extra errors in the conversion from the RGBW to the Bayer CFA. The Bayer CFA based conversion shows less aliasing artifacts in the circle region than the proposed RGBW CFA based conversion in the first row of Figure 11. However, in the vertical stripe region, the result with the proposed method is better than those using other methods in terms of both the aliasing and the noise artifacts as can be seen in the second row of Figure 11.

Next, for an assessment in low light condition, we obtained the full-resolution R, G, B, and W images using the four different optical filters with an exposure time of $1/60$ s at 1 lux illumination. After that, we sampled the R, G, B, and W pixels corresponding to the Bayer CFA and the proposed RGBW CFA patterns. The image acquired at 1 lux illumination is very dark as shown in Figure 12a. The energy levels of the R, G, B, and W channels are different due to the different spectral responses of illumination and sensor sensitivity. Therefore, we first multiplied a large number (420) to the pixels of the acquired image to stretch the ranges in the pixel values, and then applied a white balancing as a pre-process to obtain Figure 12b. We performed the gray world method [35] which is one of the simplest white balancing method, but other methods [35,36,37,38,39,40] can be applied as well with little difference in the reconstruction performance. Figure 12b illustrates the result and its corresponding histogram after rescaling the intensity range and applying white balancing to the acquired image. The color interpolation is then performed on this white-balanced image, resulting in the color image shown in Figure 12c.

In Figure 13, we compared the results of the proposed method with the images reconstructed from the Bayer CFA sampled images using the state-of-the-art Bayer denoising methods [28,29]. The methods in [28,29] utilize variations of the BM3D(Block-matching and 3D filtering) method for the denoising of the Bayer-patterned image. We show that in low illumination, the proposed method results in better reconstructed images even without any extra sophisticated denoising methods as in [28,29] which is due to the nature of the proposed CFA having a very large number of W pixels. In the case with 20 dB and 23 dB noise simulations, the BM3D denoising removes the noise well, resulting in a reconstructed image of higher SNR values than that obtained by the proposed method not using a sophisticated denoising method. To make a fair comparison, we also applied the BM3D method to the W channel as a post-denoising method. We cannot apply the exactly same BM3D method as in [28,29] to our method, because they are mainly designed for the Bayer pattern, which is different from the proposed RGBW pattern. Therefore, we applied the BM3D denoising [41] on the W channel, and afterward used it as the reference channel. We call the modified proposed method with BM3D denoising the ‘proposed + BM3D’ method. The input parameter for the denoising methods in [28,29,41] is the level of noise, for which we give the standard deviation of the noise. The standard deviation of noise in the RGB channels is 0.25 (40 in 8 bit system), and that of the noise in the W channel is 0.06 (16 in 8 bit system) in 1 lux low light condition.

Figure 13 compares the results reconstructed by the proposed RGBW CFA, the Bayer CFAs [7,11], the denoised Bayer CFAs [28,29], and the Sony RGBW CFA [22]. It can be seen in Figure 13, that the results obtained by the proposed method exhibit superior high frequency information compared with other results. Moreover, the objects are better identified, and the image details, such as letters and lines, are more distinguished, especially in Figure 13p compared with Figure 13a,d,g,j,m. Figure 13p has also less noise than Figure 13a,d,g in flat regions. The results with Bayer CFA denoising show a lot of noise removal in flat regions, but also have lost important detail information. The ‘proposed + BM3D’ method shows the best result both in flat and detailed regions which can be observed in Figure 13s,t,u. Here, the denoising is done on the reconstructed W channel, which is then used for the colorization of the color seeds.

For objective assessments, we used the Kodak 24 test images of size 512 × 768 and measured the FSIM [42] and PSNR of the reconstructed images, where the PSNR is given by

$$PSNR=10\times {\mathrm{log}}_{10}\{1/\mathrm{MSE}\}.$$

We made two sets of test images, one containing 20 dB noise, and the other 23 dB noise, where the noise follows a Poisson distribution as is the case in low illumination. The noise was added to the R, G, and B channels, respectively. The standard deviations of the noises in the RGB channels are 0.1 (26 in 8 bit system) and 0.07 (18 in 8 bit system), respectively. The standard deviations of the noises in the W channel are $0.1/\sqrt{3}=0.06$ (16 in 8 bit system) and $0.07/\sqrt{3}=0.04$ (10 in 8 bit system) in 20 dB and 23 dB, respectively. This is due to the fact that the W channel is a superposition of the R, G, and B channels. The standard deviations of the noises go in as the parameters to the BM3D denoising methods. After that, the pixels were sampled according to the Bayer CFA, the Sony RGBW CFA, and the proposed RGBW CFA pattern, respectively, to generate the CFA pattern images. For the proposed RGBW CFA, the numbers of the color seeds for the R, G, and B channels are 32,456, 32,550, and 32,659, respectively, which together sums up to 25% of the whole pixels ($512\times 768=393,216$), and the rest of the pixels (75%) are W pixels. Then, the images sampled by the proposed RGBW CFA were reconstructed by the colorization-based method, while the images sampled by Bayer CFA are demosaiced by the DLMMSE and RI methods. We also added the results using the methods in [28,29]. As shown in Table 1, Table 2, Table 3 and Table 4, the color images reconstructed by the proposed method show significant improvements compared with the results of the Bayer CFA and Sony RGBW CFA. Bold text in the tables are the optimum results for each Kodak images.

The PSNR is improved by an average of 3.2934 dB, 3.2938 dB, and 0.8639 dB, compared to the results of the DLMMSE, RI, and Sony RGBW CFA, respectively, with 20 dB noise. In the case of 23 dB noise, the PSNR is improved by an average of 2.7725 dB, 2.3157 dB, and 0.5639 dB, compared to the results of the DLMMSE, RI, and Sony RGBW CFA, respectively. As shown in Figure 14 and Figure 15, the visual qualities of the images obtained by the proposed method are superior to those by the Bayer CFA and Sony RGBW CFA.

The PSNR and FSIM values of the images obtained by combining the BM3D denoising are higher than the those obtained by the proposed method without BM3D denoising. However, the images reconstructed by the ‘proposed + BM3D’ method show higher FSIM values and similar PSNR values compared with the results of [28,29]. As shown in Figure 14h,p and Figure 15h,p, the results of the ‘proposed + BM3D’ show similar noise levels in flat regions, but preserve more detail information. Also the ‘proposed + BM3D’ introduces fewer aliasing artifacts in high frequency regions as shown in the window bars in Figure 14 and Figure 15. The Bayer CFA denoising methods not only remove the noise in the flat regions but also the details in high frequency regions. This is not reflected effectively in the PSNR measure, but in the FSIM measure which reflects the edge fidelity and the color difference consistency, which is the reason that the FSIM values of the ‘proposed + BM3D’ method are higher in all cases.

Figure 16 illustrates the relation between the modulation transfer function(MTF) and the aliasing artifact in the reconstructed images with the proposed method. The value of the horizontal axis in Figure 16a is equal to the line number shown in Figure 16d, which also represents the spatial frequency value. This is due to the fact that the line becomes thinner and denser as the line number increases as can be seen in the yellow boxes in Figure 16d. When the line number is sufficiently low, aliasing is weak as can be seen in Figure 16e. The aliasing artifact is strongest in the region of line 16 (Figure 16f), and again decreases in the region of line 20 due to the lens diffraction (Figure 16g) which also decreases the MTF value. Meanwhile, the MTF is greatly affected by the noise in low light conditions and color artifacts occur due to the diffusion of false signals as can be observed in Figure 16c which is obtained with low lighting, compared to Figure 16b obtained with high lighting.

In this paper, we proposed a colorization-based demosaicing method which suits well with the proposed RGBW CFA pattern, having a large number of white pixels. Using the proposed demosaicing technique, color images can be reconstructed using a small number of sensed color pixels while the majority of the pixels can sample the white channel, which makes it possible to obtain a reconstructed color image with high SNR value. Compared with the Bayer CFA based results, the image details, such as the letters or lines, are better preserved with the proposed method in low illumination condition. This is important especially for surveillance camera systems, and therefore, the proposed method can find its application in such areas.

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No. 2015R1A2A1A14000912).

Paul Oh and Sukho Lee designed the experiments, performed the experiments, analyzed the data and wrote the manuscript under the supervision of Moon Gi Kang.

The authors declare no conflict of interest.

- Bayer, B. Color Imaging Array. U.S. Patent 3,971,065, 20 July 1976. [Google Scholar]
- Kimmel, R. Demosaicing: Image reconstruction from color CCD samples. IEEE Trans. Image Process.
**1999**, 8, 1221–1228. [Google Scholar] [CrossRef] [PubMed] - Pei, S.C.; Tam, I.K. Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Trans. Circuits Syst. Video Technol.
**2003**, 13, 503–513. [Google Scholar] - Gunturk, B.K.; Altunbasak, Y.; Mersereau, R.M. Color plane interpolation using alternating projections. IEEE Trans. Image Process.
**2002**, 11, 997–1013. [Google Scholar] [CrossRef] [PubMed] - Alleysson, D.; Süsstrunk, S.; Hérault, J. Linear demosaicing inspired by the human visual system. IEEE Trans. Image Process.
**2005**, 14, 439–449. [Google Scholar] [PubMed] - Gunturk, B.K.; Glotzbach, J.; Altunbasak, Y.; Schafer, R.W.; Mersereau, R.M. Demosaicking: Color filter array interpolation. IEEE Signal Process. Mag.
**2005**, 22, 44–54. [Google Scholar] [CrossRef] - Zhang, L.; Wu, X. Color demosaicking via directional linear minimum mean square-error estimation. IEEE Trans. Image Process.
**2005**, 14, 2167–2178. [Google Scholar] [CrossRef] [PubMed] - Dubois, E. Frequency-domain methods for demosaicking of Bayer-sampled color images. IEEE Signal Process. Lett.
**2005**, 12, 847–850. [Google Scholar] [CrossRef] - Pekkucuksen, I.; Altunbasak, Y. Multiscale Gradients-Based Color Filter Array Interpolation. IEEE Trans. Image Process.
**2013**, 22, 157–165. [Google Scholar] [CrossRef] [PubMed] - Menon, D.; Calvagno, G. Color Image Demosaicking: An Overview. Signal Process. Image Commun.
**2011**, 26, 518–533. [Google Scholar] [CrossRef] - Kiku, D.; Monno, Y.; Tanaka, M.; Okutomi, M. Beyond Color Difference: Residual Interpolation for Color Image Demosaicking. IEEE Trans. Image Process.
**2016**, 25, 1288–1300. [Google Scholar] [CrossRef] [PubMed] - Monno, Y.; Kikuchi, S.; Tanaka, M.; Okutomi, M. A Practical One-Shot Multispectral Imaging System Using a Single Image Sensor. IEEE Trans. Image Process.
**2015**, 24, 3048–3059. [Google Scholar] [CrossRef] [PubMed] - Yasuma, F.; Mitsunaga, T.; Iso, D.; Nayar, S.K. Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum. IEEE Trans. Image Process.
**2010**, 19, 2241–2253. [Google Scholar] [CrossRef] [PubMed] - Prasad, D.K. Strategies for Resolving Camera Metamers Using 3+1 Channel. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 954–962. [Google Scholar]
- Schaul, L.; Fredembach, C.; Süsstrunk, S. Color image dehazing using the near-infrared. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP), Cairo, Egypt, 7–10 November 2009; pp. 1629–1632. [Google Scholar]
- Sadeghipoor, Z.; Lu, Y.M.; Süsstrunk, S. A novel compressive sensing approach to simultaneously acquire color and near-infrared images on a single sensor. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013; pp. 1646–1650. [Google Scholar]
- Rafinazari, M.; Dubois, E. Demosaicking algorithm for the Fujifilm X-Trans color filter array. In Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014; pp. 660–663. [Google Scholar]
- Tian, Q.; Lansel, S.; Farrell, J.E.; Wandell, B.A. Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (L3) method. Proc. SPIE
**2014**, 9023. [Google Scholar] [CrossRef] - Condat, L. A Generic Variational Approach for Demosaicking from an Arbitrary Color Filter Array. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP), Cairo, Egypt, 7–10 November 2009; pp. 1605–1608. [Google Scholar]
- Gu, J.; Wolfe, P.J.; Hirakawa, K. Filterbank-based universal demosaicking. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP), Hong Kong, China, 26–29 September 2010; pp. 1981–1984. [Google Scholar]
- Park, S.W.; Kang, M.G. Generalized color interpolation scheme based on intermediate quincuncial pattern. J. Electron. Imaging
**2014**, 23, 030501. [Google Scholar] [CrossRef] - Tachi, M. Image Processing Device, Image Processing Method, and Program Pertaining to Image Correction. U.S. Patent 8,314,863, 20 November 2012. [Google Scholar]
- Yamagami, T.; Sasaki, T.; Suga, A. Image Signal Processing Apparatus Having a Color Filter with Offset Luminance Filter Elements. U.S. Patent 5,323,233, 21 June 1994. [Google Scholar]
- Gindele, E.; Gallagher, A. Sparsely Sampled Image Sensing Device with Color and Luminance Photosites. U.S. Patent 6,476,865, 5 November 2002. [Google Scholar]
- Compton, J.; Hamilton, J. Image Sensor with Improved Light Sensitivity. U.S. Patent 8,139,130, 20 March 2012. [Google Scholar]
- Dougherty, G. Digital Image Processing for Medical Applications; Cambridge University Press: Cambridge, UK, 2009. [Google Scholar]
- Bottacchi, S. Noise and Signal Interference in Optical Fiber Transmission Systems: An Optimum Design Approach; Wiley: Hoboken, NJ, USA, 2008. [Google Scholar]
- Akiyama, H.; Tanaka, M.; Okutomi, M. Pseudo four-channel image denoising for noisy CFA raw data. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 4778–4782. [Google Scholar]
- Danielyan, A.; Vehvilainen, M.; Foi, A.; Katkovnik, V.; Egiazarian, K. Cross-color BM3D filtering of noisy raw data. In Proceedings of the 2009 International Workshop on Local and Non-Local Approximation in Image Processing, Tuusula, Finland, 19–21 August 2009; pp. 125–129. [Google Scholar]
- Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Trans. Graph.
**2004**, 23, 689–694. [Google Scholar] [CrossRef] - Takeda, H.; Farsiu, S.; Milanfar, P. Kernel regression for image processing and reconstruction. IEEE Trans. Image Process.
**2007**, 16, 349–366. [Google Scholar] [CrossRef] [PubMed] - Feng, X.; Milanfar, P. Multiscale Principal Components Analysis for Image Local Orientation Estimation. In Proceedings of the 36th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 3–6 November 2002; pp. 478–482. [Google Scholar]
- Freedman, D. Statistical Models : Theory and Practice; Cambridge University Press: Cambridge, UK, 2005. [Google Scholar]
- He, K.; Sun, J.; Tang, X. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 1397–1409. [Google Scholar] [CrossRef] [PubMed] - Buchsbaum, G. A spatial processor model for object colour perception. J. Frankl. Inst.
**1980**, 310, 1–26. [Google Scholar] [CrossRef] - Land, E.H.; McCann, J.J. Lightness and Retinex Theory. J. Opt. Soc. Am.
**1971**, 61, 1–11. [Google Scholar] [CrossRef] [PubMed] - Land, E. The Retinex Theory of Color Vision. Sci. Am.
**1977**, 237, 108–129. [Google Scholar] [CrossRef] [PubMed] - Barnard, K.; Cardei, V.; Funt, B. A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data. IEEE Trans. Image Process.
**2002**, 11, 972–984. [Google Scholar] [CrossRef] [PubMed] - Barnard, K.; Martin, L.; Coath, A.; Funt, B. A comparison of computational color constancy Algorithms. II. Experiments with image data. IEEE Trans. Image Process.
**2002**, 11, 985–996. [Google Scholar] [CrossRef] [PubMed] - Cheng, D.; Prasad, D.K.; Brown, M.S. Illuminant estimation for color constancy: Why spatial-domain methods work and the role of the color distribution. J. Opt. Soc. Am. A
**2014**, 31, 1049–1058. [Google Scholar] [CrossRef] [PubMed] - Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. Image Process.
**2007**, 16, 2080–2095. [Google Scholar] [CrossRef] [PubMed] - Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Trans. Image Process.
**2011**, 20, 2378–2386. [Google Scholar] [CrossRef] [PubMed]

Data | Bayer DLMMSE [7] | Bayer RI [11] | Sony RGBW [22] | Proposed | Akiyama [28] | BM3D-CFA [29] | Proposed + BM3D |
---|---|---|---|---|---|---|---|

1 | 20.6240 | 21.0253 | 22.8337 | 24.0300 | 25.7640 | 26.2185 | 26.5624 |

2 | 21.0217 | 21.5777 | 24.1440 | 23.2595 | 28.7567 | 29.0252 | 27.2122 |

3 | 20.7928 | 21.5582 | 23.5414 | 23.8333 | 30.6593 | 30.8327 | 29.2358 |

4 | 20.7814 | 21.4473 | 23.7901 | 24.1515 | 29.3016 | 29.5208 | 28.4922 |

5 | 20.9141 | 21.3347 | 22.7660 | 23.8527 | 25.2292 | 26.0602 | 26.0390 |

6 | 20.8267 | 21.2730 | 22.9500 | 24.2042 | 27.0638 | 27.2829 | 27.6730 |

7 | 20.7311 | 21.3759 | 23.4218 | 24.3754 | 29.4845 | 29.9281 | 29.4038 |

8 | 20.7274 | 20.9955 | 22.7796 | 23.9210 | 25.4553 | 26.1903 | 26.7617 |

9 | 20.6406 | 21.3839 | 23.5115 | 24.5921 | 30.5982 | 30.7339 | 30.2494 |

10 | 20.6563 | 21.3936 | 23.4530 | 24.5911 | 30.0215 | 30.1505 | 30.2441 |

11 | 20.8227 | 21.3718 | 23.2586 | 24.3298 | 27.6005 | 28.0054 | 28.1005 |

12 | 20.7841 | 21.4705 | 23.4702 | 24.5056 | 30.5373 | 30.6068 | 30.0079 |

13 | 20.6527 | 20.8826 | 21.7919 | 23.4370 | 23.9516 | 24.2884 | 24.9340 |

14 | 20.8022 | 21.2953 | 23.0835 | 23.7002 | 26.4241 | 27.0160 | 25.8387 |

15 | 21.4102 | 21.8628 | 23.9531 | 24.4306 | 29.1291 | 29.3797 | 28.2387 |

16 | 20.6903 | 21.3393 | 23.4063 | 24.5378 | 29.4499 | 29.5209 | 29.0873 |

17 | 21.0825 | 21.6626 | 23.8226 | 24.8379 | 29.2484 | 29.5851 | 29.7193 |

18 | 20.8700 | 21.4001 | 22.9114 | 23.9673 | 26.0022 | 26.5466 | 26.5046 |

19 | 20.6914 | 21.2650 | 23.2757 | 24.3600 | 28.6516 | 29.0308 | 28.7008 |

20 | 21.8899 | 22.3182 | 24.1087 | 24.8277 | 26.9488 | 27.8986 | 28.0698 |

21 | 20.6429 | 21.2543 | 23.0521 | 24.2204 | 27.4789 | 27.8329 | 28.1341 |

22 | 20.6400 | 21.2883 | 23.1219 | 24.1220 | 27.8864 | 28.2483 | 27.8049 |

23 | 20.7934 | 21.5650 | 23.3896 | 23.2165 | 31.1036 | 30.8427 | 28.3076 |

24 | 20.6561 | 21.1593 | 22.6258 | 23.8926 | 25.9713 | 26.4484 | 26.6763 |

Avg | 20.8394 | 21.3959 | 23.2693 | 24.1332 | 28.0299 | 28.3831 | 27.9999 |

Data | Bayer DLMMSE [7] | Bayer RI [11] | Sony RGBW [22] | Proposed | Akiyama [28] | BM3D-CFA [29] | Proposed + BM3D |
---|---|---|---|---|---|---|---|

1 | 23.4743 | 23.7315 | 25.2154 | 26.1998 | 27.1647 | 27.6840 | 28.2742 |

2 | 23.7555 | 24.0597 | 26.5783 | 26.0850 | 29.8295 | 30.1677 | 29.1778 |

3 | 23.6888 | 24.3995 | 26.5252 | 26.5784 | 31.9909 | 32.3835 | 31.0654 |

4 | 23.6173 | 24.1784 | 26.5210 | 26.3983 | 30.3262 | 30.6601 | 30.1931 |

5 | 23.6914 | 24.0412 | 24.9118 | 25.8992 | 27.0623 | 27.8761 | 27.7638 |

6 | 23.6902 | 24.0157 | 25.5248 | 26.3360 | 28.5588 | 28.8965 | 29.2019 |

7 | 23.6396 | 24.2284 | 26.3332 | 26.7672 | 31.2039 | 31.8072 | 31.1182 |

8 | 23.4988 | 23.6250 | 25.0019 | 26.0813 | 27.1062 | 27.8035 | 28.3955 |

9 | 23.5865 | 24.2659 | 26.5441 | 27.0729 | 32.1003 | 32.3720 | 31.6528 |

10 | 23.6309 | 24.2935 | 26.5336 | 27.1205 | 31.4922 | 31.8247 | 31.8130 |

11 | 23.6742 | 24.1131 | 25.9178 | 26.5520 | 28.9809 | 29.5593 | 29.5306 |

12 | 23.6820 | 24.3276 | 26.4528 | 26.8893 | 31.7944 | 32.1896 | 31.4717 |

13 | 23.4140 | 23.4268 | 23.5726 | 25.1309 | 25.5672 | 25.8364 | 26.3775 |

14 | 23.5546 | 23.9508 | 25.4867 | 25.7873 | 27.8713 | 28.4915 | 27.3360 |

15 | 24.0929 | 24.4846 | 26.5509 | 26.7164 | 30.4343 | 30.7897 | 29.9927 |

16 | 23.5826 | 24.1387 | 26.3258 | 27.0410 | 30.6613 | 30.9932 | 31.0541 |

17 | 23.8864 | 24.3835 | 26.5670 | 27.2550 | 30.8645 | 31.2523 | 31.3186 |

18 | 23.5707 | 24.0102 | 25.1366 | 25.9099 | 27.5905 | 28.0707 | 28.0735 |

19 | 23.5820 | 24.0696 | 26.0365 | 26.6651 | 29.8626 | 30.3696 | 30.2485 |

20 | 24.6759 | 25.0065 | 26.8145 | 27.1831 | 27.4626 | 28.4592 | 30.0495 |

21 | 23.5265 | 24.0586 | 25.6518 | 26.4513 | 28.9960 | 29.4035 | 29.6244 |

22 | 23.4976 | 24.0450 | 25.7935 | 26.3496 | 29.1417 | 29.5544 | 29.3926 |

23 | 23.6690 | 24.3757 | 26.2651 | 26.2849 | 32.3738 | 32.5457 | 30.7539 |

24 | 23.4244 | 23.8393 | 24.8499 | 25.8906 | 27.4968 | 28.0037 | 28.1189 |

Avg. | 23.6711 | 24.1279 | 25.8796 | 26.4436 | 29.4139 | 29.8748 | 29.6666 |

Data | Bayer DLMMSE [7] | Bayer RI [11] | Sony RGBW [22] | Proposed | Akiyama [28] | BM3D-CFA [29] | Proposed + BM3D |
---|---|---|---|---|---|---|---|

1 | 0.8672 | 0.8781 | 0.8914 | 0.9244 | 0.9233 | 0.9292 | 0.9524 |

2 | 0.7978 | 0.8044 | 0.8313 | 0.8701 | 0.8878 | 0.9156 | 0.9382 |

3 | 0.7502 | 0.7671 | 0.7955 | 0.8574 | 0.9330 | 0.9391 | 0.9516 |

4 | 0.7961 | 0.8107 | 0.8373 | 0.8856 | 0.9223 | 0.9339 | 0.9519 |

5 | 0.9067 | 0.9152 | 0.9190 | 0.9418 | 0.9297 | 0.9337 | 0.9595 |

6 | 0.8688 | 0.8789 | 0.8857 | 0.9234 | 0.9189 | 0.9264 | 0.9512 |

7 | 0.8290 | 0.8404 | 0.8593 | 0.9052 | 0.9441 | 0.9466 | 0.9631 |

8 | 0.8991 | 0.9078 | 0.9171 | 0.9409 | 0.9462 | 0.9500 | 0.9687 |

9 | 0.7640 | 0.7807 | 0.8045 | 0.8644 | 0.9425 | 0.9397 | 0.9549 |

10 | 0.8024 | 0.8176 | 0.8357 | 0.8900 | 0.9302 | 0.9306 | 0.9530 |

11 | 0.8406 | 0.8512 | 0.8678 | 0.9103 | 0.9206 | 0.9310 | 0.9504 |

12 | 0.7999 | 0.8140 | 0.8309 | 0.8832 | 0.9124 | 0.9285 | 0.9474 |

13 | 0.9100 | 0.9177 | 0.9178 | 0.9431 | 0.9135 | 0.9250 | 0.9499 |

14 | 0.8684 | 0.8783 | 0.8913 | 0.9241 | 0.9170 | 0.9252 | 0.9500 |

15 | 0.7889 | 0.7998 | 0.8254 | 0.8738 | 0.9293 | 0.9383 | 0.9539 |

16 | 0.7934 | 0.8084 | 0.8290 | 0.8830 | 0.9123 | 0.9201 | 0.9419 |

17 | 0.8368 | 0.8505 | 0.8689 | 0.9127 | 0.9308 | 0.9338 | 0.9564 |

18 | 0.8802 | 0.8890 | 0.8985 | 0.9261 | 0.9028 | 0.9127 | 0.9430 |

19 | 0.8295 | 0.8410 | 0.8610 | 0.9051 | 0.9201 | 0.9312 | 0.9512 |

20 | 0.7783 | 0.7891 | 0.8195 | 0.8691 | 0.9440 | 0.9472 | 0.9589 |

21 | 0.8639 | 0.8735 | 0.8824 | 0.9205 | 0.9241 | 0.9305 | 0.9541 |

22 | 0.8263 | 0.8392 | 0.8578 | 0.9011 | 0.9029 | 0.9191 | 0.9424 |

23 | 0.7393 | 0.7570 | 0.7808 | 0.8504 | 0.9522 | 0.9432 | 0.9554 |

24 | 0.8417 | 0.8533 | 0.8687 | 0.9087 | 0.9234 | 0.9299 | 0.9538 |

Avg. | 0.8283 | 0.8574 | 0.8401 | 0.9006 | 0.9243 | 0.9317 | 0.9522 |

Data | Bayer DLMMSE [7] | Bayer RI [11] | Sony RGBW [22] | Proposed | Akiyama [28] | BM3D-CFA [29] | Proposed + BM3D |
---|---|---|---|---|---|---|---|

1 | 0.9154 | 0.9221 | 0.9314 | 0.9497 | 0.9480 | 0.9513 | 0.9668 |

2 | 0.8657 | 0.8689 | 0.8924 | 0.9179 | 0.9146 | 0.9378 | 0.9535 |

3 | 0.8313 | 0.8436 | 0.8705 | 0.9079 | 0.9483 | 0.9555 | 0.9642 |

4 | 0.8650 | 0.8757 | 0.8986 | 0.9275 | 0.9426 | 0.9525 | 0.9653 |

5 | 0.9413 | 0.9465 | 0.9478 | 0.9611 | 0.9558 | 0.9579 | 0.9733 |

6 | 0.9156 | 0.9215 | 0.9304 | 0.9489 | 0.9468 | 0.9503 | 0.9656 |

7 | 0.8878 | 0.8957 | 0.9132 | 0.9386 | 0.9612 | 0.9644 | 0.9738 |

8 | 0.9345 | 0.9401 | 0.9465 | 0.9609 | 0.9635 | 0.9668 | 0.9779 |

9 | 0.8398 | 0.8522 | 0.8747 | 0.9128 | 0.9599 | 0.9595 | 0.9671 |

10 | 0.8715 | 0.8815 | 0.9007 | 0.9313 | 0.9523 | 0.9528 | 0.9681 |

11 | 0.8950 | 0.9023 | 0.9161 | 0.9407 | 0.9433 | 0.9505 | 0.9653 |

12 | 0.8689 | 0.8785 | 0.8939 | 0.9250 | 0.9345 | 0.9466 | 0.9602 |

13 | 0.9428 | 0.9471 | 0.9456 | 0.9596 | 0.9453 | 0.9500 | 0.9661 |

14 | 0.9148 | 0.9211 | 0.9307 | 0.9501 | 0.9430 | 0.9477 | 0.9645 |

15 | 0.8565 | 0.8641 | 0.8875 | 0.9176 | 0.9480 | 0.9545 | 0.9650 |

16 | 0.8634 | 0.8735 | 0.8922 | 0.9257 | 0.9374 | 0.9428 | 0.9591 |

17 | 0.8936 | 0.9020 | 0.9186 | 0.9440 | 0.9518 | 0.9529 | 0.9694 |

18 | 0.9225 | 0.9285 | 0.9354 | 0.9502 | 0.9397 | 0.9439 | 0.9615 |

19 | 0.8888 | 0.8971 | 0.9136 | 0.9386 | 0.9423 | 0.9491 | 0.9648 |

20 | 0.8510 | 0.8576 | 0.8882 | 0.9166 | 0.9586 | 0.9627 | 0.9695 |

21 | 0.9131 | 0.9194 | 0.9271 | 0.9484 | 0.9489 | 0.9521 | 0.9685 |

22 | 0.8875 | 0.8962 | 0.9124 | 0.9353 | 0.9290 | 0.9409 | 0.9579 |

23 | 0.8241 | 0.8375 | 0.8606 | 0.9050 | 0.9657 | 0.9611 | 0.9665 |

24 | 0.8982 | 0.9059 | 0.9180 | 0.9403 | 0.9473 | 0.9516 | 0.9673 |

Avg. | 0.8870 | 0.8949 | 0.9102 | 0.9356 | 0.9470 | 0.9523 | 0.9659 |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).