In typical imaging devices a color filter is placed on top of the imager making each pixel sensitive to only one color component. A color reconstruction algorithm interpolates the missing information at each location and reconstructs the full RGB image [9–11]. The color filter selects the red, green or blue component for each pixel; this arrangement is known as Bayer pattern [6]; other arrangements of CFA data take into account CMY complementary colors, but the RGB color space is the most common.

The number of green elements is twice the number of red and blue pixels due to the higher sensitivity of the human eye to the green light, which, in fact, has a higher weight when computing the luminance. The proposed filter processes raw Bayer data, providing the best performance if executed as the first algorithm of the IGP (Image Generation Pipeline). A typical image reconstruction pipeline is shown in Figure 1.

It is well known that the HVS has a different sensitivity at different spatial frequencies [28]. In areas containing mean frequencies the eye has a higher sensitivity. Furthermore, chrominance sensitivity is weaker than the luminance one.

HVS response does not entirely depend on the luminance value itself, rather, it depends on the luminance local variations with respect to the background; this effect is described by the Weber-Fechner’s law [13,29], which determines the minimum difference DY needed to distinguish between Y (background) and Y+DY. Different values of Y yield to different values of DY.

The aforementioned properties of the HVS have been used as a starting point to devise a CFA filtering algorithm. Luminance from CFA data can be extracted as explained in [30], but for our purposes it can be roughly approximated by the green channel values before gamma correction.

The filter changes its smoothing capability depending on the CFA color of the current pixel and its similarity with the neighborhood pixels.

More specifically, in relation to image content, the following assumptions are considered:

- if the local area is homogeneous, then it can be heavily filtered because pixel variations are basically caused by random noise.

A block diagram describing the overall filtering process is illustrated in Figure 2. Each block will be separately described in detail in the following sections.

The fundamental blocks of the algorithm are:

Signal Analyzer Block: computes a filter parameter incorporating the effects of human visual system response and signal intensity in the filter mask.

The data in the filter mask passes through the Signal Analyzer block that influences the filter strength in dark and bright regions (Section 3.2 for further details). The HVS value is used in combination with the output of the Texture Degree Analyzer (Section 3.4) and Noise Level Estimator (Section 3.5) to produce the similarity thresholds used to finally compute the weights assigned to the neighborhood of the central pixel (Section 3.6). The final filtered value is obtained by a weighted averaging process (Section 3.7).

As noted [31–33], it is possible to approximate the minimum intensity gap that is necessary for the eye to perceive a change in pixel values. The base sensitivity thresholds measure the contrast sensitivity in function of frequency while fixing the background intensity level. In general, the detection threshold varies also with the background intensity. This phenomenon is known as luminance masking or light adaptation. Higher gap in intensity is needed to perceive a visual difference in very dark areas, whereas for mid and high pixel intensities a small difference in value between adjacent pixels is more easily perceived by the eye [32].

It also crucial to observe that in data from real image sensors, the constant AWGN (Additive White Gaussian Noise) model does not fit well the noise distribution for all pixel values. In particular, as discussed in [34], the noise level in raw data is predominantly signal-dependent and increases as the signal intensity raises; hence, the noise level is higher in very bright areas. In [34] and [35] it is also illustrated how clipping in data is the cause of noise level underestimation; e.g., noise level for pixels close to saturation cannot be robustly tracked because the signal reaches the upper limit of the allowed bitdepth encoding.

We decided to incorporate the above considerations of luminance masking and sensor noise statistics into a single curve as shown in Figure 3. The shape of this curve allows compensating for lower eye sensitivity and increased noise power in the proper areas of the image, allowing adaptive filter smoothing capability in relation to the pixel values.

A high HVS value (HVSmax) is set for both low and high pixel values: in dark areas the human eye is less sensitive to variations of pixel intensities, whereas in bright areas noise standard deviation is higher. HVS value is set low (HVSmin) at mid pixel intensities.

As stated in Section 2.2, in order to make some simplifying assumptions, we use the same HVS curve for all CFA colour channels taking as input the pixel intensities directly from the sensor. The HVS coefficient computed by this block is used by the Texture Degree Analyzer that outputs a degree of texture taking also into account the above considerations (Section 3.4).

The proposed filter uses different filter masks for green and red/blue pixels to match the particular arrangement of pixels in the CFA array. The size of the filter mask depends on the resolution of the imager: at higher resolution a small processing window might be unable to capture significant details. For our processing purposes a 5×5 window size provided a good trade-off between hardware cost and image quality, allowing us to process images up to 5 megapixels, a resolution that is typical of high end mobile phones. Typical Bayer processing windows are illustrated in Figure 4.

The texture analyzer block computes a reference value T_d that is representative of the local texture degree. This reference value approaches 1 as the local area becomes increasingly flat and decreases as the texture degree increases (Figure 5). The computed coefficient is used to regulate the filter smoothing capability so that high values of T_d correspond to flat image areas in which the filter strength can be increased.

Depending on the color of the pixel under processing, either green or red/blue, two different texture analyzers are used. The red/blue filter power is increased by slightly modifying the texture analyzer making it less sensitive to small pixel differences (Figure 6). The texture analyzer block output depends on a combination of the maximum difference between the central pixel and the neighborhood D_max and TextureThreshold, a value that is obtained by combining information from the HVS response and noise level, as described below (2).

The green and red/blue texture analyzers are defined as follows: (1) T d   ( green ) = { 1 D max = 0 − D max TextureThreshold + 1 0 < D max ≤ TextureThreshold 0 D max > TextureThreshold T d   ( red / blue ) = { 1 D max ≤ Th R / B − ( D max − Th R / B ) ( TextureThreshold − Th R / B ) + 1 Th R / B < D max ≤ TextureThreshold 0 D max > TextureThresholdhence:

- if T_d = 1 the area is assumed to be completely flat;

The texture threshold for the current pixel, belonging to Bayer channel c (c=R,G,B), is computed by adding the noise level estimation to the HVS response (2): (2) TextureThreshold c   ( k ) = HVS weight ( k ) + NL c ( k − 1 )where NL_c denotes the noise level estimation on the previous pixel of the same Bayer color channel c(see Section 3.5) and HVS_weight (Figure 3) can be interpreted as a jnd (just noticeable difference); hence an area is no longer flat if the D_max value exceeds the jnd plus the local noise level NL.

The green texture analyzer (Figure 5) uses a stronger rule for detecting flat areas, whereas the red/blue texture analyzer (Figure 6) detects more flat areas, being less sensitive to small pixel differences below the Th_R/B threshold. The gray-scale output of the texture detection is shown in Figure 7: bright pixels are associated to high texture, dark pixels to flat areas.

In order to adapt the filter smoothing capability to the local characteristics of the image, a noise level estimation is required. The proposed noise estimation solution is pixel based and is implemented taking into account the previous estimation to calculate the current one.

The noise estimation equation is designed so that:

if the local area is completely flat (T_d = 1), then the noise level is set to D_max;

Each color channel has its own noise characteristics hence noise levels are tracked separately for each color channel. The noise level for each channel is estimated according to the following formulas: (3) NL R   ( k ) = T d   ( k ) * D max   ( k ) + [ 1 − T d   ( k ) ] * NL R   ( k − 1 ) NL G   ( k ) = T d   ( k ) * D max   ( k ) + [ 1 − T d   ( k ) ] * NL G   ( k − 1 ) NL B   ( k ) = T d   ( k ) * D max   ( k ) + [ 1 − T d   ( k ) ] * NL B   ( k − 1 )where T_d(k) represents the texture degree at the current pixel and NL_c(k−1) (c=R,G,B) is the previous noise level estimation, evaluated considering pixel of the same colour, already processed. For k = 1 the values NL_R(k−1), NL_G(k−1) and NL_B(k−1) are set to an initial low value depending on the pixel bit-depth. These equations satisfy requirements i), ii) and iii). The raster scanning order of the input image is constrained by global HW architecture. Starting from different spatial locations the noise level converges to the same values due to the presence of homogeneous areas that are, of course, prominent in almost all natural images.

The final step of the filtering process consists in determining the weighting coefficients W_i to be assigned to the neighboring pixels of the filter mask. The absolute differences D_i between the central pixel and its neighborhood must be analyzed in combination with the local information (noise level, texture degree and pixel intensities) for estimating the degree of similarity between pixel pairs (see Figure 8). As stated in Section 2.2, if the central pixel P_c belongs to a textured area, then only small pixel differences must be filtered. The lower degree of filtering in textured areas allows maintaining the local sharpness, removing only pixel differences that are not perceived by the HVS.

The process for determining the similarity thresholds and the W_i coefficients can be expressed in terms of fuzzy logic (Figure 9).

Let:

- P_c be the central pixel of the working window;

In order to obtain the W_i coefficients, each absolute difference D_i must be compared against two thresholds Th_low and Th_high that determine if, in relation to the local information, the i-th difference D_i is:

small enough to be heavily filtered,

The two thresholds can be interpreted as fuzzy parameters shaping the concept of similarity between pixel pairs. In particular, the associated fuzzy member function computes the similarity degree between the central and a neighborhood pixel.

By properly computing Th_low and Th_high, the shape of the membership function is determined (Figure 10).

To determine which of the above cases is valid for the current local area, the local texture degree is the key parameter to analyze. It is important to remember at this point that, by construction, the texture degree coefficient (T_d) incorporates the concepts of dark/bright and noise level; hence, its value is crucial to determine the similarity thresholds to be used for determining the W_i coefficients. In particular, the similarity thresholds are determined to obtain maximum smoothing in flat areas, minimum smoothing in highly textured areas, and intermediate filtering in areas containing medium texture; this can be obtained by using the following rules (4): (4) { Th low = Th high = D max   if   T d = 1 Th low = D min   if   T d = 0 Th high = D min + D max 2   if   T d = 0 D min < Th low < Th high   if   0 < T d < 1 D min + D max 2 < Th high < D max   if   0 < T d < 1

Once the similarity thresholds have been fixed, it is possible to finally determine the filter weights by comparing the D_i differences against them (Figure 10).

To summarize, the weighting coefficient selection is performed as follows. If the i-th absolute difference D_i is lower than Th_low, it is reasonable to assume that pixels P and P_i are very similar; hence the maximum degree of similarity Max_weight is assigned to P_i. On the other hand, if the absolute difference between P and P_i is greater than Th_high, it is reasonable that this difference is due to texture details, hence P_i is assigned a null similarity weight. In the remaining cases, i.e. when the i-th absolute difference falls in the interval [Th_low, Th_high], a linear interpolation between Max_weight and 0 is performed, allowing determining the appropriate weight for P_i.

Let W₁,…,W_N (N: number of neighborhood pixels) be the set of weights computed for the each neighboring element of the central pixel P_c. The final filtered value P_f is obtained by weighted average as follows (5): (5) P f = 1 N ∑ i = 1 N [ W i P i + ( 1 − W i ) P c ]

In order to preserve the original bitdepth, the similarity weights are normalized in the interval [0,1], and chosen according to equation (6): (6) W i = { 1   if   D i ≤ Th low L ( Th low , Th high )   if   Th low < D i < Th high 0   if   D i ≥ Th highwhere L(Th_low, Th_high) performs a simple linear interpolation between Th_low and Th_high as depicted in Figure 10.

A synthetic image was used to determine the amount of noise that the algorithm is capable to remove. Let us denote:

I_NOISY: Noisy CFA Pattern

According to these definitions we have:

I_NOISY − I_ORIGINAL = I_{ADDED_NOISE}

To modulate the power of the additive noise, different values of the standard deviation of a Gaussian distribution are used. Noise is assumed to be AWGN (Additive White Gaussian Noise), with zero mean.

A synthetic test image has been generated having the following properties: it is composed by a succession of stripes having equal brightness but different noise power. Each stripe is composed of 10 lines and noise is added with increasing power starting from the top of the image and proceeding downwards (Figure 11).

The graph in Figure 12 illustrates the filtering effects in terms of noise power; the x-axis represents the noise standard deviation; the y-axis shows the corresponding noise power decibels before and after filtering. The filter significantly reduces noise and gains up to 6–7dB can be obtained in terms of noise power reduction.

In order to assess the visual quality of the proposed method, we have compared it with the SUSAN (Smallest Univalue Segment Assimilating Nucleus) [37] and multistage median filters [36] classical noise reduction algorithm. This choice is motivated by considering the comparable complexity of these solutions. Though more complex recent methods for denoising image data exist [7,8,18,38] achieving very good results, they are not yet suitable for real-time implementation.

The tests were executed using two different approaches. In the first approach, the original noisy Bayer data were interpolated obtaining a noisy color image, which was splitted in its color channels; each color plane was filtered independently using SUSAN. Finally, the filtered color channels were recombined to obtain the denoised color image as sketched in Figure 13.

The second approach consists in slightly modifying the SUSAN algorithm so that it can process Bayer data. In both cases, the results of SUSAN were compared with the color-interpolated image obtained from a denoised Bayer pattern produced by the proposed method.

Figure 14 shows two of test noisy reference images acquired by a CFA image sensor (2 megapixels) after colour interpolation. Original SNR values for the two images are 30.2 dB and 47.2 dB, respectively. After filtering, the corresponding SNR values became comparable and higher for both, SUSAN and our filtering. In the first comparison test, both algorithms show very good performances; the proposed method, anyway, is capable to preserve some small details that are lost by SUSAN independent R/G/B filtering. Furthermore, processing is very fast because the method processes only one plane of image information, i.e. the CFA data. Figure 15 shows a magnified detail of Figure 14(a) and the filtering results with SUSAN and our method. Figure 16 shows how the proposed method significantly retains texture and sharpness after filtering. Figure 17 shows two different details of the noisy image in Figure 14(b) and their filtered counterparts. The homogeneous areas are heavily filtered (a), (b); on the other hand, in textured areas, the detail is well preserved (c), (d).

Finally, Figure 18 illustrates the results of the multistage median filters described in [36] compared with the proposed filter. Specifically, the multistage median-1 and multistage median-3 filter outputs were considered. The three methods work on CFA data. Figure 18 (e) shows, again, that the proposed filtering technique is able to preserve texture and sharpness very well.

In order to numerically quantify the performance of the filtering process, the standard Kodak 24 (8-bpp) [39] images have been processed with the proposed method comparing them with the outputs of SUSAN [37], Multistage median-1, Multistage median-3 algorithms [36] and the following fuzzy approaches from [25]:

- GMED: Gaussian Fuzzy Filter with Median Center

After converting each image of the set to Bayer pattern format, the simulation was performed by adding noise with increasing standard deviation to each CFA plane. In particular the following values have been used: σ = 5, 8, 10. More specifically, the aforementioned values of σ refer to the noise level in the middle of the dynamic range. To simulate a more realistic sensor noise, in fact, we followed the model described in [34,35], that allows obtaining lower noise values for dark areas and higher noise values for bright areas, according to a square root characterization of the noise. In order to exclude the effects of different color interpolations from the computation of the PSNR, the reference images were obtained following the procedure described in Figure 19(a); in this way, both images (i.e. clean and noisy) are generated using the same color interpolation algorithm.

Experiments show that the proposed method performs well in terms of PSNR compared to the algorithms used in the test (Figure 20). In order to compare the proposed method with other fuzzy approaches, we considered some methods described in [25]. The results are shown in Figure 21.