Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Abstract

It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. However, to the best of our knowledge, a systematic study to demonstrate the above statement does not exist. We present a comparative study to systematically and thoroughly evaluate the performance of demosaicing for low lighting images using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using the clean Kodak dataset containing 12 images, we first emulated low lighting images by injecting Poisson noise at two signal-to-noise (SNR) levels: 10 dBs and 20 dBs. We then created CFA 1.0 and CFA 2.0 images for the noisy images. After that, we applied more than 15 conventional and deep learning based demosaicing algorithms to demosaic the CFA patterns. Using both objectives with five performance metrics and subjective visualization, we observe that having more white pixels indeed helps the demosaicing performance in low lighting conditions. This thorough comparative study is our first contribution. With denoising, we observed that the demosaicing performance of both CFAs has been improved by several dBs. This can be considered as our second contribution. Moreover, we noticed that denoising before demosaicing is more effective than denoising after demosaicing. Answering the question of where denoising should be applied is our third contribution. We also noticed that denoising plays a slightly more important role in 10 dBs signal-to-noise ratio (SNR) as compared to 20 dBs SNR. Some discussions on the following phenomena are also included: (1) why CFA 2.0 performed better than CFA 1.0; (2) why denoising was more effective before demosaicing than after demosaicing; and (3) why denoising helped more at low SNRs than at high SNRs.

1. Introduction

The standard Bayer pattern [1], also known as color filter array (CFA) 1.0, has been widely used in many commercial cameras. As shown in Figure 1a, for each 2 × 2 block, there are two green pixels, one red pixel, and one blue pixel. Even in the Mastcam onboard the Mars rover Curiosity [2,3,4,5], the Bayer pattern has been used for the RGB images. The main reason for using the Bayer pattern is to reduce cost. Some researchers also developed image tamper detection algorithms based on demosaicing artifacts [6]. Because of the success of the Bayer pattern, a follow-up pattern, known as red-green-blue-white (RGBW) or CFA 2.0, was introduced by researchers at Kodak [7,8]. For each 4 × 4 block in a RGBW pattern (Figure 1b), there are 50% white pixels, 25% green pixels, and 12.5% red and blue pixels. In the past two decades, there are numerous other CFA patterns mentioned in [9,10,11,12,13,14,15].

Figure 1. Two color filter array (CFA) patterns. (a) Standard Bayer pattern (CFA 1.0); (b) RGBW (red-green-blue-white) (CFA 2.0).

For images collected in normal illumination conditions, our earlier papers [16] concluded that CFA 1.0 is better than CFA 2.0. In the CFA research community, one common belief is that CFA 2.0 has better performance for images taken in low lighting conditions. The argument is that, due to the presence of 50% white pixels in CFA 2.0, the signal-to-noise (SNR) of the whole image should be higher, hence the demosaicing performance of CFA 2.0 should be better for low lighting images.

From the above discussions, one may have several natural questions regarding the various CFA patterns. First, has anyone carried out a comparative study to compare CFA 1.0 and CFA 2.0 for low lighting images? To the best of our knowledge, only one paper [17] briefly mentioned that CFA 2.0 has some advantages in some images. This means that the claim that CFA 2.0 is more suitable for low lighting conditions may be based on intuition rather than on observations based on objective experiments. It will be good to have some objective measures to judge which CFA is better for low lighting conditions. Second, how can one perform objective experiments for low lighting conditions? If one collects images in low lighting conditions, then we may not have the ground truth images, which would be used to generate objective metrics. In [17], low lighting images were emulated by adding noise to clean images. It is well-known that the noise induced by low lighting conditions is called Poisson noise, which is magnitude-dependent. If one simply adds Gaussian noise to clean reference images, then the noise behavior will be very different from that of images collected in actual low lighting conditions. In [18], a procedure for adding Poisson noise is mentioned in detail, and we have adopted that procedure in this research. Third, after the demosaicing of low lighting images, the demosaiced images are still noisy. A common practice is to perform some denoising and contrast enhancement to improve the image quality. One immediate question regards where we should apply the denoising step. There are two places to introduce the denoising: after demosaicing and before demosaicing. Which one is better? Answering the above questions will have important implications in practice. First, practitioners or camera designers may design a camera in which a denoising algorithm is activated if lighting conditions are unfavorable. Second, camera manufacturers need to know where denoising should be performed if CFA 2.0 is chosen.

In this paper, we attempt to answer the questions raised earlier. In Section 2, we will briefly review a number of demosaicing algorithms for CFA 1.0 and CFA 2.0. The algorithms range from conventional to deep learning. Two out of 16 methods for CFA 1.0 are deep learning algorithms. Although there are other promising learning methods in the literature [19,20,21,22], some serious customizations may be needed. In Section 3, we will summarize a comparative study that compares the performance of CFA 1.0 and CFA 2.0 using a benchmark data set (Kodak). Noisy images emulating two low lighting conditions were generated. The noisy images have 10 dBs and 20 dBs SNR. Three case studies were performed for each CFA: (1) no denoising; (2) denoising before demosaicing; (3) denoising after demosaicing. Our first major finding is that CFA 2.0 indeed helped the demosaicing performance for both 10 dBs and 20 dBs conditions. Our second finding is that the demosaicing performance of the CFAs performs even better if denoising is applied. Our third finding is that denoising before demosaicing is better than denoising after demosaicing. Our last finding is that denoising helps demosaicing more in the 10 dBs SNR case than the 20 dBs SNR case. Some discussions are included to provide some qualitative analysis for the above findings. Section 4 concludes the paper with some further remarks and future research directions.

2. Demosaicing Algorithms

In this section, we present some algorithms for demosaicing CFA 1.0 and CFA 2.0.

2.1. Algorithms for Demosaicing CFA 1.0

The following algorithms were evaluated in our experiments and they are briefly summarized below:

• Linear Directional Interpolation and Nonlocal Adaptive Thresholding (LDI-NAT): This algorithm is simple but the non-local search is time consuming [23];
• Malvar–He–Cutler (MHC): This is the algorithm in [24]. This is the default method for demosaicing Mastcam images [2] used by NASA. The algorithm is very efficient and simple to implement;
• Directional Linear Minimum Mean Square-Error Estimation (DLMMSE): This is the Zhang and Wu algorithm in [25]. This method was investigated in Bell et al.’s paper [2];
• Lu and Tan Interpolation (LT): This is a frequency domain approach [26];
• Adaptive Frequency Domain (AFD): This is a frequency domain approach from Dubois [27]. The algorithm can also be used for other mosaicing patterns;
• Alternate Projection (AP): This is the algorithm from Gunturk et al. [28];
• Primary-Consistent Soft-Decision (PCSD): This is Wu and Zhang’s algorithm from [29];
• Alpha Trimmed Mean Filtering (ATMF): This method is from [30,31]. At each pixel location, we demosaic pixels from seven methods; the largest and smallest pixels are removed and the mean of the remaining pixels are used;
• Demosaicnet (Demonet): In [32], a feed-forward network architecture was proposed for demosaicing. There are D + 1 convolutional layers and each layer has W outputs and uses K × K size kernels. An initial model was trained using 1.3 million images from Imagenet and 1 million images from MirFlickr. Additionally, some challenging images were searched to further enhance the training model. Details can be found in [32];
• Fusion using three best (F3) [30]: The mean of pixels from demosaiced images of the three best individual methods were used;
• Bilinear: Bilinear interpolation is the simplest algorithm that uses the nearest neighbors for interpolation;
• Sequential Energy Minimization (SEM) [33]: A deep learning approach based on sequential energy minimization was proposed in [33]. The performance was reasonable, except that the computation takes a long time due to sequential optimization;
• Exploitation of Color Correlation (ECC) [34]: The authors of [34] proposed a scheme that exploits the correlation between different color channels much more effectively than some of the existing algorithm;
• Minimized-Laplacian Residual Interpolation (MLRI) [35]: This is a residual interpolation (RI)-based algorithm based on a minimized-Laplacian version;
• Adaptive Residual Interpolation (ARI) [36]: ARI adaptively combines RI and MLRI at each pixel, and adaptively selects a suitable iteration number for each pixel, instead of using a common iteration number for all of the pixels;
• Directional Difference Regression (DDR) [37]: DDR obtains the regression models using directional color differences of the training images. Once models are learned, they will be used for demosaicing.

It should be noted that F3 and ATMF are both pixel-level fusion methods. Details can be found in [30].

2.2. Algorithms for Demosaicing CFA 2.0

The baseline approach refers to a simple upsampling of the reduced resolution color image shown in Figure 2. The standard approach for CFA 2.0 is shown in Figure 2, which illustrates how to combine the interpolated luminance image with the reduced resolution color image to generate a full resolution color image.

Figure 2. Standard approach to demosaicing CFA 2.0 images. Image from [38].

In the paper [16] written by us, we proposed a pansharpening approach to demosaicing CFA 2.0. This approach is illustrated in Figure 3. The missing pixels in the panchromatic band are interpolated. At the same time, the reduced resolution CFA is demosaiced. We then apply pansharpening to generate the full resolution color image. There are many pansharpening algorithms that can be used. Principal Component Analysis (PCA) [39], Smoothing Filter-based Intensity Modulation (SFIM) [40], Modulation Transfer Function Generalized Laplacian Pyramid (GLP) [41], MTF-GLP with High Pass Modulation (HPM) [42], Gram Schmidt (GS) [43], GS Adaptive (GSA) [44], Guided Filter PCA (GFPCA) [45], PRACS [46] and hybrid color mapping (HCM) [47,48,49,50,51] have been used in our experiments. The list is a representative, if not exhaustive, set of competitive pansharpening algorithms.

Figure 3. A pansharpening approach to demosaicing CFA 2.0 images [16].

Recently, we further improved the pansharpening approach by integrating it with deep learning. This approach was summarized in a recent paper [52]. The key idea is to apply deep learning to improve two steps. The first is the demosaicing of the reduced resolution CFA (see Figure 4) via deep learning. The second is the improvement of the interpolation of the pan band via deep learning. The particular deep learning algorithm is Demonet mentioned earlier. We have seen good performance improvement.

Figure 4. A combination of deep learning and pansharpening approach to demosaicing CFA 2.0.

In addition to the above mentioned algorithms for CFA 2.0, we also applied least-squares luma-chroma demultiplexing (LSLCD) over [53] in our experiments.

We also have two fusion based algorithms known as F3 and ATMF, which were used in our earlier studies [16,30,52]. F3 fuses the three best performing algorithms and ATMF fuses seven high-performing algorithms.

It should be noted that algorithms for CFA 2.0 are much fewer than those of CFA 1.0. There may be promising machine learning algorithms [19,20,21,22] that have the potential to be applied to demosaicing of CFA 2.0.

3. Comparative Studies

In this section, we will answer the questions raised in Section 1. One of them is whether CFA 2.0 is indeed better than CFA 1.0 for low lighting conditions. The second is how to emulate low lighting images. The third is where denoising should be introduced. In short, we will answer the following question: which one of the two CFAs is the best method for images collected in low lighting conditions?

Since there are many possible algorithms for each CFA, our strategy is to first perform a comparative study for all the algorithms for each CFA using the same data set. We then compare the best methods from all the CFA studies. That is, we compare the best against the best, to select the best CFA.

3.1. Low Lighting Images and Denoising

We downloaded a benchmark data set (Kodak) from a website (http://r0k.us/graphics/kodak/) and selected 12 images, which are shown in Figure 5. It should be noted that this dataset is well-known and has been used by many authors in the demosaicing community such as [23,25,26,27,28,29]. These clean images will be used as reference images for objective performance metrics generation. Moreover, they will be used to generate noisy images that emulate low lighting conditions.

Figure 5. Twelve clean images from the Kodak dataset.

Emulating images in low lighting conditions is non-trivial. This is because noise introduced in low lighting images is known as Poisson noise. Unlike Gaussian noise, Poisson noise is amplitude dependent. That is, the amount of noise applied depends on the magnitude range of the image. To create a consistent level of noise close to our SNR levels of 10 dBs and 20 dBs, we created a loop where each image was rescaled between 1 and some number less than 255. Poisson noise was applied to each band. The rescaling was adjusted until the PSNR between the ground truth and the noisy image was as close to the desired SNR level as possible. This technique is described in [18]. The noisy images at 10 dBs and 20 dBs are shown in Figure 6 and Figure 7, respectively.

Figure 6. Twelve noisy images at 10 dB (Poisson noise).

Figure 7. Twelve noisy images at 20 dB (Poisson noise).

It should be noted that simply adding Gaussian noise to the clean image cannot emulate low lighting images. For example, we added Gaussian noise to the clean images to create images at 10 dBs SNR. The noisy images are shown in Figure 8. It can be seen the image characteristics are totally different as compared to the Poisson noisy images shown in Figure 6.

Figure 8. Noisy images using Gaussian additive noise at a signal-to-noise ratio (SNR) of 10 dBs. The image characteristics are very different from those of Poisson noise.

We adopted the well-known denoising algorithm known as BM3D (Block Matching 3D) [54] in our denoising experiments.

3.2. Performance Metrics

In this paper, we have used five performance metrics to compare the different methods and CFA patterns.

• Peak Signal-to-Noise Ratio (PSNR) [55]

PNSR is related to Root Mean Squared Error (RMSE). The RMSE of two vectorized images $S$ (ground truth) and $\widehat{S}$ (prediction) is defined as

$$\mathrm{RMSE}(S,\widehat{S})=\sqrt{\frac{1}{Z}{\displaystyle \sum _{j=1}^Z{\left(s_j-{\widehat{s}}_j\right)}^2}}$$

(1)

where Z is the number of pixels in each image. The ideal value of RMSE is 0 if the prediction is perfect. If the image pixels are expressed in doubles with values between 0 and 1, then

$$\mathrm{PSNR}=20\log (1/\mathrm{RMSE}(S,\widehat{S}))$$

(2)

A higher PSNR means better quality. A combined PSNR is the mean of the PSNRs of the R, G, B bands.

• Structural SIMilarity (SSIM)

This metric was defined in [56] to reflect the similarity between two images. The SSIM index is computed on various blocks of an image. The measure between two blocks x and y from two images can be defined as

$$SSIM(x,y)=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_x^2+c_1)({\sigma }_x^2+{\sigma }_x^2+c_2)}$$

(3)

where ${\mu }_x$ and ${\mu }_y$ are the means of blocks x and y, respectively; ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variances of blocks x and y, respectively; ${\sigma }_{xy}$ is the covariance of blocks x and y; and $c_1$ $c_2$ are small values (0.01, for instance) to avoid instability. The ideal value of SSIM is 1 for perfect prediction.

• Human Visual System (HVS) metric

The HVS metric in dB is defined as

$$HVS=20log\left(255/MS{E}_H\right)$$

(4)

where

$$MS{E}_H=K{\displaystyle \sum _{i=1}^{I-7}{\displaystyle \sum _{j=1}^{J-7}{\displaystyle \sum _{m=1}^8{\displaystyle \sum _{n=1}^8{((X{[m,n]}_{ij}-X{[m,n]}_{ij}^e)T_c[m,n])}^2}}}}$$

(5)

I and J denote image size, $K=1/[(\mathrm{I}-7)(\mathrm{J}-7)64]$, $X_{ij}$ are the discrete cosine transform (DCT) [57] coefficients of 8 × 8 image block for which the coordinates of the its upper left corner are equal to i and j, $X_{ij}^e$ are the DCT coefficients of the corresponding block in the original image, and $T_c$ is the matrix of correcting factors [58].

• HVSm (HVS with masking)

This metric is similar to HVS except that visual masking effects are taken into account. The inclusion of a block containing contrast masking is the only difference between HVS and HVSm. Details can be found in [59].

On the website of the authors of [59], there is a table containing the correlation of different metrics with human perception. For completeness, we include that table below (Table 1). It can be seen that HVSm and HVS have much higher correlation with human perception than PSNR and SSIM in terms of Spearman and Kendall correlation coefficients.

Table 1. Correlation of different metrics to human’s visual perception.

Measure	Reference	Spearman Correlation	Kendall Correlation
HVS-m	Ponomarenko, N.; et al. [59]	0.984	0.948
HVS	Egiazarian, K.; et al. [58]	0.895	0.712
NQM	Damera-Venkata, N.; et al. [60]	0.857	0.673
DCTune	Watson, A. B.; et al. [61]	0.829	0.712
UQI	Wang, Z.; Bovik, A. C. [62]	0.550	0.438
PSNR	Peak Signal to Noise Ratio [55]	0.537	0.359
SSIM	Structural similarity [56]	0.406	0.358
VIF	Sheikh, H. R.; Bovik, A. C. [63]	0.377	0.255
PQS	Miyahara, M.; Kotani, K.; Algazi, V. R. [64]	0.302	0.242

In addition to PSNR, SSIM, HVS, and HVSm, we also used CIELAB [65] for assessing demosaicing performance.

Before we summarize the detailed experimental results, we would like to use a diagram (Figure 9) to highlight the various studies and their corresponding sections.

Figure 9. Summary of all experiments and their corresponding sections.

3.3. CFA 1.0 Results

In this section, we summarize the CFA 1.0 studies for two SNRs: 10 dBs and 20 dBs. Within each SNR, we have three sub-cases. Both objective and subjective evaluations have been used in our studies.

3.3.1. 10 dBs SNR Case

We have three case studies. The first case is about demosaicing the noisy images without any denoising. The second case deals with the scenario where denoising is performed after demosaicing. The third case is to investigate the performance of denoising before demosaicing.

• Case 1: No Denoising

As mentioned earlier, we have 16 methods for demosaicing CFA 1.0, which were mentioned in Section 2.1. The F3 fusion method fuses the results of Demonet, Bilinear, and SEM, which were the best performing demosaicing methods. The ATMF fusion method uses the seven highest performing methods, which are Demonet, Bilinear, SEM, PCSD, DLMMSE, LDI, and LT. Table A1 in Appendix A summarizes the PSNR scores, which are the average of the three individual PSNR scores for R, G, and B bands, the CIELAB scores, SSIM, HVS, and HVSm metrics. One can see that all methods achieved PSNR values of around 16 dBs. All the SSIM values are low and the CIELAB scores are high (poor). The HVS and HVSm metrics are also not high.

Figure 10 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. There are some minor variations in the metrics.

Figure 10. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 1.0 pattern. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 11 shows the demosaiced results of Image 1 and Image 8. The demosaiced images have color distortion and noise.

Figure 11. Visual comparison of three high performing demosaicing algorithms for CFA 1.0 at 10 dBs SNR (Poisson noise).

In short, without denoising, all the demosaicing algorithms performed not so well at 10 dBs.

• Case 2: Denoising after Demosaicing

Here, our focus is to investigate the demosaicing performance with help from the BM3D denoising algorithm, which is applied after demosaicing is completed.

The F3 fusion method fuses the results of Demonet, Bilinear, and SEM, which were the best performing demosaicing methods in this case. The ATMF fusion method uses the seven highest performing methods, which are Demonet, Bilinear, SEM, DLMMSE, LDI, AP, and LT. Table A2 in Appendix A summarizes the PSNR, CIELAB, SSIM, HVS, and HVSm metrics. One can see that all methods achieved PSNR values of around 20 dBs, which are 4 dBs higher than those values in Table A1 in Appendix A. All the SSIM, CIELAB, HVS, and HVSm values have been improved over the no-denoising case.

Figure 12 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. All the scores have improved quite a lot, as compared to those in Figure 10.

Figure 12. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 1.0 pattern with denoising after demosaicing. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 13 shows the demosaiced results of Image 1 and Image 8. The demosaiced images look much better than those images in Figure 11. The artifacts are less noticeable after denoising.

Figure 13. Visual comparison of three demosaicing results for CFA 1.0 at 10 dBs SNR (Poisson noise). Denoising is performed after demosaicing.

• Case 3: Denoising before Demosaicing

Here, denoising was performed before demosaicing started. In other words, BM3D was applied to the CFA patterns before feeding them into the demosaicing algorithms. Intuitively, this makes more sense in practical applications because denoising should be more effective if one suppresses noise at the early stages rather than near the end of the demosaicing process.

The F3 fusion method fuses the results of Demonet, AP, and LT, which were the best performing demosaicing methods in this case. The ATMF fusion method uses the seven highest performing methods, which are Demonet, AP, LT, DLMMSE, DDR, LDI, and ECC. Table A3 in Appendix A summarizes the PSNR, CIELAB, SSIM, HVS, and HVSm metrics. One can see that all methods achieved metrics slightly better than those in Table A2 in Appendix A.

Figure 14 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. All the scores have improved slightly as compared to those in Figure 12.

Figure 14. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 1.0 pattern. Denoising is after CFA is generated and before demosaicing starts: (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 15 shows the demosaiced results of Image 1 and Image 8. The demosaiced images look much better than those images in Figure 11. However, it is hard to visually judge whether images in Figure 15 are of a better quality than those in Figure 13.

Figure 15. Visual comparison of three demosaicing results for CFA 1.0 at 10 dBs SNR (Poisson noise). Denoising is performed after CFA is generated and before demosaicing starts.

From the above studies, one can easily make two observations. First, denoising plays a very important role in enhancing the overall demosaicing performance in low lighting conditions. In terms of PSNR, the improvement exceeds 10 dBs. Second, denoising before demosaicing starts is more effective than after demosaicing. We can observe one to two dBs of performance gain in PSNR.

3.3.2. SNR at 20 dBs

One may argue that the noisy low lighting images at 10 dBs may be too extreme because people seldom take pictures without flash lights in such dark conditions. Now, we investigate the performance of CFA 1.0 in more realistic low lighting conditions of 20 dBs. Similar to Section 3.3.1, we also have three sub-cases.

• Case 1: No Denoising

We have 16 methods for demosaicing CFA 1.0. The F3 fusion method fuses the results of Demonet, ARI, and LDI, which are the best performing demosaicing methods. The ATMF fusion method uses the seven highest performing methods, which are Demonet, ARI, LDI, Bilinear, LT, MLRI, and SEM. Table A4 in Appendix B summarizes the PSNR scores, which is the average of the three individual PSNR scores for R, G, and B bands, the CIELAB scores, SSIM, HVS, and HVSm metrics. It should be noted that some methods (SFIM and HPM) did not perform well. Other methods achieved PSNR values of around 22 dBs.

Figure 16 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. There are some big variations in the metrics.

Figure 16. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 1.0 pattern without denoising. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 17 shows the demosaiced results of Image 1 and Image 8. The demosaiced images do not look good because of color distortion, noise, and contrast.

Figure 17. Visual comparison of three high performing demosaicing algorithms for CFA 1.0 at 20 dBs SNR (Poisson noise). No denoising.

In short, without denoising, all the demosaicing algorithms did not perform well at 20 dBs. That is, the demosaiced images have the same quality as the input CFAs.

• Case 2: Denoising after Demosaicing

Here, our focus is to investigate the demosaicing performance with help from the BM3D denoising algorithm, which is applied after demosaicing is completed.

The F3 fusion method fuses the results of Demonet, bilinear, and ARI, which were the best performing demosaicing methods in this case. The ATMF fusion method uses the seven highest performing methods, which are Demonet, bilinear, ARI, LDI, AP, LT, and MLRI. Table A5 in Appendix B summarizes the PSNR, CIELAB, SSIM, HVS, and HVSm metrics. One can see that all methods achieved PSNR values of around 22 dBs, which are 2 dBs higher than those values in the Table A4 in Appendix B. All the SSIM, CIELAB, HVS, and HVSm values have all been slightly improved over the no denoising case.

Figure 18 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. All the scores have improved slightly as compared to those in Figure 16.

Figure 18. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 1.0 pattern with denoising after demosaicing. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 19 shows the demosaiced results of Image 1 and Image 8. The demosaiced images look slightly better than the images in Figure 17. The artifacts are less noticeable after denoising.

Figure 19. Visual comparison of three demosaicing results for CFA 1.0 at 20 dBs SNR (Poisson noise). Denoising is performed after demosaicing.

• Case 3: Denoising before Demosaicing

Here, denoising was performed before demosaicing started. That is, BM3D was applied to the CFA patterns before feeding them into the demosaicing algorithms.

The F3 fusion method fuses the results of Demonet, DLMMSE, and AP, which were the best performing demosaicing methods in this case. The ATMF fusion method uses the seven highest performing methods, which are Demonet, DLMMSE, AP, LT, ARI, LDI, MLRI, and ECC. Table A6 in Appendix B summarizes the PSNR, CIELAB, SSIM, HVS, and HVSm metrics. One can see that all methods achieved metrics slightly better than those in Table A5 in Appendix B.

Figure 20 shows the averaged PSNR, CIELAB, SSIM, HVS, and HVSm scores of all the 16 methods. All the scores have improved slightly as compared to those in Figure 18.

Figure 20. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 1.0 pattern. Denoising is after CFA is generated and before demosaicing starts: (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 21 shows the demosaiced results of Image 1 and Image 8. The demosaiced images look much better than those images in Figure 17. However, it is hard to visually judge whether the images in Figure 15 are of better quality than those in Figure 19.

Figure 21. Visual comparison of three demosaicing results for CFA 1.0 at 20 dBs SNR (Poisson noise). Denoising is performed after CFA is generated and before demosaicing starts.

From the above studies, one can easily obtain two observations. First, denoising plays an important role in enhancing the overall demosaicing performance in low lighting conditions. In terms of PSNR, the improvement exceeds 2 dBs. Second, denoising before demosaicing starts is more effective than that of after demosaicing. We can observe one to two dBs of additional performance gain in PSNR if denoising is done before demosaicing.

3.4. CFA 2.0 Results

The objective of this section is to investigate the performance of CFA 2.0 in low lighting conditions. We have two SNR cases: 10 dBs and 20 dBs. Within each SNR case, we have three sub-cases, depending on whether denoising is applied or not.

3.4.1. SNR at 10 dBs

Here, we have three cases. The first case is about demosaicing the noisy images without any denoising. The second case deals with the scenario where the denoising is performed after demosaicing. The third case is to investigate the performance of denoising before demosaicing.

• Case 1: No Denoising

We have compared 15 methods for demosaicing CFA 2.0 pattern. Those methods are summarized in Section 2.2. The baseline refers to the bicubic interpolation of the reduced resolution color image shown in Figure 2. The F3 fusion method uses the three best performing methods, which are the Baseline, Standard, and GFPCA. ATMF uses the 7 best performing methods: Baseline, Standard, GFPCA, GSA, PCA, GS, and PRACS. From Table A7 in Appendix C, it can be seen that the averaged PSNR score of F3 yielded the best score, which is 21 dBs.

Figure 22 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them are reasonable. Figure 23 shows the demosaiced images of three methods. One can still see some noticeable artifacts.

Figure 22. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 2.0 pattern without denoising. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 23. Visual comparison of three high performing demosaicing algorithms for CFA 2.0 at 10 dBs SNR (Poisson noise). No denoising.

• Case 2: Denoising after Demosaicing

Here, denoising was applied after demosaicing. The F3 fusion method uses the three best performing methods, which were the Demonet+GFPCA, GFPCA, and LSLCD. ATMF uses the seven best performing methods: Demonet+GFPCA, GFPCA, LSLCD, Standard, PCA, GS, and PRACS. From Table A8 in Appendix C, it can be seen that the averaged PSNR score of LSLCD yielded the best score, which is more than 24 dBs. This is better than those numbers in Table A7 in Appendix C. The other metrics in Table A8 of Appendix C are all improved as well.

Figure 24 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them look much better than those in Figure 22.

Figure 24. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 2.0 pattern with denoising after demosaicing. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 25 shows the demosaiced images of three methods. It can be seen that the artifacts in Figure 23 have been reduced quite a lot. Visually speaking, F3 has minimal distortion for the fence area of Image 8.

Figure 25. Visual comparison of three demosaicing results for CFA 2.0 at 10 dBs SNR (Poisson noise). Denoising is performed after demosaicing.

• Case 3: Denoising before Demosaicing

Here, denoising was applied before demosaicing. That is, the BM3D algorithm was applied to the CFA patterns. Intuitively, denoising before demosaicing should perform better that that of after demosaicing. The F3 fusion method uses the three best performing methods, which were the Standard, Demonet + GFPCA, GSA. ATMF uses the seven best performing methods: Standard, Demonet + GFPCA, GSA, HCM, GLP, GS, and PRACS. From Table A9 in Appendix C, it can be seen that the averaged PSNR score of Demonet + GFPCA yielded the best score, which is more than 26 dBs. This is at least 2 dBs better than those numbers in Table A8 in Appendix C.

Figure 26 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them look much better than those in Figure 22 and slightly better than those in Figure 24.

Figure 26. Averaged performance metrics for all the low light images at 10 dBs SNR (Poisson noise) using CFA 2.0 pattern. Denoising is after CFA is generated and before demosaicing starts: (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 27 shows the demosaiced images of three methods, not necessarily the best performing methods. It is difficult to judge whether or not the demosaiced images in Figure 27 is better than that of Figure 25.

Figure 27. Visual comparison of three demosaicing results for CFA 2.0 at 10 dBs SNR (Poisson noise). Denoising is performed after CFA is generated and before demosaicing starts.

3.4.2. SNR at 20 dBs

There are three case studies here.

• Case 1: No Denoising

There are 15 methods. The F3 fusion method uses the three best performing methods, which were the Baseline, Standard, and GFPCA. ATMF uses the seven best performing methods: Baseline, Standard, GFPCA, GSA, GS, PRACS, and LSLCD. From Table A10 in Appendix D, it can be seen that the averaged PSNR score of F3 yielded the best score, which is slightly above 20 dBs. The other metrics are mediocre.

Figure 28 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them can be considered reasonable as demosaicing methods do not have denoising capability in general. Figure 29 shows the demosaiced images of three methods. One can see some artifacts.

Figure 28. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 2.0 pattern without denoising. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 29. Visual comparison of three high performing demosaicing algorithms for CFA 2.0 at 20 dBs SNR (Poisson noise). No denoising.

• Case 2: Denoising after Demosaicing

Here, denoising was applied after demosaicing. The F3 fusion method uses the three best performing methods, which were the Demonet + GFPCA, GFPCA, and PRACS. ATMF uses the seven best performing methods: Demonet + GFPCA, GFPCA, PRACS, GSA, PCA, GS, and LSLCD. From Table A11 in Appendix D, it can be seen that the averaged PSNR score of LSLCD yielded the best score, which is 24.391 dBs. This is better than most of the PSNR numbers in Table A10 in Appendix D, but only slightly better than the LSLCD method (24.05 dBs) in Table A8 of Appendix C (10 dBs SNR case). This means denoising has more impact for low SNR cases than high SNR cases.

Figure 30 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them look better than those in Figure 28.

Figure 30. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 2.0 pattern with denoising after demosaicing. (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 31 shows the demosaiced images of three methods. It can be seen that the artifacts in Figure 29 have been reduced. However, some artifacts are still very noticeable, especially the color distortions. This means there is still room for further improvement in the future.

Figure 31. Visual comparison of three demosaicing results for CFA 2.0 at 20 dBs SNR (Poisson noise). Denoising is performed after demosaicing.

• Case 3: Denoising before Demosaicing

Here, denoising was applied before demosaicing. That is, the BM3D algorithm was applied to the CFA patterns. The F3 fusion method uses the three best performing methods, which were the Standard, GSA, and HCM. ATMF uses the seven best performing methods: Standard, GSA, HCM, GLP, GS, and HPM. From Table A12 in Appendix D, it can be seen that the averaged PSNR score of GSA yielded the best score, which is 28.172 dBs. This is 4 dBs better than those numbers in Table A10 in Appendix D, and 2 dBs better than the best method in Table A11 in Appendix D.

Figure 32 shows the average performance metrics of PSNR, CIELAB, SSIM, HVS, and HVSm. All of them look better than those in Figure 28 and slightly better than those in Figure 30.

Figure 32. Averaged performance metrics for all the low light images at 20 dBs SNR (Poisson noise) using CFA 2.0 pattern. Denoising is after CFA is generated and before demosaicing starts: (a) PNSR; (b) CIELAB; (c) SSIM; (d) HVS and HVSm.

Figure 33 shows the demosaiced images of three methods, not necessarily the best performing methods. It is difficult to judge whether or not the demosaiced images in Figure 33 are better than those of Figure 31 because some color distortions are still present.

Figure 33. Visual comparison of three demosaicing results for CFA 2.0 at 20 dBs SNR (Poisson noise). Denoising is performed after CFA is generated and before demosaicing starts.

3.5. Best Against the Best Comparison Among the Two CFA Patterns

Now, we would like to compare the two CFA patterns. Since different algorithms were used in each CFA, we think that an appropriate way to compare the different CFAs is to compare the best against the best. That is, for each CFA, we select the best performing method and compare its results with the best performing methods in the other CFA.

We have two case studies below: 10 dB SNR and 20 dB SNR. For each SNR, we have three sub-cases: no denoising, denoising after demosaicing, and denoising before demosaicing.

3.5.1. 10 dBs SNR

Table 2 and Figure 34 summarize the average performance metrics for the 10 dBs SNR case in our earlier studies in Section 3.2 and Section 3.3 In Table 2, the name of the best performing algorithm is also included in each cell alongside the metrics. From Table 2 and Figure 34, we have the following observations:

Table 2. Comparison of CFA patterns for the various demosaicing cases at 10 dBs SNR. Bold numbers indicate the high performance CFA pattern.

Metrics	CFA	No Denoising/Best Algorithm	Denoising After Demosaicing/Best Algorithm	Denoising Before Demosaicing/Best Algorithm
PSNR (dB)	1.0	16.889/F3	20.826/F3	21.978/F3
	2.0	21.249/F3	24.050/LSLCD	26.141/Demonet+GFPCA
CIELAB	1.0	10.149/GFPCA	6.664/F3	6.545/Demonet
	2.0	6.354/GFPCA	5.516/F3	4.310/Demonet+GFPCA
SSIM	1.0	0.455/F3	0.476/ATMF	0.463/ATMF
	2.0	0.451/ATMF	0.459/LSLCD	0.467/Standard
HVS (dB)	1.0	12.285/SEM	16.229/F3	16.833/ARI
	2.0	16.531/F3	19.056/LSLCD	22.053/Demonet+GFPCA
HVSm (dB)	1.0	12.403/SEM	16.494/F3	17.116/ARI
	2.0	16.868/F3	19.568/LSLCD	23.121/Demonet+GFPCA

Figure 34. Best against the best comparison between CFA 1.0 and CFA 2.0 with and without denoising at 10 dBs SNR. (a) PSNR; (b) Cielab; (c) SSIM; (d) HVS; (e) HVSm.

• In the no denoising case, CFA 2.0 is indeed better than CFA 1.0. For instance, the PSNR gain in Figure 34a is more than 4 dBs, which is significant;
• Denoising definitely improves the demosaicing performance, regardless of where the denoising is done. For CFA 1.0, the improvement over no denoising is about 4 dBs; for CFA 2.0, the improvement is more than 3 dBs in terms of PSNR. For other metrics in Figure 34b–e, we also observe big improvements;
• Denoising before demosaicing has a better performance than that of denoising after demosaicing. For CFA 1.0, the improvement is 1.1 dBs and, for CFA 2.0, the improvement is 2.1 dBs in PSNR.

3.5.2. 20 dBs SNR

Table 3 and Figure 35 summarize the best against the best results for different CFAs under different denoising/demosaicing scenarios. We have the following observations:

Table 3. Comparison of CFA patterns for the various demosaicing cases at 20 dBs SNR. Bold numbers indicate the high performance CFA pattern.

Metrics	CFA	No Denoising/Best Algorithm	Denoising After Demosaicing/Best Algorithm	Denoising Before Demosaicing/Best Algorithm
PSNR (dB)	1.0	20.488/ATMF	22.821/F3	24.059/Bilinear
	2.0	23.290/F3	24.391/GSA	28.172/LSLCD
CIELAB	1.0	6.713/Demonet	5.256/Demonet	4.935/Demonet
	2.0	5.121/GFPCA	5.268/LSLCD	3.584/F3
SSIM	1.0	0.500/SEM	0.548/F3	0.574/F3
	2.0	0.545/Demonet+GFPCA	0.535/LSLCD	0.539/GSA
HVS (dB)	1.0	16.130/Demonet	18.204/Bilinear	19.142/Demonet
	2.0	18.646/F3	19.415/LSLCD	24.382/ATMF
HVSm (dB)	1.0	16.365/Demonet	18.734/Bilinear	19.444/ARI
	2.0	19.112/F3	19.881/LSLCD	25.516/ATMF

Figure 35. Best against the best comparison between CFA 1.0 and CFA 2.0 with and without denoising at 20-dB SNR. (a) PSNR; (b) Cielab; (c) SSIM; (d) HVS; (e) HVSm.

• In the no-denoising case, CFA 2.0 is 2.8 dBs better than CFA 1.0 in terms of PSNR (Figure 35a). Other metrics in Figure 35b–e also improved quite significantly;
• Denoising definitely helps the demosaicing performance, regardless of where the denoising is done. For CFA 1.0, the improvement is over 2 to 3.5 dBs; for CFA 2.0, the improvement is more than 1.1 to 4.8 dBs in terms of PSNR. There are also big improvements in other metrics (Figure 35b–e);
• Denoising before demosaicing has a better performance than that of denoising after demosaicing. For CFA 1.0, the improvement is 1.2 dBs and, for CFA 2.0, the improvement is close to 4 dBs in PSNR;
• Denoising helps the demosaicing performance more when the SNR is low. More than 4 dBs of gain in PSNR were observed after denoising in the 10 dBs SNR case;

3.5.3. Discussions

Here, we provide some qualitative analyses/explanations for some of those important findings in Section 3.5.1 and Section 3.5.2:

• Why denoising before demosaicing is better that that of after demosaicing:
  One intuitive explanation is that noise can be suppressed more effectively earlier rather than later. Once noise has propagated to subsequent steps in the processing pipeline, it is harder to suppress it because some steps in the demosaicing process may be nonlinear. For example, in deep learning approaches, some rectified linear units (ReLu) are inherently nonlinear. This intuition has been found to be valid in our past research on active noise suppression in noisy conditions, as well. For a NASA project on noise suppression in Space Station [66,67], we noticed that noise was suppressed more effectively near the source than farther away from the source, as there are more reflections in the far-field due to multipath propagations;
• Why CFA 2.0 is better than CFA 1.0 in low lighting conditions:
  We believe a concrete theory is needed to explain why CFA 2.0 has better performance than CFA 1.0 and this could be a good future research topic. The inventors of CFA 2.0 also did not provide a theory behind this. Intuitively, we agree with the inventors of CFA 2.0 that this must have something to do with the amount of white pixels in CFA 2.0. According to the inventors of CFA 2.0, more white pixels improve the sensitivity of the imager. We offer another analysis below.
  We use the bird image at 10 dBs condition (Image 1 in Figure 6 of our paper) as a case study. There is no denoising in the demosaicing process. Figure 36 below contains two histograms of the residual images (residual = reference − demosaiced) for CFAs 1.0 and 2.0. From this figure, it can be seen that the histogram of CFA 2.0 is centered near zero, whereas the histogram of CFA 1.0 is biased towards the right, meaning that CFA 2.0 is more accurate (close to the ground truth), because of its better light sensitivity, than CFA 1.0;
  

  Figure 36. No denoising case at 10 dBs. Error distributions of the two CFAs.
• Why denoising helps slightly more for 10 dBs case than the 20 dBs case:
  From Table 2 and 3, we noticed that the gap between denoising improvement in 10 dBs and 20 dBs is slim. However, we still noticed that denoising helps the demosaicing performance slightly more in the 10 dBs case than in the 20 dBs case. We do not have a concrete theory behind this. However, one intuitive explanation can be found using Figure 37, which is a hypothetical optimization problem. The x-axis shows the computational load and the y-axis shows the performance. This curve shows that, for the same amount of effort, the improvement in performance is higher in the early stage than the later. In other words, it is difficult to further improve once the system is already in good shape. In economics, there is a law of diminishing returns, which might be related to the case here.
  

  Figure 37. A hypothetical optimization problem.
  Although there is no physical law governing this behavior, we have seen similar observations in some engineering applications. For example, in a past paper on speech recognition [68] under noisy conditions, we noticed that the word recognition rate improves more when the SNR is low. See Table 1 in [68]. From that table, at 0 dB, the relative improvement is 140%, as compared to only 37% in the 6 dBs case. This implies that it may be easier to see improvements when a system starts from a poor condition.

4. Conclusions

In this research, we thoroughly investigated the performance of CFA 1.0 and CFA 2.0 for low lighting images. The low lighting images were emulated by introducing Poisson noise. We then applied more than 15 conventional and deep learning based algorithms to CFAs 1.0 and 2.0 using a set of emulated images at 10 dBs and 20 dBs SNR. Using both objective (five performance metrics) and subjective evaluations, we observed that the demosacing performance of CFA 2.0 is indeed better than that of CFA 1.0 in low lighting conditions. We also investigated where denoising should be performed. In our research, we experimented with two denoising scenarios: before and after demosaicing. One important observation is that denoising before demosaicing has a better performance than denoising after demosaicing. We also observed that denoising boosts the demosaicing performance more when the SNR is 10 dBs, compared to an SNR of 20 dBs.

In this paper, we have used the BM3D denoising algorithm, which is proven algorithm in the literature. In the future, other denoising algorithms may be tried. Moreover, we are exploring the possibility of incorporating CFA 2.0 in NASA’s future planetary missions to Mars and other planets. Lastly, we plan to investigate a direct approach to demosaicing CFA 2.0 using deep learning.