Low-Light Image Brightening via Fusing Additional Virtual Images

Capturing high-quality images via mobile devices in low-light or backlighting conditions is very challenging. In this paper, a new, single image brightening algorithm is proposed to enhance an image captured in low-light conditions. Two virtual images with larger exposure times are generated to increase brightness and enhance fine details of the underexposed regions. In order to reduce the brightness change, the virtual images are generated via intensity mapping functions (IMFs) which are computed using available camera response functions (CRFs). To avoid possible color distortion in the virtual image due to one-to-many mapping, a least square minimization problem is formulated to determine brightening factors for all pixels in the underexposed regions. In addition, an edge-preserving smoothing technique is adopted to avoid noise in the underexposed regions from being amplified in the virtual images. The final brightened image is obtained by fusing the original image and two virtual images via a gradient domain guided image filtering (GGIF) based multiscale exposure fusion (MEF) with properly defined weights for all the images. Experimental results show that the relative brightness and color are preserved better by the proposed algorithm. The details in bright regions are also preserved well in the final image. The proposed algorithm is expected to be useful for computational photography on smart phones.


Introduction
Exposure time and ISO values are usually carefully tuned by professional photographers to capture well-posed images under various lighting conditions [1,2]. A pairing of short exposure time and a small ISO value can be adopted to reduce motion blur and overexposed regions from an image captured under low-light or backlighting conditions [3]. Since ghost removal is the Achilles' heel of high dynamic range (HDR) imaging, this method is also efficient to capture images for HDR scenes, especially in the presence of moving objects. However, the visibility of underexposed regions in the captured image is poor. It is necessary to increase the brightness of the underexposed regions in many applications such as tracking ships [4] and capturing CT images [5].
Directly brightening the low-light image is probably the most intuitive and simplest way to improve the visibility of dark regions. Unfortunately, such an intuitive way causes another problem, i.e., relatively bright regions might be saturated and thus lose corresponding details [6]. To address the problem, many single image brightening algorithms were proposed in the past decade. A lightness-order-error measure was proposed in [7] to achieve overenhancement and preserve natural illumination. Based on an observation that an inverted low-light image looks like a haze image, Dong et al. [8] designed an interesting single image brightening algorithm by dehazing the inverted low-light images. The algorithm The IMFs are used directly to brighten pixels that are not underexposed due to reliability of the IMFs for these pixels. However, color distortions will be brought if the IMFs are directly used to brighten a pixel with one underexposed color channel. To handle this problem, a linear method will be provided to brighten the pixel with at least one underexposed color channel. In other words, the brightened pixel is obtained by multiplying the pixel by a constant ratio. The constant ratio is determined by a least square minimization problem. It is worth noting that the constant ratio is more determined by reliable pixels in order to avoid the effects of underexposed pixels. Besides the color distortion, it is also necessary to avoid amplifying noise in the underexposed regions. A simple edge-preserving denoising method is provided to address the problem. The input image is decomposed into a base layer and a detail layer by using a weighted guided image filter (WGIF), and only the base layer is brightened [16]. (a) is a low-exposed image; (b) is obtained by low-lighting image enhancement (LIME) [10]; (c) is obtained by the inaccurate intensity mapping function (IMF). Clearly, brightness change is possible issues for the former. Color distortion and noise amplification are possible issues for the latter.
All the input image and two virtual images are fused via a state-of-the-art multiscale exposure fusion algorithm, as in [22], to produce the final image. Different pixels usually have different constant ratios. Besides the measurements of contrast, exposure level, and saturation in [23], one more new weighting function is introduced in the proposed algorithm. With the new weighting function, larger weights are given to all the pixels in bright regions of the input image so as to avoid them from being overexposed in the brightened image. Experimental results show that the proposed algorithm outperforms existing single image brightening algorithms. Due to the simplicity of the proposed algorithm, it can indeed be embedded into mobile devices such as an app in smart phones. In summary, three major contributions of this paper are as follows: 1. Propose a simple method to brighten a low-light or backlighting image by fusing additional virtual images; 2. Reduce the possible color distortion of the generated virtual image due to inaccurate IMF; 3. Present a simple method to alleviate the noise in virtual image.
The rest of this paper is organized as follows. Two virtual images are generated in Section 2. All the input image and virtual images are fused together to produce the final image via a multiscale exposure fusion algorithm, described in Section 3. Experimental results are provided in Section 4 to illustrate the efficiency of the proposed algorithm. Finally, the paper is concluded in Section 5.

Generation of Two Virtual Images
In this section, two virtual images are generated and both of them are brighter than the input image. To avoid the brightness change from appearing in the brightened images, the virtual images are generated by using the IMFs that are computed from the available CRFs as follows: The IMFs between input image and virtual images are computed via CRFs. The CRFs are generated using the algorithm in [20] by using a large number of differently exposed images.
Let the CRFs f l (·) (l = 1, 2, 3), Z 2 , and Z 3 be two brightened images from the input Z 1 . Their exposure times are ∆t 1 , ∆t 2 , and ∆t 3 , respectively. Without loss of generality, ∆t 3 is equal to 4∆t 2 and ∆t 2 is equal to 4∆t 1 . As shown in [21], there is brightness order reversal if the ratios among exposure times are too large. The IMFs between input image and two virtual images can be expressed as follows: where f −1 l (·) is the inverse function of the CRF f l (·).

Generation of Two Intermediate Brightened Images via IMFs
Let the input image be denoted as Z 1 and [Z 1,1 (p), Z 1,2 (p), Z 1,3 (p)] be a pixel for three color channels. Z 2 and Z 3 are two virtual images which are brightened from the input image Z 1 . Let the IMF from the c-th color channel of image Z 1 to the c-th color channel of image Z i be denoted as Λ 1,i,c (z). It can be easily verified that the function Λ 1,i,c (z) has the following feature: The function Λ 1,i,c (z) is a one-to-many mapping when the value of z is smaller than a threshold ξ L and a one-to-one mapping if the value of z is above the threshold [16]. Here, the value of ξ L depends on the quality of the camera. The value of ξ L is small for a high-quality camera.
Case 2: Some of Z 1,c (p)(1 ≤ c ≤ 3) is (are) less than ξ L . The IMFs are not reliable. The color distortion will be brought to the two virtual images if the IMFs are directly used to generate the brightened pixels as in the Case 1. Noise will also be amplified. A piecewise constant method will be first provided to brighten pixels in the underexposed regions of the input image, i.e., when the IMFs are not reliable. An edge-preserving decomposition method will then be provided to avoid noise from being amplified in the brightened images.

Brightening Pixels in Underexposed Regions
A brightening algorithm under the assumption of the linear IMFs generates a brightened pixel by multiplying a pixel to be brightened by a constant. Such a constant brightening algorithm can avoid the color distortion while it causes the brightness change. On the other hand, a brightening algorithm based on the general IMFs can avoid the brightness change while it causes the color distortion for pixels in underexposed regions of the input image. In this subsection, the advantages of both the algorithms are combined to design an algorithm to brighten the pixels in the underexposed region.
Let pixel Z 1 (p) be a pixel with a channel value less than ξ L . The pixel Z 1 (p) will be brightened as γ 1 Z 1 (p) and γ 2 Z 1 (p), respectively.
The values of γ i (i = 1, 2) are obtained by solving the following optimization problems: arg min where the functionw(z) is defined according to different reliability for different pixels. The lower the pixel value, the higher the probability of being affected by noise, which is more likely to bring color distortion in the enhanced image. Therefore, the confidence interval of the pixel with a low value is smaller than the confidence interval of the pixel with a high value. It is defined as where ξ U and ξ L are positive constants to determine the reliability of pixel. Their values are chosen by the quality of fused image. The function h(z) is defined as Consider the pixel [Z 1,1 (p), Z 1,2 (p), Z 1,3 (p)] T in the input image. It can be shown from Equation (2) that the brightness of the corresponding brightened pixel in the first virtual image is determined by [Λ 1,2,1 ((Z 1,1 (p)), Λ 1,2,2 (Z 1,2 (p)), Λ 1,2,3 (Z 1,3 (p))] T and in the second virtual image by [Λ 1,3,1 ((Z 1,1 (p)), Λ 1,3,2 (Z 1,2 (p)), Λ 1,3,3 (Z 1,3 (p))] T . In other words, the brightness is determined by the general IMF-based brightening method. As such, the brightness change can be avoided. To further avoid the color distortion, the brightened pixels are generated by using the linear IMF-based method, i.e., the corresponding brightened pixels are respectively. The resultant brightening algorithm is a piecewise-constant brightening algorithm.
Equation (2) is a classic least squares problem. The values of γ i (i = 1, 2) can be computed through the method of finding the extremum of partial differential equation. Consequently, the γ i (i = 1, 2) are computed as and the brightened pixels are given as Clearly, the values of γ i are usually different for different pixels. Therefore, the pixels in underexposed regions are brightened using a different method from that in [3,14]. As a result, both the brightness change and the color distortion are being avoided from appearing in the two brightened images.

Noise Reduction of Brightened Images
A dark pixel includes noise. If the pixel Z 1 (p) is directly brightened asγ i Z 1 (p)(i = 1, 2), the noise is amplified in the brightened pixels. In this subsection, a simple method is provided to address the problem. To reduce noise while preserving details, the pixel Z 1 (p) is decomposed as by using an edge-preserving smoothing filter, such as the WGIF in [24]. Here, Z 1,b (p) and Z 1,d (p) are the base layer and the detail layer of pixel Z 1 (p), respectively. The values ofγ i (i = 1, 2) are obtained by minimizing the following cost function: Similar to Equation (2), the values ofγ i (i = 1, 2) are computed as and the brightened pixels are then given as Since the noise is almost included in the detail layer, the noise is avoided from being amplified by the brightened method (10).

Weights of Three Images
The input image and two virtual images look like three differently exposed images, but they are different from a set of differently exposed images which is captured by a digital camera. Therefore, a different method is adopted to define weights for the three images.
As pointed out in the introduction, a possible issue for the single image brightening is that relatively bright regions might be overexposed in the final image because of the brightening. Clearly, this can be avoided if the details in the relatively bright regions in the final image are from the corresponding regions in the input image. To achieve this objective, two different functions are adopted to define the weights for the three images.
One function is used to determine amplification factors of all the pixels in the first image. To brighten the dark regions while preserving the details in the bright regions, the bright regions are assigned high weights in the low-light image. The weight is defined as where the function h 3 (z) is defined as and η L and η U are two constants. The other is to measure contrast, exposure level, and color saturation of all the pixels in the three images and it is defined the same as in [23]: where w c (Z i (p)), w s (Z i (p)) and w e (Z i (p)) measure contrast, color saturation, and exposure level of pixel Z i (p), respectively. Particularly, let Y i be the luminance component of the image Z i (i = 1, 2, 3). The weight of the pixel Z 1 (p) is given as and the weight of the pixel Z i (p)(i = 2, 3) is given as It can be shown in Equations (14) and (15) that the weight of a bright pixel in the input image is much larger than a bright pixel in the two virtual images. As such, the details in the relatively bright regions in the final image are usually from the corresponding regions in the input image. The relatively bright region in the input image is avoided from being overexposed in the final image.

Fusion of Three Images via a GGIF-Based MEF
To get the good fused image, in this subsection, the three images are fused via a gradient domain guided image filtering (GGIF) based on multiexposure images fusion (MEF) [22]. The weight in each level of pyramid is smoothed by the GGIF. To reduce the noise, the guided image is luminance component of input image.
Let W (l) i (i = 1, 2, 3) be weight in l-level Gaussian pyramid and GGIF be GG. The weight in l-level is computed as where Y i (l) is the l-level Gaussian pyramid of the luminance component of the image Z i . The l-level fused image is presented as .
The final fused image can be reconstructed from Z i (l) . Overall, the proposed single image brightening algorithm is summarized as Algorithm 1.

Algorithm 1: Single image brightening and the corresponding CRFs
Input: a low-light image Output: a brightened image Step 1 Compute IMFs from the available CRFs Step 2 Generate two virtual images with larger exposure times Case1 the value of pixel above 5 do Brighten the pixel via Equation (17) end Case2 the value of pixel below 5 do 1. Constructing weight matrix by Equation (2) 2. Decompose input image as base layer and detail layer using WGIF 3. Compute brighten constant γ i using Equation (8) 4. Brighten the pixel using Equation (9) end Step3 Fuse the input image and two virtual images to produce the final image using Equations (10)-(16).

Experimental Results
In this section, extensive experimental results are provided to verify the performance of the proposed algorithm. All images are captured by us using a Nikon D7200. It should be noted that the differently exposed images are also captured at the same time.

Generation of Virtual Images
In this subsection, the effect of exposure ratios of two virtual images on the quality of enhanced image is analyzed. For the same scene, four groups of enhanced images are generated by fusing virtual images with different exposure ratios. In addition, the quality of different exposure images is also discussed. The ground truths of enhanced image are nine real exposure images and the ground truth of virtual image is the real image with the same exposure time as virtual image. The SSIM (Structural Similarity) index maps are shown in Figure 2. It demonstrates that the higher the exposure ratio of the virtual image to the dark image, the lower the quality of the virtual image is. However, it is found from Figure 3 that the quality of the image quality improved as the virtual image exposure ratios increased. This is because the virtual image with a small exposure ratio is unable to enhance the details in underexposed regions. In the rest of this section, two virtual images are generated with exposure ratios of 4 and 16.

Difference Choices of Parameters
In this subsection, the suitable parameters are selected by a well-used image quality, i.e., the MEF-SSIM in [25]. The test images are three images of different static scenes captured by ourselves. It is worth noting that the exposure ratio of the input image and two virtual images is 1:4:16. Here, the two images with the same exposure times as those of virtual images are also captured for each input image.
When analyzing the influence of ξ U and ξ L , the MEF-SSIM is adopted to assess the quality of the virtual images with the largest exposure time. Specially, the reference image is a real image with the same of exposure time as the virtual image. The 14 pairs of (ξ U ,ξ L ) and the corresponding MEF-SSIM are shown in Figure 4. Due to the high quality of the camera, the MEF-SSIM of three images decline when ξ L is below 5, as demonstrated in Figure 4. During the determination of ξ L , the value of ξ U has little effect on the MEF-SSIM. In this paper, the parameters are set as ξ L = 5, ξ U = 60.   Then, the influence of different pairs of (η U ,η L ) is studied on the fused image. Specially, the reference images are three real, different exposure images with the same exposure times as the three images for fusion. The results of MEF-SSIM are shown in Figure 4. When the values of η L and η U fluctuate by a small amount, the result will not be affected too much, as shown in Figure 4. In this paper, their values are selected as η L = 128, η U = 160.

Comparison of the Proposed Algorithm with Existing Ones
In this subsection, we compare our algorithm with seven state-of-the-art image enhancement algorithms, including two data-driven algorithms [18,19] and five model-driven algorithms-LIME [10], NPE [7], LECARM [15], SNIE [13], and Dong [8]. All the test images are taken by the Nikon D7200.
Firstly, the possible color distortion of eight algorithms is analyzed. Since a different light source may introduce luminance change that can affect the overall fluctuation of RGB values in the enhanced image, instead of calculating the absolute difference of RGB value, the angular color between two color vectors respectively produced by enhanced image and ground truth is computed as in [26,27]. As the wavelengths of purple light and blue light are short, the color of image will be biased toward yellow or even red if the exposure time is small. So, the values of pixels on the color checker board are mostly around 128 in the ground truth. Without loss of generality, there is a color checker board in every captured image, and a different color of this board can be regarded as ground truth. The color distortion is expressed as where Z n is the RGB value regarded as the one of the color vectors generated by one enhanced image, and Z n gt is the corresponding color vector from the color checker board. It is worth noting that the color vector is averaged around a 3 × 3 block of the pixel. The 24 color errors of eight algorithms are shown in Figure 5. From the box plots, our median value is the smallest and the maximum value and the minimum value are also the smallest. This means that the 24 errors of our enhanced image follow a narrow distribution and the average is the smallest. It further implies that the proposed algorithm outperforms other seven algorithms in keeping color fidelity. The comparison of the enhanced color checker board is also shown in Figure 6. The figure indicates that the Dong [8] and NPE [7,19] suffer from serious color distortion in multiple color blocks. The LIME [10] makes the image bright enough, but the enhanced image looks too sharp. The LECARM [15] and SNIE [13,18] try reducing distortion at the expense of brightness, however, the first two find it difficult to distinguish the colors of the first three blocks of the last line. By contrast, our algorithm can effectively maintain the authenticity of colors.
Next, the contrast distortion is analyzed in terms of PCQI (Patch-Based Contrast Quality Index) [28], which can predict the overall and local contrast quality of enhanced images. The ground truth is generated by fusing nine different exposure images of the same scene by Kou algorithm [22]. These nine real, different exposure images are taken with a camera from low-exposure to high-exposure, one EV value for each image interval. The assessments of each enhanced algorithm are shown in Table 1. Our algorithm first ranks by contrast quality.   Figure 6. Brighten the color checker board. (a) The ground truth of color checker board, (b) is by using NPE [7], (c) is by using SNIE [13], (d) is by using LIME [10], (e) is by using Dong [8], (f) is by using LECARM [15], (g) is by using [18], (h) is by using [19], and (i) is by using our algorithms. The MEF-SSIM is adopted to further evaluate the performance of eight algorithms. Due to the lack of ground truth, the reference images for the MEF-SSIM are nine real, different exposure images. The nine images contain more information than one dark image, so it is fairer to use this evaluation strategy. The assessment is shown in Table 2. Clearly, our algorithm outperforms all the seven state-of-art algorithms on average. Besides the objective evaluation, partial enlargements of brighten images are presented in Figure 7 and the large images are also given in Figure 8. Clearly, all eight algorithms can indeed brighten low-light images. However, it can be shown from the second row of Figure 7 that the enhanced image by the LIME [10] looks too sharp. It is demonstrated in the first row of Figure 7 that the LECARM [15] and Dong [8] produce ring artifacts in the enhanced images. Furthermore, there are both color and lightness distortions in the enhanced images by the Dong [8], SNIE [13], and NPE [7,19], as illustrated in the last row of Figure 7. In Figure 8, the enhanced images by the LIME [10] are too sharp and there is color distortion in the first column. There is slight color distortion in the enhanced image by the SINE [13] and the brightness of the enhanced images needs to be improved. All the problems are overcome by the LECARM [15] and the proposed algorithm. Since the proposed algorithm is on top of the CRF, the color is preserved better. This implies that the CRF can be applied to improve the quality of enhanced images. shows results by using NPE [7], (c) shows results by using SNIE [13], (d) shows results by using LIME [10], (e) shows results by using Dong [8], (f) shows results by using LECARM [15], (g) is by using [18], (h) is by using [19], and (i) is by using our algorithms.

Efficiency of Noise Reduction via WGIF-Based Smoothing Technique
In this subsection, efficiency of the noise reduction is evaluated. When the pixel value is less than 5, the constant amplification will lead to noise amplification, as shown in Figure 9.
To reduce the amplified noise and preserve the detail, the base layer of the input image is amplified by Formula (9) while the detail layer is not amplified. As shown in the enlarged yellow block, the noise is indeed reduced in the dark regions, such as the black regions on the panda doll and the shadow of the branches. Moreover, the details of the branches and panda doll have not been weakened. It is worth noting that the BM3D was used in the LIME [10] to reduce the effect of noise. The complexity of the BM3D could be an issue. A maximum exposure ratio was defined in [15] to address the effect of noise. However, setting the maximum exposure ratio will lead to the decline of visibility in the very dark regions. It can even reduce visibility in other regions if it is not defined properly.
It should be pointed out that the WGIF-based smoothing could be omitted if a stack of images is captured. Frame averaging could be used to reduce noise in the dark regions by properly detecting moving regions. If all the corresponding raw images are available, it is not difficult to detect the moving regions.

Comparison of Running Time
The running time of these six algorithms is also compared by testing three image sizes on an AMD 2600X CPU. The results are presented as a bar graph, shown in Figure 10. It should be noted that the two data-driven algorithms are not compared because of the limitations of our hardware. It can be seen in the bar graph that the Dong is the fastest, and our algorithm ranks fourth. Compared with the LECARM and LIME, the running time of our algorithm decreases rapidly with the decrease of image size because the MEF in [22] calculates weights in every image scale. The percentages on the running time of different components in our algorithm is shown in the right. It can be found that the MEF in [22] takes up most of the time. Luckily, the MEF in [22] can be replaced by the simpler MEF in [14] with negligible loss of the MEF-SSIM. Figure 8. The brightened night images of eight algorithms. The first row shows the input images, the second row is performed using NPE [7], the third row using SNIE [13], the fourth row using LIME [10], the fifth row using Dong [8], the sixth row using LECARM [15], the seventh row using [18], the eighth row using [19], and the last row is by our algorithm. Figure 9. Results on using weighted guided image filter (WGIF)-based noise reduction when the value of a pixel is below 5, and the results of directly amplifying pixels whose value is below 5 by using constant. The first row shows no denoised images, and the second row shown denoised images. Compare the amplified red blocks in the images. The method can effectively reduce noise and preserve details.

Limitation of the Proposed Algorithm
The proposed algorithm outperforms the other five algorithms. However, the proposed algorithm assumes that the accurate CRFs are available. This is not an issue if the proposed algorithm is embedded in a model device. However, this assumption is not true for those images that are downloaded from the Internet. It is very difficult to estimate the accurate CRFs if there is only one image. If the CRFs are not accurate, the IMF between the input image and virtual images are not accurate too. In other words, the brightened image could be a little poor, such as the color distortion in Figure 11. This problem will be studied in our future research. Figure 11. Compare the brightened image (the second row of images is a supermarket in China) by using inaccurate camera response functions (CRFs) and accurate CRFs. The first column shows input images; the second column shows brightened images by using inaccurate CRFs; the last column shows brightened images by using accurate CRFs.

Conclusions and Future Remarks
In this paper, we proposed an effective single image brightening algorithm. The proposed algorithm is supposed to be embedded in mobile devices such as digital cameras and smart phones. Two virtually brightened images are generated via intensity mapping functions (IMFs) which are computed by the camera response functions (CRFs). The input image and two virtual images are fused to preserve the natural structure. Since the three images are different from the three differently exposed images that are captured by a camera directly, the weights of these images are specially designed so as to preserve details in bright regions of the input image in the final image. Experimental results show that the proposed algorithm outperforms existing algorithms. The proposed algorithm can be embedded in mobile devices such as digital cameras and smart phones as an app.
It should be pointed out that the CRFs are assumed to be available for the proposed algorithm. Unfortunately, this is not always true. As shown in [29,30], a learning-based method could be adopted to address this issue. The proposed algorithm also assumes that details in bright regions are captured well in the input image. It is also important to enhance details in the bright regions if this assumption is not true. The concept of local inverse tone mapping in [31] could be borrowed to study this problem, and the details could also be preserved using the idea in [32]. The proposed algorithm can be improved by using a data-driven approach. All these problems will be investigated in our future research.

Conflicts of Interest:
The authors declare no conflicts of interest.