Retinex-Based Relighting for Night Photography

Abstract

The lighting up of buildings is one form of entertainment that makes a city more colorful, and photographers sometimes change this lighting using photo-editing applications. This paper proposes a method for automatically performing such changes that follows the Retinex theory. Retinex theory indicates that the complex scenes caught by the human visual system are affected by surrounding colors, and Retinex-based image processing uses these characteristics to generate images. Our proposed method follows this approach. First, we propose a method for extracting a relighting saliency map using Retinex with edge-preserving filtering. Second, we propose a sampling method to specify the lighting area. Finally, we composite the additional light to match the human visual perception. Experimental results show that the proposed sampling method is successful in keeping the illuminated points in bright locations and equally spaced apart. In addition, the proposed various diffusion methods can enhance nighttime skyline photographs with various expressions. Finally, we can add in a new light by considering Retinex theory to represent the perceptual color.

1. Introduction

Lighting up buildings is a form of entertainment that adds color to cities. This lighting can be physically changed, but it is not easy to change the lighting on a building after taking a photograph of it. After the image is captured, photographers can change the lighting of the photo by using photo-editing applications, such as Adobe Photoshop. However, creating illuminated photography manually is a time-consuming and challenging task.

In this paper, we automatically generate and change the illuminated light in an image based on the Retinex theory [1,2,3], which is based on the human visual system (HVS). Generally, colors in images are determined by interactions between lights and illuminated surfaces. This interaction depends on the reflectivity and shape of the subject. Therefore, we should analyze this information to create natural lighting; however, understanding this information from photographs alone is a complex problem.

Retinex theory shows that HVS does not perceive absolute brightness but rather relative brightness, capturing the brightness perceived simultaneously with the ambient brightness. There are various multimedia applications based on this relation: low-light enhancement [4,5,6,7,8,9,10], high-dynamic range imaging [11], haze removal [12,13], underwater image enhancement [14], shadow removal [15], denoising [16], medical imaging [17,18], and face recognition [19].

Low-light enhancement is the application most related to the proposed method. For the enhancement of low-light images, one method is to emphasize very dark areas by accurately separating the illumination from the reflected light. Because of the multiplicative nature of the illumination and reflectance relationship, poor estimation accuracy for one will result in worse accuracy for the other. Recent approaches [5,6,7,8,20] solve this problem using complex solutions. While early studies for low-light enhancement are computationally lightweight, the reflectance component is directly separated from the illumination and regarded as the enhanced output, which often causes over-enhancement [4]. Current algorithms [5,6,7,8,20] estimate the illumination and reflectance components by adding the prior constraints, and then they obtain the enhanced low-light image by multiplying the reflectance component with the adjusted illumination component.

Our research target is relighting nighttime skyline shots as a new multimedia application using Retinex theory. The difference in our approach from low-light enhancement is the dynamic range required for the final image. Low-light enhancement requires a wider dynamic range to make dark areas visible, while our application only requires a narrow dynamic range since only bright areas will change illumination, not dark areas, limiting the number of re-lit regions. Thus, the transformation of illuminated intensity can be restricted to a linear transformation instead of a non-linear one. The final image can be directly transformed with the linear transformation from the input image and illuminating the color map in the Lab color space. The additional illuminating color map in the final image is artificial; thus, the map does not require an estimation of the original physical illumination. Our method estimates only perceptual reflectance, which indicates the additional illuminating location. Only a rough shape, such as an edge, is needed to estimate reflectance; thus, our approach is lightweight and does not cause significant degradation.

Our approach has three steps. First, we generate the reflectance field using single-scale Retinex (SSR) [21], and then the reflectance is convoluted by min-filtering and joint bilateral filtering [22,23]. All filters can be a O(1) constant-time property for each filtering radius: min-filtering [24], Gaussian filtering [25], and joint bilateral filtering [26]. Recent low-light enhancement approaches [5,6,7,8,20] assume that an inverted low-light image looks like a hazy image, and the inverted image can be enhanced via the haze-removal approach [27]. With this flow, they use min-filtering and guided image filtering [28,29] as an edge-preserving filter as an analogy to the proposed method. Second, we generate several light source points on the reflectance field based on a probabilistic model with blue noise sampling, which is often used in computer graphics [30]. Next, we generate light shapes based on various spread functions with or without edge-preserving factors. Finally, we relight photographs using the input images and generated illuminating factors.  Figure 1 presents an overview of the proposed flows. The main contributions are as follows:

• We propose a new Retinex-based application of relighting night photography, i.e., changing illumination based on the HVS.
• We propose a light source sampling method based on blue noise sampling, which is used in computer graphics. In addition, we propose various light shape generation methods: a point spread function with or without an edge-preserving factor and a hyperbola function to produce a searchlight effect (i.e., a directional light to illuminate a certain location).
• We propose a method with a constant time property for the convolution radius. The independency is realized by using constant time filters in all processes in our method.

Figure 1. Overview of the proposed method.

Figure 1. Overview of the proposed method.

2. Retinex Theory

2.1. Overview

Land introduced the Retinex theory in 1964 to model how the HVS perceives a scene [1]. The HVS does not perceive absolute brightness but rather relative brightness. The fundamental observation is the insensitivity of human visual perception to a slowly varying illumination in a Mondrian-like scene. Recent reviews are presented in [31,32]. However, the goal has since shifted toward reflectance–illumination decomposition, which is picked up from the Retinex model.

There are various approaches to the decomposition problem of illumination and reflectance: threshold-based (PDE) Retinex [33,34], reset-based (random walk) Retinex [35], center-surround Retinex [3,21,36,37], and variational Retinex [31,38,39], and more recently, convolutional neural networks used for Retinex [40,41]. In another context, this separation is called intrinsic image decomposition [42,43,44,45], where the intrinsic image is the reflectance image in the Retinex.

The Retinex model involves the idea that the captured image $\mathit{f}$ is the product of an illumination map $\mathit{L}$ and a reflectance map $\mathit{R}$ of the object [46]. It is defined as follows:

$$\mathit{f}=\mathit{L}\odot \mathit{R},$$

(1)

where the operator ⊙ indicates the Hadamard product. The reflectance $\mathit{R}$ is the intrinsic feature of the observed scene, which is not affected by the illumination component $\mathit{L}$ as the extrinsic property. We can measure $\mathit{f}$ but not directly capture $\mathit{L}$ and $\mathit{R}$. All symbols used in this paper are reviewed in Appendix A.

2.2. Single-Scale Retinex

Land introduced the center/surround Retinex [3]. It is based on the idea that brightness is calculated as the ratio of the value of a pixel to the mean value of the surrounding points and that these surrounding points have a density proportional to the inverse of the squared distance. This operation is similar to high-pass filtering, and Jobson et al. applied it using the Gaussian convolution [21,36,37]. Here, we define one version of this filter called single-scale Retinex (SSR). Smoothing filters can compute the surrounding field on each pixel, such as the mean of the neighboring pixels for mean filtering.

Let a D-dimensional, B-tone image be $\mathit{f}:\mathcal{S}\mapsto \mathcal{R}$, where $\mathcal{S}\subset {\mathbb{R}}^D$ is the spatial domain, $\mathcal{R}\subset {[0,B-1]}^d$ is the range domain, and d is the color range dimension (generally, $D=2$, $B=256$, and $d=3$), respectively. The SSR output image, which is the reflectance, $\mathit{R}:\mathcal{S}\mapsto \mathcal{R}$, is defined by

$${\mathit{R}}_{\mathit{p}}=\frac{{\mathit{f}}_{\mathit{p}}}{g\ast {\mathit{f}}_{\mathit{p}}},$$

(2)

where $\mathit{p}\in \mathcal{S}$ is a pixel position. Equation (2) is based on the center-surrounding model of Retinex theory [21,36,37]. ${\mathit{f}}_{\mathit{p}}$ and ${\mathit{R}}_{\mathit{p}}$ are intensity vectors of the input and output images, respectively. The operator ∗ indicates a convolution, and here, we use Gaussian filtering (GF) as a traditional SSR representation defined by

$$g\ast {\mathit{f}}_{\mathit{p}}=\frac{{\sum }_{\mathit{q}\in {\mathsf{\Omega }}_{\mathit{p}}}exp\left(-\frac{{\parallel \mathit{p}-\mathit{q}\parallel }_2^2}{2{\sigma }^2}\right){\mathit{f}}_{\mathit{q}}}{{\sum }_{\mathit{q}\in {\mathsf{\Omega }}_{\mathit{p}}}exp\left(-\frac{{\parallel \mathit{p}-\mathit{q}\parallel }_2^2}{2{\sigma }^2}\right)},$$

(3)

where ${\mathsf{\Omega }}_{\mathit{p}}\subset \mathcal{S}$ is the set of the neighboring pixels of $\mathit{p}$ and $\sigma$ is a smoothing parameter for the Gaussian distribution. Equation (2) means that the algorithm satisfies color constancy.

In SSR, a higher $\sigma$ results in an output image with better color fidelity but a lower dynamic range. A smaller $\sigma$ leads to an enhanced dynamic range but greater color distortion. For better balancing, the multi-scale Retinex (MSR) is also introduced [36]; however, MSR requires complex parameter tuning [47]. The simpler SSR was used because there is no need to consider the color distortion of the reflectance in this paper, as will be discussed later.

3. Proposed Method

This paper proposes an algorithm that partially relights nighttime skyline photographs and determines the positions and shapes of light sources. The proposed method has four steps: (1) extracting an illuminating saliency map, (2) sampling lighting points from the saliency map as a 2D probability distribution, (3) diffusing the lighting points for generating omnidirectional/directional lighting area, and (4) relighting based on Retinex enhancement. The following sections describe the detailed information for each step.

3.1. Illuminating Saliency Map

For illuminating images, lighting source points should be in human–visual saliency areas. The well-known saliency map [48] indicates that the human eyes focus on distinctive areas in the natural scene. For relighting night photography, we assume that the light source points should be located in high-reflectance areas. Therefore, the saliency map for this relighting is based on the reflectance map.

A simple solution uses reflectance images directly. However, there are some half pixels due to blurring or object boundaries. When the new lighting point is in a mixed area, lights spread across both foreground and background objects. Therefore, we erode the reflectance image and diffuse it using edge-preserving filtering to avoid this problem.

First, we convert the image to grayscale and apply SSR to extract the salient light areas in the input image. Examples of input color and grayscale images, $\mathit{L}$, and $\mathit{R}$ from SSR are shown in  Figure 2,  Figure 3a, and  Figure 3b, respectively. To compute SSR, we use a sliding-DCT-based convolution [25,49,50,51], which has a constant-time property per pixel for convolution. The FFT computational order is $O(logN)$ per pixel, where N is the number of image pixels, while that of sliding-DCT is $O\left(1\right)$. SSR is known to produce white shadows near the boundaries of objects, i.e., halo artifacts. The halo causes problems in the next step of dithering. Therefore, we erode the reflectance area to extract the more pronounced light component without the halo effect by min-convolution and iterative joint bilateral filtering (JBF) [22,23].

Next, we erode the reflectance image $\mathit{R}$, which is defined as follows:

$${\mathit{R}}_{\mathit{p}}^{min}=\underset{\mathit{q}\in {\mathsf{\Omega }}_{\mathit{p}}^{min}}{inf}{\mathit{R}}_{\mathit{q}},$$

(4)

where ${\mathsf{\Omega }}_{\mathit{p}}^{min}$ is a set for the convolution area, also called min-convolution. We use OpenCV’s implementation, which is $O\left(1\right)$ order per pixel and is based on recursive filtering [24].

After the min-convolution, we diffuse the processed reflectance map using an iterative version of JBF. JBF is a variant of bilateral filtering (BF) [52], and the difference between BF and JBF is only the guidance signal for measuring color distance. BF measures the color distance for kernel weight from the filtering target of the input signal, while JBF measures it from an additional guide signal. Let the guide signal be the input $\mathit{f}$ and the t-th iterative input and output signal be ${{\mathit{S}}_{\mathit{p}}}^t$ and ${{\mathit{S}}_{\mathit{p}}}^{t+1}$, respectively. Iterative JBF is defined by

$${{\mathit{S}}_{\mathit{p}}}^{t+1}=\frac{{\sum }_{\mathit{q}\in {\mathsf{\Omega }}_{\mathit{p}}^{\mathrm{jbf}}}{w}_s(\mathit{p},\mathit{q})w_r({\mathit{f}}_{\mathit{p}},{\mathit{f}}_{\mathit{q}}){\mathit{S}}_{\mathit{q}}^t}{{\sum }_{\mathit{q}\in {\mathsf{\Omega }}_{\mathit{p}}^{\mathrm{jbf}}}{w}_s(\mathit{p},\mathit{q})w_r({\mathit{f}}_{\mathit{p}},{\mathit{f}}_{\mathit{q}})},$$

(5)

where $S^1={\mathit{R}}^{min}$, $w_s:\mathcal{S}\times \mathcal{S}\mapsto \mathbb{R}$ is a spatial kernel, and $w_r:\mathcal{R}\times \mathcal{R}\mapsto \mathbb{R}$ is a range kernel. The kernels are based on the Gaussian distribution:

$$w_s(\mathit{p},\mathit{q})=exp\left(-\frac{{\parallel \mathit{q}-\mathit{p}\parallel }_2^2}{2{\sigma }_s^2}\right),w_r({\mathit{f}}_{\mathit{p}},{\mathit{f}}_{\mathit{q}})=exp\left(-\frac{\parallel {\mathit{f}}_{\mathit{q}}-{\mathit{f}}_{\mathit{p}}{\parallel }_2^2}{2{\sigma }_r^2}\right),$$

(6)

where ${\sigma }_s\in {\mathbb{R}}_{+}$ is a spatial scale and ${\sigma }_r\in {\mathbb{R}}_{+}$ is a range scale. We denote the final output as $\mathit{S}={\mathit{S}}^{t+1}$, where t is the number of JBF iterations, and we normalize $\mathit{S}$ from 0 to 1 for the next step. The resulting output is shown in Figure 3c. Note that we accelerate JBF based on the constant-time BF [26,53] with the constant-time GF. GF, min-convolution, and JBF are constant for the filtering radius. That is, all processing is a constant time for each filtering radius.

Figure 4a, which is multiplied by SSR as a reflectance image (Figure 3b) and uses the GF image as an illuminance image (Figure 3a), is completely the same as  Figure 2 in grayscale. In comparison, Figure 4b, which is multiplied by $\mathit{R}$ of the proposed method (Figure 3c) and using the GF image (Figure 3c), is not the same as the input image, as shown in  Figure 2. However, this is not a problem since this $\mathit{R}$ is not used for the final output.

3.2. Lighting Points Sampling

We select N light source points based on $\mathit{S}$ as a 2D probability distribution, where the normalized saliency map is ${\mathit{S}}_{\mathit{p}}\mapsto {[0,1]}^1$. In addition, the samples must have a greater distance from each other, called blue noise sampling [30]. This paper introduces an efficient sampling method from $\mathit{S}$ with dithering and blue noise sampling.

First, we use dithering $\mathit{S}$ to extract the candidates of samples. Dithering is a blue noise sampling method [54,55]; however, it cannot control the number of samples. To change the number of samples, intensity remapping for dithering has been proposed [56]. Ideally, in dithering, the number of samples is the summation of the probability; $N=\sum \mathit{S}$. Thus, a remapping function for S can control the number of samples. Lou et al. [56] used the clipping linear function $U(h,v)=min(hx,1)$. Given the desired number of samples $N_d$, the input $\mathit{S}$ is remapped and dithered to obtain the appropriate samples by setting the most suitable v that satisfies equation $N_d=\sum U(\mathit{S},v)$. The detailed information is shown in [56]. Note that the number of sampling points $N_d$ is not the final number. We regard the dithering samples as candidates’ points because dithering does not have randomness, i.e., fixed samples. Therefore, we select the final sampling points from the dithered samples, i.e., oversampled dithering samples. An example of the dithering output is shown in  Figure 5a.

After performing remapped dithering, the oversampled points $N_d$ are randomly resampled based on blue noise sampling to reduce the number of the last samples $N_l$. This task is equivalent to selecting light sources from $N_d$. In the selection process, the distance between the points should be as far apart as possible to illuminate a wide area.

The resampling is performed according to  Figure 6 and Algorithm 1. First, we select a point randomly at lines 5 and 6 in Algorithm 1. Next, we find a point that has a distance within the range $r_l$ and maximum distance $\gamma r_l$ from the already sampled points while maximizing the remaining points that can be sampled as much as possible, which is shown in line 7 to line 23. Then, we iterate the sampling process. If the condition is not met, we retry the process by reducing $r_l$ to make it easier to meet the condition that is represented from lines 24 to 26. After processing, we can select the light source point set of $\mathbb{A}$ from the dithering point set $\mathbb{D}$, where the number of samples is $N_l=\left|\mathbb{D}\right|$.  Figure 5 shows examples of the proposed sampling method. All points are sampled at a certain distance and at approximately equal intervals.

Algorithm 1: Sample light source from dithering points

3.3. Diffusing Lighting Source Points

To define the illuminating area from the sampling points, we propose two types of diffusion: omnidirectional and directional. Omnidirectional diffusion is based on the point spread function (PSF) of filtering. Directional diffusion uses a rotated hyperbolic function to represent searchlight-like lighting, a cropped set of an omnidirectional PSF. The two types of diffusion are used according to how we want to render the image.

Let a PSF for the n-th sampling point be $P^n:\mathcal{S}\mapsto {[0,1]}^1$, i.e., normalized 0 to 1 in grayscale. The illuminating color map $\mathit{F}:\mathcal{S}\mapsto {[0,B-1]}^3(B=256)$ is the product sum of each PSF and user-specific color vectors and is defined as follows:

$${\mathit{F}}_{\mathit{p}}=\sum _{n\in \mathbb{A}}{P}_{\mathit{p}}^n{\mathit{C}}^n,$$

(7)

where $\mathbb{A}$ is the set of light source points and ${\mathit{C}}^n\in {\mathcal{R}}^3$ is a user-determined color vector (3 channels) for the n-th sampling point, e.g., red $[255,0,0]$, green $[0,255,0]$, and blue $[0,0,255]$. The following describes PSF generation methods for omnidirectional and directional diffusion.

3.3.1. Omnidirectional Diffusion

First, we explain omnidirectional diffusion. A primary method of omnidirectional diffusion uses the PSF of Gaussian filtering (GF). This kernel spreads light evenly in all directions. However, PSF of GF is not a sufficient method because it produces light outside of the target object. Therefore, we additionally use the PSF of edge-preserving filters: JBF, domain transform filtering (DTF) [57], and adaptive manifolds filtering (AMF) [58]. The PSF of edge-preserving filters depends not only on a spatial kernel, such as GF, but also on a range kernel. The kernel of joint edge-preserving filtering using the input image as a guide image can suppress light diffusion at the object boundary of the reflectance guide.

For the range kernel, JBF uses a Gaussian distribution with the Euclidean distance defined in Equation (6). The DTF range kernel uses the geodesic distance of image edges instead of the Euclidean distance, and the AMF range kernel uses the mixed Euclidean–geodetic distance. The core ideas are defined as follows:

$$w_r^{\mathrm{DTF}}({\mathit{f}}_{\mathit{p}},{\mathit{f}}_{\mathit{q}})=exp\left(-\frac{\parallel {\mathit{f}}_{\mathit{q}}-{\mathit{f}}_{\mathit{p}}{\parallel }_{\mathrm{geo}}^2}{2{\sigma }_r^2}\right),w_r^{\mathrm{AMF}}({\mathit{f}}_{\mathit{p}},{\mathit{f}}_{\mathit{q}})=exp\left(-\frac{\parallel {\mathit{f}}_{\mathit{q}}-{\mathit{f}}_{\mathit{p}}{\parallel }_{\mathrm{mix}}^2}{2{\sigma }_r^2}\right),$$

(8)

where ${\parallel ·\parallel }_{\mathrm{geo}}$ indicates geodesic distance and ${\parallel ·\parallel }_{\mathrm{mix}}$ indicates the mixed Euclidean–geodetic distance. The geodesic distance considers changes in the value along the path. The definition is as follows:

$$\parallel {\mathit{f}}_{\mathit{p}}-f_{\mathit{q}}{\parallel }_{\mathrm{geo}}={\int }_{\mathit{p}}^{\mathit{q}}\left|{\mathit{f}}_o^{\prime }\right|do,$$

(9)

where ${\mathit{f}}_o^{\prime }$ is a differential value along the path from $\mathit{p}$ to $\mathit{q}$ and $|·|$ is the absolute function. The mixed Euclidean–geodetic distance is the blended value between Euclidean and geodetic distances:

$$\parallel {\mathit{f}}_{\mathit{p}}-{\mathit{f}}_{\mathit{q}}{\parallel }_{\mathrm{mix}}=\mathrm{blend}(\parallel {\mathit{f}}_{\mathit{p}}-{\mathit{f}}_{\mathit{q}}{\parallel }_{\mathrm{geo}},\parallel {\mathit{f}}_{\mathit{p}}-{\mathit{f}}_{\mathit{q}}\parallel )$$

(10)

where $\mathrm{blend}$ is a blending function. The actual response is different from these functions because of the approximation processes: separable approximation and range value approximation. See the papers [57,58] for the actual definitions.

3.3.2. Directional Diffusion

The diffusion by GF or JBF can be realized by filtering multiple source points as input images or convoluting a kernel point by point. Since the number of sample points is tiny relative to the input image, the latter definition runs faster than the former methods. In the case of DTF and AMF, these kernels cannot be defined directly; thus, they can only be generated by convolving the sample points as an image. However, both are efficient filters that run in constant-time relative to the kernel radius.

Next, we introduce directional diffusion. The directionality can be created by cropping the region of PSF with a rotational quadratic function. The internal set in the rotational quadratic function $Q_{\mathit{p}}$ is defined by

$$Q_{\mathit{p}}=\left\{(i,j)\mid \sqrt{{(i-x)}^2+{(j-y)}^2}\le \frac{{\parallel ai+bj+c\parallel }_2}{\sqrt{a^2+b^2}}\right\},$$

(11)

where $(i,j)\in \mathcal{S}$ and $\mathit{p}=(x,y)\in \mathcal{S}$ are the s PSF pixel position and a sampling point, respectively. In this case, the sampling point is a focusing point, and the line, $ai+bj+c=0$, is a directrix of the quadratic function. A result of applying Equation (11) to a white image is shown in Figure 7. The white area in Figure 7 is the set of $Q_{\mathit{p}}$.

Finally, directional light is generated based on any PSF by cropping the parabola function:

$$P_{\mathit{p}}^{{}^{\prime }n}=\left\{\begin{array}{cc}{\mathrm{P}}_{\mathit{p}}^n\hfill & \exists \mathit{p}\in Q_{\mathit{p}}\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

Since it is a cropping function, any PSF used in (7) is possible: GF, BF, or any other convolutional PSF.

3.4. Retinex-Based Image Relighting

The output image $\mathit{J}:\mathcal{S}\mapsto \mathcal{R}$ consists of luminance and reflectance factors, and our retouching aims to change the image’s illumination; thus, we change the luminance $\mathit{L}$ by a luminance remapping function M while keeping the reflectance $\mathit{R}$:

$$\mathit{J}=M\left(\mathit{L}\right)\odot \mathit{R},\mathrm{where}M\left(\mathit{L}\right)=(l+k\mathit{F})\odot \mathit{L}.$$

(13)

The function M is based on the illuminated map $\mathit{F}$ computed in the previous section. Equation (13) shows that the lighting map boosts the luminance $\mathit{F}$ by multiplying a constant scalar factor k, linearly darkening the factor $l\le 1$. When $l=0.5$, the default lightness is half. Plugging in the Retinex relation of Equation (1), we can simplify Equation (13) as follows:

$$\mathit{J}=(l+k\mathit{F})\odot \mathit{L}\odot \mathit{R}=(l+k\mathit{F})\odot \mathit{L}\odot \frac{\mathit{f}}{\mathit{L}}=(l+k\mathit{F})\odot \mathit{f}.$$

(14)

In our condition, we assume that illumination signals change linear properties; thus, the illumination component $\mathit{L}$ can be removed in the final representation since the illumination components of $\mathit{L}$ are canceled, i.e., there is no illumination $\mathit{L}$ and reflectance $\mathit{R}$ in (14). If we change $\mathit{L}$ using a non-linear function, e.g., Gamma correction, the luminance component is required to recover the final image $\mathit{J}$. In this case, we should consider the color distortion problem [4], as in low-light enhancement.

The parameters l and k have essential roles in Retinex theory. Based on Retinex theory, human perception is affected by the surrounding colors. For example, in Figure 8, the two left gray backgrounds and the two right blue backgrounds, the RGB of the orange on the left side of each pair is $(231,153,99)$ and that of the orange on the right side is $(207,125,70)$. In the images with blue backgrounds, the actual brightness of the oranges on the left and right sides is different, but they appear to be the same color. On the other hand, on the gray backgrounds, the oranges appear to have a greater difference in brightness than on the blue backgrounds, making it difficult to see the same color. This effect plays an important role in Retinex-based image processing. The dynamic range of an output image on display is generally 0 to 255. Therefore, when light is added to the original image, for example, the green light may not appear green due to the upper limit of the dynamic range (Figure 9a,d). To eliminate this effect, we reduce the brightness of the surroundings to make them appear relatively brighter (Figure 9b,c,e). Figure 9b,e are darkened in the RGB color space. Figure 9b,e are darkened in the Lab color space. In this paper, we used the second method and controlled for it using l of Equation (14).

Our human senses are also affected by the colors of our surroundings. In other words, the surrounding bright colors affect the new color to be added. Figure 10 shows the simulated example of the effect. The target pink colors have the same RGB values; however, these are affected by surrounding colors, and thus, these have different values. For example, it looks orange when the target color on the right-most side is enclosed by yellow. Therefore, we control the color using the Lab color space.

Equation (14) is changed to handle the Lab color space:

$$\mathit{J}=(1+k\mathit{F})\mathrm{Lab}2\mathrm{RGB}\left(\mathit{l}{\mathit{f}}_{Lab}\right),$$

(15)

where ${\mathit{f}}_{Lab}$ is the $\mathit{f}$ in the Lab color space, and $\mathrm{Lab}2\mathrm{RGB}(·)$ is a function that reconverts the Lab color space to RGB color space. Note that the vector $\mathit{l}=\{l_l,l_a,l_b\}$ has a similar role as a scalar l in Equation (14). The vector is extended to control the Lab values separately. If the elements in $\mathit{l}$ for a and b channels components in the Lab color space are small, the output image has low color saturation.

Figure 11 shows the effect of surrounding colors by changing the parameter $\mathit{l}$. When $l_a$ and $l_b$ have low values, the color saturation values of the surrounding color are low. While these have high values, the color saturation values are high. The newly added light will be affected by the surrounding color situation and will change; thus, the user needs to make the desired changes.

4. Experimental Results

We performed two experiments: a comparison of the sampling methods and a visual comparison of the relighting results. We used eight images for the experiments, as shown in Figure 12.

4.1. Sampling Method Comparison

We verified the effectiveness of the sampling methods. In this experiment, we measured the probability that the sampled point was in the lighted object. We performed the following steps to determine whether light source points are inside or outside the lighted objects. First, we converted the input image into the 8-bit grayscale image and extracted its reflectance with thresholding by a threshold value T, where $T=50$ for excluding outliers such as holes, windows, and the sky. For each image, Gaussian noises (mean: 0, standard deviation: 5) were added, and we performed the experiment 100 times to generate randomness. We experimentally compared state-of-the-art Retinex algorithms (URetinex-Net [59] and LR3M [6]) and saliency map algorithms (OpenCV implementation and SaliencyELD [60]) as sampling distributions for dithering. In addition, we handle random sampling as a baseline. The random sampling results can be directly computed as a percentage of the number of thresholds satisfied. Otherwise, it finds the percentage of dithering points below the threshold value.

Table 1. Probability that the points are out of the object (8-bit luminance value threshold: 50). Probability of random sampling points is the percentage above or below the threshold. The other probabilities are the percentage of cases where the dithering points of the input image are above or below the threshold. The top score for each image is shown in boldface.

Image	a	b	c	d	e	f	g	h	Ave.
Random	0.8609	0.7173	0.9084	0.8896	0.8163	0.8566	0.8637	0.7697	0.8353
URetinex-Net [59]	0.5159	0.4322	0.8307	0.5318	0.5179	0.6167	0.6596	0.3889	0.5617
LR3M [6]	0.7539	0.488	0.8846	0.8482	0.6364	0.8077	0.8114	0.5941	0.7280
Saliency (OpenCV)	0.3831	0.4790	0.5980	0.2976	0.5483	0.5108	0.4966	0.5993	0.4891
SaliencyELD [60]	0.4230	0.2832	0.4301	0.1945	0.3101	0.4858	0.3074	0.3216	0.3445
Proposed	0.1618	0.3275	0.3727	0.1364	0.2494	0.2159	0.2556	0.2842	0.2504

shows the mean ratio. Random sampling tends to have outside points since the sampling is not affected by luminance. The latest Retinex algorithm not only selects bright areas well, but samples the dark areas well too because these algorithms are designed to enhance dark images; they are thus unsuitable for this application. On the other hand, the proposed method can suppress light outside the object by using min-filtering and JBF. The latest saliency algorithms work better than the competitive Retinex algorithms. The proposed method works best because the algorithm is designed for the proposed new application of night image relighting.

4.2. Visual Comparison

In this section, we visually compare each method. First, we compared with a light that spreads omnidirectionally from a point source using four different filters: GF (Figure 13a), JBF (Figure 13b), DTF (Figure 13c), and AMF (Figure 13d). Each filter diffuses a light source, and JBF, DTF, and AMF use the input image as a guided image. GF diffused regardless of the object boundary and even illuminated dark areas (Figure 13e). On the other hand, edge-preserving filters had weights depending on the difference in the guide image to prevent color leaking between different objects. JBF is characterized as being able to spread object awareness over a wide area (Figure 13f). In addition, JBF can also be spread out similar to GF regardless of the object by making the range smoothing parameter ${\sigma }_r$ infinitely large. DTF and AMF are characterized to illuminate the light more locally (Figure 13b,g,h). Each edge-preserving filter, JBF, DTF, and AMF, suppresses subnormal numbers to accelerate their filtering speeds [61], and the slower method of JBF is also accelerated by vectorization [62].

Next, we compared the directional light generation algorithms with the PSFs of GF and JBF using the SSR reflectance and the proposed reflectance. Figure 14 shows the diffusing lights and the rendering results. With the SSR reflectance, the sampling points tend to be located outside of the subject; thus, light leaks out of the object. The JBF PSF can reduce leakage to some extent, but it is challenging when completely outside the points. With the proposed reflectance, the sampling points concentrate on the subject, and the JBF PSF further concentrates the light within the subject. The other rendering results by the proposed reflectance, and JBF with changing color are shown in Figure 15 (omnidirectional) and Figure 16 (directional). The expanded images of Figure 15 and Figure 16 are shown in Figure 17.

5. Conclusions

In this paper, we proposed a relighting method for night photography in the form of a new Retinex-based application. We proposed a method for selecting points of relighting and synthesizing them using human perceptual characteristics according to Retinex theory. In addition, multiple diffusion methods were proposed to enable a variety of representations, such as omnidirectional diffusion and directional diffusion with and without various edge-preserving properties.