Color Constancy Based on Local Reflectance Differences

Abstract

Color constancy is used to determine the actual surface color of the scene affected by illumination so that the captured image is more in line with the characteristics of human perception. The well-known Gray-Edge hypothesis states that the average edge difference in a scene is achromatic. Inspired by the Gray-Edge hypothesis, we propose a new illumination estimation method. Specifically, after analyzing three public datasets containing rich illumination conditions and scenes, we found that the ratio of the global sum of reflectance differences to the global sum of locally normalized reflectance differences is achromatic. Based on this hypothesis, we also propose an accurate color constancy method. The method was tested on four test datasets containing various illumination conditions (three datasets in a single-light environment and one dataset in a multi-light environment). The results show that the proposed method outperforms the state-of-the-art color constancy methods. Furthermore, we propose a new framework that can incorporate current mainstream statistics-based color constancy methods (Gray-World, Max-RGB, Gray-Edge, etc.) into the proposed framework.

1. Introduction

Color constancy can ensure that the perception of object color is relatively stable under different illumination conditions, and it is a characteristic of the human color perception system [1,2,3]. For example, whether a piece of white paper is in the outdoor sunlight or in the dim candle light indoors, we can always restore its original white color in our minds. With the development of optics and material technology [4,5,6], the application of digital cameras is becoming more and more extensive. In the digital world, color constancy plays a vital role in areas such as object recognition and tracking, scene analysis, and image-based localization, etc. For example, in the field of autonomous driving, color constancy algorithms ensure that objects in a scene captured under different illumination conditions have the same appearance, thereby improving the robustness of target (pedestrian, vehicle, etc.) recognition and tracking.

Prevailing color constancy methods are mainly divided into the following two categories: learning-based methods and statistics-based methods. Learning-based color constancy methods include the following two categories: (1) Gamut mapping color constancy. (2) Learn a color constancy model from the training datasets. The gamut mapping [7,8] color constancy method is based on the property that humans can only observe a limited number of colors for a natural image under a given illumination. A typical color gamut is the set of all RGB values of a typical light source (typically a white light source). In RGB space, this canonical gamut proved to be a convex hull [7]. This approach computes the changes that transfer the recorded color gamut into the canonical gamut, allowing us to determine the hue of the light source. Barnard et al. [8] have demonstrated that the gamut mapping method outperforms the Gray-World. The Gray-World assumes that the mean value of the average reflection of light by natural scenes is a constant value close to “grey”. Finlayson improved the gamut mapping algorithm by limiting the transformations to chromaticity space, which means that only the illumination corresponding to the existing illumination is allowed [9]. The above improved algorithm is called GCIE, which can be regarded as a robust improvement method to remove the limitation in the diagonal model of illumination variation. The illumination estimation accuracy of the color constancy method of gamut mapping is highly dependent on the proposed assumptions. Once the assumptions do not meet the actual application scenarios, the color constancy performance degrades severely.

The learning-based color constancy method obtains the illumination estimation model through continuous iterative learning from a large amount of given training datasets. Learning-based color constancy methods usually first extract intrinsic properties of natural images as features (e.g., edges, histograms, chromaticity and semantic information of brightest colors, etc.), and then study the complex relationship between features and illumination. Color cat [10] utilizes the linear regression relationship between illumination and histogram to achieve illumination estimation. Corrected moments [11] also show that color moments used as features provide satisfactory illumination estimation performance with least squares training. Learning-based models [12,13], Bayesian-based color constancy [14,15], exemplar-based methods [16], biologically-inspired models [17,18], high-level information-based methods [19,20], and physics-based models [21,22] are commonly used examples of learning-based color constancy. In recent years, with the rapid development of deep neural network technology, the performance of color constancy methods based on convolutional neural network models has been continuously improved [23]. However, its practical application is limited due to the large amount of parameters and overloaded redundant features.

Statistics-based methods put greater attention on the correlation between illumination and surface reflectance. Buchsbaum [24] introduced the Gray-World hypothesis, which assumes that the average reflectance in a scene under a neutral light source is achromatic. The color constancy method based on the Gray-World assumption can calculate the mean value of the three RGB channels and eliminate the influence of ambient light as much as possible. The color constancy method based on the gray world assumption works well when the color components of the image are relatively uniform. However, once the color distribution of the image is uneven, the effect drops sharply. White-Patch [25] assumes that perfect reflections lead to a maximum response in the RGB channels, i.e., taking the maximum value of RGB as the value of white. However, White-Patch-based color constancy methods fail when the scene is flooded with a large number of monochromatic colors. Grey-Edge is another popular color constancy method [26,27], which implies that the average reflectance differences in a scene is achromatic. Both of the low-level statistical methodologies listed above can be merged in a single framework:

$${\left({\displaystyle \int \left|\frac{{\partial }^n{f}_c^{\sigma }\left(x\right)}{\partial x^n}\right|}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c$$

(1)

where the image $f_c\left(x\right)$ was captured by the camera. $f_c^{\sigma }\left(x\right)=f_c\left(x\right)\otimes G^{\sigma }$,$G^{\sigma }$ denotes a Gaussian filter with standard deviation $\sigma$. $k$ functions as a scaling coefficient that varies according to the scene observed. The constant $k$ is between 0 and 1, and color constancy based on this equation indicates that the $p-\mathrm{th}$ Minkowski norm of the $n-\mathrm{th}$ order derivative in a scene is achromatic. $e_c$ is the estimated lighting.

In this paper, we propose a novel low-level statistics-based method as a final extension of the above-mentioned framework described in Equation (1). Inspired by locally normalized reflectance estimation [28] and Gray-Edge hypothesis, to get the locally normalized reflectance differences, we partition the reflectance difference picture into $J$ non-overlapped patches of equal size and divide each reflectance difference by the local maximum inside the non-overlapping local patch. For the obtained color-biased image $f_c\left(x\right)$, the proposed algorithm is first used to determine the light source estimate $e_c$. Then, color constancy is achieved by transforming image $f_c\left(x\right)$ according to illumination estimate $e_c$ into a photograph taken under a standard light source.

The main contributions of the paper can be summarized as follows:

(1) The relationship between the total of reflectance differences and the sum of locally normalized reflectance differences is exploited. After analyzing the statistics of three datasets containing different lighting conditions and scenes, we found that the ratio of the global sum of reflectance differences to the global sum of locally normalized reflectance differences is achromatic. Based on this finding, we propose a more accurate color constancy method for recovering the true color of the scene.

(2) We propose a new framework that incorporates color constancy methods such as Gray-World, maximum RGB, and Gray-Edge. We will also show how that Grey-World, White-Patch, Grey-Edge, and Local-Surface [29] can all be incorporated into the proposed framework of color constancy.

(3) The experiment demonstrates the feasibility and effectiveness of the proposed method when facing scenarios with single or multiple illuminations. In particular, the experimental results on the HDR test set show that the color constancy method proposed by us is superior to the comparison algorithm, showing that the pro-posed method can restore the actual color more accurately for different scenes. We also incorporated a clustering algorithm to improve the results under multiple illuminations.

The rest of the paper is organized as follows: Section 2 presents the proposed algorithm in detail, Section 3 tests the performance of the proposed algorithm on four commonly used datasets, and finally, Section 4 summarizes and further discusses future research work.

2. Proposed Method

Assuming the scene is illuminated uniformly by a single light source $I\left(\lambda \right)$, such as outdoor lighting, the image $f_c\left(x\right)$ captured by the camera pipeline model are represented in the following form:

$$f_c\left(x\right)={\displaystyle {\int }_{\omega }I\left(\lambda \right)R\left(\lambda ,x\right)S_c\left(x\right)d\lambda }$$

(2)

where $c\in \left\{R,G,B\right\}$ are color channels of the camera sensor, $x$ is the spatial coordinate, wavelength of the light is represented as $\lambda$. $R\left(\lambda ,x\right)$ denotes the surface reflectance, $\omega$ is the visible spectrum. $S_c\left(\lambda \right)=\left[S_R\left(\lambda \right),S_G\left(\lambda \right),S_B\left(\lambda \right)\right]$ is the camera sensitivity, and under the diagonal transform assumption [7], the observed light source color $e_c\in \left\{e_R,e_G,e_B\right\}$ can be calculated as:

$$e_c=\frac{f_c\left(x\right)}{R_c\left(x\right)}$$

(3)

Figure 1 is a flowchart of the proposed color constancy approach, the details of which are described in the following sections.

2.1. Local Normalized Surface Reflectance Differences

This section will explain the meaning of local normalized surface reflectance differences. According to the paper [26], we can calculate the differences in the image $f_{c,X}\left(x\right)$ by the formula:

$$f_{c,X}\left(x\right)=\left|\frac{{\partial }^n{f}_c\left(x\right)}{\partial x^n}\right|$$

(4)

The entire region of the differences picture $f_{c,X}\left(x\right)$ is divided into $J$ equal-sized non-overlapping patches. Let

$$L_{c,X,j}\left(x\right)=\frac{f_{c,X,j}\left(x\right)}{f_{c,X,j}\left(x_{j,\max }\right)}$$

(5)

where $x_{j,\max }$ denotes the spatial location of the pixel with the maximum intensity in the $J-\mathrm{th}$ local region:

$$x_{j,\max }=\arg \max \left\{f_{c,X,j}\left(x\right),x=1,2,\cdots ,n\right\}$$

(6)

$f_{c,X,j}\left(x\right)$ is the edge intensity of the pixel at $x$ position, which is normalized by the maximum value of the edge value in the $j-\mathrm{th}$ local image patch.

The reflectance differences in a scene can be represented as $R_{c,j}^X\left(x\right)$, then local normalized reflectance differences $L_{c,X,j}\left(x\right)$ can be represented as:

$$L_{c,X,j}\left(x\right)=\frac{f_{c,X,j}\left(x\right)}{f_{c,X,j}\left(x_{j,\max }\right)}=\frac{e_c·R_{c,j}^X\left(x\right)}{e_c·R_{c,j}^X\left(x_{j,\max }\right)}=\frac{R_{c,j}^X\left(x\right)}{R_{c,j}^X\left(x_{j,\max }\right)}$$

(7)

2.2. Hypothesis Validation

In this section, the validation process of the hypothesis in this paper is presented, and it is shown the illuminant estimate $e_c$ can be obtained accurately calculated by dividing the total of edges ${\displaystyle {\int }_{\Omega }{f}_{c,X}\left(x\right)}dx$ by the sum of local normalized reflectance differences ${\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)dxdj}}$, where $\Omega$ denotes the overall image area, $J$ represents the total number of the local regions inside the image, and $\xi$ indicates the space of the $j-\mathrm{th}$ local area.

Let $\stackrel{\vee }{R}$ represent the ratio of total of edges ${\displaystyle {\int }_{\Omega }{f}_{c,X}\left(x\right)}dx$ and the sum of local normalized reflectance differences ${\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)dxdj}}$:

$$\stackrel{\vee }{R}=\frac{{\displaystyle {\int }_{\Omega }{f}_{c,X,}\left(x\right)dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)dxdk}}}=e_c\frac{{\displaystyle {\int }_{\Omega }{R}_c^X\left(x\right)dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)dxdk}}}=e_c\frac{R{D}_{c,s}}{R{D}_{c,sln}}$$

(8)

where

$$R{D}_{c,s}={\displaystyle {\int }_{\Omega }{R}_c^X\left(x\right)}dx$$

(9)

$$R{D}_{c,sln}={\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }\frac{e_c·R_{c,j}^X\left(x\right)dx}{e_c·R_{c,j}^X\left(x_{j,max}\right)dx}}}={\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }\frac{R_{c,j}^X\left(x\right)dx}{R_{c,j}^X\left(x_{j,max}\right)dx}}}={\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)}dxdj}$$

(10)

In order to exploit the relationship between $R{D}_{c,s}$ and $R{D}_{c,sln}$, we used the Gehler-Shi [30] reprocessed dataset containing 568 images and the NUS dataset [31] of 1737 images taken with eight cameras for color checking, and SFU dataset [13] contains 105 high dynamic range images of indoor and outdoor areas. The datasets above mentioned provide illumination of each raw image, so that the no color-biased image of each color-biased image can be obtained, and then we can compute the values of $R{D}_{c,s}$ and $R{D}_{c,sln}$ respectively.

As shown in Figure 2, columns (a), (b) and (c) are the $\frac{R{D}_{c,s}}{R{D}_{c,sln}}$ statistics on the three datasets of NUS, Gehler-Shi and SFU, respectively. The scatter points in each sub-graph represent the $\frac{R{D}_{c,s}}{R{D}_{c,sln}}$ information of one of the three RGB channels of a picture. For example, the three plots in the first column of Figure 2 depict the 1737 ratios of $R{D}_{c,s}$ and $R{D}_{c,sln}$ in each of the three color channels in the NUS dataset, where each discrete point corresponds to a picture in the NUS dataset, and $\frac{R{D}_{c,s}}{R{D}_{c,sln}}$ with $c\in \left\{R,G,B\right\}$. In detail, $\frac{R{D}_{R,s}}{R{D}_{R,sln}}$ denotes the ratio of $R{D}_{c,s}$ and $R{D}_{c,sln}$ in red color channel of the image.

As we can see from Figure 2, it is obvious that most of the scattered points are distributed along the diagonal line. Therefore, we can get the formula:

$$\frac{R{D}_{R,s}}{R{D}_{R,sln}}=\frac{R{D}_{G,s}}{R{D}_{G,sln}}=\frac{R{D}_{B,s}}{R{D}_{B,sln}}=k$$

(11)

where $k$ is a constant. For uniform illumination, the light source color can be computed by:

$$\begin{array}{ll}\frac{{\displaystyle {\int }_{\Omega }{f}_{c,X}\left(x\right)}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }\frac{f_{c,X}\left(x\right)}{f_{c,X}\left(x_{j,max}\right)}}dxdj}}& =\frac{e_c{\displaystyle \int R_c^X\left(x\right)dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }\frac{e_c·R_{c,j}^X\left(x\right)}{e_c·R_{c,j}^X\left(x_{j,max}\right)}}dxdj}}\\ & =\frac{e_c{\displaystyle \int R_c^X\left(x\right)dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{L}_{c,X,j}\left(x\right)}dxdj}}\\ & =k{e}_c\end{array}$$

(12)

Based on the above inference. Given a color-biased image $f_c\left(x\right)$ as input, $\stackrel{\vee }{R}$ can be determined by Equation (8), $k$ functions as a scaling coefficient that varies according to the scene observed. Given that $k$ is the same for all color channels $c\in \left\{R,G,B\right\}$, based on Equation (11) and Equation (3), we don’t have to find the actual value of $k$ because it can be negated by using the normalized form of $e_c$ as the final result of the illumination.

2.3. Expanded into a Unified Framework

In this section, we will show than we propose a new framework that can incorporate several important statistics-based color constancy algorithms (Grey-World, maximum RGB, Grey-Edge, etc.).

Our proposed method, like the previous framework Equation (1), can well be modified to merge into the Minkowski norm:

$${\left(\frac{{\displaystyle \int {\left(f_{c,X}^{n,\sigma }\left(x\right)\right)}^{p}dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{\left(L_{c,X,j}^{n,\sigma }\left(x\right)\right)}^{p}dxdj}}}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c^{n,p,\sigma ,J}$$

(13)

where

$$f_{c,X}^{n,\sigma }\left(x\right)=\left|\frac{{\partial }^n{f}_c^{\sigma }\left(x\right)}{\partial x^n}\right|=\left|\frac{{\partial }^n{f}_c\left(x\right)\otimes G^{\sigma }}{\partial x^n}\right|$$

(14)

$$L_{c,X,j}^{n,\sigma }\left(x\right)=\frac{f_{c,X}^{\sigma ,n}\left(x\right)}{f_{c,X}^{\sigma ,n}\left(x_{j,max}\right)}$$

(15)

$f_{c,X}^{n,\sigma }\left(x\right)$ denotes the spatial derivatives of order $n$. The Gaussian filter $G^{\sigma }$ with standard deviation $\sigma$ is also introduced in order to exploit the local correlation. The Minkowski norm $p$ determines the relative weights of the multiple measures used to estimate the final illuminant color.

What is more,

Table 1. A summary of the different illuminant estimations methods.

Methods	Symbol	Equation
Grey-World	$e_c^{0,1,0,N}$	$\left({\displaystyle \int f_c\left(x\right)dx}\right)=k{e}_c$
White-Patch	$e_c^{0,\infty ,0,N}$	${\left({\displaystyle \int {\left|f_c\left(x\right)\right|}^{\infty }dx}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\infty $}\right.}=k{e}_c$
Grey-Edge	$e_c^{1,p,\sigma ,N}$	${\left({\displaystyle \int {\left|f_{c,X}^{\sigma }\left(x\right)\right|}^{p}dx}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c$
Local-Surface	$e_c^{0,1,0,J}$	$\left(\frac{{\displaystyle \int \left|f_c\left(x\right)\right|dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }\left|L_{c,j}\left(x\right)\right|dxdj}}}\right)=k{e}_c$
Local-Edge	$e_c^{1,p,\sigma ,J}$	${\left(\frac{{\displaystyle \int {\left(f_{c,X}^{1,\sigma }\left(x\right)\right)}^{p}dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{\left(L_{c,X,j}^{1,\sigma }\left(x\right)\right)}^{p}dxdj}}}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c$
2nd order Local-Edge	$e_c^{2,p,\sigma ,J}$	${\left(\frac{{\displaystyle \int {\left(f_{c,X}^{2,\sigma }\left(x\right)\right)}^{p}dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{\left(L_{c,X,j}^{2,\sigma }\left(x\right)\right)}^{p}dxdj}}}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c$

demonstrates that both Grey-World, White-Patch, Grey-Edge, and Local-Surface methods are the extreme case of Equation (13). For example, for color constancy method 2nd order Local-Edge, ${\left(\frac{{\displaystyle \int {\left(f_{c,X}^{2,\sigma }\left(x\right)\right)}^{p}dx}}{{\displaystyle {\int }_1^J{\displaystyle {\int }_{\xi }{\left(L_{c,X,j}^{2,\sigma }\left(x\right)\right)}^{p}dxdj}}}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}=k{e}_c$, $e_c=e_c^{2,p,\sigma ,J}$, that is, $J$ represents the total number of the local regions inside the image, the spatial derivatives is 2, $p$ represents Minkowski norm and $\sigma$ represents the standard deviation of the Gaussian filter. Similarly, the remaining color constancy methods in Table 1 can also be incorporated into the unified framework we propose.

3. Experimental Results

The previous section provided a generic formulation of color illuminant estimates using low-level image features. In this section, the proposed approach is evaluated on four benchmark datasets, one indoor light source dataset [32], two for the real-world dataset (SFU indoor dataset and Grey-ball dataset) [13,30], and one for the HDR light source dataset [33]. The light source color of the scene is given as extra data for both datasets.

The angular error is used as the color constancy error metric [34]:

$$angularerror=co{s}^{-1}\left(\stackrel{\wedge }{e_l}·\stackrel{\wedge }{e_e}\right)$$

(16)

where the $\stackrel{\wedge }{e_l}$ indicates the actual light source and the $\stackrel{\wedge }{e_e}$ indicates the estimated light source. Smaller angular errors indicate more accurate color constancy results. In order to evaluate the proposed algorithm more objectively, five metrics including median, mean, trimean, max angular error are used to measure the accuracy of the color constancy of the method.

The proposed method in the paper is a low-level based method, so we compared it to the following low-level based methods: Inverse-Intensity Chromaticity Space (IICS) [22], Grey-World [24], White-Patch [25], Shade of Grey [35], General Grey World [34], 1-st order Grey-Edge, 2-nd order Grey-Edge [26], Local Surface Reflectance [29]. It also includes the most advanced methods available, such as pixel-based Gamut Mapping [7], edge-based Gamut Mapping [36], Spatio-Spectral Statistics [37], Weighted Grey Edge [27], SVR Regression [38], Natural Image Statistics [18], Exemplar-based method [16], Bayesian [15], and Thin-plate Spline Interpolation [39].

3.1. Parameter Setting and Analysis

As previously stated in Section 2, the proposed method should select the best parameter values. In our model there are a total of four variables, because of the computational limitation, only 1st-order and 2nd-order are discussed, and the remaining variables are only three, which are Minkowski norm $p$, scale $\sigma$, and number of the local regions $J$. Empirically, we traverse the indoor image dataset to determine the optimal parameters. The final selected parameters are shown in

Table 2. Parameter settings in the proposed 1st-order Local edge and 2nd-order Local edge color constancy methods. Noted that the parameters are empirical.

Methods	Minkowski ($\mathit{p}$)	Scale ($\mathit{\sigma }$)	Region Number ($\mathit{J}$)
1st-order Local edge	10	4	16
2nd-order Local edge	10	6	16

. As shown in Table 2, both Minkowski norm $p$ and the local regions $J$ adopt the same setting in both 1st-order Local edge and 2nd-order Local edge methods. We set scale $\sigma =4$ and scale $\sigma =6$ in the 1st-order Local edge and 2nd-order Local edge color constancy methods, respectively.

3.2. Indoor Dataset

The angular error results of various models on the SFU indoor dataset are presented in

Table 3. The performance of different color constancy methods on the SFU indoor dataset.

Methods	Median	Mean	Trimean	Best-25%	Worst-25%
IICS	8.2	15.5	12.0	2.2	40.0
Grey-World	7.0	9.8	8.1	0.90	23.3
White-Patch	6.5	9.1	7.6	1.8	20.9
Shade of Grey	3.7	6.4	5.0	0.6	16.4
General Grey World	3.3	5.4	4.1	0.5	13.7
1st-order Grey-Edge	3.20	5.6	4.2	1.0	14.0
2nd-order Grey-Edge	2.7	5.2	4.3	1.2	13.7
Local Surface Reflectance	2.4	5.7	4.1	0.5	15
Pixel-based Gamut	2.3	3.7	2.7	0.5	9.3
Edge-based Gamut	2.3	3.9	2.8	0.6	16.08
Bayesian	3.67	2.73	2.91	0.82	8.21
NIS	3.71	2.60	2.84	0.79	8.47
19-Edge Corrected-moment	2.0	2.6	2.25	0.68	7.08
Proposed 1st-order Local Edge	2.0	2.52	2.31	0.46	6.78
Proposed 2nd-order Local Edge	1.9	2.50	2.23	0.45	6.79

. A total of 321 linear photos were captured in the laboratory under 11 various illumination conditions in the SFU indoor dataset [32].

It can be seen from Table 3 that our model performed well when compared to other models on a variety of measures. Specifically, our proposed 2nd-order Local Edge color constancy method achieves the smallest angular error on the four metrics of median, mean, trimean and best-25%, and the proposed 1st-order Local Edge method on the worst-25% metric, it achieves better results than the comparison algorithm. Table 3 shows that the proposed color constancy method can achieve more accurate lighting color estimation results than the comparison algorithm in indoor lighting environments.

3.3. Real-World Dataset

The Gehler-Shi dataset includes 568 linear natural photos [15,30], all of which were captured in RAW format with a DSLR camera, with no color correction. As in many prior studies, the 24 patch color checkboard in every image of the dataset was disguised for illuminant estimation.

Our approaches were then tested on the SFU Grey-Ball dataset [13], which comprises 11,346 non-linear photos. This dataset has been treated in camera using complicated processing, making it impossible to derive an exact illuminant estimate. Before the experiment, we masked out the gray balls in each photo for unbiased evaluation.

Table 4. Performance of different color constancy methods on the Gehler-Shi dataset.

Methods	Median	Mean	Trimean	Best-25%	Worst-25%
IICS	13.6	13.6	13.4	9.5	18.0
Grey-World	6.3	6.4	6.3	2.3	10.6
White-patch	5.7	7.5	6.7	1.5	16.1
Shade of Grey	4.0	4.9	4.4	1.1	10.2
General Grey World	3.5	4.7	4.0	1.0	10.2
1st-order Grey-Edge	4.7	5.9	5.4	1.6	11.9
2nd-order Grey-Edge	4.5	5.3	4.9	1.9	10.0
Local Surface Reflectance	2.6	3.4	2.9	0.8	7.2
Pixel-based Gamut	2.4	4.2	3.3	0.5	11.2
Edge-based Gamut	5.6	6.7	6.0	2.0	13.5
SVR Regression	6.7	8.1	7.4	3.3	14.9
Bayesian	3.5	4.8	4.1	1.3	10.5
Exemplar-based	2.3	2.9	2.5	0.8	6.0
NIS	3.1	4.2	3.5	1.0	9.2
19-Edge Corrected-moment	2.0	2.8	2.25	0.68	7.08
Proposed 1st-order Local edge	1.8	2.04	2.31	0.66	5.89
Proposed 2nd-order Local edge	1.75	1.97	2.28	0.65	5.80

shows the results on this color check dataset, and

Table 5. Performance of different color constancy methods on the SFU Grey-Ball dataset.

Methods	Median	Mean	Best-25%	Worst-25%
IICS	5.6	6.6	1.8	13.3
Grey-World	7.0	7.9	2.2	15.2
White-Patch	5.3	6.8	1.2	14.7
Shade of Grey	5.3	6.1	1.8	11.9
General Grey World	5.3	6.1	1.8	11.9
1st-order Grey-Edge	4.7	5.9	1.6	11.9
2nd-order Grey-Edge	4.8	6.1	1.6	12.4
Local Surface Reflectance	5.1	6.0	-	11.9
Pixel-based Gamut	5.8	7.1	1.7	14.7
Edge-based Gamut	5.8	6.8	1.9	13.5
Exemplar-based	3.4	4.4	1.0	9.4
NIS	3.9	5.2	1.2	11.1
Proposed 1st-order Local edge	3.8	4.36	0.9	8.89
Proposed 2nd-order Local edge	3.76	4.32	0.87	8.89

lists the results of the SFU grey-ball dataset. In general, among all models, our method shows the best color constancy accuracy. As can be seen from Table 4, the 2nd-order Local edge method we proposed achieved the lowest values on the five metrics of median, mean, trimean, best-25% and worst-25%. Lower values for the above five metrics indicate more accurate color constancy. Therefore, the 2nd-order Local edge method we proposed is superior to Grey-World, White-patch, Shade of Grey, Grey-Edge, Local Surface Reflectance, Pixel-based Gamut, Edge-based Gamut on the Gehler-Shi dataset test set, SVR Regression, Bayesian, Exemplar-based and NIS color constancy methods. Specifically, it can be seen from Table 5 that on the SFU Grey-Ball dataset, our proposed 2nd-order Local edge method achieves optimal results on the three metrics of mean, best-25% and worst-25%. And it is only 0.36 higher than the best performing Exemplar-based method on the median metric.

Figure 3 demonstrates the results on sample photos from the color checker dataset. It can be seen from Figure 3 that the proposed color constancy method can well restore the real color of the scene on the Gehler-Shi test data set. For the scenes shown in the first and third rows in Figure 3, the White-Patch, Shade of Gray and Gray-Edge methods can hardly restore the actual color of the scene, and the results have no obvious improvement compared to the original input. Gray-World has achieved better color constancy results than the above methods in all scenes, but it still has a certain degree of color cast. Our proposed method achieves the smallest angular error on all test scenarios.

3.4. SFU HDR Dataset

Our approach was then tested on a dataset with an HDR light source [33], which includes 105 high-quality images captured under indoor and outdoor light sources.

The performance statistics for several methods on the SFU HDR dataset are shown in

Table 6. Performance of different color constancy methods on the SFU HDR image dataset.

Methods	Median	Mean	Worst-25%
Grey-World	7.4	8.0	19.5
White-Patch	3.9	6.3	-
Shade of Grey	3.9	5.7	12.9
Grey-Edge	3.8	6.0	13.8
Proposed 1st-order Local edge	2.9	4.6	10.6
Proposed 2nd-order Local edge	2.9	4.5	10.4

. When compared to other models on this dataset, our model performs well on a variety of metrics. It can be seen from Table 6 that on the SFU HDR image dataset, the 2nd-order Local edge method we proposed achieved the best results on the three metrics of median, mean and worst-25%, and the proposed 1st-order Local edge algorithm achieves as good results as the 2nd-order Local edge on the median metric. The proposed methods are significantly better than the second-ranked Grey-Edge algorithm. Table 6 shows that the color constancy method we proposed can also accurately restore the true color of the scene in the HDR scene with rich light sources both indoors and outdoors.

4. Conclusions

In this paper, inspired by the Gray-Edge hypothesis, our statistical experiments on the three public datasets of NUS, Gehler-Shi and SFU show that the ratio of the global sum of reflection differences to the global sum of locally normalized reflection differences is achromatic. Based on the above conclusion, we propose a new method for more accurate color constancy. Qualitative and quantitative results of the proposed color constancy method on multiple test datasets containing different lighting scenarios demonstrate its effectiveness. In particular, the experimental results on the HDR test set show that the color constancy method proposed by us is superior to the comparison algorithm, showing that the proposed method can restore the actual color more accurately for different scenes. Additionally, we propose a new framework that can incorporate current mainstream statistics-based color constancy methods (Gray-World, Max-RGB, Gray-Edge, etc.) into the proposed framework. A limitation of our research work is that the four parameters in the proposed color constancy algorithm are empirical rather than adaptive. Our future research work is to combine the statistical information of the image with the convolutional neural network to design a network structure with more accurate color constancy.