Green Stability Assumption: Unsupervised Learning for Statistics-Based Illumination Estimation

Abstract

In the image processing pipeline of almost every digital camera, there is a part for removing the influence of illumination on the colors of the image scene. Tuning the parameter values of an illumination estimation method for maximal accuracy requires calibrated images with known ground-truth illumination, but creating them for a given sensor is time-consuming. In this paper, the green stability assumption is proposed that can be used to fine-tune the values of some common illumination estimation methods by using only non-calibrated images. The obtained accuracy is practically the same as when training on calibrated images, but the whole process is much faster since calibration is not required and thus time is saved. The results are presented and discussed. The source code website is provided in Section Experimental Results.

1. Introduction

Regardless of the influence of the scene illumination, the human visual system can recognize object colors through its ability known as color constancy [1]. In the image processing pipeline of almost every digital camera, there is also a part dedicated to computational color constancy [2]. It first estimates the scene illumination and then uses it to chromatically adapt the image i.e., to correct the colors. For a more formal problem statement, an often used image $\mathbf{f}$ formation model written under Lambertian assumption is given as [3]

$$f_c\left(\mathbf{x}\right)={\int }_{\omega }I(\lambda ,\mathbf{x})R(\lambda ,\mathbf{x}){\rho }_c\left(\lambda \right)d\lambda ,$$

(1)

where $c\in \{R,G,B\}$ is a color channel, $\mathbf{x}$ is a given image pixel, $\lambda$ is the wavelength of the light, $\omega$ is the visible spectrum, $I(\lambda ,\mathbf{x})$ is the spectral distribution of the light source, $R(\lambda ,\mathbf{x})$ is the surface reflectance, and ${\rho }_c\left(\lambda \right)$ is the camera sensitivity of color channel c. Assuming uniform illumination for the sake of simplicity makes it possible to remove $\mathbf{x}$ from $I(\lambda ,\mathbf{x})$ and then the observed light source color is given as

$$\mathbf{e}=\left(\begin{array}{c}{e}_R\\ e_G\\ e_B\end{array}\right)={\int }_{\omega }I\left(\lambda \right)\mathbf{\rho }\left(\lambda \right)d\lambda .$$

(2)

The direction of $\mathbf{e}$ provides enough information for successful chromatic adaptation [4]. Still, calculating $\mathbf{e}$ is an ill-posed problem because only image pixel values $\mathbf{f}$ are given, while both $I\left(\lambda \right)$ and $\rho \left(\lambda \right)$ are unknown. The solution to this problem is to make additional assumptions. Different assumptions have given rise to numerous illumination estimation methods that can be divided into two main groups. First of these groups contains low-level statistics-based methods such as White-patch [5,6] and its improvements [7,8,9], Gray-world [10], Shades-of-Gray [11], Grey-Edge (1st and 2nd order) [12], Weighted Gray-Edge [13], using bright pixels [14], and using bright and dark colors [15]. The second group includes learning-based methods such as gamut mapping (pixel, edge, and intersection based) [16], using neural networks [17], using high-level visual information [18], natural image statistics [19], Bayesian learning [20], spatio-spectral learning (maximum likelihood estimate, and with gen. prior) [21], simplifying the illumination solution space [22,23,24,25], using color/edge moments [26,27], using regression trees with color distribution statistics features [28], performing various kinds of spatial localizations [29,30], and using convolutional neural networks [31,32,33].

Statistics-based illumination estimation methods are less accurate than the learning-based ones, but they are faster and simpler to implement in embedded systems, which is one of the reasons for their widespread usage [34] that justifies additional research related to them. In this paper, the so-called green stability assumption is proposed that can be used to fine-tune the values of the parameters of statistics-based methods by using only non-calibrated images without known ground-truth illumination. It is based on observations about the behaviour of the green chromaticity of ground-truth illuminations. The obtained accuracy is practically the same as when using calibrated training images, but the whole process is much faster and it can be directly applied in practice.

The paper is structured as follows: Section 2 briefly describes the best-known statistics-based methods and the most commonly used error statistics, Section 3 proposes the green stability assumption, Section 4 presents the results, and, finally, Section 5 concludes the paper.

2. Best-Known Statistics-Based Methods

2.1. Definition

Some of the best-known statistics-based illumination estimation methods are centered around the Gray-world assumption and its extensions. Under this assumption, the average scene reflectance is achromatic [10] and $\mathbf{e}$ is obtained as

$$\frac{\int \mathbf{f}\left(x\right)dx}{\int dx}=k\mathbf{e},$$

(3)

where $k\in [0,1]$ is the reflectance amount with 0 meaning no reflectance and 1 meaning total reflectance. By adding the Minkowski norm p to Equation (3), the Gray-world method is generalized into the Shades-of-Gray method [11]:

$${\left(\frac{\int {\left(\mathbf{f}\left(x\right)\right)}^{p}dx}{\int dx}\right)}^{\frac{1}{p}}=k\mathbf{e}.$$

(4)

Having $p=1$ results in Gray-world, while $p\to \infty$ results in White-patch [5,6]. In [12], Equation (4) was extended to the general Gray-world through local smoothing:

$${\left(\frac{\int {\left({\mathbf{f}}^{\sigma }\left(x\right)\right)}^{p}dx}{\int dx}\right)}^{\frac{1}{p}}=k\mathbf{e},$$

(5)

where ${\mathbf{f}}^{\sigma }=\mathbf{f}\ast {\mathbf{G}}^{\sigma }$ and ${\mathbf{G}}^{\sigma }$ is a Gaussian filter with standard deviation $\sigma$. Another extension is the Grey-edge assumption, under which the scene reflectance differences calculated with derivative order n are achromatic [12] so that

$${\left(\int \left|\frac{{\partial }^n{\left({\mathbf{f}}^{\sigma }\left(x\right)\right)}^p}{{\partial }^{n}x}\right|dx\right)}^{\frac{1}{p}}=k\mathbf{e}.$$

(6)

The described Shades-of-Gray, general Gray-world, and Gray-edge methods have parameters and the methods’ accuracy depends on how these parameters are tuned. Nevertheless, in the literature, it often appears as if they require no training [3,15], which is then said to be an advantage. It may be argued that the parameter values are in most cases the same, but this is easily disproved. In [15] for methods mentioned in this section, the best fixed parameter values were given for ten different datasets. These values are similar for some datasets, but overall they span two orders of magnitude. With such high differences across datasets in mind, it is obvious that the parameter values have to be learned. Therefore, later in Section 4 in experimental results, the originally reported results are also compared to the newly calculated revisited results where the accuracy was obtained by performing learning and cross-validation. Although the mentioned methods’ accuracy is lower than the one of state-of-the-art methods, they are still widely used because of their simplicity. Empirical evidence for this was given in, e.g., Ref. [34], where the claim that “most commercial cameras are doing white balance based on the best-known Gray-World assumption” has been successfully used for forensic tasks.

2.2. Error Statistics

The accuracy of illumination estimation methods is generally expressed in form of illumination estimation error. From various proposed illumination estimation accuracy measures [35,36,37], the angular error is most commonly used. It represents the angle between the illumination estimation vector and the ground-truth illumination. All angular errors obtained for a given method on a chosen dataset are usually summarized by different statistics. Because of the non-symmetry of the angular error distribution, the most important of these statistics is the median angular error [38]. It was shown that angular errors below $3^{\circ }$ can be considered acceptable [39,40]. The ground-truth illuminations of benchmark dataset images are obtained by reading off the color of neutral areas on calibration objects that are previously placed in the image scene, e.g., a gray ball or a color checker, which is effectively the illumination color. During testing, these objects are masked out to prevent possible bias.

3. The Proposed Assumption

3.1. Practical Application

The methods mentioned in Section 2 are some of the most widely used illumination estimation methods [34] and this means that their parameters should preferably be appropriately fine-tuned before putting them in production. The best way to do this is to use a benchmark dataset, but, because of dependence of Equation (2) on $\rho \left(\lambda \right)$, a benchmark dataset would be required for each used camera sensor. Since putting the calibration objects into image scenes and later extracting the ground-truth illumination is time-consuming, it would be better to perform some kind of unsupervised learning on non-calibrated images i.e., to learn from images for which the ground-truth illumination is unknown.

3.2. Motivation

When for a dataset the ground-truth illuminations are unknown, an alternative is to make assumptions about the nature of illumination estimations produced by statistics-based methods when their parameters are fine-tuned and then to meet the conditions of the assumptions. When considering the nature of illumination estimations, a good starting point is the observation that some statistics-based illumination estimations appear “to correlate roughly with the actual illuminant” [26]. Figure 1 shows this for the images of the GreyBall dataset [41].

The points in Figure 1 can be considered to occupy a space around a line in the $rb$-chromaticity [22], which is connected to the fact that the green chromaticity of the ground-truth illuminations is relatively stable and similar for all illuminations. For the GreyBall dataset, the standard deviations of the red, green, and blue chromaticity components of the ground-truth illuminations are $0.0723,0.0106$, and $0.0750$, respectively, and similar results are obtained for all other datasets. For Shades-of-Gray illumination estimations shown in Figure 1, the red, green, and blue chromaticity components of the ground-truth illuminations are $0.0842,0.0253$, and $0.0770$, respectively, which means that there is also a trend of green chromaticity stability, although the standard deviation is greater than in the case of ground-truth illumination. This means that, if a set of illumination estimations is to resemble the set of ground-truth illuminations, the estimations’ green chromaticity standard deviation should also be smaller and closer to the one of the ground-truth. As a matter of fact, if, for example, the Shades-of-Gray illumination estimations for $p=2$ and $p=15$ shown in Figure 2 are compared, the standard deviations of their green chromaticities are $0.0253$ and $0.0158$, respectively, while their median angular errors are ${6.2}^{\circ }$ and ${5.3}^{\circ }$, respectively. Similar behaviour where lower green chromaticity standard deviation is to some degree followed by lower median angular error can be seen on all datasets and for all methods that were mentioned and described in Section 2.

Such behaviour can also be explained theoretically. Namely, due to the constraint that the sum of the chromaticity components has to be 1, the chromaticity dimensionality is 2. As for the illumination chromaticities, they mostly come either from outdoor sources such as the sun or the sky reflection, or from an artificial source such as incandescent light bulbs, and in such cases their chromaticities should ideally follow the Planckian locus i.e., the curve of the black body color, which reduces the dimensionality of illumination chromaticity to 1. Since even for non-ideal real-world image illumination chromaticities there is a strong linear connection between the red and blue chromaticities [22,23,42], the green chromaticity is consequently supposed to be more or less similar among various illuminations. Furthermore, the more accurate some illumination estimations are, the more similar they should be to the ground-truth illuminations and therefore they should also be expected to have a smaller deviation of the green chromaticity. One possible application of this would be to blindly choose between two sets of parameter values for a given illumination estimation method on a given set of images without known ground-truth illumination. Namely, the parameter values for which the method will produce illumination estimations with smaller deviation of green chromaticity should be preferred in light of the laid out theoretical background. This is effectively unsupervised learning since it is not required to know the ground-truth illumination for the training images. Since camera sensors are supposed to be linear at their initial stages, the described behaviour should hold for any of them.

To check if this holds in practice, another larger experiment was conducted on the GreyBall dataset [41]. The goal of the experiment was to find additional empirical clues about the connection between the change in standard deviation of the green chromaticity and the change in median angular error for a given set of illumination estimations. The gist of the experiment can be summarized in the following steps: (1) select several illumination estimation methods to be tested; (2) for each of the methods, select several sets of possible parameter values; (3) for each combination of a method and a set of its parameter values, calculate the set of illumination estimations for images in the GreyBall dataset; (4) for each of these sets of illumination estimations, calculate the median angular error and the standard deviation of the green chromaticities; (5) for every possible pair of the same sets of illumination estimations, calculate the differences between their median angular errors and standard deviations of the green chromaticities and use them as coordinates to plot a point. If there is a connection between the changes in median angular error and the standard deviation of the green chromaticities, there is a good chance that it may be visible after plotting the points for all mentioned pairs. For additional clarification, the experiment can be stated more formally. First, for each method $M\in \mathbb{M}$, where $\mathbb{M}$ contains all methods from Section 2, the Cartesian product of discrete sets of evenly spread values for individual parameters of M was calculated to get n tuples ${\mathbf{p}}_M^{\left(i\right)},i\in \{1,2,\cdots ,n\}$. Gray-world and White-patch have no parameters, but they were implicitly included as special cases of Shades-of-Gray. Second, each ${\mathbf{p}}_M^{\left(i\right)}$ was used to set the parameter values of M and then M was applied to all images of the GreyBall dataset to obtain an illumination estimation for each of them. Third, for these illumination estimations, the standard deviation of their green chromaticities ${\sigma }_i$ and their median angular error $m_i$ were calculated. Fourth, for every of $\left(\genfrac{}{}{0pt}{}{n}{2}\right)$ possible pairs of indices $i,j\in \{1,2,\cdots ,n\}$ such that $i<j$ a new difference pair $\{\Delta {\sigma }_k,\Delta m_k\}$ was calculated such that $\Delta {\sigma }_k={\sigma }_i-{\sigma }_j$ and $\Delta m_k=m_i-m_j$. Finally, all such difference pairs created for all $M\in \mathbb{M}$ were put together into set of pairs $\mathbb{P}$. If members of pairs in $\mathbb{P}$ are interpreted as coordinates, then their plot is shown in Figure 3.

3.3. Green Stability Assumption

The value of Pearson’s linear correlation coefficient for the points in Figure 3 is $0.7408$, which indicates a strong positive linear relationship [43]. In other words, the difference between the standard deviations of green chromaticities of illumination estimations produced by the same method when using different parameter values is strongly correlated to the difference between median angular errors of these illumination estimations. The same experiment was repeated for each of the NUS datasets [15] and the resulting correlations are given in

Table 1. Correlation between difference in green chromaticity standard deviation and difference in median angular errors for NUS datasets [15].

Dataset	C1	C2	Fuji	N52	Oly	Pan	Sam	Sony
Correlation	0.9255	0.6381	0.8977	0.9443	0.8897	0.9644	0.8902	0.9095

. For all datasets except for Canon2 (C2), the correlation is significantly high. After analysing the case of C2, the main culprit for its lower correlation was found to be the performance of the general Gray-world algorithm, whose accuracy for higher values of $\sigma$ did not change with same dynamic as was the case for other datasets. This is, however, not critical because it represents only a single case of slightly lower correlation.

Based on this empirical results and observations, it is possible to introduce the green stability assumption: the parameter values for which a method’s illumination estimations’ green chromaticity standard deviation is lower simultaneously lead to lower illumination estimation errors. Like many other assumptions, this assumption does also not always hold, but it can still be useful in cases when the ground-truth illuminations for a set of images taken with a given sensor are not available. These images should also be taken under similar illuminations as the mentioned datasets that were used for empirical results. An example of a failure case would be if, for example, the illumination was mostly composed of artificial lights whose colors are spread across the whole chromaticity plane, which would also lead to significant differences in green chromaticity values, but such cases happen only very rarely.

For a specific case when the parameter values of a chosen method are fine-tuned and only non-calibrated images are available, the green stability assumption can be expressed more formally. If n is the number of images in the training set, ${\mathbf{p}}_i$ is the i-th vector of parameter values, ${\mathbf{e}}^{i,j}$ is the method’s illumination estimation obtained for the j-th image when ${\mathbf{p}}_i$ is used for parameter values, $e_G^{i,j}$ is the green component of ${\mathbf{e}}^{i,j}$, and $\overline{e_G^i}$ is the mean green component of illumination estimations for all images obtained with parameters ${\mathbf{p}}_i$, then, under the green stability assumption, the index $i^{\ast }$ of such ${\mathbf{p}}_{i^{\ast }}$ that should result in minimal angular errors is obtained as

$$i^{\ast }=\underset{i}{\arg \min }\sqrt{\frac{{\sum }_{j=1}^n{\left(e_G^{i,j}-\overline{e_G^i}\right)}^2}{n-1}}.$$

(7)

Since Equation (7) performs minimization of standard deviation, it can also be written without the square and the denominator.

4. Experimental Results

4.1. Experimental Setup

The following benchmark datasets have been used to demonstrate the difference between previously reported and newly calculated accuracy results for methods mentioned in Section 2 and to test the effectiveness of the proposed green stability assumption: the GreyBall dataset [41], its approximated linear version, and eight linear NUS dataset [15]. The ColorChecker dataset [20,44] was not used because of its confusing history of wrong usage [45] despite warnings from leading experts [46]. Except the original GreyBall dataset, all other contain linear images, which is preferred because illumination estimation is in cameras usually performed on linear images [2] similar to the model described by Equation (1).

The tested methods include all the ones from $\mathbb{M}$. During cross-validation on all datasets the same folds were used as in other publications. The source code for recreating the numerical results given in the following subsection is publicly available at [47].

4.2. Accuracy

Table 2. Combined accuracy on eight NUS datasets (lower Avg. is better). The used format is the same as in [30].

Algorithm	Mean	Med.	Tri.	Best 25%	Worst 25%	Avg.
Originally Reported Results
Shades-of-Gray [11]	3.67	2.94	3.03	0.98	7.75	3.01
General Gray-World [4]	3.20	2.56	2.68	0.85	6.68	2.63
1st-order Gray-Edge [12]	3.35	2.58	2.76	0.79	7.18	2.67
2nd-order Gray-Edge [12]	3.36	2.70	2.80	0.89	7.14	2.76
Revisited Results
Shades-of-Gray [11]	3.48	2.63	2.81	0.81	7.62	2.76‘
General Gray-World [4]	3.37	2.49	2.61	0.73	7.58	2.61
1st-order Gray-Edge [12]	3.12	2.19	2.39	0.71	7.11	2.42
2nd-order Gray-Edge [12]	3.15	2.23	2.42	0.74	7.13	2.46
Green Stability Assumption Results
Shades-of-Gray [11]	3.44	2.65	2.81	0.83	7.41	2.75
General Gray-World [4]	3.40	2.63	2.76	0.77	7.42	2.69
1st-order Gray-Edge [12]	3.29	2.36	2.55	0.79	7.36	2.58
2nd-order Gray-Edge [12]	3.29	2.44	2.59	0.83	7.30	2.63

,

Table 3. Accuracy on the original GreyBall dataset (lower median is better).

Method	Mean ${(}^{\circ })$	Median ${(}^{\circ })$	Trimean ${(}^{\circ })$
Originally Reported Results
Shades-of-Gray [11]	6.14	5.33	5.51
General Gray-World [4]	6.14	5.33	5.51
1st-order Gray-Edge [12]	5.88	4.65	5.11
2nd-order Gray-Edge [12]	6.10	4.85	5.28
Revisited Results
Shades-of-Gray [11]	7.80	7.15	7.21
General Gray-World [4]	7.61	6.85	6.92
1st-order Gray-Edge [12]	6.14	5.32	5.49
2nd-order Gray-Edge [12]	6.89	5.84	6.06
Green Stability Assumption Results
Shades-of-Gray [11]	6.80	5.30	5.77
General Gray-World [4]	6.80	5.30	5.77
1st-order Gray-Edge [12]	5.97	4.64	5.10
2nd-order Gray-Edge [12]	6.69	5.17	5.72

and

Table 4. Accuracy on the linear GreyBall dataset (lower median is better).

Method	Mean ${(}^{\circ })$	Median ${(}^{\circ })$	Trimean ${(}^{\circ })$
Originally Reported Results
Shades-of-Gray [11]	11.55	9.70	10.23
General Gray-World [4]	11.55	9.70	10.23
1st-order Gray-Edge [12]	10.58	8.84	9.18
2nd-order Gray-Edge [12]	10.68	9.02	9.40
Revisited Results
Shades-of-Gray [11]	13.32	11.57	12.10
General Gray-World [4]	13.69	12.11	12.55
1st-order Gray-Edge [12]	11.06	9.54	9.81
2nd-order Gray-Edge [12]	10.73	9.21	9.49
Green Stability Assumption Results
Shades-of-Gray [11]	12.68	10.50	11.25
General Gray-World [4]	12.68	10.50	11.25
1st-order Gray-Edge [12]	13.41	11.04	11.87
2nd-order Gray-Edge [12]	12.83	10.70	11.44

show the previously reported accuracies, the newly recalculated accuracies, and the accuracies obtained by using the green stability assumption. The results clearly confirm the potential and the practical applicability of the green stability assumption. Namely, the results obtained by using the the ground-truth and the results obtained without having the ground-truth i.e., by applying the proposed assumption are very similar. As a matter of fact, in some cases, using the proposed green stability assumption produces results that are even more accurate than when using ground-truth illumination. This also demonstrates the success of the proposed approach to unsupervised learning for most common low-level statistics-based illumination estimation methods.

5. Conclusions

In most relevant papers, the accuracy results for some of the most widely used statistics-based methods were calculated without cross-validation. Here, it was shown that cross-validation is needed and the accuracy results were revisited. When statistics-based methods are fine-tuned, the best way to do this is by using images with known ground-truth illumination. Based on several observations and empirical evidence, the green stability assumption has been proposed that can successfully be used to fine-tune the parameters of most common statistics-based methods when only non-calibrated images without ground-truth illumination are available. This makes the whole fine-tuning process much simpler, faster, and more practical. It is also an unsupervised learning approach to color constancy. In the future, other similar bases for further assumptions that enable unsupervised learning for color constancy will be researched.