^{*}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (

It has now been 20 years since the seminal work by Finlayson

Our visual system has a striking ability in allowing us to deal with color. However, we are far from fully understanding its behavior. To gain an insight into how our visual system works, some assumptions are often made: first, it is assumed that there is a single illuminant in the scene which is spatially uniform, and second, it is assumed that objects are flat, coplanar, and Lambertian, i.e., their reflectances are diffuse and independent from the angle of view.

Following these assumptions light energy reaching our eye depends on the spectral power distribution of the illuminant (

This color signal is weighted along the visible spectrum

From this equation, we can see that when looking at a white piece of paper in sunset the values captured by our eye are reddish as a result of the illumination. In contrast, when looking at a white piece of paper on a cloudy day these values are bluish. However, we perceive the piece of paper as approximately white in both cases. This property of our visual system is called color constancy.

Color constancy is usually defined as the effect whereby the perceived or apparent color of a surface remains constant despite changes in intensity and spectral composition of the illumination [

von Kries [

This model is called von Kries adaptation. The idea of an individual gain for each photoreceptor was adopted early in computer vision and the gains were computed based on the scene, e.g., assuming that the scene had a mean value of grey [

Even though this model can predict the data very well in natural environments, there is still no agreement in the literature as to how the gain values are computed from the image statistics of the stimulus [

For this reason, further research on the neural mechanisms that underline color constancy has been conducted over the years [

Computational color constancy does not aim to recover the perceived image, but an estimation of the surface reflectances of the scene; Therefore, it changes the paradigm upon which human color constancy is built. In other words, while human color constancy relies on the perception of the colors, computational color constancy relies on the absolute color values of the objects viewed under a canonical (usually white) illuminant, without considering how the image is perceived by an observer. Vazquez-Corral

Therefore, computational color constancy is treated from a mathematical point of view. Let us suppose we have an object with reflectance _{i}^{1}(λ), ^{2}(λ) (let us suppose the illuminant is uniform). Then, the response of the object recorded by the camera sensor

Marimont and Wandell [

Early research in computational color constancy [^{1,2} is a 3 × 3 diagonal matrix.

Many cameras do not have sensors that match the above specifications. Therefore, different methods have tried to search for a linear combination of the original sensor responses in order to force them to accomplish the diagonal model. Mathematically, this linear combination will be the one accomplishing

Finlayson

An example of the use of spectral sharpening can be seen in ^{−1}. The different

In this section we will show a review of different methods used to achieve spectral sharpening.

Finlayson

Let us suppose that reflectances are three dimensional and illuminants two dimensional (the other case is analogous). In this case any reflectance can be decomposed as
_{j}_{1}, _{2}, _{3}] is a coefficient vector in this basis. Let us define Λ^{k}_{i}

As illuminants are two dimensional they need a second illuminant ^{2}(λ) independent from the canonical one ^{c}^{2}. This second lighting matrix is some linear transform (^{2} = ^{c}^{2}[Λ^{c}^{−1}.

As ^{2}(λ) and ^{c}^{c}^{c}^{e}^{c}^{2}(λ) can be written as
^{−1}^{c}^{e}

Finlayson _{1}, λ_{2}]. The resulting sensor _{1}, λ_{2}] and _{1}, λ_{2}] in relation to the rest of the spectrum.

To solve the problem for all the spectra Finlayson and co-authors defined

Mathematically, they defined a _{α}

They took partial derivatives over the vector

In parallel, they differentiated _{λ∈}_{ω}^{2} = 1. Rearranging _{λ∈Φ}[^{2}.

Sensors with negative values are physically impossible. For this reason Pearson and Yule [_{1} or _{2} norm and can be performed both in the sensors themselves or in the sharpening matrix coefficients. All these methods can be solved by either linear or quadratic programming. Here we report the different methods presented.

_{1}- _{1} Constrained coefficients:

_{1}- _{1} Constrained sensors:

_{2}- _{2} Constrained coefficients:

_{2}_{2} Constrained sensors

_{2}_{1} Constrained coefficients

_{2}- _{1} Constrained sensors

Information about the illuminants and reflectances that are more representative in natural scenes is available from multiple sources [

Finlayson ^{1} and ^{2} as 3 × ^{1} and ^{2}.

^{1,2} in a least-squares sense. This can be done by the Moore-Penrose inverse
^{+} represents the pseudoinverse [^{1}[^{2}]^{+}.

Some years later, Barnard

Chong _{i}_{i}_{=1,⋯,}_{I}_{j}_{j}_{=1,⋯,}_{J}_{k}_{k}_{=1,⋯,}_{K}

Chong

They rewrote ^{−1}.

In order to solve _{kij}_{kij}

This method has the drawback of local convergence, that is, the result obtained can be a local minima. Also, TALS needs initialization values for two of their three matrices.

In [

All the previous methods were defined in order to help solving for diagonal color constancy. But, the sharpening matrices related to these methods are not the only ones, there are sharpening matrices that have been derived from psychophysical experiments in chromatic adaptation. Chromatic adaptation matrices also represent sharp sensors [

The Von Kries chromatic adaptation transform is usually defined by the Hunt, Poynton and Estevez transform [

The Bradform transform was defined by Lam [

The Fairchild transform was suggested by Mark Fairchild [

The

Bianco and Schettini [

They defined an objective function with two competing terms

The term _{est}_{med}

This first optimization might, however, incur in negative values of the resulting sensors. For this reason they defined another objective function
_{PC}

Spherical sampling [

Mathematically, let us represent our original camera sensors ^{t}

From this basis

Effectiveness of sharpening matrices has usually been evaluated by least-squares as follows. Let us denote an observed color by

Let us note that if the transformation

Mathematically, if we select a canonical illuminant ^{c}

This formula has been widely used to compare spectral sharpening methods and was already included in Finlayson

The formula presented in

The original aim of spectral sharpening was to achieve diagonal color constancy. Over the years, spectral sharpening has proven beneficial for a number of purposes, some far removed from the original aim. In this section we review some of these new applications. They are presented graphically in

Section 2.4 shows that chromatic adaptation transforms can be understood as spectral sharpening, therefore it is a straightforward idea to use spectral sharpening techniques for handling corresponding colors data and chromatic adaptation.

Finlayson and Drew [

Human perception is not linear but colorimetric spaces are. In other words, when we work in RGB or XYZ spaces, a Euclidean distance

To overcome this issue CIE proposed the CIELab and CieLuv color spaces [^{D}^{65} the XYZ color value of a particular patch under the ^{e}^{D}^{65} and
_{ϵ}

The matrix

Foster and co-authors [

Finlayson ^{c}^{e}

Chong

Therefore, the definition of the color space parametrization

The authors showed the advantages of this new space in two common image processing tasks: image segmentation and Poisson editing [

Drew and Finlayson [

First, they defined a set of color signals, from which they obtained a n-dimensional (_{2} − _{2} sensor-based with positivity. Then, any light or reflectance can be expressed as a coefficient vector in this last basis

Later on, Finlayson _{ϵ}

Finlayson

Finlayson

Recently, Marin-Franch and Foster [

Philipona and O'Regan [^{s}_{i}

They repeated this procedure for ^{s}^{s}^{s}^{s}^{s} and
^{s} are the 3 ×x 3 matrices of the eigenvectors and eigenvalues respectively.

Philipona and O'Regan selected the eigenvalues
^{s} as the surface descriptor, from where they were able to precisely predict color naming, unique hues, hue equilibrium and hue cancellation data.

Building upon Philipona and O'Regan's model, Vazquez-Corral

Sensor sharpening was developed 20 years ago to achieve computational color constancy using a diagonal model. Over this period, spectral sharpening has proven important for solving other problems unrelated to its original aim.

In this paper, we have explained some of the differences between human and computational color constancy: human color constancy relies on the perception of the colors while computational color constancy relies on the absolute color values of the objects viewed under a canonical illuminant.

We have reviewed different methods used to obtain spectrally sharpened sensors, dividing them into perfect sharpening, sensor-based sharpening, sharpening with data, spherical sharpening, and chromatic adaptation transforms.

We have also described different research lines where sharpened sensors have proven useful: chromatic adaptation, color constancy in perceptual spaces, relational color constancy, perceptual-based definition of color spaces, multispectral processing without the use of all the spectra, shadow removal, extraction of information from an image, estimation of the color names and unique hues presented in the human visual system, and estimation of the hue cancellation and hue equilibrium phenomena.

This work was supported by European Research Council, Starting Grant ref. 306337, and by Spanish grants ref. TIN2011-15954-E, and ref. TIN2012-38112.

The authors declare no conflict of interest.

Example of color constancy. We are able to perceive the t-shirt of the man in the right as yellow, but, when looking at it in isolation the color of the t-shirt appears green.

Original camera sensors (

Example of diagonal color constancy using spectral sharpening for five different methods. The original image (sensors) is converted by a linear matrix

Hierarchy for the selection of a spectral sharpening method. The decision should take into account two aspects: the final goal pursued and the availability of spectral data.

Hierarchy of sensor sharpening applications grouped by research field. Each application is linked to a section in this paper.