Per-Pixel Manifold-Based Color Calibration Technique

Abstract

In this paper, we present a method for obtaining a manifold color correction transform for multiview images. The method can be applied in various scenarios, for correcting the colors of stitched images, adjusting the colors of images obtained in different lighting conditions, and performing virtual view synthesis based on images taken by different cameras or in different conditions. The provided derivation allows us to use the method to correct regular RGB images. The provided solution is specified as a transform matrix that provides the pixel-specific color transformation for each pixel and therefore is more general than the methods described in the literature, which only provide the transformed images without explicitly providing the transform. By providing the transform for each pixel separately, we can introduce a smoothness constraint based on the transformation similarity for neighboring pixels, a feature that is not present in the available literature.

1. Introduction

In many applications, we use multiple acquisition devices to record images of the same object. This results in multiple images of the same object with different gamut/color characteristics. Even a single camera can capture images with different colorings due to changing weather conditions. This variation affects image or object recognition algorithms, panorama creation, image stitching, virtual view creation, or Neural Radiance Field (NeRF) [1,2] estimation. To address these differences, color calibration is necessary. It allows us to find a mapping of colors from one image to another. Various methods exist to achieve this mapping or transformation. Nearly all of them assume constant color transformation across the whole image. In this paper, we are presenting a novel manifold color calibration method that allows per-pixel color transformation estimation between pairs of images, even when in partially occluded image pairs.

For example, in low-light conditions, we can take a dark color image and a near-infrared image of the same object [3], or take a shot with and without a flash [4] and then try to combine both of the images together.

Another example would be applications in view synthesis [5,6], where we can create a non-existing view of the scene from the viewpoint of so-called virtual cameras based on two (or more) images captured by real cameras recording the given scene. Even if both cameras are identical (which is not always possible), they can produce images with different color characteristics. This could happen even due to the different viewpoints from which the images were taken.

Although the problem is the most obvious for multi-camera scenarios, using a single camera can also lead to the different colorings of the captured images, for example, due to the changing weather conditions. In applications such as image/object recognition, this could significantly influence the performance of the algorithms used.

Yet another example is panorama creation or/and image stitching [7]. Where we can stitch or combine multiple images captured either by the same image sensor or by multiple sensors into one big continuous image.

In all the abovementioned situations, we can color calibrate the images in order to minimize the artifacts occurring due to the color mismatch. This can be performed by finding the mapping of colors from one image to the other.

There already exist many methods of finding such mappings or transformations, which map the color of all pixels from one image to the other.

The significance of color correction manifests itself in many applications, where different methods were demonstrated to effectively improve the performance of the whole system. One such example is multiview video compression, as evidenced in [8,9,10,11]. In those papers, the authors demonstrate an improvement in the coding of the multiview video after applying a preprocessing step of color matching of the sequences due to the improved accuracy of motion compensation and inter-view prediction.

The methods used to perform the color correction that is described in the literature can be based on histogram equalization [11,12,13]. A popular method is also using a single scaling factor, correction coefficient, or a polynomial to correct for the color differences [14,15,16,17]. A low-dimensional matrix correction can also be used [18]. Another way to color correct the images is to use the color transfer technique [19]. In recent years, neural networks have been used to perform similar tasks and can be used to perform color transfer as well; an example of this color transfer method can be found in [20].

In practice, it is important to not only correct the colors of the images but also maintain sufficient quality of the corrected images without degrading the contrast of the processed images [21]. Thus, the correction algorithm should also consider the texture/contrast information in the processed images.

Another way of performing the color correction is to solve a manifold optimization problem defined globally for the entire image. Such an approach can be found in [8,22,23]. The authors of [8,23] define color correction as an optimization problem with priors both from the source image and from the target image. From the source image, the priors are color characteristics and color distribution, while from the target image, the priors tend to preserve the spatial and temporal structure of the image, after color correction. Similarly, the paper [22] introduces a method for global optimization of pixel values for large baseline multiview sequences, preserving the structure and spatio-temporal consistency.

The paper [24] introduces a parametric linear method and a nonparametric nonlinear method to deal with different color changes between pairs of images. The method proposed is a modified Laplacian Eigenmaps, which is a nonlinear manifold learning approach. The used goal function for color transfer is a quadratic cost function with a quadratic regularizer. The described method only searches for image sample values of a target image, matching the color characteristics of a source image.

In many applications, only a fraction of the pixels from one image directly correspond to any other image. For example, in image stitching, images only partially overlap. Similarly, in view synthesis, synthesized images from a virtual camera contain holes caused by disocclusions, where some parts of the scene are not visible due to perspective changes between the source view and the virtual view created with the source view data. Thus, only part of the real image has corresponding pixels in the virtual image. Yet we can obtain color transformation for all of the pixels in the images in order to seamlessly combine them together.

Color characteristics can vary across images due to effects like ambient occlusion, nonuniform lighting, shadows, and non-Lambertian reflections. Most color calibration techniques do not work reliably for these cases, only averaging color distribution and color characteristics across entire images. But in many applications, we can retain that non-uniformity during the color transformation from one image to the other, since they are inherent for some objects and materials. We can preserve these features for a color-corrected view.

To obtain a proper transformation that preserves the aforementioned effects, one needs to estimate per-pixel color transformation from one image to the other. An important factor of novelty of the work presented in this paper is the fact that the optimization algorithm does not work on the pixel values themselves, but rather on the transformations applied to those pixels. In this way, our method can use optimization constraints that do not refer to pixel values, but rather to the transformations that are applied to the pixel values. In this way, it is possible to obtain a smooth transformation field that can better capture the specific properties of a given scene. Additionally, the presented method includes parameters that can be used to adjust the smoothness of the transformation manifold. We also provide values of the parameters that were found to provide good results for all the tested sequences.

In this paper, we describe such a method and provide experimental results obtained with the use of the method.

In this paper, we first present the formal description of the optimization problem in a simple abstract space and then provide an analytical solution for this space.

From there, we move on to a more complex case, where the problem is defined in a 2D grid, normally used in image and video processing. We present the problem formulation in this 2D grid space and provide a solution for such a space.

The final part of derivation considers the occlusions and non-overlapping parts of the scene, the phenomena usually encountered in practical applications. The solution provided for the 2D grid is therefore modified to deal with occlusions and provide accurate transformation for those parts of the image, even in case of the absence of correspondence data.

The rest of the paper is organized as follows: Section 2 defines formally the actual problem, Section 3 provides an analytical solution, Section 4 describes the same problem in a 2D grid that is commonly used in digital images, Section 5 provides the solution for a 2D grid, Section 6 provides information about the application for occluded parts of the virtual image, Section 7 provides experimental results performed with the use of the described method, and Section 8 summarizes the entire paper with conclusions.

2. Problem Definition

Let us assume we have two images, left and right, that represent the same viewpoint of a scene. Let us assume that $L^p={\left[\begin{array}{cccccc}{l}_1^p& l_2^p& \cdots & l_i^p& \cdots & l_n^p\end{array}\right]}^T$ and $R^p={\left[\begin{array}{cccccc}{r}_1^p& r_2^p& \cdots & r_i^p& \cdots & r_n^p\end{array}\right]}^T$ are n-component vectors of colors of pixel p in the left and right images. The n denotes the number of components in an image increased by 1 since the operations are performed in homogenous coordinates, which offer more flexibility over regular cartesian coordinates. Therefore, commonly n will be 4 for homogenous coordinates of RGB color space.

We would like to estimate a manifold of per-pixel transformation $A^p$ such that for all pixels p

$$R^p=A^p·L^p$$

(1)

where

$$A^p=\left[\begin{array}{cccccc}{a}_{11}^p& a_{12}^p& \cdots & a_{1j}^p& \cdots & a_{1n}^p\\ a_{21}^p& a_{22}^p& \cdots & a_{2j}^p& \cdots & a_{2n}^p\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{i1}^p& a_{i2}^p& \cdots & a_{ij}^p& \cdots & a_{in}^p\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{n1}^p& a_{n2}^p& \cdots & a_{nj}^p& \cdots & a_{nn}^p\end{array}\right]$$

(2)

For an RGB image, the matrix $A^p$ will be a 4 × 4 (since we are operating in homogenous coordinates, as explained before), and there is a p such matrix, one for every pixel in the image (p is the number of pixels in the image). Please note that at this stage, the matrix is agnostic to the way the pixels are organized in the image since p is merely an index.

In the further part of the derivation, we will continue to use n as the size of the matrix A^p since the method can be applied not only to RGB images but potentially also to multispectral and hyperspectral images.

The problem considered in this paper is to find the transformation $A^p$ for a given pair of images, e.g., coming from a pair of cameras in stereoscopic applications.

3. Derivation of the Solution

Because the system of Equation (1) for every pixel $p$ is under-determined, we can define the minimization problem with a goal function $S$. We can transform color $L^p$ of left image pixel $p$ through $A^p$ as close as possible to the color $R^p$ of the right image pixel with the same pixel index $p$

$$S_1={\displaystyle \sum _p}{‖A^p·L^p-R^p‖}^2$$

(3)

where $‖·‖$ is the Euclidean norm.

Moreover, we can have a smooth manifold $A^p$, which means that the two transformations $A^p$ and $A^q$ for two neighboring pixels $p$ or $q$ should be similar. It means that both transformations applied to one (any) of those pixels should give the same result (or at least as close as possible). We denote $o$ as any of the two pixels $p$ or $q$

$$S_2={\displaystyle \sum _{p,q}}{‖A^p·L^o-A^q·L^o‖}^2$$

(4)

So, finally, we can minimize the difference (3). We compute the derivative of (3) with respect to every element $a_{ij}^p$ of manifold $A^p$

$${\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({\displaystyle {\sum }_{p^{\prime }}}{‖A^{p\prime }·L^{p\prime }-R^{p\prime }‖}^2\right)}{\partial a_{ij}^p}}$$

(5)

All derivatives with respect to $a_{ij}^p$ from the sum in (5) except $p^{\prime }=p$ are equal to $0$ from which follows that

$${\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({‖A^p·L^p-R^p‖}^2\right)}{\partial a_{ij}^p}}$$

(6)

Lemma 1.

Assume some vector $B={\left[\begin{array}{cccccc}{b}_1& b_2& \cdots & b_i& \cdots & b_n\end{array}\right]}^T$ then the square of Euclidean norm is:

$${‖B‖}^2={\left(\sqrt{{\displaystyle \sum _{i=1}^n}{b}_i^2}\right)}^2={\displaystyle \sum _{i=1}^n}{b}_i^2=B^T·B$$

(7)

So, any derivative of a square of Euclidean norm of this vector can be expressed as:

$${\displaystyle \frac{\partial {‖B‖}^2}{\partial ·}}={\displaystyle \frac{\partial \left(B^T·B\right)}{\partial ·}}={\displaystyle \frac{\partial \left({\displaystyle {\sum }_{i=1}^n}{b}_i^2\right)}{\partial ·}}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^n}2·b_i·\partial b_i}{\partial ·}}=2·B^T·{\displaystyle \frac{\partial B}{\partial ·}}$$

(8)

Using Lemma 1 in (6), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({‖A^p·L^p-R^p‖}^2\right)}{\partial a_{ij}^p}}=2·{\left(A^p·L^p-R^p\right)}^T·{\displaystyle \frac{\partial \left(A^p·L^p-R^p\right)}{\partial a_{ij}^p}}\\ =2·{\left(A^p·L^p-R^p\right)}^T·{\displaystyle \frac{\partial A^p}{\partial a_{ij}^p}}·L^p\end{array}$$

(9)

Lemma 2.

Let us define a vector $M_i={\left[\begin{array}{cccccc}0& 0& \cdots & 1& \cdots & 0\end{array}\right]}^T$ in which all components are 0 except for $i$-th, which is 1. Now, we can calculate the derivative of matrix $A$ with respect to its element $a_{ij}$

$$\begin{array}{l}{\displaystyle \frac{\partial A}{\partial a_{ij}}}={\displaystyle \frac{\partial \left[\begin{array}{cccccc}{a}_{11}& a_{12}& \cdots & a_{1j}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2j}& \cdots & a_{2n}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{i1}& a_{i2}& \cdots & a_{ij}& \cdots & a_{in}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nj}& \cdots & a_{nn}\end{array}\right]}{\partial a_{ij}}}\\ =\left[\begin{array}{cccccc}{\displaystyle \frac{\partial a_{11}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{12}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{1j}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{1n}}{\partial a_{ij}}}\\ {\displaystyle \frac{{\partial a}_{21}}{\partial a_{ij}}}& {\displaystyle \frac{{\partial a}_{22}}{{\partial a}_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{2j}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{2n}}{\partial a_{ij}}}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\partial a_{i1}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{i2}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{ij}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{in}}{\partial a_{ij}}}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\partial a_{n1}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{n2}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{nj}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{nn}}{\partial a_{ij}}}\end{array}\right]\end{array}$$

(10a)

Because all matrix components are independent of  $a_{ij}$ except $a_{ij}$ element, so

$$\begin{array}{l}{\displaystyle \frac{\partial A}{\partial a_{ij}}}=\left[\begin{array}{cccccc}{\displaystyle \frac{\partial a_{11}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{12}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{1j}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{1n}}{\partial a_{ij}}}\\ {\displaystyle \frac{\partial a_{21}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{22}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{2j}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{2n}}{\partial a_{ij}}}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\partial a_{i1}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{i2}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{ij}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{in}}{\partial a_{ij}}}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\partial a_{n1}}{\partial a_{ij}}}& {\displaystyle \frac{\partial a_{n2}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{nj}}{\partial a_{ij}}}& \cdots & {\displaystyle \frac{\partial a_{nn}}{\partial a_{ij}}}\end{array}\right]=\left[\begin{array}{cccccc}0& 0& \cdots & 0& \cdots & 0\\ 0& 0& \cdots & 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 1& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& \cdots & 0\end{array}\right]\\ =\left[\begin{array}{c}0\\ 0\\ ⋮\\ 1\\ ⋮\\ 0\end{array}\right]·{\left[\begin{array}{c}0\\ 0\\ ⋮\\ 1\\ ⋮\\ 0\end{array}\right]}^T=M_i·M_j^T\end{array}$$

(10b)

So, using Lemma 2 in (9), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}=2·{\left(A^p·L^p-R^p\right)}^T·{\displaystyle \frac{\partial A^p}{\partial a_{ij}^p}}·L^p\\ =2·{\left(A^p·L^p-R^p\right)}^T·M_i·M_j^T·L^p\\ =2·M_i^T·\left(A^p·L^p-R^p\right)·M_j^T·L^p\\ =2·\left(M_i^T·A^p·L^p-M_i^T·R^p\right)·M_j^T·L^p\end{array}$$

(11)

Lemma 3.

Let us define a vector $M_i={\left[\begin{array}{cccccc}0& 0& \cdots & 1& \cdots & 0\end{array}\right]}^T$ in which all components are 0 except for $i$-th, which is 1. Now, we can define the selection of $i$-th component as:

$$M_i^T·L=\left[\begin{array}{cccccc}0& 0& \cdots & 1& \cdots & 0\end{array}\right]·{\left[\begin{array}{cccccc}{l}_1& l_2& \cdots & l_i& \cdots & l_n\end{array}\right]}^T=l_i$$

(12)

where L is an $n$ component vector.

$$M_i^T·A=\left[\begin{array}{cccccc}0& 0& \cdots & 1& \cdots & 0\end{array}\right]·\left[\begin{array}{cccccc}{a}_{11}& a_{12}& \cdots & a_{1j}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2j}& \cdots & a_{2n}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{i1}& a_{i2}& \cdots & a_{ij}& \cdots & a_{in}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nj}& \cdots & a_{nn}\end{array}\right]=a_{i:}$$

(13)

where $A$ is $n$ by $n$ matrix and $a_{i:}$ mean $i$-th row of a matrix $A$.

Using Lemma 3 on (11), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}=2·\left(M_i^T·A^p·L^p-M_i^T·R^p\right)·M_j^T·L^p=2·\left(a_{i:}^p·L^p-r_i^p\right)·l_j^p\\ =2·\left({\displaystyle {\displaystyle \sum }_k}{a}_{ik}^p·l_k^p-r_i^p\right)·l_j^p=2·{\displaystyle {\displaystyle \sum }_k}{a}_{ik}^p·l_k^p·l_j^p-2·r_i^p·l_j^p\end{array}$$

(14)

Similarly, we would like to minimize (4). So, we compute the derivative of (4) with respect to every element $a_{ij}^p$ of manifold $A^p$

$${\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({\displaystyle {\sum }_{p^{\prime },q\prime }}{‖A^{p\prime }·L^o-A^{q\prime }·L^o‖}^2\right)}{\partial a_{ij}^p}}$$

(15)

All derivatives with respect to $a_{ij}^p$ from the sum in (15) except for the pair $p^{\prime }=p$ and $q^{\prime }=q$ are equal to $0$, so

$${\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({‖A^p·L^o-A^q·L^o‖}^2\right)}{\partial a_{ij}^p}}$$

(16)

Using Lemma 1 in (16), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}={\displaystyle \frac{\partial \left({‖A^p·L^o-A^q·L^o‖}^2\right)}{\partial a_{ij}^p}}=2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial \left(A^p·L^o-A^q·L^o\right)}{\partial a_{ij}^p}}\\ =2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial A^p}{\partial a_{ij}^p}}·L^o\end{array}$$

(17)

Using Lemma 2 in (17), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}=2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial A^p}{\partial a_{ij}^p}}·L^o=2·{\left(A^p·L^o-A^q·L^o\right)}^T·M_i·M_j^T·L^o\\ =2·M_i^T·\left(A^p·L^o-A^q·L^o\right)·M_j^T·L^o=2·\left(M_i^T·A^p·L^o-M_i^T·A^q·L^o\right)·M_j^T·L^o\end{array}$$

(18)

Using Lemma 3 on (18), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}=2·\left(M_i^T·A^p·L^o-M_i^T·A^q·L^o\right)·M_j^T·L^o\\ =2·\left(a_{i:}^p·L^o-a_{i:}^q·L^o\right)·l_j^o\\ =2·\left({\displaystyle \sum _k}{a}_{ik}^p·l_k^o-{\displaystyle \sum _k}{a}_{ik}^q·l_k^o\right)·l_j^o\\ =2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^o·l_j^o-2·{\displaystyle \sum _k}{a}_{ik}^q·l_k^o·l_j^o\end{array}$$

(19)

Similarly, we compute the derivative of (4) with respect to every element $a_{ij}^q$ of manifold $A^p$

$${\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}={\displaystyle \frac{\partial \left({\displaystyle {\sum }_{p^{\prime },q\prime }}{‖A^{p\prime }·L^o-A^{q\prime }·L^o‖}^2\right)}{\partial a_{ij}^q}}$$

(20)

All derivatives with respect to $a_{ij}^q$ from the sum in (20) except for the pair $p^{\prime }=p$ and $q^{\prime }=q$ are equal to $0,$ so

$${\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}={\displaystyle \frac{\partial \left({‖A^p·L^o-A^q·L^o‖}^2\right)}{\partial a_{ij}^q}}$$

(21)

Using Lemma 1 in (21), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}={\displaystyle \frac{\partial \left({‖A^p·L^o-A^q·L^o‖}^2\right)}{\partial a_{ij}^q}}=2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial \left(A^p·L^o-A^q·L^o\right)}{\partial a_{ij}^q}}\\ =-2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial A^q}{\partial a_{ij}^q}}·L^o\end{array}$$

(22)

Using Lemma 2 in (22), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}=-2·{\left(A^p·L^o-A^q·L^o\right)}^T·{\displaystyle \frac{\partial A^q}{\partial a_{ij}^q}}·L^o=-2·{\left(A^p·L^o-A^q·L^o\right)}^T·M_i·M_j^T·L^o\\ =-2·M_i^T·\left(A^p·L^o-A^q·L^o\right)·M_j^T·L^o=-2·\left(M_i^T·A^p·L^o-M_i^T·A^q·L^o\right)·M_j^T·L^o\end{array}$$

(23)

Using Lemma 3 on (23), we have

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}=-2·\left(M_i^T·A^p·L^o-M_i^T·A^q·L^o\right)·M_j^T·L^o=-2·\left(a_{i:}^p·L^o-a_{i:}^q·L^o\right)·l_j^o\\ =-2·\left({\displaystyle \sum _k}{a}_{ik}^p·l_k^o-{\displaystyle \sum _k}{a}_{ik}^q·l_k^o\right)·l_j^o\\ =-2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^o·l_j^o+2·{\displaystyle \sum _k}{a}_{ik}^q·l_k^o·l_j^o\end{array}$$

(24)

In order to estimate manifold $A^p$, we create a system of equations for pixel $p$

$$\begin{array}{c}{\displaystyle \frac{\partial S_1}{\partial a_{11}^p}}=2·{\displaystyle \sum _k}{a}_{1k}^p·l_k^p·l_1^p-2·r_1^p·l_1^p=0\\ {\displaystyle \frac{\partial S_1}{\partial a_{12}^p}}=2·{\displaystyle \sum _k}{a}_{1k}^p·l_k^p·l_2^p-2·r_1^p·l_2^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{1j}^p}}=2·{\displaystyle \sum _k}{a}_{1k}^p·l_k^p·l_j^p-2·r_1^p·l_j^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{1n}^p}}=2·{\displaystyle \sum _k}{a}_{1k}^p·l_k^p·l_n^p-2·r_1^p·l_n^p=0\\ {\displaystyle \frac{\partial S_1}{\partial a_{21}^p}}=2·{\displaystyle \sum _k}{a}_{2k}^p·l_k^p·l_1^p-2·r_2^p·l_1^p=0\\ {\displaystyle \frac{\partial S_1}{\partial a_{22}^p}}=2·{\displaystyle \sum _k}{a}_{2k}^p·l_k^p·l_2^p-2·r_2^p·l_2^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{2j}^p}}=2·{\displaystyle \sum _k}{a}_{2k}^p·l_k^p·l_j^p-2·r_2^p·l_j^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{2n}^p}}=2·{\displaystyle \sum _k}{a}_{2k}^p·l_k^p·l_n^p-2·r_2^p·l_n^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{i1}^p}}=2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^p·l_1^p-2·r_i^p·l_1^p=0\\ {\displaystyle \frac{\partial S_1}{\partial a_{i2}^p}}=2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^p·l_2^p-2·r_i^p·l_2^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{ij}^p}}=2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^p·l_j^p-2·r_i^p·l_j^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{in}^p}}=2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^p·l_n^p-2·r_i^p·l_n^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{n1}^p}}=2·{\displaystyle \sum _k}{a}_{nk}^p·l_k^p·l_1^p-2·r_n^p·l_1^p=0\\ {\displaystyle \frac{\partial S_1}{\partial a_{n2}^p}}=2·{\displaystyle \sum _k}{a}_{nk}^p·l_k^p·l_2^p-2·r_n^p·l_2^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{nj}^p}}=2·{\displaystyle \sum _k}{a}_{nk}^p·l_k^p·l_j^p-2·r_n^p·l_j^p=0\\ ⋮\\ {\displaystyle \frac{\partial S_1}{\partial a_{nn}^p}}=2·{\displaystyle \sum _k}{a}_{nk}^p·l_k^p·l_n^p-2·r_n^p·l_n^p=0\end{array}$$

(25)

We can express (25) as

$${LL}^p·{AA}^p={RL}^p$$

(26)

where ${AA}^p$ is a vector composed of all coefficients of the matrix $A^p$, and vector ${RL}^p$ is defined as follows

$${AA}^p=\left[\begin{array}{c}{a}_{11}^p\\ a_{12}^p\\ ⋮\\ a_{1j}^p\\ ⋮\\ a_{1n}^p\\ a_{21}^p\\ a_{22}^p\\ ⋮\\ a_{2j}^p\\ ⋮\\ a_{2n}^p\\ ⋮\\ a_{i1}^p\\ a_{i2}^p\\ ⋮\\ a_{ij}^p\\ ⋮\\ a_{in}^p\\ ⋮\\ a_{n1}^p\\ a_{n2}^p\\ ⋮\\ a_{nj}^p\\ ⋮\\ a_{nn}^p\end{array}\right]{RL}^p=\left[\begin{array}{c}2·r_1^p·l_1^p\\ 2·r_1^p·l_2^p\\ ⋮\\ 2·r_1^p·l_j^p\\ ⋮\\ 2·r_1^p·l_n^p\\ 2·r_2^p·l_1^p\\ 2·r_2^p·l_2^p\\ ⋮\\ 2·r_2^p·l_j^p\\ ⋮\\ 2·r_2^p·l_n^p\\ ⋮\\ 2·r_i^p·l_1^p\\ 2·r_i^p·l_2^p\\ ⋮\\ 2·r_i^p·l_j^p\\ ⋮\\ 2·r_i^p·l_n^p\\ ⋮\\ 2·r_n^p·l_1^p\\ 2·r_n^p·l_2^p\\ ⋮\\ 2·r_n^p·l_j^p\\ ⋮\\ 2·r_n^p·l_n^p\end{array}\right]$$

(27)

And the matrix ${LL}^p$ is a block-based matrix composed of ${ll}^p$ defined as follows

$${ll}^p=\left[\begin{array}{cccccc}2·l_1^p·l_1^p& 2·l_2^p·l_1^p& \cdots & 2·l_k^p·l_1^p& \cdots & 2·l_n^p·l_1^p\\ 2·l_1^p·l_2^p& 2·l_2^p·l_2^p& \cdots & 2·l_k^p·l_2^p& \cdots & 2·l_n^p·l_2^p\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 2·l_1^p·l_j^p& 2·l_2^p·l_j^p& \cdots & 2·l_k^p·l_j^p& \cdots & 2·l_n^p·l_j^p\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 2·l_1^p·l_n^p& 2·l_2^p·l_n^p& \cdots & 2·l_k^p·l_n^p& \cdots & 2·l_n^p·l_n^p\end{array}\right]$$

(28)

$${LL}^p=\left[\begin{array}{cccccc}{ll}^p& 0& \cdots & 0& \cdots & 0\\ 0& {ll}^p& \cdots & 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & {ll}^p& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& \cdots & {ll}^p\end{array}\right]$$

(29)

For a manifold composed of $P$ pixels, we can define the system of Equation (25) using the Formula (26) as

$$\mathcal{L}\mathcal{L}·ÅÅ=\mathcal{R}\mathcal{L}$$

(30)

where $ÅÅ$ and $\mathcal{R}\mathcal{L}$ are a concatenation of ${AA}^p$ and ${RL}^p$ for all pixels in the image

$$ÅÅ=\left[\begin{array}{c}{AA}^1\\ {AA}^2\\ ⋮\\ {AA}^p\\ ⋮\\ {AA}^P\end{array}\right]\mathcal{R}\mathcal{L}=\left[\begin{array}{c}{RL}^1\\ {RL}^2\\ ⋮\\ {RL}^p\\ ⋮\\ {RL}^P\end{array}\right]$$

(31)

And $\mathcal{L}\mathcal{L}$ can be defined as a block matrix composed of ${LL}^p$

$$\mathcal{L}\mathcal{L}=\left[\begin{array}{cccccc}{LL}^1& 0& \cdots & 0& \cdots & 0\\ 0& {LL}^2& \cdots & 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & {LL}^p& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& \cdots & {LL}^P\end{array}\right]$$

(32)

Similarly, in order to estimate manifold $A^p$ based on $S_2$, we create a system of Equations (19) and (24) for pixel $p$ and q

$$\begin{array}{l}{\displaystyle \frac{\partial S_2}{\partial a_{ij}^p}}=2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^o·l_j^o-2·{\displaystyle \sum _k}{a}_{ik}^q·l_k^o·l_j^o=0\\ {\displaystyle \frac{\partial S_2}{\partial a_{ij}^q}}=2·{\displaystyle \sum _k}{a}_{ik}^q·l_k^o·l_j^o-2·{\displaystyle \sum _k}{a}_{ik}^p·l_k^o·l_j^o=0\end{array}$$

(33)

Using a definition of ${LL}^p$ given in (29), we can rewrite Equation (33) as

$$\left[\begin{array}{cc}{LL}^o& -{LL}^o\\ -{LL}^o& {LL}^o\end{array}\right]·\left[\begin{array}{c}{AA}^p\\ {AA}^q\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

(34)

The exact formulation of $\mathcal{L}\mathcal{L}$ matrix for all of the $P$ pixels depends strongly on the relationship between the position of pixels $p$ and $q$.

4. Formulation in 2D Pixel Grid

Suppose we have an image of size $W$ by $H,$ and we can indicate the pixel $p$ position by its $u,v$ coordinates in the image. Moreover, suppose that we introduce 2 smoothness constraints: one vertical and one horizontal. It means that pixel $p_v$ and $q_v$ have two neighboring pixels in which the $v$ coordinate differs only by 1.

$$\begin{array}{cc}{p}_v=u,v& q_v=u,v+1\end{array}$$

(35)

Similarly, pixel $p_h$ and $q_h$ have two neighboring pixels whose u coordinate differs only by 1.

$$\begin{array}{cc}{p}_h=u,v& q_h=u+1,v\end{array}$$

(36)

Moreover, we will assume that condition (34) is reflective and must hold for both pixels $p$ and $q$, so:

$$\begin{array}{cc}o=p& o=q\end{array}$$

(37)

will lead to

$$\left[\begin{array}{cc}{LL}^p& -{LL}^p\\ -{LL}^p& {LL}^p\end{array}\right]·\left[\begin{array}{c}{AA}^p\\ {AA}^q\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

(38)

$$\left[\begin{array}{cc}{LL}^q& -{LL}^q\\ -{LL}^q& {LL}^q\end{array}\right]·\left[\begin{array}{c}{AA}^p\\ {AA}^q\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

(39)

Combining both (38) and (39) together results in

$$\left[\begin{array}{cc}{LL}^p+{LL}^q& -\left({LL}^p+{LL}^q\right)\\ -\left({LL}^p+{LL}^q\right)& {LL}^p+{LL}^q\end{array}\right]·\left[\begin{array}{c}{AA}^p\\ {AA}^q\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

(40)

After making such assumptions, we can provide an explicit form of $ÅÅ$, $\mathcal{R}\mathcal{L}$ vectors, and $\mathcal{L}\mathcal{L}$ matrix for any given pixel $p=u,v$ based on (31), (32), and (40). As the conditions $S_1$ and $S_2$ need to be satisfied simultaneously, we introduce two coefficients $\alpha$ and $\beta$ to control the strength of the smoothness prior to $S_2$ in both directions on the 2D grid. The values of coefficients $\alpha$ and $\beta$ were experimentally adjusted for universally good results among the tested sequences. The value of 0.1 was found to work well in our experiments, independent of the sequence contents and scene complexity. It needs to be stressed, however, that different resolutions of the sequences may require a slight modification of the values, but we found during the experiments that the method is not sensitive to the actual value of the coefficients $\alpha$ and $\beta$.

$${ÅÅ}^{p=u,v}=\left[\begin{array}{c}0\\ 0\\ ⋮\\ 0\\ {AA}^{p=u,v}\\ 0\\ 0\\ ⋮\\ 0\\ 0\\ 0\\ ⋮\\ 0\end{array}\right]{\mathcal{R}\mathcal{L}}^{p=u,v}=\left[\begin{array}{c}0\\ 0\\ ⋮\\ 0\\ {RL}^{p=u,v}\\ 0\\ 0\\ ⋮\\ 0\\ 0\\ 0\\ ⋮\\ 0\end{array}\right]$$

(41)

$${\mathcal{L}\mathcal{L}}^{p=u,v}=\left[\begin{array}{ccccccccccccc}0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& {LL}^{p=u,v}+\alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)+\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& -\alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& 0& \cdots & 0& -\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& \cdots & 0\\ 0& 0& \cdots & 0& -\alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& \alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& -\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& 0& \cdots & 0& \beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\end{array}\right]$$

(42)

The final form of Equation (43) for all of the pixels is just a sum of (41) and (42)

$$\mathcal{L}\mathcal{L}·ÅÅ=\mathcal{R}\mathcal{L}$$

(43)

$$\begin{array}{c}\mathcal{L}\mathcal{L}={\displaystyle \sum _p^P}{\mathcal{L}\mathcal{L}}^p\\ \mathcal{R}\mathcal{L}={\displaystyle \sum _p^P}{\mathcal{R}\mathcal{L}}^p\\ ÅÅ={\displaystyle \sum _p^P}{ÅÅ}^p\end{array}$$

(44)

5. Solution on a 2D Grid

The solution of obtained Equation (43) is simply given by

$$ÅÅ={\mathcal{L}\mathcal{L}}^{-1}·\mathcal{R}\mathcal{L}$$

(45)

However, solving for $ÅÅ$ on an average image is not an easy task. For a typical RGB image of resolution 1920 × 1080 vectors $\mathcal{R}\mathcal{L}$ have the size of $4·4·1920·1080=\mathrm{33,177,600}$ and the square matrix $\mathcal{L}\mathcal{L}$ is of that same size. The main problem is to inverse the $\mathcal{L}\mathcal{L}$ matrix, mainly because it is so enormous.

Fortunately, matrix $\mathcal{L}\mathcal{L}$ is also vastly sparse with most of its coefficients located along the first, second, and W-diagonal (where W stands for the width of the image under consideration). There are many algorithms for efficient inversing (or pseudo-inversing) such huge sparse matrices.

In the experiments further on we have used Transpose-Free Quasi-Minimal Residual Algorithm [25,26,27].

After solving Equation (43) for $ÅÅ,$ we obtained an estimation of the manifold of per-pixel transformation $A^p$ allowing per-pixel color transformation between both of the images.

This allows the transformation of one image into a color space of the other, retaining its local color characteristics. After transformation, depending on the use case, one can directly output the transformed image as the final result, as can be performed in the “flash–no flash” application, merge them via some kind of blending or averaging, or inpaint one of the images by the content of the other or stitch them together to create a larger panorama.

6. Notes on Partially Overlapping Images

In the case of partially overlapping images like in the case of image stitching or view synthesis, a form of the ${\mathcal{L}\mathcal{L}}^{p=u,v}$ matrix and the ${\mathcal{R}\mathcal{L}}^{p=u,v}$ vector needs to be adjusted. Without loss of generality, let us assume that the left image is more complete, i.e., it has pixels in places where the right image does not. So, there are two possibilities: either left and right images do not have a given pixel or just the right pixel is not available. In both cases, for these pixels’ positions, condition $S_1$ cannot be used due to a lack of explicit pixel correspondence. In the second case, depending on the surroundings, we have pixel color values in the left image that can be used in condition $S_2$ for smooth manifold estimation.

For the first case, ${\mathcal{L}\mathcal{L}}^{p=u,v}$ is as shown in (46)

$${\mathcal{L}\mathcal{L}}^{p=u,v}=\left[0\right]$$

(46)

In the second case, the ${\mathcal{L}\mathcal{L}}^{p=u,v}$ is given as (47)

$${\mathcal{L}\mathcal{L}}^{p=u,v}=\left[\begin{array}{ccccccccccccc}0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& \alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)+\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& -\alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& 0& \cdots & 0& -\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& \cdots & 0\\ 0& 0& \cdots & 0& -\alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& \alpha ·\left({LL}^{p=u,v}+{LL}^{q_h=u+1,v}\right)& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& -\beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& 0& \cdots & 0& \beta ·\left({LL}^{p=u,v}+{LL}^{q_v=u,v+1}\right)& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & 0& 0& 0& 0& \cdots & 0& 0& 0& \cdots & 0\end{array}\right]$$

(47)

7. Experiments

The foundation of our experiments was the reference virtual view synthesis software VSRS [28]. It should be noted that VSRS does not include any color correction algorithm. Our algorithm was implemented within this software package.

We modified the post-processing only after the forward depth map projection step and the backward texture warping step had been carried out, and before the inpainting step.

For our testing, we employed several well-known and recognizable multiview test sequences equipped with high-quality depth maps. This set includes linear and arc multiview content. Used multiview test sequences exhibit a whole range of effects like ambient occlusions, nonuniform lighting, shadows, and non-Lambertian reflections.

Table 1. Test sequences used in the experiment.

Sequence	Resolution	Left Reference View	Right Reference View	Virtual View
Poznań Street [29]	1920 × 1080	3	5	4
Poznan Fencing 2 [30]	1920 × 1080	4	6	5
Poznan Blocks [30]	1920 × 1080	4	6	5
Poznan Carpark [29]	1920 × 1080	3	5	4
Ballet [31]	1024 × 768	3	5	4
Breakdancer [31]	1024 × 768	3	5	4
Soccer [32]	1392 × 1136	3	5	4

lists the sequences used in the experiment.

We rendered virtual views using the original VSRS and VSRS implementing our method. The virtual views were generated from data from the two views specified as the right and left views in Table 1. The set of data consists of the images, depth maps, and camera parameters. The virtual view matched the position of the third camera used during sequence acquisition.

The quality of the resulting virtual view has been measured as a luminance PSNR value of the virtual view generated with respect to the view recorded by the real camera at the same spatial position. The numerical results are presented in

Table 2. Quality (PSNR) of the virtual views.

	Quality (PSNR) Without Color Correction [dB]	Quality (PSNR) with Proposed Color Correction [dB]	Quality (PSNR) Improvement [dB]
Poznan Street	35.48	36.23	0.75
Poznan Fencing	29.62	31.23	1.61
Poznan Blocks	29.60	32.43	2.83
Poznan Carpark	36.87	37.02	0.15
Ballet	31.05	31.94	0.89
Breakdancer	31.47	31.83	0.46
Soccer	35.00	36.03	1.03

. A demonstration of the improvement provided by our method over the original method is shown in Figure 1, where details of the virtual views are compared.

It can clearly be seen that the presented method significantly improves the quality of a virtual view by providing consistent colors of the pixels from different views. The artifacts stemming from slight differences between the color characteristics of the cameras and/or lighting conditions are reduced significantly.

The PSNR value measured against a real view is a commonly used measure of quality for virtual view synthesis research. For color correction algorithms, however, a more significant figure of merit would be the comparison between the virtual view synthesized only from the left and only from the right view.

Table 3. Mean values of SSD for the virtual view generated from the left view compared to the virtual view generated from the right view.

Sequence	Without Color Correction	Global Color Transformation From [33]	Proposed Per-Pixel Color Transformation
Poznań Street	1.6496	1.4576	0.0304
Poznan Fencing 2	4.4137	4.0283	0.0504
Poznan Blocks	2.0739	1.9387	0.0977
Poznan Carpark	1.7684	1.5984	0.0779
Ballet	1.3975	1.1293	0.0600
Breakdancer	1.4839	1.3938	0.2880
Soccer	1.1830	1.0283	0.2735

shows the values of the error (measured as a mean value of SSD for pixel values) for the virtual view generated from the left view calculated against the virtual view generated from the right view for three cases: no color correction applied, the global color correction applied according to the method described in [33], and the proposed method. A visual comparison of the differences between the virtual view generated from the left view and the virtual view generated from the right view is presented in Figure 2 and Figure 3. The proposed method outperforms significantly the global color correction method.

In order to test our method in a flash/no-flash scenario, we use the images from [4], as supplied in [34]. The proposed method of estimating per-pixel color transfer manifold can be used to correct images captured with flash to its low-light ambient counterpart, to match the no-flash colors (Figure 4).

As can be seen, the proposed method is able to perform the color correction of the flash image so that it matches the colors of the image taken without flash. It can be seen that the artifacts common to images taken with flash, as non-Lambertian reflections are correctly processed.

8. Summary

In the paper, we have presented a new formulation of the per-pixel color calibration between images. The presented method can be applied to images with an arbitrary number of channels due to the flexibility and generalization provided by our derivation. The proposed algorithm is based on per-pixel manifold color transformation estimation. The proposed algorithm estimated a smooth manifold of color transformations across the whole image, even in the case when not all pixels have correspondence. The method allows for color profile mapping in difficult areas, like occluded regions of images, in shadows, and in the presence of reflections and a non-uniform lighting. The proposed method provides a gain in objective quality measure PSNR of 1.10 dB on average over seven different test sequences. The proposed method provided improvement for all tested sequences; the quality increase was in the range from 2.83 dB to 0.15 dB. The explanation of the differences may be attributed qualitatively to the different lighting conditions. Smaller improvements were observed for outdoor sequences recorded during a cloudy day, with no bright sun, and for sequences with uniform ambient lighting. For sequences with artificial light, the improvement was significantly larger. The subjective quality increase is also significant, as presented by the images in Figure 1 and Figure 4.

Much more improvement due to the use of the proposed method is observed when SSD between the virtual view synthesized from the left view is compared to the view synthesized from the right view. Here, when compared to the no color correction scenario, the reduction was more than 20-fold, with a maximum of 87-fold SSD reduction. When compared to a scenario where a competing method [33] is used, the SSD reduction is from 19.8-fold to 79.9-fold.

The presented method is envisaged to be applied as a preprocessing step in a post-production phase, since the computational complexity is significant, the most time-consuming part being the large sparse matrix inversion. While real-time image processing was beyond the scope of this work, it is worth noting that faster matrix inversion methods exist, which would reduce the computational complexity of our method.

The method significantly improves the quality of the virtual view synthesized from the multi-camera setups by applying color correction to all original views. It can be used in immersive video applications, where it can greatly reduce the color mismatch between camera views. It is also useful for single images in the scenario, flash–no flash images, as evidenced in the experiments.