DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces

Abstract

This paper presents a photometric stereo (PS) method based on the dichromatic reflectance model (DRM) using colour images. The proposed method estimates surface orientations for surfaces with non-Lambertian reflectance using diffuse-specular separation and contains two steps. The first step, referred to as diffuse-specular separation, initialises surface orientations in a specular invariant colour subspace and further separates the diffuse and specular components in the RGB space. In the second step, the surface orientations are refined by first initialising specular parameters via solving a log-linear regression problem owing to the separation and then fitting the DRM using Levenburg-Marquardt algorithm. Since reliable information from diffuse reflection free from specularities is adopted in the initialisations, the proposed method is robust and feasible with less observations. At pixels where dense non-Lambertian reflectances appear, signals from specularities are exploited to refine the surface orientations and the additionally acquired specular parameters are potentially valuable for more applications, such as digital relighting. The effectiveness of the newly proposed surface normal refinement step was evaluated and the accuracy in estimating surface orientations was enhanced around $30\%$ on average by including this step. The proposed method was also proven effective in an experiment using synthetic input images comprised of twenty-four different reflectances of dielectric materals. A comparison with nine other PS methods on five representative datasets further prove the validity of the proposed method.

1. Introduction

Photometric stereo (PS) estimates surface orientations using images captured from a fixed viewpoint under various illuminations and is especially powerful in acquisition of fine surface details at pixel level [1]. Surface orientation is important in a variety of fields, such as geometry segmentation for three-dimensional (3D) object recognition [2] and digital re-rendering in computer graphics. Surface geometries, which can be obtained via integrating surface orientations, have also been proven useful for applications, such as industrial quality control and reverse engineering. Due to the strength of PS and the importance of surface orientation acquisition, PS has drawn increasing interests since its debut [3]. However, making PS for a general real scene remains challenging due to the diverse reflectance properties of different materials that appear non-Lambertian [4]. This has given rise to the need for reliable estimation of surface orientations for a wide range of non-Lambertian reflectance, which essentially requires a proper imaging photometry model characterizing the forward problem and a subsequent PS method that inversely derives surface orientations.

Existing PS methods dealing with non-Lambertian reflectance can be classified into three categories. The first approximates surface reflectance using analytical bidirectional reflectance distribution function (BRDF) [5] and formulates the estimation of surface orientations as a nonlinear fitting problem. Nayar et al. [6] derived surface orientations using the Torrance-Sparrow BRDF [7] and incorporated extended sources to ensure sufficient information from specularities. Georghiades [8] inverted the same BRDF to simultaneously estimate surface orientations and resolve the generalized bas-relief ambiguity. Goldman et al. [9] assumed that general material reflectance can be represented by a convex combination of fundamental materials characterized by the Ward BRDF [10] and recovered surface orientations, fundamental material BRDFs and weight maps simultaneously for further scene editing purpose. Methods in this category exploiting information from surface reflectance are capable to derive not only surface orientations but also the other parameters in the analytical BRDFs, allowing more functionalities, such as digital relighting and material classification. However, due to the nonlinearity of analytical BRDFs and larger number of parameters to be estimated, these methods are sensitive to initialisations, numerically unstable under heavily corrupted outliers (e.g., shadows), and inapplicable when the number of observations is limited.

The second category of methods infers surface orientations through adopting the general properties of BRDF, such as isotropy, monotonicity and reciprocity. Alldrin et al. [11] developed a non-parametric PS method using bi-variate approximation of the isotropy property. Higo et al. [12] analyzed the general BRDF constraints and employed the BRDF properties of monotonicity, isotropy and visibility to vote for the most possible surface orientations for single-lobed reflectance. Shi et al. [13] proposed a bi-polynomial representation for low-frequency reflectance that was especially adaptable to the inverse problem as PS, while Ikehata et al. [14] developed another general isotropic BRDF as sum of lobes with unknown center directions. Methods in this category capitalize on the most fundamental properties of BRDF and therefore have the potential to deal with a broader range of reflectance. However, these methods are only capable to derive surface orientations with limited other functionalities and require an even larger set of observations compared with the first category. To widen the applicability, recent years have seen non-parametric BRDFs based on machine learning. Santo et al. [15,16] used a deep neural network for the first time whereas Taniai and Maehara [17] estimated surface normals and BRDFs by unsupervised learning. Ikehata [18] estimated surface normals more straightforward by deriving the so-called observation maps and using convolutional neural networks. Such data-driven approaches clearly improves the accuracy as observations directly create the BRDF or estimate surface normals. The use of a significantly large set of observations, which is key to the accuracy, and the expensive training effort using the observations make the approaches outside the scope of this paper.

Methods in the third category assume that non-Lambertian effects appear sparsely among observations and treat them as outliers. A substantial corpus of methods rely on robust statistical techniques for outlier rejection. Wu et al. [19] formulated the PS with outlier rejection as a global rank minimization problem, while Ikehata et al. [20] employed sparse Bayesian regression instead. Barsky et al. [21] initiated another line of PS researches using colour images where they first identified the significance of using specular colour in the dichromatic reflectance model (DRM) [22] for specularity rejection. Using the cue from known specular colour under the same theoretical foundation, Zickler et al. [23] derived a PS method in a novel two-dimensional (2D) specular invariant colour subspace. The major advantages of methods in the third category are their robustness and requirement for less images, while they are inefficient in the presence of dense non-Lambertian effects.

This paper presents a colour PS method to estimate surface orientations using diffuse-specular separation dealing with non-Lambertian reflectance. The proposed method models the colour imaging photometry using DRM as the forward problem and inverts it following a two-step procedure, the diffuse-specular separation and the surface normal refinement. The first step, initialising surface orientations using known specular colour similarly to [23], separates the diffuse and specular components in the RGB space and identifies outliers to reject in the UV space. In the second step, the parameters characterizing the specularity are initialised via solving a log-linear regression problem and the surface orientations are finally refined by fitting the nonlinear DRM using Levenburg-Marquardt algorithm (LMA). The proposed method robustly initialises surface orientations in a specular-free colour subspace and further fits for the nonlinear DRM with a newly derived parameter initialisation strategy. The proposed method preserves the advantage of robustness as the third category of methods, while it also benefits from the separated specular component to tackle with dense non-Lambertian reflectance as the first category. Furthermore, DRM parameters besides the surface orientations can be additionally obtained where dense non-Lambertian reflectance appears, making more potential applications feasible, such as digital relighting and material classification.

The rest of the paper is organized as follows. The next section presents the problem formulation of colour PS incorporating DRM and the relevant PS methods. Section 3 first overviews the flow and original contribution of the proposed colour DRM-based PS method and then, elucidates the specific two steps in the subsequent subsections. It follows the comprehensive evaluations on surface orientation estimation using both synthetic and real images in Section 4, while the last section summarizes the conclusions and proposes the future works.

2. Colour PS Incorporating Dichromatic Reflectance Model

2.1. Generic Colour PS Problem Formulation

Figure 1 shows the schematic diagram of the hardware components of the sensor and the necessary data for the generic colour PS. The hardware components are an RGB camera and a set of directional point sources whose positions are priorly known. Assume that ambient light is completely blocked, the colour PS requires that the RGB camera captures a colour image for the target surface with one source lit at a time. Given the N colour images illuminated under N different sources, the objective of the colour PS is to identify surface orientations pixel-wise. More specifically, given the colour image irradiance measurements at pixel $\left(i,j\right)$, ${\left\{{\mathbf{e}}_k^{i,j*}\right\}}_{k\in \left[1,N\right]}$, under the unit illumination directions, ${\left\{{\overline{\mathbf{l}}}_k^{i,j}\right\}}_{k\in \left[1,N\right]}$, estimated from the light positions, the colour PS aims to derive the unit surface normal, ${\overline{\mathbf{n}}}^{i,j}$, by minimizing the sum of N corresponding square residuals:

$${ϵ}^{i,j}=\sum _{k=1}^N{\left[{\left({\mathbf{e}}_k^{i,j*}\right)}^T\mathbf{w}-f\left({\overline{\mathbf{n}}}^{i,j},{\overline{\mathbf{l}}}_k^{i,j}\right)\right]}^2\to \underset{{\overline{\mathbf{n}}}^{i,j}}{min}.$$

(1)

where $\mathbf{w}$ is a vector of weights for colour channels whereas f is a forward function of ${\overline{\mathbf{n}}}^{i,j}$ having ${\overline{\mathbf{l}}}_k^{i,j}$ as parameters, which corresponds to the weighted measurements. Note that $\overline{\left(\right)}$ and ${\left(\right)}^{*}$ mean the unit version and the measurement of $\left(\right)$ respectively throughout the paper.

Figure 1. Schematic diagram for generic colour PS.

2.2. Colour Image Formation Model

Figure 2 shows the coordinate setups that are necessary for the colour PS to model the imaging geometry and the light configuration. The coordinate frames are the 3D camera frame, $\left\{C\right\}$, the 2D pixel frame, $\left\{P\right\}$, and the world frame, $\left\{W\right\}$, where its XY plane is known and referred to as the reference plane. These coordinate frames follow the convention given in [24]. A 3D surface point, ${}^{\left\{C\right\}}\tilde{\mathbf{X}}$, is projected to be a 2D image point, ${}^{\left\{P\right\}}\mathbf{x}$, in the RGB camera using the perspective projection as:

$${}^{\left\{P\right\}}\mathbf{x}=\frac{1}{{}^{\left\{C\right\}}Z}{\mathbf{M}}_p\widehat{\mathbf{K}}{}^{\left\{C\right\}}\tilde{\mathbf{X}}=\frac{1}{{}^{\left\{C\right\}}Z}\left[\begin{array}{ccc}{m}_x& 0& o_x\\ 0& m_y& o_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}f& 0& 0\\ 0& f& 0\\ 0& 0& 1\end{array}\right]{}^{\left\{C\right\}}\tilde{\mathbf{X}},$$

(2)

where $m_x$ and $m_y$ are each the number of pixels per unit distance along x and y direction respectively. $o_x$ and $o_y$ are the principal point location in $\left\{P\right\}$, and f is the effective focal length. Note that ${}^{\left\{P\right\}}\mathbf{x}={\left[{}^{\left\{P\right\}}x,{}^{\left\{P\right\}}y,1\right]}^{\top }$ and ${}^{\left\{C\right\}}\tilde{\mathbf{X}}={\left[{}^{\left\{C\right\}}X,{}^{\left\{C\right\}}Y,{}^{\left\{C\right\}}Z\right]}^{\top }$. ${}^{\left\{P\right\}}\mathbf{x}$ is rasterized into a regular grid in the camera as:

$$\left[\begin{array}{c}j\\ i\end{array}\right]=\mathrm{ceil}\left(\left[\begin{array}{c}{}^{\left\{P\right\}}x+0.5\\ {}^{\left\{P\right\}}y+0.5\end{array}\right]\right),$$

(3)

where $\mathrm{ceil}(·)$ denotes the ceiling operator. $\left\{W\right\}$ and $\left\{C\right\}$ are related with a 3D rigid transformation: ${}^{\left\{C\right\}}\tilde{\mathbf{X}}={\mathbf{R}}_W{}^{\left\{W\right\}}\tilde{\mathbf{X}}+{\mathbf{t}}_W$.

Figure 2. Coordinate setups for modeling imaging geometry and light configuration.

Let the kth light position be ${}^{\left\{C\right\}}{\tilde{\mathbf{S}}}_k={\left[{}^{\left\{C\right\}}{X}_k,{}^{\left\{C\right\}}{Y}_k,{}^{\left\{C\right\}}{Z}_k\right]}^{\top }$. ${}^{\left\{C\right\}}{\tilde{\mathbf{S}}}_k$ is associated in the spherical space as:

$$\left\{\begin{array}{c}{}^{\left\{C\right\}}{X}_k={}^{\left\{C\right\}}{r}_{k}sin{}^{\left\{C\right\}}{\theta }_{k}cos{}^{\left\{C\right\}}{\varphi }_k\hfill \\ {}^{\left\{C\right\}}{Y}_k={}^{\left\{C\right\}}{r}_{k}sin{}^{\left\{C\right\}}{\theta }_{k}sin{}^{\left\{C\right\}}{\varphi }_k\hfill \\ {}^{\left\{C\right\}}{Z}_k={}^{\left\{C\right\}}{r}_{k}cos{}^{\left\{C\right\}}{\theta }_k,\hfill \end{array}\right.$$

(4)

where ${}^{\left\{C\right\}}{r}_k$, ${}^{\left\{C\right\}}{\theta }_k$ and ${}^{\left\{C\right\}}{\varphi }_k$ are the radius, zenith and azimuth angle, respectively. The N lights are configured such that they have the same zenith angle and radius, and uniformly distributed azimuth angles around the optical axis, i.e., ${}^{\left\{C\right\}}{\theta }_k=\theta$, ${}^{\left\{C\right\}}{r}_k=r$ and ${}^{\left\{C\right\}}{\varphi }_k=\frac{{360}^{\circ }(k-1)}{N}+{\varphi }_0$.

Having established the imaging geometry and light configuration, the imaging photometry is then modeled using the physically-based DRM [22]. The DRM properly characterizes reflectance for dielectric materials [25] and is represented by a linear combination of the diffuse and specular components. Assume that pixel $\left(i,j\right)$ is shadow-free, the image irradiance for colour channel c after light strength normalisation and vignetting correction is written as:

$$e_c^{i,j}=e_{d,c}^{i,j}+e_{s,c}^{i,j}=d_c^{i,j}{f}_d^{i,j}+s_c{f}_s^{i,j},$$

(5)

where $e_{d,c}^{i,j}$ and $e_{s,c}^{i,j}$ are the diffuse and specular components, and $d_c^{i,j}$ and $s_c$ are diffuse and specular colours which are wavelength-dependent and represented by:

$$\left\{\begin{array}{c}{d}_c^{i,j}={{\displaystyle \int }}_0^{\infty }\mathsf{\Phi }\left(\lambda \right)R^{i,j}\left(\lambda \right)C_c\left(\lambda \right)d\lambda ,\hfill \\ S_c={{\displaystyle \int }}_0^{\infty }\mathsf{\Phi }\left(\lambda \right)C_c\left(\lambda \right)d\lambda .\hfill \end{array}\right.$$

(6)

In the equation, $\lambda$ is the wavelength, $\mathsf{\Phi }\left(\lambda \right)$ is the source spectral power density (SPD), $R^{i,j}\left(\lambda \right)$ is the spectral body reflectance, and $C_c\left(\lambda \right)$ is the camera spectral sensitivity for the colour channel. $f_d^{i,j}$ and $f_s^{i,j}$ represent the diffuse and specular geometrical scaling factors that are characterized by BRDF.

Since the scaling factors are illumination-dependent, let the image irradiance Equation (5) generalized in the RGB colour space be with the kth illumination as:

$${\mathbf{e}}_k^{i,j}={\mathbf{e}}_{d,k}^{i,j}+{\mathbf{e}}_{s,k}^{i,j}={\overline{\mathbf{d}}}_{RGB}^{i,j}{f}_{d,k}^{i,j}+{\overline{\mathbf{s}}}_{RGB}{f}_{s,k}^{i,j}=\left[\begin{array}{c}{\overline{d}}_R^{i,j}\\ {\overline{d}}_G^{i,j}\\ {\overline{d}}_B^{i,j}\end{array}\right]f_{d,k}^{i,j}+\left[\begin{array}{c}{\overline{s}}_R\\ {\overline{s}}_G\\ {\overline{s}}_B\end{array}\right]f_{s,k}^{i,j},$$

(7)

where ${\overline{\mathbf{d}}}_{RGB}^{i,j}={\left\{d_c^{i,j}\right\}}_{c=R,G,B}$ and ${\overline{\mathbf{s}}}_{RGB}={\left\{s_c\right\}}_{c=R,G,B}$ are the unit diffuse and specular colour, respectively. The diffuse geometrical scaling factor is characterized with the kth unit illumination direction ${\overline{\mathbf{l}}}_k^{i,j}$ by Lambertian model as:

$$f_{d,k}^{i,j}=k_d^{i,j}{\left({\overline{\mathbf{n}}}^{i,j}\right)}^{\top }{\overline{\mathbf{l}}}_k^{i,j},$$

(8)

where $k_d^{i,j}$ is the diffuse reflectance factor. Since the surface deviation is small compared with the light-surface distance, ${\overline{\mathbf{l}}}_k^{i,j}$ can be approximately derived using a vector from the point intersected by ray $\left(i,j\right)$ and the reference plane to ${}^{\left\{C\right\}}{\tilde{\mathbf{S}}}_k$. One of the common options of the specular geometrical scaling factor, on the other hand, is a model using the Blinn-Phong BRDF [26] because of its computational efficiency and high performance in characterizing a wide range of isotropic reflectance [27]:

$$f_{s,k}^{i,j}=k_s^{i,j}{\left({\left({\overline{\mathbf{n}}}^{i,j}\right)}^T{\overline{\mathbf{h}}}_k^{i,j}\right)}^{{\beta }^{i,j}},$$

(9)

where $k_s^{i,j}$ is the specular reflectance factor, ${\overline{\mathbf{h}}}_k^{i,j}$ is the unit half vector, and ${\beta }^{i,j}$ is the shininess coefficient. The unit half vector is written as:

$${\overline{\mathbf{h}}}_k^{i,j}=\frac{{\overline{\mathbf{l}}}_k^{i,j}+{\overline{\mathbf{v}}}^{i,j}}{\parallel {\overline{\mathbf{l}}}_k^{i,j}+{\overline{\mathbf{v}}}^{i,j}\parallel },$$

(10)

where ${\overline{\mathbf{v}}}^{i,j}$ is the unit viewer direction and determined by:

$$\left\{\begin{array}{c}{\mathrm{v}}^{i,j}=-{\left[\frac{j-o_x}{m_x},\frac{i-o_y}{m_y},f\right]}^{\mathrm{𐊗}},\hfill \\ {\overline{\mathrm{v}}}^{i,j}={\mathrm{v}}^{i,j}/\Vert {\mathrm{v}}^{i,j}\Vert .\hfill \end{array}\right.$$

(11)

2.3. Colour PS Methods Using DRM

Existing colour PS methods using DRM contain two major variants, representative works of which were developed by Barsky et al. [21] and by Zickler et al. [23]. The PS method of Barsky finds the unit surface normal ${\overline{\mathbf{n}}}^{i,j}$ and the diffuse reflectance factor $k_d^{i,j}$ such that the residual between the colour image irradiances weighted by ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and the diffuse geometrical scaling factor is minimised:

$${ϵ}^{i,j}=\sum _{k=1}^{N_b}{\left[{\left({\mathbf{e}}_k^{i,j*}\right)}^{\top }{\overline{\mathbf{d}}}_{RGB}^{i,j}-f_{d,k}^{i,j}\right]}^2=\sum _{k=1}^{N_b}{\left[{\left({\mathbf{e}}_k^{i,j*}\right)}^{\top }{\overline{\mathbf{d}}}_{RGB}^{i,j}-k_d^{i,j}{\left({\overline{\mathbf{n}}}^{i,j}\right)}^{\top }{\overline{\mathbf{l}}}_k^{i,j}\right]}^2\to \underset{{\overline{\mathbf{n}}}^{i,j},k_d^{i,j}}{min},$$

(12)

where $N_b(\le N)$ is the number of specular-free and shadow-free images. The reason for the use of $N_b$ instead of N is that the Barsky’s PS method assumes that non-Lambertian reflectance appears sparsely among observations, which should be rejected as outliers. As the first process for outlier rejection, the direct principal component analysis (DPCA) [28] on ${\left[{\mathbf{E}}_{RGB}^{i,j*}\right]}^{\top }\left[{\mathbf{E}}_{RGB}^{i,j*}\right]$ is performed to estimate ${\overline{\mathbf{d}}}_{RGB}^{i,j}$, where $\left[{\mathbf{E}}_{RGB}^{i,j*}\right]={\left[{\mathbf{e}}_1^{i,j*},{\mathbf{e}}_2^{i,j*},\dots ,{\mathbf{e}}_N^{i,j*}\right]}^{\top }$. Given ${\overline{\mathbf{s}}}_{RGB}$ and ${\overline{\mathbf{d}}}_{RGB}^{i,j}$, the method then determines $f_{s,k}^{i,j}$ as:

$$f_{s,k}^{i,j}=\frac{{\left({\mathbf{e}}_k^{i,j*}\right)}^T{\overline{\mathbf{s}}}_{RGB}-\left({\left({\mathbf{e}}_k^{i,j*}\right)}^{\top }{\overline{\mathbf{d}}}_{RGB}^{i,j}\right)\left({\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^{\top }{\overline{\mathbf{s}}}_{RGB}\right)}{1-{\left({\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^T{\overline{\mathbf{s}}}_{RGB}\right)}^2}.$$

(13)

A threshold on $f_{s,k}^{i,j}$ is finally used to determine pixels to be rejected as outliers, and this subsequently determines $N_b$.

The objective function (12) of this PS method is not different from that of the conventional Lambertian-based PS method [3]. Given ${\left\{{\overline{\mathbf{l}}}_k^{i,j}\right\}}_{k\in \left[1,M\right]}$a priori, ${\overline{\mathbf{n}}}^{i,j}$ together with $k_d^{i,j}$, which minimizes the objective function (12), is analytically determined as:

$$\left\{\begin{array}{c}{k}_d^{i,j}=∥{\left({\left[{\mathbf{L}}^{i,j}\right]}^{\top }\left[{\mathbf{L}}^{i,j}\right]\right)}^{-1}{\left[{\mathbf{L}}^{i,j}\right]}^{\top }\left[{\mathbf{E}}_{RGB}^{i,j*}\right]{\overline{\mathbf{d}}}_{RGB}^{i,j}∥,\hfill \\ {\overline{\mathbf{n}}}^{i,j}=\frac{{\left({\left[{\mathbf{L}}^{i,j}\right]}^{\top }\left[{\mathbf{L}}^{i,j}\right]\right)}^{-1}{\left[{\mathbf{L}}^{i,j}\right]}^{\top }\left[{\mathbf{E}}_{RGB}^{i,j*}\right]{\overline{\mathbf{d}}}_{RGB}^{i,j}}{k_d^{i,j}},\hfill \end{array}\right.$$

where $\left[{\mathbf{L}}^{i,j}\right]={\left[{\overline{\mathbf{l}}}_1^{i,j},{\overline{\mathbf{l}}}_2^{i,j},\dots ,{\overline{\mathbf{l}}}_{N_b}^{i,j}\right]}^{\top }$ and $\mathrm{rank}\left(\left[{\mathbf{L}}^{i,j}\right]\right)=3$. While the effect of specularity rejection through the use of ${\overline{\mathbf{s}}}_{RGB}$ is exhibited, the estimation of ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ using DPCA is erroneous due to the existence of specularities. Furthermore, all the specularities are rejected as outliers though useful information may be contained.

Unlike Barsky’s PS method, which performs PS in the RGB colour space, the PS method of Zickler estimates surface orientations in the SUV colour space, or the UV colour space as S is not considered:

$$\begin{array}{c}\hfill {ϵ}^{i,j}=\sum _{k=1}^N{\left[{\left({\mathbf{e}}_{UV,k}^{i,j*}\right)}^T{\overline{\mathbf{d}}}_{UV}^{i,j}-{\alpha }^{i,j}{\left({\overline{\mathbf{n}}}^{i,j}\right)}^T{\overline{\mathbf{l}}}_k^{i,j}\right]}^2\to \underset{{\overline{\mathbf{n}}}^{i,j}}{min},\end{array}$$

(14)

where ${\alpha }^{i,j}$ is a scaling factor. The UV colour subspace includes more observations than the RGB colour space in surface orientation estimation. To operate in the SUV space, the colour image irradiance is transformed from RGB to SUV as:

$${\mathbf{e}}_{SUV,k}^{i,j}\triangleq {\mathbf{e}}_k^{i,j}={\mathbf{R}}_s{\mathbf{e}}_k^{i,j}={\overline{\mathbf{d}}}_{SUV}^{i,j}{f}_{d,k}^{i,j}+{\overline{\mathbf{s}}}_{SUV}{f}_{s,k}^{i,j},$$

(15)

where ${\overline{\mathbf{d}}}_{SUV}^{i,j}={\left[{\overline{d}}_U^{i,j},{\overline{d}}_V^{i,j},{\overline{d}}_S^{i,j}\right]}^T$, ${\overline{\mathbf{s}}}_{SUV}={\left[0,0,1\right]}^T$, and ${\mathbf{R}}_s\in SO\left(3\right)$ is any transformation that yields ${\overline{\mathbf{s}}}_{SUV}={\mathbf{R}}_s{\overline{\mathbf{s}}}_{RGB}={\left[0,0,1\right]}^T$. UV forms a specular-free colour subspace that is invariant to $f_{s,k}^{i,j}$. The Zickler’s PS method first estimates ${\overline{\mathbf{d}}}_{UV}^{i,j}$ of unit length using DPCA on ${\left[{\mathbf{E}}_{UV}^{i,j*}\right]}^{\top }\left[{\mathbf{E}}_{UV}^{i,j*}\right]$. It then becomes the conventional Lambertian-based PS problem and the solution is obtained in the similar form to Equation (14). Compared with the Barsky’s PS, the Zickler’s PS utilizes more observations and advantageously performs PS in the specular-free UV colour subspace. Shape information from specularities is however neglected again, and the method can only provide diffuse component up to the normalised RGB space. The diffuse-specular separation in the RGB space remains open.

3. DRM-Based Colour PS Method Using Diffuse-Specular Separation

3.1. Overview of the Proposed Color PS Method

Figure 3 shows the flow and original contributions of the proposed DRM-based colour PS method using the diffuse-specular separation. The proposed method consists of two processes: Step 1, which is diffuse-specular separation using a PS, and Step 2, which is the surface normal refinement. Step 1 identifies good approximate surface normals, whereas Step 2 further refines and finalizes them with maximum accuracy. The steps are marked by red boxes.

Figure 3. Flows and the original contribution of the DRM-based colour PS.

Step 1 is composed of four sub-processes: diffuse colour estimation, diffuse-specular separation in RGB space, PS in UV space and the outlier estimation. After completing shadow rejection as a preprocess, and generating the shadow-free image irradiance matrix $\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]$, Step 1 begins with the diffuse colour estimation using the robust principal component analysis (RPCA) proposed in this paper and derives ${\overline{\mathbf{d}}}_{RGB}^{i,j}$. Note that $\underline{\left(\right)}$ is a quantity of $\left(\right)$ after shadow rejection. The diffuse color estimation also outputs a specularity map, which describes the distribution of specular reflection. The diffuse-specular separability is then checked for each pixel by deriving the diffuse-specular chromatic angle ${\psi }^{i,j}$ in RGB space. For separable pixels, the proposed method performs PS in the specular-free UV colour subspace similarly to Zickler’s PS method and derives the initial guess of the unit surface normal ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$ and the diffuse reflectance factor ${\left(k_d^{i,j}\right)}_1$ similarly to Barsky’s PS. ${\left(\right)}_1$ means the initial guess of $\left(\right)$. In parallel, the specular geometrical scaling factors, ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$, are derived through the diffuse-specular separation process using Equation (13) for the subsequent surface normal refinement. Observations not in the specularity map are clamped to zero in ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$. The surface normals and the other parameters are then refined by fitting the nonlinear DRM using LMA with regularisation.

In Step 2, the initial guess of the specular parameters, ${\left(k_s^{i,j}\right)}_1$ and ${\left({\beta }^{i,j}\right)}_1$, are drived from ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$ and ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$. Finally, PS for the surface normal refinement derives ${\overline{\mathbf{n}}}^{i,j}$ and $k_d^{i,j}$ with enhanced accuracy while additionally updating $k_s^{i,j}$ and ${\beta }^{i,j}$. The strength of the proposed method lies in this PS with the diffuse-specular separation. For pixels where sparse non-Lambertian reflectances appear, the first step is sufficient to reliably estimate the surface orientations. To tackle with more general problems with dense non-Lambertian reflectances, the second step is additionally required to exploit surface normal information from specularities in the DRM with the designed parameter initialisation strategy. The rest of this section elucidates the proposed method in detail.

3.2. Diffuse-Specular Separation

3.2.1. Diffuse Color Estimation

The proposed RPCA derives ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and the specularity map as follows:

1.

Perform principle component analysis (PCA) for ${\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]}^{\top }\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]$ to estimate ${\overline{\mathbf{d}}}_{RGB}^{i,j}$;

2.

Compute residual matrix: $\left[{\mathbf{R}}_d^{i,j}\right]=\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]{\overline{\mathbf{d}}}_{RGB}^{i,j}{\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^{\top }-\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]$.

3.

Compute residual vector: ${\mathbf{r}}_d^{i,j}=\sqrt{{\left({\mathbf{r}}_{d,1}^{i,j}\right)}^{\circ 2}+{\left({\mathbf{r}}_{d,2}^{i,j}\right)}^{\circ 2}+{\left({\mathbf{r}}_{d,3}^{i,j}\right)}^{\circ 2}}$, where $\left[{\mathbf{R}}_d^{i,j}\right]=\left[{\mathbf{r}}_{d,1}^{i,j},{\mathbf{r}}_{d,2}^{i,j},{\mathbf{r}}_{d,3}^{i,j}\right]$ and $\circ 2$ denotes the hadamard square.

4.

If the mean of ${\mathbf{r}}_d^{i,j}$, $\mathrm{mean}\left({\mathbf{r}}_d^{i,j}\right)$, is smaller than the threshold $T_d$, terminate RPCA and output the current estimate of ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and the specularity map. Otherwise, find the element that provides the maximal value of $\left(\frac{{\mathbf{r}}_d^{i,j}-\mathrm{mean}\left({\mathbf{r}}_d^{i,j}\right)}{\mathrm{std}\left({\mathbf{r}}_d^{i,j}\right)}\right)$ and register $(i,j)$ in the specularity map. Remove the corresponding row vector in $\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]$ and repeat from step 1.

The RPCA eliminates image irradiances that are inappropriate for diffuse colour estimation.

The validity of the RPCA can be explained as follows. The shadow-free image irradiance matrix $\left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]$ is expanded from Equation (7) and decomposed as

$$\begin{array}{ccc}\hfill \left[{\underline{\mathbf{E}}}_{RGB}^{i,j*}\right]& =& {\underline{\mathbf{f}}}_d^{i,j}{\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^{\top }+{\underline{\mathbf{f}}}_s^{i,j}{\left({\overline{\mathbf{s}}}_{RGB}\right)}^{\top }\hfill \\ & =& \left[\begin{array}{c}{f}_{d,1}^{i,j}\\ f_{d,2}^{i,j}\\ \dots \\ f_{d,N_p}^{i,j}\end{array}\right]\left[\begin{array}{ccc}{\overline{d}}_R^{i,j}& {\overline{d}}_G^{i,j}& {\overline{d}}_B^{i,j}\end{array}\right]+\left[\begin{array}{c}{f}_{s,1}^{i,j}\\ f_{s,2}^{i,j}\\ \dots \\ f_{s,N_p}^{i,j}\end{array}\right]\left[\begin{array}{ccc}{\overline{s}}_R& {\overline{s}}_G& {\overline{s}}_B\end{array}\right]\hfill \end{array}$$

(16)

where most entries in ${\underline{\mathbf{f}}}_s^{i,j}$ are near-zero since images are often near specular-free. In such a case, a non-zero residual occurs if ${\underline{\mathbf{f}}}_s^{i,j}$ contains non-negligible entries. The RPCA algorithms robustly and accurately estimate ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ by keeping rejecting specularities until the residual becomes smaller than $T_d$. Unlike Barsky’s PS method, which identifies specularities after ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimation, the proposed algorithms derive ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and the specularity map simultaneously.

While the originality and superiority of the proposed RPCA algorithms have been identified, the performance of the RPCA algorithms depends on the value of $T_d$. If $T_d$ is too large, the RPCA algorithms are tolerant to specularities and perform similarly to the DPCA algorithms. If $T_d$ is too small, the RPCA algorithms reject innocent pixels as specularities and make images sensitive to noise.

3.2.2. Diffuse-Specular Separability Check

Having ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimated, the diffuse-specular chromatic angle, used to perform the separability check, is derived as

$$cos{\psi }^{i,j}={\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^T{\overline{\mathbf{s}}}_{RGB},$$

(17)

Asymptotically, the chromatic angle gives

$$\underset{{\psi }^{i,j}\to 0}{lim}{\overline{\mathbf{d}}}_{RGB}^{i,j}={\overline{\mathbf{s}}}_{RGB}$$

(18)

This means that ${\psi }^{i,j}$ should be sufficiently larger than 0 in order for ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and ${\overline{\mathbf{s}}}_{RGB}$ to be distinct and separable. The threshold of the chromatic angle to determine the separability is $T_c$. If ${\psi }^{i,j}\ge T_c$, ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and ${\overline{\mathbf{s}}}_{RGB}$ are sufficiently distinct and pixel $\left(i,j\right)$ is considered as separable. Similarly to $T_d$, $T_c$ should be chosen carefully. A small $T_c$ results in yielding false separation. Once ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and ${\overline{\mathbf{s}}}_{RGB}$ have been identified, ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$ is then initialised using Equation (13).

3.2.3. PS in UV Space

Since the PS is performed in the UV colour space, the unit diffuse colour ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ should be converted into that in UV colour space ${\overline{\mathbf{d}}}_{UV}^{i,j}$. The PS in UV color space allows more observations as described in the Zickler’s PS method. With ${\mathbf{R}}_s$, the estimated ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ is transformed to the SUV space as ${\widehat{\mathbf{d}}}^{i,j}={\left[{\widehat{d}}_U^{i,j},{\widehat{d}}_V^{i,j},{\widehat{d}}_S^{i,j}\right]}^{\top }$. The diffuse color in UV space ${\widehat{\mathbf{d}}}_{UV}^{i,j}$ is then given by

$${\widehat{\mathbf{d}}}_{UV}^{i,j}={\left[{\widehat{d}}_U^{i,j},{\widehat{d}}_V^{i,j}\right]}^{\top }={\kappa }_z^{i,j}{\overline{\mathbf{d}}}_{UV}^{i,j}$$

(19)

where ${\overline{\mathbf{d}}}_{UV}^{i,j}={\left[{\overline{d}}_U^{i,j},{\overline{d}}_V^{i,j}\right]}^{\top }$ is a unit vector, and

$${\kappa }_z^{i,j}=\sqrt{{\left({\widehat{d}}_U^{i,j}\right)}^2+{\left({\widehat{d}}_V^{i,j}\right)}^2}>0.$$

(20)

Image irradiances of separable pixels are also transformed to the SUV colour space as $\left[{\underline{\mathbf{E}}}_{SUV}^{i,j}\right]$ using the same ${\mathbf{R}}_s$. $\left[{\underline{\mathbf{E}}}_{UV}^{i,j}\right]$ is then formed by picking the first two columns of $\left[{\underline{\mathbf{E}}}_{SUV}^{i,j}\right]$. Due to the presence of image noise, the PS in UV space is modified from Equation (15) to reject noise-corrupted image irradiances as outliers using studentised residuals [29]:

$$\begin{array}{ccc}\hfill {ϵ}^{i,j}& =& \sum _{k=1}^{N_p}{\left[{\left({\mathbf{e}}_{UV,k}^{i,j*}\right)}^{\top }{\overline{\mathbf{d}}}_{UV}^{i,j}-{\left({\kappa }_z^{i,j}\right)}^2{f}_{d,k}^{i,j}\right]}^2\hfill \\ & =& \sum _{k=1}^{N_p}{\left[{\left({\mathbf{e}}_{UV,k}^{i,j*}\right)}^{\top }{\overline{\mathbf{d}}}_{UV}^{i,j}-{\left({\kappa }_z^{i,j}\right)}^2{\left(k_d^{i,j}\right)}_1{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1^{\top }{\overline{\mathbf{l}}}_k^{i,j}\right]}^2\hfill \\ & \to & \underset{{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1,{\left(k_d^{i,j}\right)}_1}{min},\hfill \end{array}$$

(21)

where $N_p$ is the number of shadow-free images, so $N_b\le N_p\le N$. The derivation of ${\left(k_d^{i,j}\right)}_1$ in addition to ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$ similarly to the Barsky’s PS method enables full recovery of the colour diffuse component. $N_p$ is used instead of N and $N_b$ because shadow-free images are effective for reliability.

After the PS, ${\overline{\mathbf{n}}}^{i,j}$ and $k_d^{i,j}$ are initialised as ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$ and ${\left(k_d^{i,j}\right)}_1$ respectively. The diffuse component matrix, $\left[{\mathbf{E}}_{d,RGB}^{i,j}\right]$, is then re-rendered as:

$$\left[{\mathbf{E}}_{d,RGB}^{i,j}\right]=max\left({\left(k_d^{i,j}\right)}_1\left[{\mathbf{L}}^{i,j}\right]{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1,{\mathbf{0}}_N\right){\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^T,$$

(22)

where ${\mathbf{0}}_N$ is an $N\times 1$ vector with all zero entries. The specular component matrix, $\left[{\mathbf{E}}_{s,RGB}^{i,j}\right]$, is obtained by $max\left({\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1{\overline{\mathbf{s}}}_{RGB}^T,{\mathbf{0}}_N\right)$. With the separated colour diffuse and specular components, the proposed method allows the functionality of specular removal and intrinsic image decomposition.

3.2.4. Outlier Estimation

Let the residual vector ${\underline{\mathbf{r}}}^{i,j}$ be written as:

$${\underline{\mathbf{r}}}^{i,j}=\left[{\underline{\mathbf{E}}}_{UV}^{i,j}\right]{\overline{\mathbf{d}}}_{UV}^{i,j}-\left[{\underline{\mathbf{L}}}^{i,j}\right]{\mathbf{n}}^{i,j}=\left[{\underline{\mathbf{E}}}_{UV}^{i,j}\right]{\overline{\mathbf{d}}}_{UV}^{i,j}-\left[{\underline{\mathbf{L}}}^{i,j}\right]{\rho }^{i,j}{\overline{\mathbf{n}}}^{i,j},$$

(23)

where ${\mathbf{n}}^{i,j}={\rho }^{i,j}{\overline{\mathbf{n}}}^{i,j}$ is the scaled normal, $\left[{\underline{\mathbf{L}}}^{i,j}\right]={\left[{\overline{\mathbf{l}}}_1^{i,j},{\overline{\mathbf{l}}}_2^{i,j},\dots ,{\overline{\mathbf{l}}}_M^{i,j}\right]}^T$, and ${\rho }^{i,j}=k_d^{i,j}{\kappa }_z^{i,j}$. The hat matrix $\left[{\mathbf{H}}_a^{i,j}\right]$ [29] is represented by:

$$\left[{\underline{\mathbf{H}}}_a^{i,j}\right]=\left[{\underline{\mathbf{L}}}^{i,j}\right]{\left({\left[{\underline{\mathbf{L}}}^{i,j}\right]}^T\left[{\underline{\mathbf{L}}}^{i,j}\right]\right)}^{-1}{\left[{\underline{\mathbf{L}}}^{i,j}\right]}^T.$$

(24)

Diagonal entries in $\left[{\underline{\mathbf{H}}}_a^{i,j}\right]$ are leverages and the kth leverage is denoted as $h_{a,k}^{i,j}$. These leverages quantify the influence that the observed $\left[{\underline{\mathbf{E}}}_{UV}^{i,j}\right]{\overline{\mathbf{d}}}_{UV}^{i,j}$ on their predicted values $\left[{\underline{\mathbf{L}}}^{i,j}\right]{\mathbf{n}}^{i,j}$. Each entry in the studentized residual vector ${\widetilde{\underline{\mathbf{r}}}}^{i,j}$ is then approximated using:

$${\tilde{r}}_k^{i,j}=\frac{r_k^{i,j}}{\sqrt{{\left(MSE\right)}^{i,j}\left(1-h_{a,k}^{i,j}\right)}},$$

(25)

where ${\left(\mathrm{MSE}\right)}^{i,j}$ represents the mean squared error of ${\underline{\mathbf{r}}}^{i,j}$. If any entry in $|{\widetilde{\underline{\mathbf{r}}}}^{i,j}|$ exceeds $T_o$, this entry is estimated as outliers. The proposed PS rejects outliers until no more can be detected or ${\left(\mathrm{MSE}\right)}^{i,j}$ is smaller than a tolerance, $T_m$. With the general rule of thumb in detecting outliers using studentised residuals, $T_o$ and $T_m$ can be chosen as $2.5$ and $9{\sigma }_n^2$, where ${\sigma }_n$ is the standard deviation of additive white Gaussian noise that can be estimated using the method given by [30].

3.3. Surface Normal Refinement

3.3.1. Specular Parameter Initialisation

If dense non-Lambertian reflectance appears at pixel $\left(i,j\right)$, which implies that more than one entry in ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$ are in the specularity map, surface normal refinement is necessary since the sparse non-Lambertian reflectance assumption in the diffuse-specular separation is violated. With the estimated ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$, the next objective is to initialise the specular parameters, ${\left(k_s^{i,j}\right)}_1$ and ${\left({\beta }^{i,j}\right)}_1$. The specific cost functional is given by:

$${ϵ}_s^{i,j}=\sum _{k=1}^{N_q}{\left({\left(f_{s,k}^{i,j}\right)}_1-{\left(k_s^{i,j}\right)}_1{\left({\left({\overline{\mathbf{h}}}_k^{i,j}\right)}^{\top }{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1\right)}^{{\left({\beta }^{i,j}\right)}_1}\right)}^2\to \underset{{\left(k_s^{i,j}\right)}_1,{\left({\beta }^{i,j}\right)}_1}{min},$$

(26)

where $N_q$ is the number of specularity maps showing pixel $\left(i,j\right)$ and $N_q\ge 2$. Equation (26) suggests a nonlinear least-squares problem, while it can be manipulated to the natural logarithm domain as a linear problem given by:

$${ϵ}_{ls}^{i,j}=\sum _{k=1}^{N_q}{\left(ln{\left(f_{s,k}^{i,j}\right)}_1-ln{\left(k_s^{i,j}\right)}_1-{\left({\beta }^{i,j}\right)}_{1}ln\left({\left({\overline{\mathbf{h}}}_k^{i,j}\right)}^{\top }{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1\right)\right)}^2\to \underset{{\left(k_s^{i,j}\right)}_1,{\left({\beta }^{i,j}\right)}_1}{min}.$$

(27)

Let ${\left({\underline{\underline{\mathbf{f}}}}_s^{i,j}\right)}_1$ with $N_q$ rows consist of entries in ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$ in the specularity map and ${\left[\underline{\underline{\mathbf{H}}}\right]}^{i,j}={\left[{\overline{\mathbf{h}}}_1^{i,j},{\overline{\mathbf{h}}}_2^{i,j},...,{\overline{\mathbf{h}}}_{N_q}^{i,j}\right]}^T$ be the matrix comprised of the corresponding unit half vectors. Then, the solution of $ln{\left(k_s^{i,j}\right)}_1$ and ${\left({\beta }^{i,j}\right)}_1$ to Equation (27) is given analytically by:

$$\left[\begin{array}{c}{\left({\beta }^{i,j}\right)}_1\\ ln{\left(k_s^{i,j}\right)}_1\end{array}\right]={\left({\left[{\underline{\underline{\mathbf{S}}}}^{i,j}\right]}^{\top }\left[{\underline{\underline{\mathbf{S}}}}^{i,j}\right]\right)}^{-1}{\left[{\underline{\underline{\mathbf{S}}}}^{i,j}\right]}^{\top }ln{\left({\underline{\underline{\mathbf{f}}}}_s^{i,j}\right)}_1,$$

(28)

where

$$\left[{\underline{\underline{\mathbf{S}}}}^{i,j}\right]=\left[ln\left({\left[\underline{\underline{\mathbf{H}}}\right]}^{i,j}{\left({\overline{\mathbf{n}}}^{i,j}\right)}_1\right),{\mathbf{1}}_{N_q}\right]$$

(29)

and ${\mathbf{1}}_{N_q}$ is a $N_q\times 1$ vector with all entries equal to 1.

3.3.2. Surface Normal Refinement in DRM

All parameters in the DRM have been initialised up to this point. The LMA is then employed to iteratively refine these parameters by solving the optimization problem:

$$\begin{array}{c}\hfill {ϵ}^{i,j}=\sum _{k=1}^N\left({\left({\mathbf{e}}_k^{i,j}\right)}^{\top }{\overline{\mathbf{s}}}_{RGB}-k_d^{i,j}{\left({\overline{\mathbf{n}}}^{i,j}\right)}^{\top }{\overline{\mathbf{l}}}_k^{i,j}{\left({\overline{\mathbf{d}}}_{RGB}^{i,j}\right)}^{\top }{\overline{\mathbf{s}}}_{RGB}-k_s^{i,j}{\left({\left({\overline{\mathbf{n}}}^{i,j}\right)}^{\top }{\overline{\mathbf{h}}}_k^{i,j}\right)}^{{\beta }^{i,j}}\right.\\ \hfill {\left.+T_{\alpha }\left(1-{\left({\overline{\mathbf{n}}}^{i,j}\right)}^{\top }{\overline{\mathbf{n}}}^{i,j}\right)\right)}^2\to \underset{{\overline{\mathbf{n}}}^{i,j},k_d^{i,j},k_s^{i,j},{\beta }^{i,j}}{min},\end{array}$$

(30)

where the first three terms are from the generalised formulation, and, the fourth regularization term with constant $T_{\alpha }$ is added. The regularisation term reduces the change of surface normal refinement from the initial guess of ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$ and prevents overfitting. Larger value of $T_{\alpha }$ strengthens the robustness but limits the capability of surface normal refinement.

The strength of the proposed PS method lies in the determination of the surface normals while separating the diffuse and specular components in RGB space, eliminating outliers and then tuning the diffuse and specular reflectance factors simultaneously. The determination considering diffuse and specular reflections simultaneously makes the surface normal estimation more accurately. In addition, the proposed method can be used for specular removal [31] and intrinsic image decomposition [32]. Parameters characterizing $f_{s,k}$ are determined at pixels, so the proposed method could also be used for digital relighting and material classification where dense non-Lambertian reflectance appears.

4. Performance Evaluation on Surface Orientation Estimation

4.1. Evaluations Using Synthetic Images

The first experiment aims to evaluate the effectiveness of surface normal refinement for dense non-Lambertian reflectance using synthetic input images. A scene with six different-coloured spheres were rendered under 32 illuminants using the Blinn-Phong model. The sphere was adopted as the scene geometry since it samples all the possibilities of surface orientations for the visible surface. Different colours of the spheres were introduced for more comprehensive evaluations of the proposed PS method on surfaces with various spectral reflectances. The 32 illuminants were chosen to provide a sufficient number of pixels with dense non-Lambertian reflectance for evaluation. ${\overline{\mathbf{s}}}_{RGB}$ was set at ${\left[0.5774,0.5774,0.5774\right]}^T$. Three colours of the spheres were red, green and blue with the same ${\psi }^{i,j}$ of $57.{74}^{\circ }$, while the other three spheres were yellow, cyan and magenta with ${\psi }^{i,j}$ of $35.{26}^{\circ }$. The 32 light positions were configured with $r=442$ mm and $\theta ={20}^{\circ }$. The reference plane was located where ${\mathbf{R}}_W=diag\left(\left[1,-1,-1\right]\right)$ and ${\mathbf{t}}_W={\left[0,0,678\right]}^T$. $k_d^{i,j}$, $k_s^{i,j}$ and ${\beta }^{i,j}$ were set the same across the field of view as $0.4$, $0.2$ and 100, respectively. Additive Gaussian noise was introduced with ${\sigma }_n=0.02$. The image irradiance under the first illuminant is shown by Figure 4a. The parameters of the proposed PS method were configured as: $T_d=0.01$, $T_c=5^{\circ }$, $T_o=2.5$, $T_m=0.0036$, $T_{\alpha }=3$, $T_x=0.02$ and $T_y=0.02$. 100 repeated tests were conducted and the mean value was adopted for evaluation.

Figure 4. Evaluation of surface normal refinement: (a) Image irradiance under the first illuminant; (b) Angular error of surface orientations without surface normal refinement in degrees; (c) Angular error of surface orientations with surface normal refinement in degrees; (d) Improvement of surface orientation estimation by including surface normal refinement in percentage.

Figure 4b shows the angular error of surface orientation estimation without surface normal refinement. As is shown, the central regions of the spheres where dense non-Lambertian reflectances appear have larger error. By including the proposed surface normal refinement step, the angular errors of surface orientations around the detected dense non-Lambertian reflectance region, shown by a ring, are significantly reduced as shown by Figure 4c. This is where the reflectance parameters are most tuned through the surface normal refinement. Errors in such regions are even smaller than those with sparse non-Lambertian reflectance, which further enhances that making specularities as meaningful signals is beneficial for surface orientation estimation. Figure 4d demonstrates the improvement percentage for quantitative evaluation. The improvement percentage is defined reduction of the error divided by the error without surface normal refinement. The accuracy of surface orientations was enhanced by $34.33\%$ in median and $32.25\%$ on average. The first and third quantile values of the improvement were $15.76\%$ and $54.23\%$, respectively. The performance improvements onto the six different-coloured spheres are shown in Table 1. The improvements were obvious for all spheres is significant, implying that the method was applicable to a wide range of surfaces with different spectral reflectances. For spheres that had larger chromatic angle, ${\psi }^{i,j}$, the ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimations were more accurate. The better estimated ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ led to the more accurate recovery of ${\left({\underline{\mathbf{f}}}_s^{i,j}\right)}_1$, resulting in the more reliable initialisation of ${\left({\overline{\mathbf{n}}}^{i,j}\right)}_1$. Such initialisations of ${\overline{\mathbf{n}}}^{i,j}$ ultimately affected the different improvement performances. Overall, the accuracy was enhancement by including the additional surface normal refinement step and for spectral reflectances with larger chromatic angles, the improvement was more phenomenal.

Table 1. Improvement on different-coloured spheres.

Index	Colour	${\mathit{\psi }}^{\mathit{i},\mathit{j}}$	Error of ${\overline{\mathbf{d}}}_{{\mathit{RGB}}^{\mathit{i},\mathit{j}}}$	Mean Improvement
1	red	$57.{74}^{\circ }$	$1.{15}^{\circ }$	$23.39\%$
2	yellow	$35.{26}^{\circ }$	$1.{33}^{\circ }$	$32.85\%$
3	green	$57.{74}^{\circ }$	$1.{15}^{\circ }$	$23.42\%$
4	cyan	$35.{26}^{\circ }$	$1.{32}^{\circ }$	$32.04\%$
5	blue	$57.{74}^{\circ }$	$1.{15}^{\circ }$	$23.15\%$
6	magenta	$35.{26}^{\circ }$	$1.{33}^{\circ }$	$32.24\%$

In the second experiment, the performance improvement due to surface normal refinement was verified onto a wide range of reflectances. Experimental settings and parameters of the proposed method were the same as the first experiment except that different values of $k_d^{i,j}$, $k_s^{i,j}$ and ${\beta }^{i,j}$ were applied to represent various surface reflectances when generating synthetic images. The ratio of ${\kappa }_{ds}^{i,j}=k_d^{i,j}/k_s^{i,j}$ represents the relative strength between the diffuse and specular components, whereas ${\beta }^{i,j}$ indicates the width of the specular lobe. The mean, median, first and third quantile values of the improvement were adopted for evaluation and the angular error of ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimation was used for analyses. As shown from Table 2, the improvements on all the different reflectances were significant and more obvious for larger value of ${\beta }^{i,j}$ and ${\kappa }_{ds}^{i,j}$. Larger value of ${\beta }^{i,j}$ suggesting narrower specular lobe made the ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimation more accurate since the sparse non-Lambertian reflectance assumption was valid. Similarly, larger value of ${\kappa }_{ds}^{i,j}$ indicating stronger diffuse components made better ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimation due to stronger inliers. More accurate ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ estimation led to bigger improvement from the surface normal refinement. In summary, with the additional surface normal refinement step, the accuracy of surface orientation estimation is enhanced by around $30\%$ on average and the improvement is more obvious for reflectance that has narrower specular lobe and stronger diffuse component.

Table 2. Evaluation of the improvement for different reflectances.

${\mathit{k}}_{\mathit{d}}/{\mathit{k}}_{\mathit{s}}$	$\mathit{\beta }$	Mean	Median	First Quantile	Third Quantile	Error of ${\overline{\mathbf{d}}}_{\mathit{RGB}}$
$0.4/0.2$	100	$32.25\%$	$34.33\%$	$15.76\%$	$54.23\%$	$1.{23}^{\circ }$
$0.4/0.4$	100	$31.14\%$	$32.97\%$	$15.21\%$	$53.75\%$	$1.{58}^{\circ }$
$0.4/0.8$	100	$30.48\%$	$32.77\%$	$14.27\%$	$52.90\%$	$2.{34}^{\circ }$
$0.4/0.2$	20	$27.49\%$	$26.70\%$	$6.80\%$	$51.66\%$	$2.{99}^{\circ }$
$0.4/0.4$	20	$25.92\%$	$25.98\%$	$6.67\%$	$51.25\%$	$5.{06}^{\circ }$
$0.4/0.8$	20	$25.02\%$	$24.07\%$	$6.57\%$	$50.81\%$	$8.{30}^{\circ }$

The third experiment aims to evaluate the overall performance of the proposed colour PS method onto a wide variety of dielectric material reflectances. Twenty-four material BRDFs divided into six categories were evaluated, where the different BRDFs were provided from the MERL database [33]. Experimental settings and the method parameters were exactly the same as the first experiment except that the BRDFs were replaced. The image irradiance for the first six materials under the first illuminant is shown in Figure 5a. The angular errors of the surface orientation estimation are given in Figure 5b. As is shown, the error is not only larger at the near central regions where specularities overlap but also at the boundaries. Boundary pixels in the spheres have shallow grazing angles and the error is due to the lack of modeling of the fresnel reflection in the Blinn-Phong model. Figure 5c shows the performance of surface normal estimation on the 24 materials using box-and-whisker plot. The red and black dot represent the mean and median value, respectively. The lower and upper bound of the box indicates the first and third quantile values. The six colours of the boxes suggest the six material categories. As is shown, the proposed method can accurately estimate ${\overline{\mathbf{n}}}^{i,j}$ for most dielectric materials within 5 degrees on average. The estimation performances on phenolic, plastic and rubber were consistent, whereas those on the wood stain, fabric and acrylic were also acceptable with two exceptions, the violet acrylic and the green fabric. The degradation for violet acrylic was due to its wide specular lobe with small ${\kappa }_{ds}^{i,j}$, while the reason for green fabric was because of its small ${\psi }^{i,j}$.

Figure 5. (a) Image irradiance for the first six material BRDFs under the first illuminant; (b) Angular error of surface orientation estimation for the first six material BRDFs; (c) Angular error of surface orientation for the twenty-four dielectric materals.

4.2. Evaluations Using Real Images

The proposed method was evaluated on five different datasets comprised of real images in the DiLigenT database [4]. Parameters of the proposed method were set the same as the first experiment. ${\overline{\mathbf{s}}}_{RGB}$ was estimated as ${\left[0.5774,0.5774,0.5774\right]}^T$. From the first to fifth column in Figure 6, the results on BUDDHA, BEAR, POT2, READING and GOBLET are respectively shown. Figure 6a shows the image irradiance under the first illuminant, while Figure 6b,c demonstrate the estimated normal map and the angular error of surface orientation estimation, respectively. As is shown from READING, the proposed method is only feasible for surfaces whose ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and ${\overline{\mathbf{s}}}_{RGB}$ are distinct. Figure 6c shows that the dense non-Lambertian problem where the specularity overlaps is mostly solved for BUDDHA, BEAR and POT2, while deficits still exist on GOBLET, which is because the DRM is not proper to characterize metallic reflectance.

Figure 6. Evaluations on surface oreintation estimation: (a) Image irradiance under the first illuminant; (b) Estimated normal map; (c) Angular error of surface orientations; (d) Angular error of surface orientations on five datasets using ten different PS methods.

The results were compared with those of nine other PS methods which are listed in Table 3. The results from the nine methods were provided from the DiLigenT database and these methods were representative and well-recognized. It is to be noted that the results of the recent machine learning based techniques reviewed in Introduction are not shown for comparison since the effect of the physics based models is investigated in this paper. Due to the data availability, refs. [21,23] were not involved in the comparison, while the essence of their works in using DRM with known ${\overline{\mathbf{s}}}_{RGB}$ was well-inherited by the proposed method.

Table 3. Comparative studies of PS methods.

Method Index	1	2	3	4	5	6	7	8	9	10
Reference	[3]	[9]	[11]	[12]	[19]	[20]	[13]	[34]	[14]	proposed

Figure 6d shows the results from ten PS methods on the five datasets. The X-axis represents the method index and the Y-axis indicates the angular error of ${\overline{\mathbf{n}}}^{i,j}$. The five colours of the bar represent the different datasets. The angular error of ${\overline{\mathbf{n}}}^{i,j}$ is also demonstrated using the box-and-whisker plot similar to Figure 5. As is shown, the proposed method performs well in estimating surface orientations for all the five datasets except GOBLET. The results suggest that modeling reflectance using DRM is proper for surfaces made of dielectric materials. The strength of the proposed method lies on the reliable estimation of ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ with simultaneous specularity detection. The advantage is also accredited by exploiting ${\overline{\mathbf{n}}}^{i,j}$ not only from the diffuse components but also from the specularities owing to the diffuse-specular separation. In spite of its advantages, the proposed method is not effective for metallic surfaces compared with methods, such as [14]. The limitation originates from the fundamental assumption of DRM that ${\overline{\mathbf{s}}}_{RGB}$ are the same across the surface. This assumption is violated for metallic surfaces since the wavelength and geometry exhibit inter-dependency.

5. Conclusions and Future Works

A PS method using colour images dealing with non-Lambertian reflectance has been proposed. The method formulates the imaging photometry using DRM with known specular colour. It extracts surface orientations not only from the diffuse components but also specularities owing to the diffuse-specular separation. Introducing the additional surface refinement step using information from specularities, the proposed method particularly improves the accuracy for surface orientation estimation at pixels where dense non-Lambertian reflectance appear. The simultaneously acquired specular parameters can be applied for more potential functionalities, such as digital relighting and material classification.

From the experiment of validating the effectiveness of the newly proposed surface normal refinement step on surface where dense non-Lambertian reflectance appears, the results indicate that with the additional step, the accuracy is enhanced by around $30\%$ on average and the improvement is more phenominal for surfaces with larger chromatic angles, stronger diffuse components and narrower specular lobes due to the better estimated unit diffuse colour. The result investigating the proposed method applicability on 24 different reflectances of dielectric materials suggests that the average angular error of surface orientation estimation for most materials are within $5^{\circ }$ and bigger errors occur at surface patches with shallower grazing angles due to the limitation of the Blinn-Phong model that does not incorporate the fresnel reflection. From the systematic comparison on five datasets with nine other representative methods, the proposed method shows its descent performance on reflectances of dielectric materials and degradation on metallic surface due to the limitation of DRM. The result has also demonstrated that the proposed method is only feasible for surfaces whose diffuse and specular colour are distinct.

This paper presented the first set of results to estimate surface orientations using colour images for non-Lambertian reflectance, while it can be extended in a variety of ways. First, other analytical BRDFs can be employed instead of the Blinn-Phong model to better account for the fresnel reflection. Second, machine learning based models can be incorporated to learn and reduce errors of analytical models. Further, an alternative method can be proposed specifically for surfaces whose diffuse and specular colours are close to extend the method’s applicability. Making the method adaptive is one possible solution. Last but not least, it is also of particular interest to infer surface properties from the estimated specular parameters for the purpose of material classification.