# DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces

^{1}

^{2}

^{*}

^{†}

^{‡}

## Abstract

**:**

## 1. Introduction

## 2. Colour PS Incorporating Dichromatic Reflectance Model

#### 2.1. Generic Colour PS Problem Formulation

#### 2.2. Colour Image Formation Model

#### 2.3. Colour PS Methods Using DRM

## 3. DRM-Based Colour PS Method Using Diffuse-Specular Separation

#### 3.1. Overview of the Proposed Color PS Method

#### 3.2. Diffuse-Specular Separation

#### 3.2.1. Diffuse Color Estimation

- 1.
- Perform principle component analysis (PCA) for ${\left(\right)}^{{\underline{\mathbf{E}}}_{RGB}^{i,j*}}\top $ to estimate ${\overline{\mathbf{d}}}_{RGB}^{i,j}$;
- 2.
- Compute residual matrix: $\left(\right)open="["\; close="]">{\mathbf{R}}_{d}^{i,j}{\overline{\mathbf{d}}}_{RGB}^{i,j}{\left(\right)}^{{\overline{\mathbf{d}}}_{RGB}^{i,j}}\top $.
- 3.
- Compute residual vector: ${\mathbf{r}}_{d}^{i,j}=\sqrt{{\left(\right)}^{{\mathbf{r}}_{d,1}^{i,j}}\circ 2+{\left(\right)}^{{\mathbf{r}}_{d,2}^{i,j}}\circ 2}+{\left(\right)}^{{\mathbf{r}}_{d,3}^{i,j}}\circ 2$, where $\left(\right)open="["\; close="]">{\mathbf{R}}_{d}^{i,j}$ and $\circ 2$ denotes the hadamard square.
- 4.
- If the mean of ${\mathbf{r}}_{d}^{i,j}$, $\mathrm{mean}\left({\mathbf{r}}_{d}^{i,j}\right)$, is smaller than the threshold ${T}_{d}$, terminate RPCA and output the current estimate of ${\overline{\mathbf{d}}}_{RGB}^{i,j}$ and the specularity map. Otherwise, find the element that provides the maximal value of $\left(\right)$ and register $(i,j)$ in the specularity map. Remove the corresponding row vector in $\left(\right)$ and repeat from step 1.

#### 3.2.2. Diffuse-Specular Separability Check

#### 3.2.3. PS in UV Space

#### 3.2.4. Outlier Estimation

#### 3.3. Surface Normal Refinement

#### 3.3.1. Specular Parameter Initialisation

#### 3.3.2. Surface Normal Refinement in DRM

## 4. Performance Evaluation on Surface Orientation Estimation

#### 4.1. Evaluations Using Synthetic Images

#### 4.2. Evaluations Using Real Images

## 5. Conclusions and Future Works

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Ackermann, J.; Goesele, M. A survey of photometric stereo techniques. J. Found. Trends Comput. Graph. Vis.
**2015**, 9, 149–254. [Google Scholar] [CrossRef] - Klasing, K.; Althoff, D.; Wollherr, D.; Buss, M. Comparison of surface normal estimation methods for range sensing applications. In Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan, 12–17 May 2009; pp. 3206–3211. [Google Scholar]
- Woodham, R.J. Photometric method for determining surface orientation from multiple images. Opt. Eng.
**1980**, 19, 139–144. [Google Scholar] [CrossRef] - Shi, B.; Zhe, W.; Mo, Z.; Duan, D.; Yeung, S.; Tan, P. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. In Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 3707–3716. [Google Scholar]
- Shell, J.R. Bidirectional Reflectance: An Overview with Remote Sensing Applications & Measurement Recommendations. 2004, pp. 37–50. Available online: http://web.gps.caltech.edu/~vijay/Papers/BRDF/shell-04.pdf (accessed on 4 January 2022).
- Nayar, S.K.; Ikeuchi, K.; Kanade, T. Determining shape and reflectance of hybrid surfaces by photometric sampling. IEEE Trans. Robot. Autom.
**1990**, 6, 418–431. [Google Scholar] [CrossRef] - Torrance, K.E.; Sparrow, E.M. Theory for off-specular reflection from roughened surfaces. J. Opt. Soc. Am.
**1967**, 57, 1105–1114. [Google Scholar] [CrossRef] - Georghiades, A. Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo. In Proceedings of the Proceedings Ninth IEEE International Conference on Computer Vision, Nice, France, 13–16 October 2003; pp. 816–823. [Google Scholar]
- Goldman, D.B.; Curless, B.; Hertzmann, A.; Seitz, S.M. Shape and spatially-varying BRDFs from photometric stereo. IEEE Trans. PAMI
**2010**, 32, 1060–1071. [Google Scholar] [CrossRef] [PubMed] - Ward, G.J. Measuring and modeling anisotropic reflection. In SIGGRAPH92: Proceedings of the 19th Annual Conference and Exhibition on Computer Graphics and Interactive Techniques; Association for Computing Machinery: New York, NY, USA, 1992; pp. 265–272. [Google Scholar]
- Alldrin, N.G.; Zickler, T.; Kriegman, D.J. Photometric stereo with non-parametric and spatially-varying reflectance. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; pp. 1–8. [Google Scholar]
- Higo, T.; Matsushita, Y.; Ikeuchi, K. Consensus photometric stereo. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 1157–1164. [Google Scholar]
- Shi, B.; Tan, P.; Matsushita, Y.; Ikeuchi, K. Bipolynomial modeling of low-frequency reflectances. IEEE Trans. PAMI
**2014**, 36, 1078–1091. [Google Scholar] [CrossRef] [PubMed] - Ikehata, S.; Aizawa, K. Photometric stereo using constrained bivariate regression for general isotropic surfaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 23–28. [Google Scholar]
- Santo, H.; Samejima, M.; Sugano, Y.; Shi, B.; Matsushita, Y. Deep photometric stereo network. In Proceedings of the International Workshop on PBDL in Conjunction with IEEE ICCV, Venice, Italy, 22–29 October 2017; pp. 1–9. [Google Scholar]
- Santo, H.; Samejima, M.; Sugano, Y.; Shi, B.; Matsushita, Y. Deep photometric stereo networks for determining surface normal and reflectances. IEEE Trans. Pattern Anal. Mach. Intell.
**2022**, 44, 114–128. [Google Scholar] [CrossRef] [PubMed] - Taniai, T.; Maehara, T. Neural inverse rendering for general reflectance photometric stereo. In Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018. [Google Scholar]
- Ikehata, S. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–18. [Google Scholar]
- Wu, L.; Ganesh, A.; Shi, B.; Matsushita, Y.; Wang, Y.; Ma, Y. Robust photometric stereo via low-rank matrix completion and recovery. In Proceedings of the 10th Asian Conference on Computer Vision, Queenstown, New Zealand, 8–12 November 2010; pp. 703–717. [Google Scholar]
- Ikehata, S.; Wipf, D.; Matsushita, Y.; Aizawa, K. Robust photometric stereo using sparse regression. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 318–325. [Google Scholar]
- Barsky, S.; Petrou, M. The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Trans. PAMI
**2003**, 25, 1239–1252. [Google Scholar] [CrossRef] [Green Version] - Shafer, S.A. Using colour to separate reflection components. Color Res. Appl.
**1985**, 10, 210–218. [Google Scholar] [CrossRef] [Green Version] - Zickler, T.; Mallick, S.P.; Kriegman, D.J.; Belhumeur, P.N. Color subspace as photometric invariants. Int. J. Comput. Vis.
**2008**, 79, 13–30. [Google Scholar] [CrossRef] [Green Version] - Hartley, R.; Zisserman, A. Chapter 6: Camera Models. In Multiple View Geometry; Cambridge University Press: Cambridge, UK, 2003; pp. 153–177. [Google Scholar]
- Angelopoulou, E. Specular highlight detection based on the fresnel reflection coefficient. In Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, 14–21 October 2007; pp. 1–8. [Google Scholar]
- Blinn, J.F. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th Annual Conference on Computer Graphics and Interactive Techniques, San Jose, CA, USA, 20–22 July 1977; pp. 192–198. [Google Scholar]
- Ngan, A.; Dur, F.; Matusik, W. Experimental validation of analytical BRDF models. In Proceedings of the ACM SIGGRAPH 2004 Sketches, Los Angeles, CA, USA, 8–12 August 2004; p. 90. [Google Scholar]
- Kao, Y.H.; Roy, B.V. Directed Principal Component Analysis. Oper. Res.
**2014**, 62, 957–972. [Google Scholar] [CrossRef] [Green Version] - Pope, A.J. The Statics of Residuals and the Detection of Outliers; NOAA Technical Report; 1976; Volume 1, p. 65. Available online: https://www.ngs.noaa.gov/PUBS_LIB/TRNOS65NGS1.pdf (accessed on 4 January 2022).
- Liu, C.; Freeman, W.T.; Szeliski, R.; Kang, S.B. Noise estimation from a single image. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA, 17–22 June 2006; pp. 901–908. [Google Scholar]
- Artusi, A.; Banterle, F.; Chetverikov, D. A survey of specularity removal methods. Comput. Graph. Forum
**2011**, 30, 2208–2230. [Google Scholar] [CrossRef] - Barrow, H.; Tanenbaum, J. Recovering intrinsic scene characteristic from images. Proc. Comput. Vis. Syst.
**1978**, 2, 3–26. [Google Scholar] - Matusik, W.; Pfister, H.; Br, M.; McMillan, L. A Data-Driven Reflectance Model. ACM Trans. Graph.
**2003**, 22, 759–769. [Google Scholar] [CrossRef] - Shi, B.; Tan, P.; Matsushita, Y.; Ikeuchi, K. Elevation angle from reflectance monotonicity: Photometric stereo for general isotropic reflectances. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; pp. 455–468. [Google Scholar]

**Figure 4.**Evaluation of surface normal refinement: (

**a**) Image irradiance under the first illuminant; (

**b**) Angular error of surface orientations without surface normal refinement in degrees; (

**c**) Angular error of surface orientations with surface normal refinement in degrees; (

**d**) Improvement of surface orientation estimation by including surface normal refinement in percentage.

**Figure 5.**(

**a**) Image irradiance for the first six material BRDFs under the first illuminant; (

**b**) Angular error of surface orientation estimation for the first six material BRDFs; (

**c**) Angular error of surface orientation for the twenty-four dielectric materals.

**Figure 6.**Evaluations on surface oreintation estimation: (

**a**) Image irradiance under the first illuminant; (

**b**) Estimated normal map; (

**c**) Angular error of surface orientations; (

**d**) Angular error of surface orientations on five datasets using ten different PS methods.

Index | Colour | ${\mathit{\psi}}^{\mathit{i},\mathit{j}}$ | Error of ${\overline{\mathbf{d}}}_{{\mathit{RGB}}^{\mathit{i},\mathit{j}}}$ | Mean Improvement |
---|---|---|---|---|

1 | red | $57.{74}^{\circ}$ | $1.{15}^{\circ}$ | $23.39\%$ |

2 | yellow | $35.{26}^{\circ}$ | $1.{33}^{\circ}$ | $32.85\%$ |

3 | green | $57.{74}^{\circ}$ | $1.{15}^{\circ}$ | $23.42\%$ |

4 | cyan | $35.{26}^{\circ}$ | $1.{32}^{\circ}$ | $32.04\%$ |

5 | blue | $57.{74}^{\circ}$ | $1.{15}^{\circ}$ | $23.15\%$ |

6 | magenta | $35.{26}^{\circ}$ | $1.{33}^{\circ}$ | $32.24\%$ |

${\mathit{k}}_{\mathit{d}}/{\mathit{k}}_{\mathit{s}}$ | $\mathit{\beta}$ | Mean | Median | First Quantile | Third Quantile | Error of ${\overline{\mathbf{d}}}_{\mathit{RGB}}$ |
---|---|---|---|---|---|---|

$0.4/0.2$ | 100 | $32.25\%$ | $34.33\%$ | $15.76\%$ | $54.23\%$ | $1.{23}^{\circ}$ |

$0.4/0.4$ | 100 | $31.14\%$ | $32.97\%$ | $15.21\%$ | $53.75\%$ | $1.{58}^{\circ}$ |

$0.4/0.8$ | 100 | $30.48\%$ | $32.77\%$ | $14.27\%$ | $52.90\%$ | $2.{34}^{\circ}$ |

$0.4/0.2$ | 20 | $27.49\%$ | $26.70\%$ | $6.80\%$ | $51.66\%$ | $2.{99}^{\circ}$ |

$0.4/0.4$ | 20 | $25.92\%$ | $25.98\%$ | $6.67\%$ | $51.25\%$ | $5.{06}^{\circ}$ |

$0.4/0.8$ | 20 | $25.02\%$ | $24.07\%$ | $6.57\%$ | $50.81\%$ | $8.{30}^{\circ}$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Li, B.; Furukawa, T.
DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces. *J. Imaging* **2022**, *8*, 40.
https://doi.org/10.3390/jimaging8020040

**AMA Style**

Li B, Furukawa T.
DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces. *Journal of Imaging*. 2022; 8(2):40.
https://doi.org/10.3390/jimaging8020040

**Chicago/Turabian Style**

Li, Boren, and Tomonari Furukawa.
2022. "DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces" *Journal of Imaging* 8, no. 2: 40.
https://doi.org/10.3390/jimaging8020040