# Quantified, Interactive Simulation of AMCW ToF Camera Including Multipath Effects

^{*}

## Abstract

**:**

## 1. Introduction

- Enhancement of the Reflective Shadow Map (RSM) algorithm [6] for GPU-based, interactive, single-bounce, image-space, multipath interference simulation.
- BRDF-based reflection simulation for measured real-world materials.
- Extension of the simulation model to include realistic electronic and optical shot noise.

- Measurement of isotropic BRDF at 850 nm wavelength for several specified materials that can be purchased worldwide and thus can be used to reproduce scenes reliably.
- Quantitative evaluations of the proposed simulator based on AMCW ToF camera acquisition of real-world reference scenes. We clearly see the improved ToF simulation results of our single-bounce approach over direct simulation in terms of quality, and over higher-order global illumination simulation [4] in terms of computational performance.
- Publicly available simulator, BRDF data including references to material vendors, geometry of the reference scenes, and real AMCW ToF camera measurements, in order to promote further activities in quantitative evaluation of AMCW ToF simulation.

## 2. Related Work

## 3. Time-of-Flight Simulation

#### 3.1. Direct Light Propagation

#### 3.2. Sensor Pixel Model

#### 3.3. Multipath Simulation

#### 3.4. Noise Model

**SSE**$=0.0391$ and (root mean square error)

**RMSE**$=0.0112$. Applying the model is done by transforming the charge values from Equations (13) and (14) into the FT domain, computing the Gaussian parameters by evaluating the variance curve for the given intensity, generating the noise value using a random number and the variance, and, finally, back-transforming this value in the original domain of the charge values. This noise value is then added to the noise-free charge value in order to get the final charge value.

## 4. NIR BRDF Measurements

#### 4.1. Measurement Setup

#### 4.2. Extrapolating BRDF Measurements

## 5. Results

#### 5.1. BRDF Measurement

#### Open Science

#### 5.2. Simulator Evaluation

**Corner**is a simple corner scene without an additional cube, in the

**CornerCube**scene an additional cube is placed directly in the corner, and in the

**CornerCubeShift**the cube is shifted by 10 cm from each corner wall. We have setup the three scenes with

**Material #1**and

**Material #5**.

**CamCube**, while the simulation results are labeled as

**SimDirect**(only direct reflection is simulated) and

**SimSingle**(additional single-bounce indirect reflections are simulated). Furthermore, we add the ground truth depth information for comparison (

**GroundTruth**). Our evaluation indirectly compares to Lambers et al. [2], as

**SimDirect**essentially is the approach in Lambers et al. [2] enhanced with the noise model described in Section 3.4 and BRDF-based reflection (instead of Lambertian).

**GroundTruth**, the

**CamCube**measurements, and the

**SimDirect**and

**SimSingle**simulations. Figure 7 and Figure 8 give additional insight into range simulation results by showing the signed differences between the simulation and the

**CamCube**measurements for the scenes

**CornerCube, CornerCubeShift**and the explicit range values along row 100 for all three scenes, respectively. As expected, the real ToF data exhibits significant multipath effects in all three scenes. Considering the simulation without multipath component (

**SimDirect**), the resulting range values are close to the ground truth depth. This is consistent with the ToF measurement principle, which explicitly considers direct reflection only. Table 2 states all error values for all scenes and material with respect to the measured

**CamCube**data. Especially for scenes

**Corner**and

**CornerCube**, our approach outperforms

**SimDirect**since it captures multipath effects in the corners. In

**CornerCubeShifted**, our approach still decreases the errors by more than $50\%$. In summary, we find that adding single-bounce indirect reflections (

**SimSingle**) significantly improves the simulation results with respect to the

**CamCube**measurements. This is especially the case for the

**Corner**and the

**CornerCube**scenes. For the

**CornerCubeShift**scene, however, the deviation between the ToF measurement

**CamCube**and the simulation including single-bounce reflections

**SimSingle**still deviate, mainly in the visual corners between the base corner and the inserted, shifted cube.

**CornerCube**scene, where have placed an aluminum cube with an edge length of 5 cm into the glossy corner (

**Material #1**). In our simulation, we have used a Cook-Torrance BRDF to model the reflection behavior of the aluminum cube. In this highly reflective scenario, the multipath effects have a strong influence on the

**CamCube**measurements (see range values in row 108, Figure 9 right). Here, the cube nearly vanishes in the distance measurements between pixels 90 and 110.

**SimSingle**, in comparison, cannot capture this camera behavior, as no higher order multipath effects are simulated.

**SimSingle**simulations and the real ToF measurements

**CamCube**are comparable, whereas the noise level for the direct simulation

**SimDirect**is higher. This is due to the fact that the total amount of charge is lower and that, in this case, the additive Poisson noise is with a fixed amplitude. Thus, the relative impact of the noise is higher in the case of the lower overall charge in the

**SimDirect**simulation. This effect gets very apparent for flat incident angle with respect to the direct light–surface impact .

## 6. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## References

- Kolb, A.; Barth, E.; Koch, R.; Larsen, R. Time-of-Flight cameras in computer graphics. Comput. Graph. Forum
**2010**, 29, 141–159. [Google Scholar] [CrossRef] - Lambers, M.; Hoberg, S.; Kolb, A. Simulation of Time-of-Flight sensors for evaluation of chip layout variants. IEEE Sens.
**2015**, 15, 4019–4026. [Google Scholar] [CrossRef] - Nair, R.; Meister, S.; Lambers, M.; Balda, M.; Hofmann, H.; Kolb, A.; Kondermann, D.; Jähne, B. Ground truth for evaluating time of flight imaging. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications; Springer: Berlin/Heidelberg, Germany, 2013; pp. 52–74. [Google Scholar]
- Meister, S.; Nair, R.; Kondermann, D. Simulation of Time-of-Flight sensors using global illumination. In Vision, Modeling & Visualization; The Eurographics Association: Geneve, Switzerland, 2013. [Google Scholar]
- Keller, M.; Kolb, A. Real-time simulation of time-of-flight sensors. Simul. Model. Pract. Theory
**2009**, 17, 967–978. [Google Scholar] [CrossRef] - Dachsbacher, C.; Stamminger, M. Reflective shadow maps. In Proceedings of the Symposium on Interactive 3D Graphics and Games, Washington, DC, USA, 3–6 April 2005; pp. 203–231. [Google Scholar]
- Schmidt, M.; Jähne, B. A physical model of time-of-flight 3D imaging systems, including suppression of ambient light. In Dynamic 3D Imaging; Springer: Berlin, Germany, 2009; pp. 1–15. [Google Scholar]
- Ritschel, T.; Dachsbacher, C.; Grosch, T.; Kautz, J. The state of the art in interactive global illumination. Comput. Graph. Forum
**2012**, 31, 160–188. [Google Scholar] [CrossRef] - Matusik, W.; Pfister, H.; Brand, M.; McMillan, L. A Data-Driven Reflectance Model. ACM Trans. Graph.
**2003**, 22, 759–769. [Google Scholar] [CrossRef] - Choe, G.; Narasimhan, S.G.; So Kweon, I. Simultaneous estimation of near IR BRDF and fine-scale surface geometry. In Proceedings of the IEEE 2016 IEEE Conference on Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2452–2460. [Google Scholar]
- Mutny, M.; Nair, R.; Gottfried, J.M. Learning the correction for Multi-Path deviations in Time-of-Flight Cameras. arXiv, 2015; arXiv:1512.04077. [Google Scholar]
- Lange, R.; Seitz, P. Solid-state time-of-flight range camera. IEEE J. Quantum Electron.
**2001**, 37, 390–397. [Google Scholar] [CrossRef] - Keller, A. Instant radiosity. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), Los Angeles, CA, USA, 3–8 August 1997; pp. 49–56. [Google Scholar]
- Conde, M.H.; Zhang, B.; Kagawa, K.; Loffeld, O. Low-light image enhancement for multiaperture and multitap systems. IEEE Photonics J.
**2016**, 8, 1–25. [Google Scholar] [CrossRef] - White, G.C.; Bennetts, R.E. Analysis of frequency count data using the negative binomial distribution. Ecology
**1996**, 77, 2549–2557. [Google Scholar] [CrossRef] - Freeman, M.F.; Tukey, J.W. Transformations related to the angular and the square root. Ann. Math. Stat.
**1950**, 21, 607–611. [Google Scholar] [CrossRef] - Schlick, C. An Inexpensive BRDF Model for Physically-based Rendering. Comput. Graph. Forum
**1994**, 13, 233–246. [Google Scholar] [CrossRef] - Achutha, S. BRDF Acquisition with Basis Illumination; Chapter 2.2 “BRDF Acquisition”. Ph.D. Thesis, The University of British Columbia, Vancouver, BC, Canada, 2006. [Google Scholar]
- Li, H.; Foo, S.C.; Torrance, K.E.; Westin, S.H. Automated three-axis gonioreflectometer for computer graphics applications. Opt. Eng.
**2006**, 45, 043605. [Google Scholar] [CrossRef] - Panzer, J.; Ponteggia, D. Inverse Distance Weighting for Extrapolating Balloon-Directivity-Plots; AES Convention 13; Audio Engineering Society: New York, NY, USA, 2011. [Google Scholar]
- All-Foam Products Company. Available online: http://products.allfoam.com/ (accessed on 20 December 2017).
- Evonik Cyro LCC. Available online: http://www.plexiglas-shop.com/ (accessed on 20 December 2017).
- Lindner, M.; Schiller, I.; Kolb, A.; Koch, R. Time-of-Flight Sensor Calibration for Accurate Range Sensing. Comput. Vis. Image Underst.
**2010**, 114, 1318–1328. [Google Scholar] [CrossRef] - Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI)
**2000**, 22, 1330–1334. [Google Scholar] [CrossRef] - Lindner, M. Calibration and Realtime Processing of Time-of-Flight Range Data. Ph.D. Thesis, University of Siegen, Computer Graphics Group, Siegen, Germany, 2010. [Google Scholar]

**Figure 1.**Scheme of a AMCW ToF camera including the pixel layout and the optically active pixel area ${A}_{S}$ (gray) with the two readout circuits A (red) and B (green) (

**a**); direct illumination and one path of single-bounce indirect illumination in the AMCW ToF simulation (

**b**).

**Figure 2.**Plot of noise model that describes the Gaussian variance as function of the mean value (transformed intensity) in Freeman–Tukey space with

**SSE**$=0.0391$ and

**RMSE**$=0.0112$.

**Figure 3.**The schematic gonioreflectometer measurement setup (

**left**) and a photo of the real setup (

**right**).

**Figure 4.**BRDF raw data and IDW interpolation results for materials #1, #5, #6 and #8 acquired for an incident light angle ${\theta}_{i}={30}^{\circ}$. The blue ray indicates the incident light direction, the red ray the ideally reflected incident light direction and the red curve the BRDF value related to the corresponding ray from the center to a point of this curve.

**Figure 5.**The geometry of our box scene (light brown) with the additional cube (blue) (

**a**); a photo of the AMCW ToF measurement setup (

**b**); and the positioning of the calibration pattern for ToF camera pose estimation (

**c**).

**Figure 6.**Range image comparison:

**GroundTruth**(left) and

**CamCube**(mid-left) compared with simulation using direct illumination (

**SimDirect**, mid-right) and single bounce reflection (

**SimSingle**, right) for the three test scenes

**Corner**(rows 1,4),

**CornerCube**(rows 2,5) and

**CornerCubeShift**(rows 3,6) for

**Material # 1**and

**Material # 5**.

**Figure 7.**Signed difference images to the

**CamCube**measurement for

**CornerCube**and

**CornerCubeShift**:

**GroundTruth**(Left) and the simulation with direct illumination (

**SimDirect**, middle) and with single bounce reflection (

**SimSingle**, Right) for

**Material #1**(top row) and

**Material #5**(bottom).

**Figure 8.**Range comparison for scan lines 100 for the three test scenes

**Corner**(

**Left**),

**CornerCube**(

**Middle**) and

**CornerCubeShift**(

**Right**) for

**Material #1 and Material #5**.

**Figure 9.**Evaluation of the corner scene with an aluminum cube (extraction of pixel regions $[70,130]\times [70,130]$): this scene comprises a significantly larger amount of multipath effects that cannot be fully covered by our single-bounce simulation method.

**Table 1.**Materials used for BRDF measurement. The PLEXIGLAS${}^{\circledR}$ provides more specular reflection, and the PVC rigid foam is rather diffuse.

Mat. No. | PLEXIGLAS${}^{\circledR}$ (Glossy) | Mat. No. | Guttagliss PVC (Diffuse) |
---|---|---|---|

1 | XT (allround), White WN297 GT | 5 | Rigid Foam, White |

2 | XT (allround), Red 3N570 GT | 6 | Rigid Foam, Red |

3 | XT (allround), Green 6N570 GT | 7 | Rigid Foam, Green |

4 | XT (allround), Blue 5N870 GT | 8 | Rigid Foam, Blue |

9 | Rigid Foam, Yellow | ||

10 | Rigid Foam, Gray |

**Table 2.**Evaluation of error for all simulation methods and scenes with respect to to the measured

**CamCube**data. For each method you can see the the mean-absolute-error (

**MAE**), mean-squared-error, (

**MSE**) and the root-mean-squared-error (

**RSME**).

Corner | CornerCube | CornerCubeShifted | |||||
---|---|---|---|---|---|---|---|

Material #1 | Material #5 | Material #1 | Material #5 | Material #1 | Material #5 | ||

GroundTruth | |||||||

MAE | 0.1001 | 0.0998 | 0.0831 | 0.0790 | 0.0969 | 0.0897 | |

MSE | 0.0103 | 0.0105 | 0.0071 | 0.0068 | 0.0106 | 0.0091 | |

RMSE | 0.1017 | 0.1025 | 0.0853 | 0.0823 | 0.1029 | 0.0956 | |

SimDirect | |||||||

MAE | 0.0885 | 0.0884 | 0.0728 | 0.0688 | 0.0846 | 0.0777 | |

MSE | 0.0084 | 0.0085 | 0.0079 | 0.0069 | 0.0086 | 0.0074 | |

RMSE | 0.0916 | 0.0924 | 0.0888 | 0.0829 | 0.0927 | 0.0859 | |

SimSingle | |||||||

MAE | 0.0238 | 0.0200 | 0.0194 | 0.0138 | 0.0396 | 0.0342 | |

MSE | 0.0007 | 0.0005 | 0.0006 | 0.0003 | 0.0023 | 0.0017 | |

RMSE | 0.0270 | 0.0226 | 0.0233 | 0.0171 | 0.0476 | 0.0417 |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bulczak, D.; Lambers, M.; Kolb, A.
Quantified, Interactive Simulation of AMCW ToF Camera Including Multipath Effects. *Sensors* **2018**, *18*, 13.
https://doi.org/10.3390/s18010013

**AMA Style**

Bulczak D, Lambers M, Kolb A.
Quantified, Interactive Simulation of AMCW ToF Camera Including Multipath Effects. *Sensors*. 2018; 18(1):13.
https://doi.org/10.3390/s18010013

**Chicago/Turabian Style**

Bulczak, David, Martin Lambers, and Andreas Kolb.
2018. "Quantified, Interactive Simulation of AMCW ToF Camera Including Multipath Effects" *Sensors* 18, no. 1: 13.
https://doi.org/10.3390/s18010013