# Toward 3D Reconstruction of Outdoor Scenes Using an MMW Radar and a Monocular Vision Sensor

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. System Model

**Figure 1.**Sensors system geometry: ${R}_{c}$ and ${R}_{r}$ are the camera and radar frames, respectively. Polar coordinates ${m}_{r}(\alpha ,r)$ of the target are provided by the radar data, but not the elevation angle. The light ray L and the projected point p in the image ${I}_{c}$ are shown together with the horizontal radar plane.

## 3. Three-Dimensional Reconstruction

**Figure 2.**An illustration of the elevation map generation exploiting radar and vision complementarity.

**Figure 3.**In order to achieve the 3D reconstruction, three preliminary steps must be carried out: simultaneous data acquisition by the sensors, the estimation of the transformation between the sensor frames and the extraction and matching of features from the camera image and the radar panoramic. (

**a**) Data acquisition; (

**b**) System calibration; (

**c**) Features extraction and matching.

**Figure 4.**The 3D reconstructed point Q is the intersection of light ray L and the sphere C at α. ${m}_{r}$ is the projected 2D point on the horizontal radar plane, and $VP$ is the vertical plane of the target at α.

## 4. System Calibration

#### 4.1. Related Work

#### 4.2. The Proposed Calibration Method

#### 4.3. Inter-Distance Constraint

**Figure 5.**The triangle formed by $M1$, $M2$ (3D points in the camera frame) and ${O}_{c}$ is shown. ${d}_{12}$ is the Euclidean distance between $M1$ and $M2$ that is supposed to be known, and ${D}_{1}$, ${D}_{2}$ are their depths relative to ${O}_{c}$.

#### 4.4. Relaxation of the Inter-Distance Constraint

**Figure 6.**The displacement of the system around a fixed scene gives more geometric equations. An illustration of this process is shown. The matrix ${A}_{k}$ represent the transformation between one position and another.

## 5. Uncertainty Analysis

**Figure 7.**Calibration error with respect to the number of points.

**Left column**: translation error in meters;

**Right column**: rotation error in radians. The graphs show the mean and the standard deviation of RMSE upon six iterations. The number of matches is increased by a step of one from three to 30, and the noise level is: ±2 p, ±${2}^{\circ}$ for α, ± 2 cm for r. (

**a**) First calibration method; (

**b**) Second calibration method.

**Figure 8.**Calibration error with respect to the noise level.

**Left**: translation error in meters;

**Right**: rotation error in radians. The graphs show the mean and the standard deviation of RMSE upon six reiterations with 10 matches used. (

**a**) First calibration method; (

**b**) Second calibration method.

**Figure 9.**Reconstruction error with respect to the noise level. The error is in meters relative to the point depths r. The mean and standard deviation of the RMSE, over 50 reconstructed points, are shown.

**Figure 10.**Reconstruction error with respect to the baseline with a noise level corresponding to ±2 p, ±${2}^{\circ}$ for α and ±2 cm for r. The error is in meters relative to the point depths (r). The mean and standard deviation over 50 reconstructed points are shown.

**Figure 11.**Illustration of the baseline effect on the reconstruction error. (

**a**) Intersection uncertainty using a narrow baseline; (

**b**) Intersection uncertainty using a wide baseline.

**Figure 12.**The effect of the baseline is illustrated. The intersection of the uncertainty regions of each sensor projection is also shown. (

**a**) Short baseline; (

**b**) Wide baseline; (

**c**) Wide baseline.

**Figure 13.**The intersection of the uncertainty regions of each sensor’s projection: (

**a**) the ideal case of the geometric reconstruction; (

**b**) introducing uncertainty regions of each sensor to the geometric model; (

**c**) the error intersection region; (

**d**) uncertainity intersection region in respect to the baseline for the stereo reconstruction method.

**Figure 14.**Reconstruction error with respect to the mean depth of the targets. The error is in meters. The mean and standard deviation of the RMSE, over 50 reconstructed points, are shown. (

**a**) Noise added corresponding to 0 p, ±${2}^{\circ}$ for α and ±2cm for r; (

**b**) Noise added corresponding to ±2 p, ${0}^{\circ}$ for α and 0 cm for r.

**Figure 15.**Illustration of the baseline effect on the reconstruction error. (

**a**) Noise added only to radar data; (

**b**) Noise added only to camera data; (

**c**) Noise added to both camera and radar data.

**Figure 16.**Reconstruction error with respect to the mean depth of the targets with a noise level corresponding to ±2 p, ±${2}^{\circ}$ for α and ±2 cm for r. The error is in meters. The mean and standard deviation of the RMSE, over 50 reconstructed points, are shown.

## 6. Experimental Results

Camera Characteristics | |
---|---|

Sensor technology | CMOS |

Sensor size | $4.512\times 2.880$ mm |

Pixel size | $0.006$ mm |

Resolution in pixel $(h\times v)$ | $752\times 480$ |

Focal distance | 8 mm |

Viewing angle | $43\phantom{\rule{0.166667em}{0ex}}\times \phantom{\rule{0.166667em}{0ex}}$${25}^{\circ}$ |

Radar Characteristics | |

Carrier frequency | 24 GHz |

Antenna gain | 20 dB |

Range | 3 to 100 m |

Angular resolution | ${4}^{\circ}$ |

Distance resolution | 1 m |

Distance precision | 0.02 m |

Viewing angle | $360\phantom{\rule{0.166667em}{0ex}}\times \phantom{\rule{0.166667em}{0ex}}$${20}^{\circ}$ |

**Figure 17.**Radar and camera system. The image to the right presents the zoom in of the sensors system (the radar to the right and the camera to the left).

**Figure 18.**Center detection of targets in both the camera image and radar panoramic. (

**a**) Radar target extraction; (

**b**) Camera target extraction.

**Figure 19.**An image and a panoramic of targets. The targets are numbered from 1 to 8: one Luneburg lens and seven trihedral corners. The yellow crosses indicate the centers of the targets. The variations of amplitude of the reflected signal are introduced by the nature and orientation of each target. Manually-extracted matches between the image and the PPI are shown.

**Figure 20.**Reconstruction results using the inter-distance constraint for calibration. (

**a**) Camera image of the eight canonical targets: one Luneburg lens and seven trihedral corners. The yellow crosses indicate the center of the targets; (

**b**) Radar image with eight canonical targets; (

**c**) The reconstruction results of both of our reconstruction methods (circular points) and the stereo head method as the ground truth (squared points). The radar position is indicated by the letter R.

Target Coordinates in Meters with the Stereo Head (Ground Truth) | ||||||||
---|---|---|---|---|---|---|---|---|

X | 5.41 | 7.45 | 7.47 | 7.44 | 7.53 | 4.59 | 12.53 | 13.01 |

Y | 1.32 | 1.45 | 0.70 | −0.03 | −0.78 | −0.68 | 2.24 | −1.03 |

Z | −0.38 | 0.05 | 0.02 | −0.01 | −0.06 | −0.92 | −1.05 | 0.33 |

Targets Coordinates in Meters with the Developed Method | ||||||||

X | 5.50 | 7.55 | 7.53 | 7.46 | 7.51 | 4.57 | 12.64 | 12.98 |

Y | 0.83 | 0.86 | 0.11 | −0.62 | −1.37 | −1.12 | 1.40 | −1.89 |

Z | −0.35 | 0.05 | 0.01 | −0.03 | −0.08 | −0.89 | −1.15 | 0.21 |

Error in Meters (Euclidean Distance) | ||||||||

0.49 | 0.60 | 0.59 | 0.59 | 0.59 | 0.44 | 0.85 | 0.87 | |

Error mean in meters = 0.63 | ||||||||

Error standard deviation in meters = 0.15 |

Targets Coordinates in Meters with the Stereo Head (Ground Truth) | ||||||||
---|---|---|---|---|---|---|---|---|

X | 14.21 | 14.40 | 14.58 | 16.85 | 17.91 | 18.55 | 20.04 | 21.70 |

Y | −3.08 | −0.02 | 2.05 | −4.53 | 0.53 | 3.88 | −3.21 | −0.15 |

Z | −0.19 | −0.18 | −0.17 | −1.10 | 0.28 | −0.90 | 0.98 | −0.61 |

Targets Coordinates in Meters with the Developed Method | ||||||||

X | 12.57 | 12.56 | 12.53 | 15.20 | 15.94 | 16.59 | 18,56 | 19,52 |

Y | −3.09 | 0.03 | 2.09 | −4.59 | 0.55 | 3.96 | −3,20 | −0,058 |

Z | 0.09 | 0.14 | 0.19 | −1.03 | 0.35 | −0.85 | 0,88 | −0,79 |

Error in Meters (Euclidean Distance) | ||||||||

0.05 | 0.04 | 0.06 | 0.05 | 0.05 | 0.03 | 0.08 | 0.11 | |

Error- mean in meters = 0.058 | ||||||||

Error- standard deviation in meters = 0.024 |

**Figure 21.**Top line: Camera images of the eight canonical targets. middle line: Radar images with eight canonical targets. The reconstruction results from both our reconstruction methods (circular points) and the stereo head method as the ground truth (squared points). The radar position is indicated by the letter R. (

**a**) First position; (

**b**) Second position; (

**c**) Third position; (

**d**) First position; (

**e**) Second position; (

**f**) Third position; (

**g**) Reconstruction results.

**Figure 22.**Results of a reconstructed urban scene using the camera/radar system and the second calibration method. The results are enhanced with texture mapping (this figure is clearer in color). (

**a**) Camera image of an urban scene; (

**b**) Segmented image (polygons are shown in red); (

**c**) Part of the radar image of the same scene; (

**d**) Segmented radar image; (

**e**) Results of the reconstruction using Delaunay triangulation; (

**f**) Enhanced results with texture; (

**g**) Another view of the 3D results; (

**h**) Another view of the 3D results.

**Figure 23.**Results of a reconstructed urban scene using the camera/radar system, and the second calibration method. The results are enhanced with texture mapping (this figure is clearer in color). (

**a**) Camera image of the an urban scene; (

**b**) Segmented camera image; (

**c**) Part of the radar image of the same scene; (

**d**) Segmented radar image; (

**e**) Results of the reconstruction using Delaunay triangulation; (

**f**) Enhanced results with texture; (

**g**) Another view of the 3D results; (

**h**) Another view of the 3D results.

## 7. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## References

- Henry, P.; Krainin, M.; Herbst, E.; Ren, X.; Fox, D. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res.
**2012**, 31, 647–663. [Google Scholar] [CrossRef] - Pfitzner, C.; Antal, W.; Hess, P.; May, S.; Merkl, C.; Koch, P.; Koch, R.; Wagner, M. 3D Multi-Sensor Data Fusion for Object Localization in Industrial Applications. In Proceedings of the 41st International Symposium on Robotics, ISR/Robotik 2014, Munich, Germany, 2–3 June 2014; pp. 1–6.
- Guivant, J.E.; Marden, S.; Pereida, K. Distributed Multi Sensor Data Fusion for Autonomous 3D Mapping. In Proceedings of the IEEE 2012 International Conference onthe Indoor Positioning and Indoor Navigation (IPIN), Sydney, Australia, 13–15 Novembre 2012; pp. 1–11.
- Bhagawati, D. Photogrammetry and 3-D Reconstruction-the State of the Art. In Proceedings of the ASPRS 2000, Washington, DC, USA, 22–26 May 2000.
- Kordelas, G.; Perez-Moneo Agapito, J.; Vegas Hernandez, J.; Daras, P. State-of-the-Art Algorithms for Complete 3D Model Reconstruction. In Proceedings of the Engage Summer School, Zermatt, Switzerland, 13–15 September 2010; pp. 1–15.
- Musialski, P.; Wonka, P.; Aliaga, D.G.; Wimmer, M.; Gool, L.; Purgathofer, W. A Survey of Urban Reconstruction. Comput. Graph. Forum
**2013**, 32, 146–177. [Google Scholar] [CrossRef] - Gallup, D.; Frahm, J.M.; Mordohai, P.; Yang, Q.; Pollefeys, M. Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8.
- Pollefeys, M.; Nistér, D.; Frahm, J.M.; Akbarzadeh, A.; Mordohai, P.; Clipp, B.; Engels, C.; Gallup, D.; Kim, S.J.; Merrell, P.; et al. Detailed Real-Time Urban 3d Reconstruction from Video. Int. J. Comput. Vis.
**2008**, 78, 143–167. [Google Scholar] [CrossRef] - Royer, E.; Lhuillier, M.; Dhome, M.; Lavest, J.M. Monocular Vision for Mobile Robot Localization and Autonomous Navigation. Int. J. Comput. Vis.
**2007**, 74, 237–260. [Google Scholar] [CrossRef] - Kim, C.; Kim, B.; Kim, H. 4D CAD Model Updating Using Image Processing-Based Construction Progress Monitoring. Autom. Constr.
**2013**, 35, 44–52. [Google Scholar] [CrossRef] - Aydin, C.C. Designing Building Façades for the Urban Rebuilt Environment with Integration of Digital Close-Range Photogrammetry and Geographical Information Systems. Autom. Constr.
**2014**, 43, 38–48. [Google Scholar] [CrossRef] - Yang, M.D.; Chao, C.F.; Huang, K.S.; Lu, L.Y.; Chen, Y.P. Image-Based 3D Scene Reconstruction and Exploration in Augmented Reality. Autom. Constr.
**2013**, 33, 48–60. [Google Scholar] [CrossRef] - Furukawa, Y.; Ponce, J. Accurate, Dense, and Robust Multiview Stereopsis. IEEE Trans. Pattern Anal. Mach. Intell.
**2010**, 32, 1362–1376. [Google Scholar] [CrossRef] [PubMed] - Zhu, Z.; Kanade, T. Modeling and Representations of Large-scale 3D Scenes. Int. J. Comput. Vis.
**2008**, 78, 119–120. [Google Scholar] [CrossRef] - Lafarge, F.; Keriven, R.; Brédif, M.; Vu, H.H. A Hybrid Multiview Stereo Algorithm for Modeling Urban Scenes. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 5–17. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zhang, Y.; Li, Q.; Lu, H.; Liu, X.; Huang, X.; Song, C.; Huang, S.; Huang, J. Optimized 3D Street Scene Reconstruction from Driving Recorder Images. Remote Sens.
**2015**, 7, 9091–9121. [Google Scholar] [CrossRef] - Rouveure, R.; Monod, M.; Faure, P. High Resolution Mapping of the Environment with a Ground-Based Radar Imager. In Proceedings of the IEEE International Radar Conference-Surveillance for a Safer World, Bordeaux, France, 12–16 October 2009; pp. 1–6.
- Smisek, J.; Jancosek, M.; Pajdla, T. 3D with Kinect. In Consumer Depth Cameras for Computer Vision; Springer: Berlin, Germany, 2013; pp. 3–25. [Google Scholar]
- Schindhelm, C.K. Evaluating SLAM Approaches for Microsoft Kinect. In Proceedings of the Eighth International Conference on Wireless and Mobile Communications, Venice, Italy, 24–29 June 2012; pp. 402–407.
- Forlani, G.; Nardinocchi, C.; Scaioni, M.; Zingaretti, P. Complete Classification of Raw LiDAR Data and 3D Reconstruction of Buildings. Pattern Anal. Appl.
**2006**, 8, 357–374. [Google Scholar] [CrossRef] - Stamos, I.; Allen, P.K. 3-D Model Construction Using Range and Image Data. Proc. IEEE Comput. Vis. Pattern Recogn.
**2000**, 1, 531–536. [Google Scholar] - Bok, Y.; Choi, D.G.; Kweon, I.S. Sensor Fusion of Cameras and a Laser for City-Scale 3D Reconstruction. Sensors
**2014**, 14, 20882–20909. [Google Scholar] [CrossRef] - Cheng, L.; Tong, L.; Chen, Y.; Zhang, W.; Shan, J.; Liu, Y.; Li, M. Integration of LiDAR Data and Optical Multi-View Images for 3D Reconstruction of Building Roofs. Opt. Lasers Eng.
**2013**, 51, 493–502. [Google Scholar] [CrossRef] - Duraisamy, P.; Jackson, S.; Namuduri, K.; Alam, M.S.; Buckles, B. Robust 3D Reconstruction Using LiDAR and N-Visual Image. Proc. SPIE
**2013**. [Google Scholar] [CrossRef] - Williams, K.; Olsen, M.J.; Roe, G.V.; Glennie, C. Synthesis of Transportation Applications of Mobile LiDAR. Remote Sens.
**2013**, 9, 4652–4692. [Google Scholar] [CrossRef] - Bertozzi, M.; Bombini, L.; Cerri, P.; Medici, P.; Antonello, P.C.; Miglietta, M. Obstacle Detection and Classification Fusing Radar and Vision. In Proceeding of the IEEE Intelligent Vehicles Symposium, Eindhoven, Netherlands, 4–6 June 2008; pp. 608–613.
- Roy, A.; Gale, N.; Hong, L. Fusion of Doppler Radar and Video Information for Automated Traffic Surveillance. In Proceedings of the 12th International Conference on IEEE Information Fusion, Seattle, WA, USA, 6–9 July 2009; pp. 1989–1996.
- Hofmann, U.; Rieder, A.; Dickmanns, E.D. Radar and Vision Data Fusion for Hybrid Adaptive Cruise Control on Highways. Mach. Vis. Appl.
**2003**, 14, 42–49. [Google Scholar] [CrossRef] - Wang, T.; Zheng, N.; Xin, J.; Ma, Z. Integrating Millimeter Wave Radar with a Monocular Vision Sensor for on-Road Obstacle Detection Applications. Sensors
**2011**, 11, 8992–9008. [Google Scholar] [CrossRef] [PubMed] - Bombini, L.; Cerri, P.; Medici, P.; Alessandretti, G. Radar-Vision Fusion for Vehicle Detection. Int. Workshop Intell. Transp.
**2006**, 3, 65–70. [Google Scholar] - Sugimoto, S.; Tateda, H.; Takahashi, H.; Okutomi, M. Obstacle Detection Using Millimeter-Wave Radar and Its Visualization on Image Sequence. Int. Conf. Pattern Recogni.
**2004**, 3, 342–345. [Google Scholar] - El-Natour, G.; Ait-Aider, O.; Raphael, R.; Francois, B.; Faure, P. Sensor Fusion of Cameras and a Laser for City-Scale 3D Reconstruction. Int. Conf. Comput. Vis. Theory Appl.
**2015**, 14, 20882–20909. [Google Scholar] - Skolnik, M.I. Introduction to Radar Systems; CRC Press: Boca Raton, FL, USA, 2001. [Google Scholar]
- Bouguet, J.Y. Camera Calibration Toolbox for Matlab. Available online: http://www.vision.caltech.edu/bouguetj/calib_doc/ (accessed on 30 September 2015).
- Kennedy, E.S. A Fifteenth-Century Planetary Computer: al-Kāshī’s Tabaq al-Manāteq. II. Longitudes, Distances, and Equations of the Planets. Isis
**1952**, 43, 42–50. [Google Scholar] [CrossRef] - Harris, C.; Stephens, M. A Combined Corner and Edge Detector. Alv. Vis. Conf.
**1988**, 15, 50. [Google Scholar]

© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Natour, G.E.; Ait-Aider, O.; Rouveure, R.; Berry, F.; Faure, P.
Toward 3D Reconstruction of Outdoor Scenes Using an MMW Radar and a Monocular Vision Sensor. *Sensors* **2015**, *15*, 25937-25967.
https://doi.org/10.3390/s151025937

**AMA Style**

Natour GE, Ait-Aider O, Rouveure R, Berry F, Faure P.
Toward 3D Reconstruction of Outdoor Scenes Using an MMW Radar and a Monocular Vision Sensor. *Sensors*. 2015; 15(10):25937-25967.
https://doi.org/10.3390/s151025937

**Chicago/Turabian Style**

Natour, Ghina El, Omar Ait-Aider, Raphael Rouveure, François Berry, and Patrice Faure.
2015. "Toward 3D Reconstruction of Outdoor Scenes Using an MMW Radar and a Monocular Vision Sensor" *Sensors* 15, no. 10: 25937-25967.
https://doi.org/10.3390/s151025937