# Airborne Optical Sectioning

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Wide Sythetic Aperture Sampling

**r**captured through the synthetic aperture that intersects the aperture plane at coordinates

**u**,

**v**and the freely adjustable synthetic focal plane at coordinates

**s**,

**t**. To compute the surface color

**R**of each point on the focal plane, we computationally integrate and normalize all sampled rays

_{s}_{,t}**r**

_{s}_{,t,u,v}of the synthetic aperture (with directional resolution

**U**,

**V**) that geometrically intersect at

**s**,

**t**:

**N**), which in this case is the ratio of the height of the drone

_{s}**h**(distance between the aperture plane and focal plane) to the diameter of the synthetic aperture (size of the sampled area).

**N**results in a shallow depth of field and consequently in a strong point spread of out-of-focus occluders over a large region of the entire integral image. Images of focused points remain concentrated in small spatial regions that require an adequate spatial resolution to be resolved.

_{s}**f**(number of sample points per unit area) at the focal plane can be determined by back-projecting the focal plane onto the imaging plane of the drone’s camera, as illustrated in Figure 1d. Assuming a pinhole camera model and parallel focal and image planes, this leads to:

_{s}**FOV**is the field of view of the drone’s camera,

_{c}**n**is its resolution (number of pixels on the image sensor), and

**h**is the height of the drone. Note, that in practice, the spatial sampling resolution is not uniform, as the focal plane and image plane are not necessarily parallel, and imaging with omnidirectional cameras is not uniform on the image sensor. Furthermore, the sampling resolution is affected by the quality of the drone’s pose estimation. Appendix A explains how we estimate the effective spatial sampling resolution that considers omnidirectional images and pose-estimation errors.

#### 2.2. Wide Synthetic Aperture Visualization

**FOV**, a ray through its projection center and a pixel at coordinates

_{v}**x**,

**y**in its image plane can be defined (blue in Figure 2a). This ray intersects the adjusted focal plane at coordinates

**s**,

**t**. With Equation (1), we now integrate all rays

**r**

_{s}_{,t,u,v}from

**s**,

**t**through each sample at coordinates

**u**,

**v**in the synthetic aperture plane that intersect the virtual camera’s circular aperture area

**A**of defined radius. The resulting surface color integral

_{v}**R**

_{s}_{,t}is used for the corresponding pixel’s color. This is repeated for all pixels in the virtual camera’s image plane to render the entire image. Interactive control over the virtual camera’s parameters allows real-time renderings of the captured scene by adjusting perspective, focus, and depth of field.

**x**,

**y**-registered, shallow depth-of-field images with varying focus). For every pixel at

**x**,

**y**in the virtual camera’s image plane, we now determine the optimal focus through all slices of the focal stack by maximizing the Sobel magnitude computed within a 3 × 3 pixel neighborhood. The surface color at the optimal focus is then used for the pixel’s color. Repeating this for all pixels leads to an image that renders the ground target within the desired focal range with a large depth of field, while occluders outside are rendered with a shallow depth of field.

## 3. Results

^{2}on the ground (see Appendix A for details) from an altitude of 35 m with a low-cost drone that uses a fixed omnidirectional camera. Since only 80° of the camera’s 178°

**FOV**was used, just a fraction of the sensor resolution was utilized (3.5 MP of a total of 14 MP in our example). In the 60 × 60 m region of interest shown in Figure 4b (corresponding to

**FOV**= 80° and

_{c}**n**= 3.5 MP), we find a ten times denser spatial sampling resolution than in the LiDAR example shown in Figure 4e (i.e., 74 samples/m

^{2}compared to 8 samples/m

^{2}). A total of 100 focal slices were computed and processed for achieving the results shown in Figure 4b.

## 4. Discussion and Conclusions

^{2}from an altitude of 40 m. The sampling precision is affected by the precision of the drone’s inertial measurement unit (IMU) and Global Positioning System (GPS) signal, which is required to integrate the LiDAR scan lines during flight. While IMUs suffer from drift over time, GPS is only moderately precise. In contrast, pose estimation for AOS is entirely based on computer vision and a dense set of visually registered multi-view images of many hundreds of overlapping perspectives. It is therefore more stable and precise. Since AOS processes unstructured records of perspective images, GPS and IMU data is only used for navigating the drone and does not affect the precision of pose estimation.

## Supplementary Materials

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A

^{2}for an altitude of

**h**= 35 m and the

**n**= 3.5 MP fraction of the image sensor that was affected by the scene portion of interest (

**FOV**= 80°). This implies that the size of each pixel on the focal plane was

_{c}**p**= 3.18 × 3.18 cm. The average lateral pose estimation error

**e**is determined by COLMAP as the average difference between back-projected image features after pose estimation and their original counterparts in the recorded images. For experiment 2, this was 1.33 pixels on the image plane (i.e.,

**e**= 4.23 cm on the focal plane). Thus, the effective size

**p’**of a sample on the focal plane that considers both the projected pixel size

**p,**and a given pose-estimation error

**e**, is

**p’**=

**p**+

**2e**(cf. Figure A1). For experiment 2, this leads to

**p’**= 3.18 cm + 2 × 4.23 cm = 11.64 cm, which is equivalent to an effective spatial sampling resolution of $f{\prime}_{s}$ = 74 samples/m

^{2}.

**Figure A1.**The size

**p**of a pixel projected onto the focal plane at the ground surface increases to

**p’**by twice the average pose-estimation error

**e**in both lateral directions. This reduces the effective spatial sampling resolution.

**p**= 1.74 × 1.74 cm (assuming a 30 × 30 m region of interest) and the pose-estimation error was

**e**= 1.6 cm (0.94 pixels on the image plane). This leads to an effective sample size of

**p’**= 5.03 cm and an effective spatial sampling resolution of $f{\prime}_{s}$ = 396 samples/m

^{2}. As expected, the checker size of 3 × 3 cm can still be resolved in single raw images captured by the drone, but only the 5 × 5 cm checker size can be resolved in the full-aperture AOS visualization, which combines 135 raw images at a certain pose-estimation error.

**Figure A2.**A multi-resolution chart (

**a**) attached to a building and captured with a 30 m diameter synthetic aperture from a distance of 20 m. (

**b**) A single raw video image captured with the drone. (

**c**) A contrast enhanced target close-up captured in the raw image. (

**d**) A full-aperture AOS visualization focusing on the target. (

**e**) A contrast enhanced target close-up in the AOS visualization. All images are converted to grayscale for better comparison.

## Appendix B

^{3}(maximum) opaque voxels at various densities (simulating the occlusion by trees and other vegetation). The simulated spatial resolution (i.e., image resolution) was 512

^{2}pixels. The simulated directional resolution (i.e., number of recorded images) was either 3

^{2}, 9

^{2}, or 27

^{2}.

^{2}× 9

^{2}in this example) and at full synthetic aperture (i.e., 30 m diameter).

**Figure A3.**AOS simulation of increasing occlusion density above ground target at a 512

^{2}× 9

^{2}sampling resolution and a full synthetic aperture of 30 m diameter. Projected occlusion voxels are green. The numbers indicate the structural similarity index (Quaternion SSIM [36]) and the PSNR with respect to the ground truth (gt).

^{2}, 9

^{2}, and 27

^{2}perspectives) at a constant occlusion density (50% in this example).

**Figure A4.**AOS simulation of various synthetic aperture diameters (pinhole = single drone recording, 15 m, and 30 m) and directional sampling resolutions within the selected synthetic aperture (3

^{2}, 9

^{2}, and 27

^{2}perspectives). The occlusion density is 50%. Projected occlusion voxels are green. The numbers indicate the structural similarity index (Quaternion SSIM [38]) and the PSNR with respect to the ground truth.

## References

- Rempel, R.C.; Parker, A.K. An information note on an airborne laser terrain profiler for micro-relief studies. In Proceedings of the Symposium Remote Sensing Environment, 3rd ed.; University of Michigan Institute of Science and Technology: Ann Arbor, MI, USA, 1964; pp. 321–337. [Google Scholar]
- Nelson, R. How did we get here? An early history of forestry lidar. Can. J. Remote Sens.
**2013**, 39, S6–S17. [Google Scholar] [CrossRef] - Sabatini, R.; Richardson, M.A.; Gardi, A.; Ramasamy, S. Airborne laser sensors and integrated systems. Prog. Aerosp. Sci.
**2015**, 79, 15–63. [Google Scholar] [CrossRef] - Kulawardhana, R.W.; Popescu, S.C.; Feagin, R.A. Airborne lidar remote sensing applications in non-forested short stature environments: A review. Ann. For. Res.
**2017**, 60, 173–196. [Google Scholar] [CrossRef] - Synge, E. XCI. A method of investigating the higher atmosphere. Lond. Edinb. Dublin Philos. Mag. J. Sci.
**1930**, 9, 1014–1020. [Google Scholar] [CrossRef] - Vasyl, M.; Paul, F.M.; Ove, S.; Takao, K.; Chen, W.B. Laser radar: Historical prospective—From the East to the West. Opt. Eng.
**2016**, 56, 031220. [Google Scholar] [CrossRef] - Behroozpour, B.; Sandborn, P.A.M.; Wu, M.C.; Boser, B.E. Lidar System Architectures and Circuits. IEEE Commun. Mag.
**2017**, 55, 135–142. [Google Scholar] [CrossRef] - Du, B.; Pang, C.; Wu, D.; Li, Z.; Peng, H.; Tao, Y.; Wu, E.; Wu, G. High-speed photon-counting laser ranging for broad range of distances. Sci. Rep.
**2018**, 8, 4198. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Chase, A.F.; Chase, D.Z.; Weishampel, J.F.; Drake, J.B.; Shrestha, R.L.; Slatton, K.C.; Awe, J.J.; Carter, W.E. Airborne LiDAR, archaeology, and the ancient Maya landscape at Caracol, Belize. J. Archaeol. Sci.
**2011**, 38, 387–398. [Google Scholar] [CrossRef] - Khan, S.; Aragão, L.; Iriarte, J. A UAV–lidar system to map Amazonian rainforest and its ancient landscape transformations. Int. J. Remote Sens.
**2017**, 38, 2313–2330. [Google Scholar] [CrossRef] - Inomata, T.; Triadan, D.; Pinzón, F.; Burham, M.; Ranchos, J.L.; Aoyama, K.; Haraguchi, T. Archaeological application of airborne LiDAR to examine social changes in the Ceibal region of the Maya lowlands. PLoS ONE
**2018**, 13, 1–37. [Google Scholar] [CrossRef] [PubMed] - Maltamo, M.; Naesset, E.; Vauhkonen, J. Forestry Applications of Airborne Laser Scanning: Concepts and Case Studies. In Managing Forest Ecosystems; Springer: Amsterdam, The Netherlands, 2014. [Google Scholar]
- Sterenczak, K.; Ciesielski, M.; Balazy, R.; Zawiła-Niedzwiecki, T. Comparison of various algorithms for DTM interpolation from LIDAR data in dense mountain forests. Eur. J. Remote Sens.
**2016**, 49, 599–621. [Google Scholar] [CrossRef] [Green Version] - Chen, Z.; Gao, B.; Devereux, B. State-of-the-Art: DTM Generation Using Airborne LIDAR Data. Sensors
**2017**, 17, 150. [Google Scholar] [CrossRef] [PubMed] - Nagai, M.; Chen, T.; Shibasaki, R.; Kumagai, H.; Ahmed, A. UAV-Borne 3-D Mapping System by Multisensor Integration. IEEE Trans. Geosci. Remote Sens.
**2009**, 47, 701–708. [Google Scholar] [CrossRef] [Green Version] - Lin, Y.; Hyyppa, J.; Jaakkola, A. Mini-UAV-Borne LIDAR for Fine-Scale Mapping. IEEE Geosci. Remote Sens. Lett.
**2011**, 8, 426–430. [Google Scholar] [CrossRef] - Favorskaya, M.; Jain, L. Handbook on Advances in Remote Sensing and Geographic Information Systems: Paradigms and Applications in Forest Landscape Modeling. In Intelligent Systems Reference Library; Springer International Publishing: New York, NY, USA, 2017. [Google Scholar]
- Kwon, S.; Park, J.W.; Moon, D.; Jung, S.; Park, H. Smart Merging Method for Hybrid Point Cloud Data using UAV and LIDAR in Earthwork Construction. Procedia Eng.
**2017**, 196, 21–28. [Google Scholar] [CrossRef] - Chiang, K.W.; Tsai, G.J.; Li, Y.H.; El-Sheimy, N. Development of LiDAR-Based UAV System for Environment Reconstruction. IEEE Geosci. Remote Sens. Lett.
**2017**, 14, 1790–1794. [Google Scholar] [CrossRef] - Wallace, L.; Lucieer, A.; Malenovský, Z.; Turner, D.; Vopenka, P. Assessment of Forest Structure Using Two UAV Techniques: A Comparison of Airborne Laser Scanning and Structure from Motion (SfM) Point Clouds. Forests
**2016**, 7, 62. [Google Scholar] [CrossRef] - Guo, Q.; Su, Y.; Hu, T.; Zhao, X.; Wu, F.; Li, Y.; Liu, J.; Chen, L.; Xu, G.; Lin, G.; et al. An integrated UAV-borne lidar system for 3D habitat mapping in three forest ecosystems across China. Int. J. Remote Sens.
**2017**, 38, 2954–2972. [Google Scholar] [CrossRef] - Streibl, N. Three-dimensional imaging by a microscope. J. Opt. Soc. Am. A
**1985**, 2, 121–127. [Google Scholar] [CrossRef] - Conchello, J.A.; Lichtman, J.W. Optical sectioning microscopy. Nat. Methods
**2005**, 2, 920–931. [Google Scholar] [CrossRef] [PubMed] - Qian, J.; Lei, M.; Dan, D.; Yao, B.; Zhou, X.; Yang, Y.; Yan, S.; Min, J.; Yu, X. Full-color structured illumination optical sectioning microscopy. Sci. Rep.
**2015**, 5, 14513. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Ryle, M.; Vonberg, D.D. Solar Radiation on 175 Mc./s. Nature
**1946**, 158, 339. [Google Scholar] [CrossRef] - Wiley, C.A. Synthetic aperture radars. IEEE Trans. Aerosp. Electron. Syst.
**1985**, AES-21, 440–443. [Google Scholar] [CrossRef] - Moreira, A.; Prats-Iraola, P.; Younis, M.; Krieger, G.; Hajnsek, I.; Papathanassiou, K.P. A tutorial on synthetic aperture radar. IEEE Geosci. Remote Sens. Mag.
**2013**, 1, 6–43. [Google Scholar] [CrossRef] [Green Version] - Ouchi, K. Recent Trend and Advance of Synthetic Aperture Radar with Selected Topics. Remote Sens.
**2013**, 5, 716–807. [Google Scholar] [CrossRef] [Green Version] - Li, C.J.; Ling, H. Synthetic aperture radar imaging using a small consumer drone. In Proceedings of the 2015 IEEE International Symposium on Antennas and Propagation USNC/URSI National Radio Science Meeting, Vancouver, BC, Canada, 19–25 July 2015; pp. 685–686. [Google Scholar]
- Baldwin, J.E.; Beckett, M.G.; Boysen, R.C.; Burns, D.; Buscher, D.; Cox, G.; Haniff, C.A.; Mackay, C.D.; Nightingale, N.S.; Rogers, J.; et al. The first images from an optical aperture synthesis array: Mapping of Capella with COAST at two epochs. Astron. Astrophys.
**1996**, 306, L13–L16. [Google Scholar] - Turpin, T.M.; Gesell, L.H.; Lapides, J.; Price, C.H. Theory of the synthetic aperture microscope. In Proceedings of the SPIE’s 1995 International Symposium on Optical Science, Engineering, and Instrumentation, San Diego, CA, USA, 9–14 July 1995; Volume 2566. [Google Scholar]
- Levoy, M.; Zhang, Z.; Mcdowall, I. Recording and controlling the 4D light field in a microscope using microlens arrays. J. Microsc.
**2009**, 235, 144–162. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Vaish, V.; Wilburn, B.; Joshi, N.; Levoy, M. Using plane + parallax for calibrating dense camera arrays. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 27 June–2 July 2004; Volume 1, pp. 2–9. [Google Scholar]
- Levoy, M.; Hanrahan, P. Light Field Rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’96, New Orleans, LA, USA, 4–9 August 1996; ACM: New York, NY, USA, 1996; pp. 31–42. [Google Scholar]
- Isaksen, A.; McMillan, L.; Gortler, S.J. Dynamically reparameterized light fields. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’00, 2000, New Orleans, LA, USA, 23—28 July 2000; ACM: New York, NY, USA, 2000; pp. 297–306. [Google Scholar] [Green Version]
- Schoenberger, J.L.; Frahm, J.-M. Structure-from-Motion Revisited. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar] [CrossRef]
- Pentland, A.P. A new sense for depth of field. IEEE Trans. Pattern Anal. Mach. Intell.
**1987**, 4, 523–531. [Google Scholar] [CrossRef] - Kolaman, A.; Yadid-Pecht, O. Quaternion Structural Similarity: A New Quality Index for Color Images. IEEE Trans. Image Process.
**2012**, 21, 1526–1536. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**For light detection and ranging (LiDAR) (

**a**) a single ray is sufficient to sample the depth of a point on the ground surface, while photometric stereo (

**b**) requires a minimum of two views and relies on a dense set of matchable image features. In airborne optical sectioning (AOS), for each point on the synthetic focal plane we directionally integrate many rays over a wide synthetic aperture to support optical sectioning with a low synthetic f-number (

**c**). The spatial sampling density at the focal plane depends on the field of view, the resolution of the drone’s camera, and the height of the drone relative to the focal plane (

**d**).

**Figure 2.**For visualization, we interactively define a virtual camera (green triangle) by its pose, size of its aperture, field of view, and its focal plane. Novel images are rendered by ray integration (Equation (1)) for points

**s**,

**t**at the focal plane (black circle) that are determined by the intersection of rays (blue) through the camera’s projection center (grey circle) and pixels at

**x**,

**y**in the camera’s image plane (blue circle) within its field of view

**FOV**. Only the rays (green) through

_{v}**u**,

**v**at the synthetic aperture plane that pass through the virtual camera’s aperture

**A**, are integrated. While (

_{v}**a**) illustrates the visualization for a single focal plane, (

**b**) shows the focal slicing being applied to increase the depth of field computationally.

**Figure 3.**A densely forested patch is captured with a circular synthetic aperture of 30 m diameter (

**a**), sampled with 231 video images (

**b**) from a camera drone at an altitude of 20 m above ground. A dense 3D point cloud reconstruction from these images (

**c**) took 8 h to compute with photogrammetry. (

**d**) A single raw video image captured through the narrow-aperture omnidirectional lens of the drone. After image rectification, the large depth of field does not reveal features on the ground that are occluded by the trees (

**e**). Increasing the aperture computationally decreases the depth of field and significantly blurs out-of-focus features (

**f**). Shifting focus computationally towards the ground slices optically through the tree branches (

**f**) makes a hidden ground target visible (

**g**,

**h**). Synthetic aperture rendering was possible in real-time after 15 min of preprocessing (image rectification and pose estimation). 3D visualization results are shown in Supplementary Materials Video S1.

**Figure 4.**Ruins of an early 19th century fortification tower near Linz, Austria, as seen from the air (

**a**). The structure of the remaining inner and outer ring walls and a trench become visible after optical sectioning (

**b**). Image preprocessing (rectification and pose estimation) for AOS took 23 min. Photometric 3D reconstruction took 15 h (without tree removal) and does not capture the site structures well (

**c**). Remains of the site as seen from the ground (

**d**). An LiDAR scan with a resolution of 8 samples/m

^{2}(

**e**) does not deliver surface color but provides consistent depth. The synthetic aperture for AOS in this experiment was a 90° sector of a circle with 50 m radius and was sampled with 505 images at an altitude of 35 m above the ground (

**f**,

**g**). For our camera drone, this leads to effectively 74 samples/m

^{2}on the ground. 3D visualization results are shown in Supplementary Materials Video S2.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kurmi, I.; Schedl, D.C.; Bimber, O.
Airborne Optical Sectioning. *J. Imaging* **2018**, *4*, 102.
https://doi.org/10.3390/jimaging4080102

**AMA Style**

Kurmi I, Schedl DC, Bimber O.
Airborne Optical Sectioning. *Journal of Imaging*. 2018; 4(8):102.
https://doi.org/10.3390/jimaging4080102

**Chicago/Turabian Style**

Kurmi, Indrajit, David C. Schedl, and Oliver Bimber.
2018. "Airborne Optical Sectioning" *Journal of Imaging* 4, no. 8: 102.
https://doi.org/10.3390/jimaging4080102