Design and Analysis of a Single—Camera Omnistereo Sensor for Quadrotor Micro Aerial Vehicles (MAVs) ^†

Abstract

We describe the design and 3D sensing performance of an omnidirectional stereo (omnistereo) vision system applied to Micro Aerial Vehicles (MAVs). The proposed omnistereo sensor employs a monocular camera that is co-axially aligned with a pair of hyperboloidal mirrors (a vertically-folded catadioptric configuration). We show that this arrangement provides a compact solution for omnidirectional 3D perception while mounted on top of propeller-based MAVs (not capable of large payloads). The theoretical single viewpoint (SVP) constraint helps us derive analytical solutions for the sensor’s projective geometry and generate SVP-compliant panoramic images to compute 3D information from stereo correspondences (in a truly synchronous fashion). We perform an extensive analysis on various system characteristics such as its size, catadioptric spatial resolution, field-of-view. In addition, we pose a probabilistic model for the uncertainty estimation of 3D information from triangulation of back-projected rays. We validate the projection error of the design using both synthetic and real-life images against ground-truth data. Qualitatively, we show 3D point clouds (dense and sparse) resulting out of a single image captured from a real-life experiment. We expect the reproducibility of our sensor as its model parameters can be optimized to satisfy other catadioptric-based omnistereo vision under different circumstances.

1. Introduction

Micro aerial vehicles (MAVs), such as quadrotor helicopters, are popular platforms for unmanned aerial vehicle (UAV) research due to their structural simplicity, small form factor, vertical take-off and landing (VTOL) capability, and high omnidirectional maneuverability. In general, UAVs have plenty of military and civilian applications, such as target localization and tracking, 3-dimensional (3D) mapping, terrain and infrastructural inspection, disaster monitoring, environmental and traffic surveillance, search and rescue, deployment of instrumentation, and cinematography, among other uses. However, MAVs have size, payload, and on-board computation limitations, which involve the use of compact and lightweight sensors. The most commonly used perception sensors on MAVs are laser scanners and cameras in various configurations such as monocular, stereo, or omnidirectional. We present a vision-based omnidirectional stereo (omnistereo) sensor motivated by several aspects of MAV robotics.

1.1. Sensor Motivation

We justify the need for the proposed omnistereo sensor after observing two basic differences in the sensor requirements between MAVs and ground vehicles:

• Size and payload—In MAV applications, the sensor’s physical dimensions and weight are always a great concern due to payload constraints. Generally, MAVs require fewer and lighter sensors that are compactly designed, while larger robots (including high-payload UAVs) have greater freedom of sensor choice.
• Field-of-view (FOV)—Due to their omnidirectional motion model, MAVs require a simultaneous observation of the 3D surroundings. Conversely, most ground robots can safely rely upon narrow vision as their motion control on the plane is more stable.

1.2. Existing Range Sensors for MAVs

In addition to specifying our sensor requirements, it is important to note the most prevalent robot range sensors used today by MAVs and their limitations. For example, lightweight 2.5D laser scanners can accurately measure distances at fast rates, however, their instantaneous sensing is limited to plane sweeps, which in turn require the quadrotor to move vertically in order to generate 3D maps or to foresee obstacles and free space during navigation. More recently, 3D laser rangefinders and LiDARs are being developed, such as the sensor presented in [1], but this one is not compact enough for MAVs. Another disadvantage of laser-based technologies is their active sensing nature, which requires more power to operate and their measurements are more vulnerable to detection and corruption (e.g., due to dark/reflective surfaces) than vision-based solutions. Time-of-flight (ToF) cameras as well as red, green, blue plus depth (RGB-D) sensors like the Microsoft Kinect^® are also very popular for robot navigation. They have been adopted for low-sunlight conditions and mainly indoor navigation of MAVs [2] due to its structured infrared light projection and short range sensing (under 5 m) [3]. Hence, a lightweight imaging system capable of instantly providing a large field of view (FOV) with acceptable resolutions is essential for MAV applications in 3D space. These state-of-the-art sensors’ pitfalls motivate the design and analysis of our omnistereo sensor.

1.3. Related Work

Using omnidirectional images alone and motion—like the approaches taken in [4,5]—have been proposed to map and localize a robot. Omnidirectional vision using a single mirror for the flight of large UAVs was first attempted in [6]. In [7], Hrabar proposed the use of traditional horizontal stereo-based obstacle avoidance and path planing for AUVs, but these techniques were only tested in a scaled-down air vehicle simulator (AVS). Omnidirectional catadioptric cameras can be aided by structured light such as the prototypes presented in [8] and more flexible configurations demonstrated in [9]. Alternatively, stereo cameras can provide passive, instantaneous 3D information for robot mapping and navigation (including UAVs [10]). Intuitively, omnidirectional stereo (omnistereo) can be achieved through circular arrangements of multiple perspective cameras with overlapping views. Higher resolution panoramas can be achieved by rotating a linear camera as presented in [11], but this approach suffers from motion blur in dynamic environments. We point the reader to [12] for a detailed study of multiple view geometry, and [13] for a compendium of geometric computer vision concepts. Instead, our solution to omnistereo vision consists of a ‘catadioptric’ system by employing cameras and mirrors [14].

Throughout the years, [15,16,17,18,19,20] are some of the works that have applied various omnistereo catadioptric configurations for ground mobile robots. Unfortunately, these systems are not compact since they use separate camera-mirror pairs, which are known to experience synchronization issues. In [21], Yi and Ahuja described a configuration using a mirror and a concave lens for omnistereo, but it rendered a very short baseline in comparison to the two-mirror configurations. Originally, Nayar and Peri [22] studied 9 possible folded-catadioptric configurations for a single-camera omnistereo imaging system. Eventually, a catadioptric system using two hyperbolic mirrors in a vertical configuration was implemented by He et al. [23]. Their omnistereo sensor provides a lengthy baseline at the expense of a very tall system. In the past [24], we developed a novel omnistereo catadioptric rig consisting of a perspective camera coaxially-aligned with two spherical mirrors of distinct radii (in a “folded" configuration). One caveat of spherical mirrors is their non-centrality; they do not satisfy the single effective viewpoint (SVP) constraint (discussed in Section 2.2) but rather a locus of viewpoints is obtained [25].

1.4. Proposed Sensor

We design a SVP-compliant omnistereo system based on the folded, catadioptric configuration with hyperboloidal mirrors. Our approach resembles the work of Jang, Kim, and Kweon [26], who first implemented an omnistereo system using a pair of hyperbolic mirrors and a single camera. However, their sensor’s characteristics were not studied in order to justify their design parameters and capabilities, which we do in our case.

It is true that an omnidirectional catadioptric system sacrifices spatial resolution on the imaging sensor (analyzed in Section 3.4). However, our sensor offers practical advantages such as reduced cost, acceptable weight, and truly-instantaneous pixel-disparity correspondences since the same single camera-lens operates for both views, so mis-synchronization issues do not exist. In fact, we believe we are the first to present a single-camera catadioptric omnistereo solution for MAVs. The initial geometry of our model was proposed in [27]. Now, we perform an extensive analysis of our model’s parameters (Section 2) involving its geometric projection (Section 3) that are obtained as a constrained numerical optimization solution devising the sensor’s real-life application to MAVs passive range sensing (Section 4). We also show how the panoramic images are obtained, where we find correspondences and triangulate 3D points for which an uncertainty model is introduced (Section 5). Finally, we present our experimental results and evaluation for 3D sensing with the proposed omnistereo sensor (Section 6), and we discuss the future direction of our work in Section 7.

2. Sensor Design

Figure 1 shows the single-camera catadioptric omnistereo vision system that we specifically design to be mounted on top of our micro quadrotors (manufactured by Ascending Technologies [28]). It consists of (1) one hyperboloid-planar mirror at the top; (2) one hyperboloidal mirror at the bottom; and (3) a high-resolution USB camera also at the bottom (inside the bottom mirror and looking up). The components are housed and supported by a (4) transparent tube or plastic standoffs (for the real-life prototype shown in Figure 13). The choice of the hyperboloidal reflectors owes to three reasons: it is one of the four non-degenerated conic shapes satisfying the SVP constraint [29]; it allows a wider vertical FOV than elliptical and planar mirrors; and it does not require a telescopic (orthographic) lens for imaging as with paraboloidal mirrors (so our system can be downsized). In addition, the planar part of mirror 1 works as a reflex mirror, which in part reduces distortion caused by dual conic reflections. Based on the SVP property, the system obtains two radial images of the omnidirectional views in the form of an inner and an outer ring as illustrated in Figure 2a,b). Nevertheless, the unique set of parameters describing the entire system categorizes it as a “global camera model" given by [13] because changing the value of any parameter in the model affects the overall projection function of visible light rays in the scene as well as other computational imaging factors such as depth resolution and overlapping field of view, which we attempt to optimize with the following design subsections. Please, refer to Appendix A for clarification on our symbolic notation.

2.1. Model Parameters

In the configuration of Figure 3, mirror 1’s real or primary focus is ${\mathrm{F}}_1$, which is separated by a distance $c_1$ from its virtual or secondary focus, ${\mathrm{F}}_1^{\prime }$, at the bottom. Without loss of generality, we make both the camera’s pinhole and ${\mathrm{F}}_1^{\prime }$ coincide with the origin of the camera’s coordinate system, ${\mathrm{O}}_{\mathrm{C}}$. This way, the position of the primary focus, ${\mathrm{F}}_1$, can be referenced by vector ${}^{\left[C\right]}{\mathbf{f}}_1={\left[\begin{array}{ccc}0,& 0,& c_1\end{array}\right]}^{\mathsf{T}}$ in Cartesian coordinates with respect to the camera frame, $\left[C\right]$. Similarly, the distance between the foci of mirror 2, ${\mathrm{F}}_2$ and ${\mathrm{F}}_2^{\prime }$, is measured by $c_2$. Here, we use the planar (reflex) mirror of radius $r_{ref}$ and unit normal vector

$$\begin{array}{c}\hfill {}^{\left[C\right]}{\widehat{\mathbf{n}}}_{ref}=\left[\begin{array}{ccc}0,& 0,& -1\end{array}\right]\end{array}$$

(1)

in order to project the real camera’s pinhole located at ${\mathrm{O}}_{\mathrm{C}}$ as a virtual camera ${\mathrm{O}}_{\mathrm{C}}^{\prime }$ coinciding with the virtual focal point ${\mathrm{F}}_2^{\prime }$ positioned at ${}^{\left[C\right]}{\mathbf{f}}_{2v}={\left[\begin{array}{ccc}0,& 0,& d\end{array}\right]}^{\mathsf{T}}$. We achieve this by setting $d/2$ as the symmetrical distance from the reflex mirror to ${\mathrm{O}}_{\mathrm{C}}$ and from the reflex mirror to ${\mathrm{O}}_{\mathrm{C}}^{\prime }$. With respect to $\left[C\right]$, mirror 2’s primary focus, ${\mathrm{F}}_2$, results in position ${}^{\left[C\right]}{\mathbf{f}}_2={\left[\begin{array}{ccc}0,& 0,& d-c_2\end{array}\right]}^{\mathsf{T}}$. It yields the following expression for the reflective plane:

$$\begin{array}{c}\hfill {}^{\left[C\right]}{\widehat{\mathbf{n}}}_{ref}^{\mathsf{T}}{}^{\left[C\right]}\mathbf{x}=-d/2\end{array}$$

(2)

The profile of each hyperboloid is determined by independent parameters $k_1$ and $k_2$, respectively. Their reflective vertical field of view (vFOV) are indicated by angles ${\alpha }_1$ and ${\alpha }_2$. They play an important role when designing the total vFOV of the system, ${\alpha }_{sys}$, formally defined by Equation (54) and illustrated in Figure 5. Also importantly, while performing stereo vision, it is to consider angle ${\alpha }_{SROI}$, which measures the common (overlapping) vFOV of the omnistereo system. The camera’s nominal field of view ${\alpha }_{cam}$ and its opening radius $r_{cam}$ also determine the physical areas of the mirrors that can be fully imaged. Theoretically, the mirrors’ vertical axis of symmetry (coaxial configuration) produces two image points that are radially collinear. This property is advantageous for the correspondence search during stereo sensing (Section 5) with a baseline measured as

$$b=c_1+c_2-d$$

(3)

Among design parameters, we also include the total height of the system, $h_{sys}$, and weight $m_{sys}$, both being formulated in Section 2.3.

To summarize, the model has 6 primary design parameters given as a vector

$$\begin{array}{c}\hfill \mathsf{\theta }=\left[\begin{array}{cccccc}{c}_1,& c_2,& k_1,& k_2,& d,& r_{sys}\end{array}\right]\end{array}$$

(4)

in addition to by-product parameters such as

$$\begin{array}{c}\hfill \left[\begin{array}{cccccccccc}b,& h_{sys},& r_{ref},& r_{cam},& m_{sys},& {\alpha }_1,& {\alpha }_2,& {\alpha }_{sys},& {\alpha }_{SROI},& {\alpha }_{cam}\end{array}\right]\end{array}$$

In Section 4, we perform a numerical optimization of the parameters in θ with the goal to maximize the baseline, b, required for life-size navigational stereopsis. At the same time, we restrict the overall size of the rig (Section 2.3) without sacrificing sensing performance characteristics such as vertical field of view, spatial resolution, and depth resolution. In the upcoming subsections, we first derive the analytical solutions for the forward projection problem in our coaxial stereo configuration as a whole. In Section 3.2, we derive the back-projection equations for lifting 2D image points into 3D space.

2.2. Single Viewpoint (SVP) Configuration for OmniStereo

As a central catadioptric system, its projection geometry must obey the existence of the so-called single effective viewpoint (SVP). While the SVP guarantees that true perspective geometry can always be recovered from the original image, it limits the selection of mirror profiles to a set of conics. Generally, a circular hyperboloid of revolution (about its axis of symmetry) conforms to the SVP constraint as demonstrated by Baker and Nayar in [30]. Since a hyperboloidal mirror has two foci, the effective viewpoint is the primary focus $\mathrm{F}$ inside the physical mirror and the secondary (outer) focus ${\mathrm{F}}^{\prime }$ is where the centre (pinhole) of the perspective camera should be placed for depicting a scene obeying the SVP configuration discussed in this section.

First of all, a hyperboloid i can be described by the following parametric equation:

$$\begin{array}{c}\hfill \frac{{\left(z_i-z_{0_i}\right)}^2}{a_i^2}-\frac{r_i^2}{b_i^2}=1,\text{with}{a}_i=\frac{c_i}{2}\sqrt{\frac{k_i-2}{k_i}},b_i=\frac{c_i}{2}\sqrt{\frac{2}{k_i}}\end{array}$$

(5)

where $z_{0_i}=\frac{c_i}{2}$ is the offset (shift) position of the focus along the Z-axis from the origin ${\mathrm{O}}_{\mathrm{C}}$, and $r_i$ is the orthogonal distance to the axis of revolution / symmetry (i.e. the Z-axis) from a point ${\mathrm{P}}_i$ on its surface.

In fact, the position of a valid point ${\mathrm{P}}_i$ is constrained within the mirror’s physical surface of reflection, which is radially limited by $r_{i,min}$ and $r_{i,max}$, such that:

$$\begin{array}{c}\hfill r_i=\sqrt{x_i^2+y_i^2},\text{for}{r}_{i,min}\le r_i\le r_{i,max},\forall i\in \{1,2\}\end{array}$$

(6)

and $r_{1,min}=r_{ref},r_{1,max}=r_{sys},r_{2,min}=r_{cam},r_{2,max}=r_{sys}$. Observe that the radius of the system is the upper bound for both mirrors (Figure 3). In addition, the hyperboloids profiled by Equation (5) must obey the following conical constraints:

$$\begin{array}{c}\hfill \forall i\in \{1,2\}\left(c_i>0\wedge k_i>2\right)\end{array}$$

(7)

k is a constant parameter (unit-less) inversely related to the mirror’s curvature or more precisely, the eccentricity ${\epsilon }_c$ of the conic. In fact, ${\epsilon }_c>1$ for hyperbolas, yet a plane is produced when ${\epsilon }_c\to \infty$ or $k=2$.

We devise $M_i$ as the set of all the reflection points ${\mathrm{P}}_i$ with coordinates $(x_i,y_i,z_i)$ laying on the surface of the respective mirror i within bounds. Formally,

$$\begin{array}{c}\hfill M_i:=\left\{{\mathrm{P}}_i\in {\mathbb{R}}^3|\frac{{\left(z_i-z_{0_i}\right)}^2}{a_i^2}-\frac{r_i^2}{b_i^2}=1\wedge \text{Equation}(6)\wedge \text{Equation}(7)\right\}\end{array}$$

(8)

In our model, we describe both hyperboloidal mirrors, 1 and 2, with respect to the camera frame $\left[C\right]$, which acts as the common origin of the coordinate system. Therefore,

$$\begin{array}{cc}\hfill z_{0_1}& =\frac{c_1}{2}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill z_{0_2}& =d-\frac{c_2}{2}\hfill \end{array}$$

(10)

By expanding Equation (5) with their respective index terms, it becomes

$$\begin{array}{cc}\hfill {\left(z_1-\frac{c_1}{2}\right)}^2-r_1^2\left(\frac{k_1}{2}-1\right)& =\frac{c_1^2}{4}\left(\frac{k_1-2}{k_1}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\left(z_2-d+\frac{c_2}{2}\right)}^2-r_2^2\left(\frac{k_2}{2}-1\right)& =\frac{c_2^2}{4}\left(\frac{k_2-2}{k_2}\right)\hfill \end{array}$$

(12)

Additionally, we define the function ${\mathrm{f}}_{z_i}:r\mapsto z_i$ to find the corresponding $z_i$ component from a given r value as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathrm{f}}_{z_i}\left(r\right):=& \left\{\begin{array}{cc}{z}_{0_i}+{\gamma }_i\hfill & \text{if}i=1\wedge \text{Equation}\left(6\right)\hfill \\ z_{0_i}-{\gamma }_i\hfill & \text{if}i=2\wedge \text{Equation}\left(6\right)\hfill \\ None\hfill & \text{otherwise}\hfill \end{array}\right.\hfill \end{array}\end{array}$$

(13)

where ${\gamma }_i={\displaystyle \frac{a_i}{b_i}}\sqrt{b_i^2+r_i^2}$.

The inverse relation ${\mathrm{f}}_{r_i}:z\mapsto \left\{+r_i,-r_i\right\}$ can be also implemented as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathrm{f}}_{r_i}\left(z\right):=& \left\{\begin{array}{cc}\pm b_i{\Gamma }_i\hfill & \text{if}i\in \{1,2\}\wedge \text{Equation}\left(6\right)\hfill \\ None\hfill & \text{otherwise}\hfill \end{array}\right.\hfill \end{array}\end{array}$$

(14)

where ${\Gamma }_i=\sqrt{{\displaystyle \frac{{\left(z-z_{0_i}\right)}^2}{a_i^2}}-1}$, so a valid input z can be associated with both positive and negative solutions $r_i$.

2.3. Rig Size

In the attempt to evaluate the overall system size, we consider the height and weight variables due to the primary design parameters, θ.

First, the height of the system, $h_{sys}$ can be estimated from the functional relationships ${\mathrm{f}}_{z_1}$ and ${\mathrm{f}}_{z_2}$ defined in Equation (13), which can provide the respective $z-$component values at the out-most point on the mirror’s surface. More specifically, knowing $r_{sys}$, we get

$$\begin{array}{c}\hfill h_{sys}=z_{max}-z_{min}\end{array}$$

(15)

where $z_{max}={\mathrm{f}}_{z_1}\left(r_{sys}\right)$ and $z_{min}={\mathrm{f}}_{z_2}\left(r_{sys}\right)$.

The rig’s weight can be indicated by the total resulting mass of the main “tangible” components:

$$\begin{array}{c}\hfill m_{sys}=m_{cam}+m_{tub}+m_{mir}\end{array}$$

(16)

where the mass of the camera-lens combination is $m_{cam}$; the mass of the support tube $m_{tub}$ can be estimated from its cylindrical volume $V_{tub}$ and material density ${\rho }_{tub}$, and the mass due to the mirrors

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill m_{mir}& =V_{mir}{\rho }_{mir}\hfill \\ & =\left(V_1+V_{ref}+V_2\right){\rho }_{mir}\hfill \end{array}\end{array}$$

(17)

For computing the volume of the hyperboloidal shell, $V_i$ for mirror i, we apply a “ring method” of volume integration. By assuming all mirror material has the same wall thickness ${\tau }_m$, we acquire $V_i$ by integrating the horizontal cross-sections area along the $\mathrm{Z}$-axis. Each ring area depends on its outer and inner circumferences that vary according to radius $r{\mid }_z$ for a given height z. Equation (14) establishes the functional relation ${r_i}^{+}={\mathrm{f}}_{r_i}\left(z\right)$, from which we only need its positive answer. We let A be the function that computes the ring area of constant thickness ${\tau }_m$ for a variable outer radius $r_i$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill A\left(r_i\right)& =\pi r_i^2-\pi {\left(r_i-{\tau }_m\right)}^2\hfill \\ & =\pi {\tau }_m\left(2r_i-{\tau }_m\right)\hfill \end{array}\end{array}$$

(18)

We consider the definite integral evaluated in the z interval bounded by its height limits, which are correlated with its radial limits Equation (6) and can be obtained via the ${\mathrm{f}}_{z_i}$ defined in Equation (13), such that

$$\begin{array}{c}\hfill z_{i,min}={\mathrm{f}}_{z_i}\left(r_{i,min}\right)\text{and}{z}_{i,max}={\mathrm{f}}_{z_i}\left(r_{i,max}\right)\end{array}$$

(19)

Then, we proceed to integrate Equation (18), so the shell volume for each hyperboloidal mirror is defined as

$$\begin{array}{cc}\hfill V_i& ={\int }_{z_{i,min}}^{z_{i,max}}A\left(r_i\right)\mathrm{dz}\hfill \end{array}$$

(20)

Finally, since the reflex mirror piece is just a solid cylinder of thickness ${\tau }_m$, its volume is simply

$$\begin{array}{c}\hfill V_{ref}={\tau }_m\pi r_{ref}^2\end{array}$$

(21)

3. Projective Geometry

3.1. Analytical Solutions to Projection (Forward)

Assuming a central catadioptric configuration for the mirrors and camera system (Section 2.2), we derive the closed-form solution to the imaging process (forward projection) for an observable point ${\mathrm{P}}_w$, positioned in three-dimensional Euclidean space, ${\mathbb{R}}^3$, with respect to the reference frame, $\left[C\right]$, as vector ${}^{\left[C\right]}{\mathbf{p}}_w={\left[\begin{array}{ccc}{x}_w,& y_w,& z_w\end{array}\right]}^{\mathsf{T}}$. In addition, we assume all reference frames such as $\left[F_1\right]$ and $\left[F_2\right]$ have the same orientation as $\left[C\right]$.

For mathematical stability, we must constrain that all projecting world points lie outside the mirror’s volume:

$$\begin{array}{c}\hfill {\mathrm{f}}_{r_i}\left(z_w\right)<{\rho }_w,\text{where}{\rho }_w=\sqrt{x_w^2+y_w^2}\end{array}$$

(22)

where ${\mathrm{f}}_{r_i}$ is defined by Equation (14) and ${\rho }_w$ measures the horizontal range to ${\mathrm{P}}_w$.

${\mathrm{P}}_w$ is imaged at pixel position ${}^{\left[I\right]}{\mathit{m}}_1$ after its reflection as point ${\mathrm{P}}_1$ on the hyperboloidal surface of mirror 1 (Figure 4). On the other hand, the second image point’s position, ${}^{\left[C\right]}{\mathit{m}}_2$, due to reflection point ${\mathrm{P}}_2$ on mirror 2 is rather obtained indirectly after an additional point ${\mathrm{P}}_r$ is reflected at ${}^{\left[C\right]}{\mathbf{p}}_{ref}$ on the reflex mirror represented via Equation (32).

First, for ${\mathrm{P}}_w$’s reflection point via mirror 1 at position vector ${}^{\left[C\right]}{\mathbf{p}}_1$, we use ${\lambda }_1$ as the parametrization term for the line equation passing through ${\mathrm{F}}_1$ toward ${\mathrm{P}}_w$ with direction ${}^{\left[F_1\right]}{\mathbf{d}}_1={}^{\left[C\right]}{\mathbf{p}}_w-{}^{\left[C\right]}{\mathbf{f}}_1$. The position of any point ${\mathrm{P}}_1$ on this line is given by:

$${}^{\left[C\right]}{\mathbf{p}}_1={}^{\left[C\right]}{\mathbf{f}}_1+{\lambda }_1{}^{\left[F_1\right]}{\mathbf{d}}_1$$

(23)

Substituting Equation (23) into Equation (11), we obtain:

$$\begin{array}{cc}\hfill {\left({\lambda }_1(z_w-c_1)+\frac{c_1}{2}\right)}^2-\left({\lambda }_1^2{x}_w^2+{\lambda }_1^2{y}_w^2\right)\left(\frac{k_1}{2}-1\right)& \\ \hfill -\frac{c_1^2}{4}\left(\frac{k_1-2}{k_1}\right)& =0\hfill \end{array}$$

in order to solve for ${\lambda }_1$, which turns out to be

$${\lambda }_1=\frac{c_1}{\Vert {}^{\left[F_1\right]}{\mathbf{d}}_1\Vert \sqrt{k_1·(k_1-2)}-k_1\left(z_w-c_1\right)}$$

(24)

where $\Vert {}^{\left[F_1\right]}{\mathbf{d}}_1\Vert =\sqrt{x_w^2+y_w^2+{(z_w-c_1)}^2}$ is the Euclidean norm between ${\mathrm{P}}_w$ and mirror 1’s focus, ${\mathrm{F}}_1$.

In practice, we represent the reflection point’s position ${}^{\left[C\right]}{\mathbf{p}}_1$ as a matrix-vector multiplication between the $3\times 4$ transformation matrix ${\mathbf{K}}_1=\left[\begin{array}{cc}{\lambda }_1{\mathbf{I}}_{\left(3\right)},& \left(1-{\lambda }_1\right){}^{\left[C\right]}{\mathbf{f}}_1\end{array}\right]$ and the point’s position vector ${}^{\left[C\right]}{\mathbf{p}}_{w,h}={\left[\begin{array}{cccc}{x}_w,& y_w,& z_w,& 1\end{array}\right]}^{\mathsf{T}}$ in homogeneous coordinates:

$${}^{\left[C\right]}{\mathbf{p}}_1={\mathbf{K}}_1{}^{\left[C\right]}{\mathbf{p}}_{w,h}$$

(25)

Note that ${}^{\left[C\right]}{\mathbf{p}}_1$’s elevation angle, ${\theta }_1$, must be bounded as

$$\begin{array}{c}\hfill {\theta }_{1,min}\le {\theta }_1\le {\theta }_{1,max}\end{array}$$

(26)

where ${\theta }_{1,min}$ and ${\theta }_{1,max}$ are the angular elevation limits for the real reflective area of the hyperboloid.

Finally, the reflection point ${\mathrm{P}}_1$ with position ${}^{\left[C\right]}{\mathbf{p}}_1$ can now be perspectively projected as a pixel point located at ${}^{\left[I\right]}{\mathit{m}}_1={\left[\begin{array}{cc}{u}_1,& v_1\end{array}\right]}^{\mathsf{T}}$ on the image. In fact, the entire imaging process of ${\mathrm{P}}_w$ via mirror 1 can be expressed in homogeneous coordinates as:

$${}^{\left[I\right]}{\mathit{m}}_{1,h}={\zeta }_1{\mathbf{K}}_c{\mathbf{K}}_1{}^{\left[C\right]}{\mathbf{p}}_{w,h}$$

(27)

where the scalar ${\zeta }_1=1/z_1=1/\left(c_1+{\lambda }_1\left(z_w-c_1\right)\right)$ is the perspective normalizer that maps the principal ray passing through ${\mathbf{p}}_1$ onto a point ${}^{\left[C\right]}{\mathbf{q}}_1={\left[\begin{array}{ccc}{x}_{q_1},& y_{q_1},& 1\end{array}\right]}^{\mathsf{T}}$ on the normalized projection plane ${\widehat{\pi }}_{im{g}_1}$. The traditional $3\times 3$ intrinsic matrix of the camera’s pinhole model is

$$\begin{array}{c}\hfill {\mathbf{K}}_c=\left[\begin{array}{ccc}{f}_u& s& u_c\\ 0& f_v& v_c\\ 0& 0& 1\end{array}\right]\end{array}$$

(28)

in which $f_u=f/h_x$ and $f_v=f/h_y$ are based on the focal length f and the pixel dimension $(h_x,h_y)$, s is the skew parameter, and ${}^{\left[I\right]}{\mathit{m}}_c={\left[\begin{array}{cc}{u}_c,& v_c\end{array}\right]}^{\mathsf{T}}$ is the optical center position on the image $\left[I\right]$. Figure 4 illustrates the projection point $f{}^{\left[C\right]}{\mathbf{q}}_1$ on the respective image plane ${\pi }_{im{g}_1}$.

Similarly, we provide the analytical solution for the forward projection of ${\mathrm{P}}_w$ via mirror 2 by first considering the position of reflection point ${\mathrm{P}}_2$:

$${}^{\left[C\right]}{\mathbf{p}}_2={\mathbf{K}}_2{}^{\left[C\right]}{\mathbf{p}}_{w,h}$$

(29)

where ${\mathbf{K}}_2=\left[\begin{array}{cc}{\lambda }_2{\mathbf{I}}_{\left(3\right)},& \left(1-{\lambda }_2\right){}^{\left[C\right]}{\mathbf{f}}_2\end{array}\right]$ is similar to the transformation matrix ${\mathbf{K}}_1$, but obviously it now uses ${}^{\left[C\right]}{\mathbf{f}}_2$ and

$${\lambda }_2=\frac{c_2}{\Vert {}^{\left[F_2\right]}{\mathbf{d}}_2\Vert \sqrt{k_2·(k_2-2)}+k_2\left(z_w-(d-c_2)\right)}$$

(30)

with direction vector’s norm

$$\begin{array}{c}\hfill \Vert {}^{\left[F_2\right]}{\mathbf{d}}_2\Vert =\Vert {}^{\left[C\right]}{\mathbf{p}}_w-{}^{\left[C\right]}{\mathbf{f}}_2\Vert =\sqrt{x_w^2+y_w^2+{\left(z_w-(d-c_2)\right)}^2}\end{array}$$

(31)

For completeness, note that the physical projection via mirror 2 is incident to the reflex mirror at

$$\begin{array}{c}\hfill {}^{\left[C\right]}{\mathbf{p}}_{ref}={}^{\left[C\right]}{\mathbf{f}}_{2v}+{\lambda }_{ref}\left({}^{\left[C\right]}{\mathbf{p}}_2-{}^{\left[C\right]}{\mathbf{f}}_{2v}\right)\end{array}$$

(32)

where ${\lambda }_{ref}=\frac{d}{2(d-z_2)}$ according to Equation (2) in the theoretical model. Ultimately, ignoring any astigmatism and chromatic aberrations introduced by the reflex mirror, and because the same (and only) real camera with ${\mathbf{K}}_c$ is used for imaging, we obtain the projected pixel position ${}^{\left[I\right]}{\mathit{m}}_{2,h}={\left[\begin{array}{ccc}{u}_2,& v_2,& 1\end{array}\right]}^{\mathsf{T}}$:

$${}^{\left[I\right]}{\mathit{m}}_{2,h}={\zeta }_2{\mathbf{K}}_c{\mathbf{K}}_{ref}{\mathbf{K}}_2{}^{\left[C\right]}{\mathbf{p}}_{w,h}$$

(33)

where ${\zeta }_2=1/\left(d-z_2\right)$ is the perspective normalizer to find ${}^{\left[C\right]}{\mathbf{q}}_2$ on the normalized projection plane, ${\widehat{\pi }}_{im{g}_2}$.

Due to planar mirroring via the reflex mirror, ${}_{\left[C\right]}^{\left[C^{\prime }\right]}{\mathbf{K}}_{ref}$ is used to change the coordinates of ${\mathrm{P}}_2$ from $\left[C\right]$ onto the virtual camera frame, $\left[C^{\prime }\right]$, located at ${}^{\left[C\right]}{\mathbf{f}}_{2v}$. Hence,

$$\begin{array}{c}\hfill {}_{\left[C\right]}^{\left[C^{\prime }\right]}{\mathbf{K}}_{ref}=\left[\begin{array}{ccc}{\mathbf{I}}_{\left(3\right)}+2{\mathbf{D}}_{{\widehat{\mathbf{n}}}_{ref}},& & {}^{\left[C\right]}{\mathbf{f}}_{2v}\end{array}\right]\end{array}$$

(34)

where the $3\times 1$ unit normal vector of the reflex mirror plane, ${}^{\left[C\right]}{\widehat{\mathbf{n}}}_{ref}$ given in Equation (1), is mapped into its corresponding $3\times 3$ diagonal matrix ${\mathbf{D}}_{{\widehat{\mathbf{n}}}_{ref}}$, via the relationship:

$$\begin{array}{c}\hfill {\mathbf{D}}_{{\widehat{\mathbf{n}}}_{ref}}\leftarrow {\mathbf{I}}_{\left(3\right)}\mathrm{diag}\left({}^{\left[C\right]}{\widehat{\mathbf{n}}}_{ref}\right)\end{array}$$

(35)

It is convenient to define the forward projection functions ${\mathbf{f}}_{{\phi }_1}\left({}^{\left[C\right]}\mathbf{p}\right)$ and ${\mathbf{f}}_{{\phi }_2}\left({}^{\left[C\right]}\mathbf{p}\right)$ for a 3D point $\mathrm{P}$ whose position vector is known with respect to $\left[C\right]$ and which is situated within the vertical field of view ${\alpha }_i$ of mirror i (for $i\in \{1,2\}$) indicated in Figure 5. Function ${\mathbf{f}}_{{\phi }_i}\left({}^{\left[C\right]}\mathbf{p}\right)$ maps ${}^{\left[C\right]}\mathbf{p}$ to image point ${}^{\left[I\right]}{\mathit{m}}_i$ on frame $\left[I\right]$, such that ${\mathbf{f}}_{{\phi }_i}:{\mathbb{R}}^3\mapsto {\mathbb{R}}^2$,

$$\begin{array}{cc}\hfill {\mathbf{f}}_{{\phi }_i}\left({}^{\left[C\right]}\mathbf{p}\right):=& \left\{\begin{array}{cc}{}^{\left[C\right]}\mathbf{p}\stackrel{\text{Equation}\left(27\right)}{\mapsto }{}^{\left[I\right]}{\mathit{m}}_1\hfill & \text{if}i=1\wedge \text{Equations}\left(37\right)\text{and}\left(22\right)\hfill \\ {}^{\left[C\right]}\mathbf{p}\stackrel{\text{Equation}\left(33\right)}{\mapsto }{}^{\left[I\right]}{\mathit{m}}_1\hfill & \text{if}i=2\wedge \text{Equations}\left(37\right)\text{and}\left(22\right)\hfill \\ None\hfill & \text{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(36)

In fact, ${}^{\left[I\right]}{\mathit{m}}_i$ is considered valid if it is located within the imaged radial bounds, such that:

$$\begin{array}{c}\hfill {}^{\left[I_{C_i}\right]}\Vert {}^{\left[I_i\right]}{\mathit{m}}_{{\mathit{r}}_{\mathit{i},\mathit{m}\mathit{i}\mathit{n}}}\Vert \le {}^{\left[I_{C_i}\right]}{}^{\left[I\right]}{\mathit{m}}_{\mathit{i}}\le {}^{\left[I_{C_i}\right]}\Vert {}^{\left[I\right]}{\mathit{m}}_{{\mathit{r}}_{\mathit{i},\mathit{m}\mathit{a}\mathit{x}}}\Vert \end{array}$$

(37)

where the frame of reference $\left[I_{C_i}\right]$ implies that its origin is the image center ${}^{\left[I\right]}{\mathit{m}}_c={\left[\begin{array}{cc}{u}_{c_i},& v_{c_i}\end{array}\right]}^{\mathsf{T}}$ of the $\left[I_i\right]$ masked image (Figure 7). Therefore, the magnitude (norm) of any position ${}^{\left[I_{C_i}\right]}\mathit{m}$ in pixel space $\left[I_{C_i}\right]$ can be measured as

$$\begin{array}{c}\hfill {}^{\left[I_{C_i}\right]}\Vert {}^{\left[I_i\right]}\mathit{m}\Vert :=\Vert {}^{\left[I_i\right]}\mathit{m}-{}^{\left[I_i\right]}{\mathit{m}}_c\Vert =\sqrt{{(u-u_c)}^2+{(v-v_c)}^2}\end{array}$$

(38)

In particular, ${}^{\left[I_{C_i}\right]}\Vert {}^{\left[I\right]}{\mathit{m}}_{r_{i,lim}}$ is the image radius obtained from the projection ${}^{\left[I\right]}{\mathit{m}}_{r_{i,lim}}\leftarrow {\mathbf{f}}_{{\phi }_i}\left({}^{\left[C\right]}{\mathbf{p}}_{i,lim}\right)$ corresponding to a particular point coincident with the line of sight of the radial limit $r_{i,lim}$—it being either $r_{sys}$, $r_{ref}$, or $r_{cam}$ as indicated by Equation (6).

3.2. Analytical Solutions to Back Projection

The back projection procedure establishes the relationship between the 2D position of a pixel point ${}^{\left[I\right]}{\mathit{m}}_i={\left[\begin{array}{cc}u,& v\end{array}\right]}^{\mathsf{T}}$ on the image $\left[I_i\right]$ and its corresponding 3D projective direction vector ${\mathbf{v}}_i$ toward the observed point ${\mathrm{P}}_w$ in the world.

Initially, the pixel point ${}^{\left[I\right]}{\mathit{m}}_1$ (imaged via mirror 1) is mapped as ${\mathrm{Q}}_1$ onto the normalized projection plane ${\widehat{\pi }}_{im{g}_1}$ with coordinates ${}^{\left[C\right]}{\mathbf{q}}_1={\left[\begin{array}{ccc}{x}_{q_1},& y_{q_1},& 1\end{array}\right]}^{\mathsf{T}}$ by applying the inverse transformation of the camera intrinsic matrix Equation (28) as follows:

$${}^{\left[C\right]}{\mathbf{q}}_1={}_{\left[C\right]}^{\left[I\right]}{\mathbf{K}}_c^{-1}{}^{\left[I\right]}{\mathit{m}}_{1,h}=\left[\begin{array}{ccccc}\frac{1}{f_u}& & -\frac{s}{f_u{f}_v}& & \frac{s{v}_c-f_v{u}_c}{f_u{f}_v}\\ 0& & \frac{1}{f_v}& & -\frac{v_c}{f_v}\\ 0& & 0& & 1\end{array}\right]\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]$$

(39)

For simplicity, we assume no distortion parameters exist, so we can proceed with the lifting step along the principal ray that passes through three points: the camera’s pinhole ${\mathrm{O}}_{\mathrm{C}}$, point ${\mathrm{Q}}_1$ on the projection plane, and the reflection point ${\mathrm{P}}_1$ (Figure 4). The vector form of this line equation can be written as:

$$\begin{array}{cc}\hfill {}^{\left[C\right]}{\mathbf{p}}_1& ={}^{\left[C\right]}{\mathbf{o}}_c+t_1\left({}^{\left[C\right]}{\mathbf{q}}_1-{}^{\left[C\right]}{\mathbf{o}}_c\right)=t_1{}^{\left[C\right]}{\mathbf{q}}_1\hfill \end{array}$$

(40)

By substituting Equation (40) into Equation (11), we solve for the parameter $t_1$, to get

$$t_1=\frac{c_1}{k_1-\Vert {}^{\left[C\right]}{\mathbf{q}}_1\Vert \sqrt{k_1·(k_1-2)}}$$

(41)

where $\Vert {}^{\left[C\right]}{\mathbf{q}}_1\Vert =\sqrt{x_{q_1}^2+y_{q_1}^2+1}$ is the distance between ${\mathrm{Q}}_1$ and ${\mathrm{O}}_{\mathrm{C}}$.

Given ${}^{\left[F_1\right]}{\mathbf{v}}_1$ as the direction vector leaving focal point ${\mathrm{F}}_1$ toward the world point ${}^{\left[C\right]}{\mathrm{P}}_w$. Through frame transformation ${}_{\left[C\right]}^{\left[F_1\right]}{T}_1{}^{\left[C\right]}{\mathbf{p}}_{1,h}$, we get

$$\begin{array}{c}\hfill {}^{\left[F_1\right]}{\mathbf{v}}_1={}_{\left[C\right]}^{\left[F_1\right]}{T}_1{}^{\left[C\right]}{\mathbf{p}}_{1,h},\text{where}{{}_{\left[C\right]}^{\left[F_1\right]}{T}_1}_{(3\times 4)}=\left[\begin{array}{ccc}{\mathbf{I}}_{\left(3\right)},& & -{}^{\left[C\right]}{\mathbf{f}}_1\end{array}\right]\end{array}$$

(42)

for ${}^{\left[C\right]}{\mathbf{p}}_{1,h}$ as the homogeneous form of Equation (40). In fact, ${}^{\left[F_1\right]}{\mathbf{v}}_1$ provides the back-projected angles (elevation ${\theta }_1$, azimuth ${\psi }_1$) from focus ${\mathrm{F}}_1$ toward ${}^{\left[C\right]}{\mathrm{P}}_w$:

$$\begin{array}{cc}\hfill {}^{\left[F_1\right]}{\theta }_1& =arcsin\left(\frac{z_{{\mathbf{v}}_1}}{\Vert {}^{\left[F_1\right]}{\mathbf{v}}_1\Vert }\right)=arcsin\left(\frac{z_1-c_1}{\Vert {}^{\left[F_1\right]}{\mathbf{v}}_1\Vert }\right)\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill {}^{\left[F_1\right]}{\psi }_1& =arctan\left(\frac{y_{{\mathbf{v}}_1}}{x_{{\mathbf{v}}_1}}\right)=arctan\left(\frac{y_1}{x_1}\right)\hfill \end{array}$$

(44)

where $\Vert {}^{\left[F_1\right]}{\mathbf{v}}_1\Vert$ is the norm of the back-projection vector up to the mirror surface.

Using the same approach, we lift a pixel point ${}^{\left[I\right]}{\mathit{m}}_2$ imaged via mirror 2. Because the virtual camera ${\mathrm{O}}_{\mathrm{C}}^{\prime }$ located at ${}^{\left[C\right]}{\mathbf{f}}_2={\left[\begin{array}{ccc}0,& 0,& d-c_2\end{array}\right]}^{\mathsf{T}}$ uses the same intrinsic matrix ${\mathbf{K}}_c$, we can safely back-project pixel ${}^{\left[I\right]}{\mathit{m}}_2$ to ${\mathrm{Q}}_{2v}$ on the normalized projection plane ${\widehat{\pi }}_{im{g}_2}$ as follows:

$$\begin{array}{c}\hfill {}^{\left[C^{\prime }\right]}{\mathbf{q}}_{2v}={}^{\left[C\right]}{\mathbf{q}}_2={\mathbf{K}}_c^{-1}{}^{\left[I\right]}{\mathit{m}}_{2,h}\end{array}$$

(45)

where the inverse transformation of the camera intrinsic matrix ${\mathbf{K}}_c^{-1}$ is given by Equation (28). Since the reflection matrix ${\mathbf{K}}_{ref}$ defined in Equation (34) is bidirectional due to the symmetric position of the reflex mirror about $\left[C\right]$ and $\left[C^{\prime }\right]$, we can find the desired position of ${}^{\left[C\right]}{\mathbf{q}}_{2v}$ with respect to $\left[C\right]$:

$${}^{\left[C\right]}{\mathbf{q}}_{2v}={}_{\left[C^{\prime }\right]}^{\left[C\right]}{\mathbf{K}}_{ref}{}^{\left[C^{\prime }\right]}{\mathbf{q}}_{2v,h}$$

(46)

which is equivalent to ${}^{\left[C\right]}{\mathbf{q}}_{2v}={\left[\begin{array}{ccc}{x}_{q_{2v}},& y_{q_{2v}},& d-1\end{array}\right]}^{\mathsf{T}}$.

In Figure 4, we can see the principal ray that passes through the virtual camera’s pinhole ${\mathrm{O}}_{\mathrm{C}}^{\prime }$ and the reflection point ${\mathrm{P}}_2$, so this line equation can be written as:

$$\begin{array}{cc}\hfill {}^{\left[C\right]}{\mathbf{p}}_2& ={}^{\left[C\right]}{\mathbf{f}}_{2v}+t_2\left({}^{\left[C\right]}{\mathbf{q}}_{2v}-{}^{\left[C\right]}{\mathbf{f}}_{2v}\right)\hfill \end{array}$$

(47)

Solving for $t_2$ Equations (47) and (12), we get

$$t_2=\frac{c_2}{k_2-\Vert {}^{\left[C\right]}{\mathbf{q}}_2\Vert \sqrt{k_2·(k_2-2)}}$$

(48)

where $\Vert {}^{\left[C\right]}{\mathbf{q}}_2\Vert =\sqrt{x_{q_2}^2+y_{q_2}^2+1}$ is the distance between the normalized projection point ${\mathrm{Q}}_2$ and the camera ${\mathrm{O}}_{\mathrm{C}}$ while considering Equation (46). Beware that the newly found location of ${\mathrm{P}}_2$ is given with respect to the real camera frame, $\left[C\right]$.

Again, we obtain the back-projection ray

$$\begin{array}{c}\hfill {}^{\left[F_2\right]}{\mathbf{v}}_2={}_{\left[C\right]}^{\left[F_2\right]}{T}_2{}^{\left[C\right]}{\mathbf{p}}_{2,h},\text{where}{{}_{\left[C\right]}^{\left[F_2\right]}{T}_2}_{(3\times 4)}=\left[\begin{array}{ccc}{\mathbf{I}}_{\left(3\right)},& & -{}_{}^{}{\mathbf{f}}_2\end{array}\right]\end{array}$$

(49)

in order to indicate the direction leaving from the primary focus ${\mathrm{F}}_2$ toward ${\mathrm{P}}_w$ through ${\mathrm{P}}_2$. Here, the corresponding elevation and azimuth angles are respectively given by

$$\begin{array}{cc}\hfill {}^{\left[F_2\right]}{\theta }_2& =arcsin\left(\frac{z_{{\mathbf{v}}_2}}{\Vert {}^{\left[F_2\right]}{\mathbf{v}}_2\Vert }\right)=arcsin\left(\frac{d-t_2}{\Vert {}^{\left[F_2\right]}{\mathbf{v}}_2\Vert }\right)\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill {}^{\left[F_2\right]}{\psi }_2& =arctan\left(\frac{y_{{\mathbf{v}}_2}}{x_{{\mathbf{v}}_2}}\right)=arctan\left(\frac{y_2}{x_2}\right)\hfill \end{array}$$

(51)

where $\Vert {}^{\left[F_2\right]}{\mathbf{v}}_2\Vert =\sqrt{x_2^2+y_2^2+{\left(c_2-t_2\right)}^2}$ is the magnitude of the direction vector from its reflection point ${\mathrm{P}}_2$.

Like done for the (forward) projection, it is convenient to define the back-projection functions ${\mathbf{f}}_{{\beta }_1}$ and ${\mathbf{f}}_{{\beta }_2}$ for lifting a 2D pixel point ${}^{\left[I\right]}m$ within radial bounds validated by Equation (37) to their angular components ${}^{\left[F_i\right]}\left({\theta }_i,{\psi }_i\right)$ with respect to the respective foci frame $\left[F_i\right]$ (oriented like $\left[C\right]$) as indicated by Equations (43), (44), (50) and (51), such that ${\mathbf{f}}_{{\beta }_i}:{\mathbb{R}}^2\mapsto {\mathbb{R}}^2$,

$$\begin{array}{ccc}\hfill {\mathbf{f}}_{{\beta }_i}\left({}^{\left[I\right]}m\right)& :=& \left\{\begin{array}{cc}\left({}^{\left[I\right]}\mathit{m}\stackrel{\text{Equation}\left(43\right)}{\mapsto }{}^{\left[F_1\right]}{\theta }_1,{}^{\left[I\right]}\mathit{m}\stackrel{\text{Equation}\left(44\right)}{\mapsto }{}^{\left[F_1\right]}{\psi }_1\right)\hfill & \text{if}i=1\hfill \\ \left({}^{\left[I\right]}\mathit{m}\stackrel{\text{Equation}\left(50\right)}{\mapsto }{}^{\left[F_2\right]}{\theta }_2,{}^{\left[I\right]}\mathit{m}\stackrel{\text{Equation}\left(51\right)}{\mapsto }{}^{\left[F_2\right]}{\psi }_2\right)\hfill & \text{if}i=2\hfill \\ None\hfill & \neg \text{Equation}\left(37\right).\hfill \end{array}\right.\hfill \end{array}$$

(52)

3.3. Field-of-View

The horizontal FOV is clearly 360° for both mirrors. In other words, azimuths ψ can be measured in the interval $\left[0,2\pi \right)$ $\mathrm{rad}$. As discussed previously, there exists a positive correlation between the vertical field of view (vFOV) angle ${\alpha }_i$ of mirror i and its profile parameter $k_i$, such that ${\alpha }_i\to 180°$ as $k_i\to \infty$ (see Figure 9). As demonstrated in Figure 5, ${\alpha }_i$ is physically bounded by its corresponding elevation angles: ${\theta }_{i,max}$, ${\theta }_{i,min}$. Both vFOV angles, ${\alpha }_1$ and ${\alpha }_2$, are computed from their elevation limits as follows:

$$\begin{array}{cc}\hfill {\alpha }_1& ={\theta }_{1,max}-{\theta }_{1,min}\hfill \end{array}$$

(53a)

$$\begin{array}{cc}\hfill {\alpha }_2& ={\theta }_{2,max}-{\theta }_{2,min}\hfill \end{array}$$

(53b)

The overall vFOV of the system is also given from these elevation limits:

$${\alpha }_{sys}=max\left({\theta }_{1,max},{\theta }_{2,max}\right)-min\left({\theta }_{1,min},{\theta }_{2,min}\right)$$

(54)

Figure 6 highlights the the so-called common vFOV angle, ${\alpha }_{SROI}$, for the stereo region of interest (SROI) where the same point can be seen from both mirrors so point correspondences can be found (Section 5). In our model, ${\alpha }_{SROI}$ can be decided from the value of the three prevailing elevation angles (${\theta }_{1,max}$, ${\theta }_{1,min}$, and ${\theta }_{2,min}$), such that:

$${\alpha }_{SROI}={\theta }_{SROI,max}-{\theta }_{SROI,min}$$

(55)

where generally,

$$\begin{array}{cc}\hfill {\theta }_{SROI,min}& =max({\theta }_{1,min},{\theta }_{2,min})\hfill \end{array}$$

(56a)

$$\begin{array}{cc}\hfill {\theta }_{SROI,max}& =min({\theta }_{1,max},{\theta }_{2,max})\hfill \end{array}$$

(56b)

The shaded area in Figure 6 illustrates the SROI that is far-bounded by the set of triangulated points found at the maximum range due to minimum disparity $\Delta m_{12}=1\mathrm{px}$ in the discrete case (refer to Figure 17), such that

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill P_{fs}=\left\{\right.{\mathrm{P}}_w\leftarrow {\mathbf{f}}_{\Delta }(({\theta }_1,{\psi }_1),({\theta }_2,{\psi }_2))& \mid ({\theta }_1,{\psi }_1)\leftarrow {\mathbf{f}}_{{\beta }_1}\left(m_1\right)\hfill \\ & \wedge ({\theta }_2,{\psi }_2)\leftarrow {\mathbf{f}}_{{\beta }_2}\left(m_2\right)\hfill \\ & \wedge \left.\Delta m_{12}=1,\mathrm{px}\right\}\hfill \end{array}\end{array}$$

(57)

where functions ${\mathbf{f}}_{{\beta }_i}$ and ${\mathbf{f}}_{\Delta }$, are provided in Equations (52) and (89).

The SROI is near-bounded (to the $\mathrm{Z}$-axis of radial symmetry) by its vertices ${\mathrm{P}}_{n{s}_{high}}$, ${\mathrm{P}}_{n{s}_{mid}}$ and ${\mathrm{P}}_{n{s}_{low}}$, which result from the following ray-intersection cases:

(a)

${\mathrm{P}}_{n{s}_{high}}\leftarrow {\mathbf{f}}_{\Delta }(({\theta }_{1,max},{\psi }_1),({\theta }_{2,max},{\psi }_2))$

(b)

${\mathrm{P}}_{n{s}_{mid}}\leftarrow {\mathbf{f}}_{\Delta }(({\theta }_{1,min},{\psi }_1),({\theta }_{2,max},{\psi }_2))$

(c)

${\mathrm{P}}_{n{s}_{low}}\leftarrow {\mathbf{f}}_{\Delta }(({\theta }_{1,min},{\psi }_1),({\theta }_{2,min},{\psi }_2))$

where the intersection function ${\mathbf{f}}_{\Delta }$ is implemented for direction rays (or angles) as defined in the Triangulation Section 5.2.

By assuming a radial symmetry on the camera’s field of view ${\alpha }_{cam}$, it should allow for a complete view of the mirror surface at its outmost diameter of $2r_{sys}$ according to Equation (6). Substantially, as depicted in Figure 6, ${\alpha }_{cam}$ is upper-bounded by the camera hole radius $r_{cam}$ selected according to Equation (78). The following inequality constraint emerges

$$\begin{array}{c}\hfill 2arctan\left(\frac{r_{sys}}{{\mathrm{f}}_{z_1}\left(r_{sys}\right)}\right)\le {\alpha }_{cam}\le 2arctan\left(\frac{r_{cam}}{{\mathrm{f}}_{z_2}\left(r_{cam}\right)}\right)\end{array}$$

(58)

where the respective functions ${\mathrm{f}}_{z_i}$ are defined in Equation (13).

Our specific viewing requirements when mounting the omnidirectional sensor along the central axis of the quadrotor ensure that objects located at 15 cm under the rig’s base and at 1 meter away (from the central axis) can be viewed. Thus, angles ${\theta }_{1,min}$ and ${\theta }_{2,min}$ should only be large enough as to avoid occlusions from the MAV’s propellers (Figure 5) and to produce inner and outer ring images at a useful ratio (Figure 7).

3.4. Spatial Resolution

The resolution of the images acquired by our system are not space invariant. In fact, an omnidirectional camera producing spatial resolution-invariant images can only be obtained through a non-analytical function of the mirror profile as shown in [31]. In this section, we study the effect our design has on its spatial resolution as it depends on position parameters like d and $c_i$ introduced in Section 2.1 as well as a direct dependency on the characteristics (e.g., focal length f) of the camera obtaining the image.

Let ${\eta }_{cam}$ be the spatial resolution for a conventional perspective camera as defined by Baker and Nayar in [25,29]. It measures the ratio between the infinitesimal solid angle d${\omega }_i$ (usually measured in steradians) that is directed toward a point ${\mathrm{P}}_i$ at an angle ${\theta }_{i,\mathrm{pix}}$ (formed with the optical axis ${\mathrm{Z}}_{\mathrm{C}}$) and the infinitesimal element of image area d$A_{\mathrm{pix}}$ that d${\omega }_i$ subtends (as shown in Figure 8). Accordingly, we have:

$$\begin{array}{cc}\hfill {\eta }_{cam}& =\frac{{\mathrm{d}A}_{\mathrm{pix}}}{{\mathrm{d}\omega }_i}=\frac{f^2}{{cos}^3{\theta }_{i,\mathrm{pix}}}\hfill \end{array}$$

(59)

whose behavior tends to decrease as ${\theta }_{\mathrm{pix}}\to 0$, so higher resolution areas on the sensor plane continuously increase the farther away they get from the optical center imaged at ${}^{\left[I\right]}{\mathit{m}}_c$. For ease of visualization, we plot only the u pixel coordinates corresponding to the 2D spatial resolution ${\eta }_{2\mathrm{D}}$, which is obtained by projecting the solid angle Ω onto a planar angle ${\theta }_{\Omega }$ (the apex angle in 2D of the solid cone of view). This yields ${\theta }_{\Omega }=2arccos\left(1-\Omega /2\pi \right)$, and we reduce the image area into its circular diameter with $2\sqrt{A/\pi }$. Generally, our conversion from 3D spatial resolution η in $\left[{\mathrm{m}}^2/\mathrm{sr}\right]$ units to 2D proceeds as follows:

$$\begin{array}{c}\hfill {\eta }_{2\mathrm{D}}=\frac{2\sqrt{\eta /\pi }}{{\theta }_{\Omega =1\mathrm{sr}}}\end{array}$$

(60)

where ${\theta }_{\Omega =1\mathrm{sr}}\approx 1.14390752211\mathrm{rad}$. More specifically, Equation (59) is manipulated to provide ${\eta }_{i,cam}$ as the indicative of spatial resolution toward any specific point in the mirror, ${}^{\left[C\right]}{\mathrm{P}}_i\in M_i$ according to Equation (8), as follows:

$$\begin{array}{cc}\hfill {\eta }_{i,cam}& =\left\{\begin{array}{cc}{f}^2{\left(\frac{\sqrt{r_1^2+z_1^2}}{z_1}\right)}^3\hfill & \text{if}i=1\hfill \\ f^2{\left(\frac{\sqrt{r_2^2+{(d-z_2)}^2}}{d-z_2}\right)}^3\hfill & \text{if}i=2\hfill \end{array}\right.\hfill \end{array}$$

(61)

where $r_i$ is the radial length defined in Equation (6) and its associated $z_i$ coordinate, f is the camera’s focal length, and the design parameters d and $c_i$ that relate to the position of the mirror focal points ${\mathrm{F}}_i$ with respect to the camera frame $\left[C\right]$.

Thus, for a conventional perspective camera, ${\eta }_{i,cam}$ grows as ${\theta }_{i,\mathrm{pix}}\to \pi /2$ due to the foreshortening effect that stretches the image representation around the sensor plane’s periphery where spatial information gets collected onto a larger number of pixels. Therefore, image areas farther from the optical axis are considered to have higher spatial resolutions.

Baker and Nayar also defined the resolution, ${\eta }_i$, of a catadioptric sensor in order to quantify the view of the world or d${\nu }_i$, an infinitesimal element of the solid angle subtended by the mirror’s effective viewpoint ${\mathrm{F}}_i$, which is consequently imaged onto a pixel area d$A_{\mathrm{pix}}$. Again, here we provide the resolution according to our model:

$$\begin{array}{cc}\hfill {\eta }_1& =\frac{{\mathrm{d}A}_{\mathrm{pix}}}{{\mathrm{d}\nu }_1}=\left[\frac{r_1^2+{(c_1-z_1)}^2)}{r_1^2+z_1^2}\right]{\eta }_{1,cam}\hfill \end{array}$$

(62a)

$$\begin{array}{cc}\hfill {\eta }_2& =\frac{{\mathrm{d}A}_{\mathrm{pix}}}{{\mathrm{d}\nu }_2}=\left[\frac{r_2^2+{(c_2-d+z_2)}^2)}{r_2^2+{(d-z_2)}^2}\right]{\eta }_{2,cam}\hfill \end{array}$$

(62b)

for our mirror-perspective camera configuration, where ${\mathrm{O}}_{\mathrm{C}}$ is the origin of coordinates as shown in Figure 8 and ${\eta }_{i,cam}$ is given in Equation (61).

As demonstrated by the plot of Figure 12 in Section 4.2.2, ${\eta }_i$ grows accordingly towards the periphery of each mirror (the equatorial region). This aspect of our sensor design is very important because it indicates that the common field of view, ${\alpha }_{SROI}$, where stereo vision is employed (Section 5), is imaged at a relatively higher resolution than the unused polar regions closer to the optical axis (the ${\mathrm{Z}}_{\mathrm{C}}$ axis).

If we modify ${\eta }_i$ by substituting $r_i$ with its equivalent ${\mathrm{f}}_{r_i}\left(z_i\right)$ function defined in Equation (14), using mirror 1 for example, we get:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\eta }_1& =\left[\frac{{\mathrm{f}}_{r_1}^2\left(z_1\right)+{(c_1-z_1)}^2}{{\mathrm{f}}_{r_1}^2\left(z_1\right)+z_1^2}\right]{\eta }_{1,cam}\hfill \\ & =\frac{f^2\sqrt{{\mathrm{f}}_{r_1}^2\left(z_1\right)+z_1^2}\left[{\mathrm{f}}_{r_1}^2\left(z_1\right)+{(c_1-z_1)}^2\right]}{z_1^3}\hfill \end{array}\end{array}$$

(63)

which is an inherent indicative of how the resolution ${\eta }_i$ for a reflection point ${\mathrm{P}}_i$ increases with $k_i\to \infty$ (Figure 11). Conversely, the smaller the $k_i$ parameter gets (related to eccentricity as discussed in Section 2.2), the flatter the mirror becomes, so its resolution resembles more that of the perspective camera alone. Mathematically, ${lim}_{k_i\to 2}{\eta }_i\to {\eta }_{i,cam}$.

As shown in Figure 9, a smaller $k_i$ would require a wider radius $r_{sys}$ in order to achieve the same omnidirectional vertical field of view, ${\alpha }_{sys}$. Even worse, in order to image such a wider reflector, either the camera’s field of view, ${\alpha }_{cam}$, would have to increase (by decreasing the focal length f and perhaps requiring a larger camera hole $r_{cam}$ and sensor size), or the distance $c_i$ between the effective pinhole and the viewpoint would have to increase accordingly. Another consequence is the effect on the baseline b, which must change in order to maintain the same vertical field of view (Figure 10). As a result, the depth resolution of the stereo system would suffer as well.

4. Parameter Optimization and Prototyping

The nonlinear nature of this system makes it very difficult to balance among its desirable performance aspects. The optimal vector of design parameters, ${\mathsf{\theta }}^{*}$, can be found by posing a constrained maximization problem for the objective function

$$\begin{array}{c}\hfill {\mathrm{f}}_b\left(\mathsf{\theta }\right)=c_1+c_2-d\end{array}$$

(64)

which measures the baseline according to Equation (3). Indeed, the optimization problem is subject to the set of constraints C, which we enumerate in Section 4.1. Formally,

$$\begin{array}{c}\hfill {\mathsf{\theta }}^{*}=\underset{\mathsf{\theta }\in \Theta }{\mathrm{arg\; max}}{\mathrm{f}}_b\left(\mathsf{\theta }\right)\text{subject}\text{to}C\end{array}$$

(65)

where $\Theta \subseteq {\mathbb{R}}^6$ is the 6-dimensional solution space for $\mathsf{\theta }\in {\mathbb{R}}^6$ given in Equation (4) as $\mathsf{\theta }=\left[\begin{array}{cccccc}{c}_1,& c_2,& k_1,& k_2,& d,& r_{sys}\end{array}\right]$.

4.1. Optimization Constraints

We discuss the constraints that the proposed omnistereo sensor is subject to. Overall, we mainly take the following into account:

(a)

geometrical constraints — including SVP and reflex constraints described by Equations (11), (12) and (2);

(b)

physical constraints — the rig’s dimensions, which include the mirrors radii as well as by-product parameters such as system height $h_{sys}$ and mass $m_{sys}$;

(c)

performance constraints — the spatial resolution and range from triangulation determined by parameters $k_1$, $k_2$, and $c_1$; the desired viewing angles for an optimal SROI field of view, ${\alpha }_{SROI}$.

Following the design model described throughout Section 2, we now list the pertaining linear and nonlinear constraints that compose the set C. We disjoint the linear constraints in a subset $C_L$ and the non-linear constraints subset $C_{NL}$, so $C=C_L\uplus C_{NL}$. Within each subset, we generalize equality constraints as functions $\mathrm{h}:{\mathbb{R}}^6\mapsto \mathbb{R}$ that obey

$$\begin{array}{c}\hfill \mathrm{h}\left(\mathsf{\theta }\right)=0\end{array}$$

(66)

whereas inequality functions $\mathrm{g}:{\mathbb{R}}^6\mapsto \mathbb{R}$ satisfy

$$\begin{array}{c}\hfill \mathrm{g}\left(\mathsf{\theta }\right)\le 0\end{array}$$

(67)

4.1.1. Linear Constraints

We have only setup linear inequalities for constraints in $C_L$. Specifically, we require the following:

g1:

In order to set the position of ${\mathrm{F}}_2$ below the origin ${\mathrm{O}}_{\mathrm{C}}$ of the pinhole camera frame $\left[C\right]$, the focal distance $c_2$ of mirror 2 must be larger than d (distance between ${\mathrm{O}}_{\mathrm{C}}$ and ${\mathrm{F}}_{2v}$),

$$\begin{array}{c}\hfill d\le c_2\end{array}$$

(68)

g2:

Because the hyperboloidal mirror should reflect light towards its effective viewpoint ${\mathrm{F}}_1$ without being occluded by the reflex mirror, mirror 1’s focal distance, $c_1$, needs to exceed the placement of the reflex mirror,

$$\begin{array}{c}\hfill d/2\le c_1\end{array}$$

(69)

g3:

The empirical constraint

$$\begin{array}{c}\hfill {\displaystyle \frac{5}{3}}\le {\displaystyle \frac{k_2}{k_1}}\end{array}$$

(70)

pertains our rig dimensions in order to assign a greater curvature to mirror 2’s profile (located a the bottom), so its view is directed toward the equatorial region rather than up. Complementarily, this constraint flattens mirror 1’s profile, so it can possess a greater view of the ground. This curvature inequality allows the SROI to be bounded by a wider vertical field of view when the sensor must be mounted above the MAV’s propellers as depicted in Figure 5.

4.1.2. Non-Linear Constraints

For the non-linear design constraints, we establish the following inequalities:

g4:

The AscTec Pelican quadrotor has a maximum payload of 650$\mathrm{g}$ (according to the manufacturer specifications [28]). Therefore, we must satisfy the system mass computed via Equation (16), such that

$$\begin{array}{c}\hfill m_{sys}\le 650\end{array}$$

(71)

g5:

Similarly, we limit the system’s height obtained with Equation (15) by a height limit $h_{sys,max}$,

$$\begin{array}{c}\hfill h_{sys}\le h_{sys,max}\end{array}$$

(72)

For example, we set $h_{sys,max}=150\mathrm{mm}$ for the 37$\mathrm{mm}$-radius rig.

g6:

The origin of coordinates for the camera frame is set at its viewpoint, ${\mathrm{O}}_{\mathrm{C}}$. In order to fit the camera enclosure under mirror 2, it is realistic to position the focus ${\mathrm{F}}_2$ on the vertical transverse axis at more than 5$\mathrm{mm}$ away from ${\mathrm{O}}_{\mathrm{C}}$:

$$\begin{array}{c}\hfill 5\le z_{0_2}-a_2\end{array}$$

(73)

where $z_{0_2}$ is defined in Equation (10), and $a_2$ pertains to Equation (5).

Next, we determine the bounds for the limiting angles that partake in the computation of the system’s vertical field of view ${\alpha }_{sys}$, which is based on equation Equation (54). Our application has specific viewing requirements that can be achieved with the following application conditions:

g7:

Let ${\Lambda }_{1,max}={14}^{\circ }$ be an acceptable upper-bound for angle ${\theta }_{1,max}$ , such that

$$\begin{array}{c}\hfill {\theta }_{1,max}\le {\Lambda }_{1,max}\end{array}$$

(74)

g8:

Because we desire a larger view towards the ground from mirror 1, we empirically set ${\Lambda }_{1,min}=-{25}^{\circ }$ as a lower-bound for the minimum elevation ${\theta }_{1,min}$,

$$\begin{array}{c}\hfill {\Lambda }_{1,min}\le {\theta }_{1,min}\end{array}$$

(75)

g9:

In order to avoid occlusions with the MAV’s propellers while being capable to image objects located about 5 $\mathrm{cm}$ under the rig’s base and 20 $\mathrm{cm}$ away (horizontally) from the central axis, we limit mirror 2’s lowest angle by a lower-bound ${\Lambda }_{2,min}=-{14}^{\circ }$,

$$\begin{array}{c}\hfill {\Lambda }_{2,min}\le {\theta }_{2,min}\end{array}$$

(76)

Finally, we restrict the radius of the system, $r_{sys}$, to be identical for both hypeboloids by satisfying the following equality condition:

h1:

With functions ${\mathrm{f}}_{r_1}$ and ${\mathrm{f}}_{r_2}$ defined in Equation (14), we set

$$\begin{array}{cc}\hfill r_{sys}=r_{i,max}& ={\mathrm{f}}_{r_i}\left(z_{i,max}\right),\forall i\in \{1,2\}\hfill \end{array}$$

where we imply that $z_{i,max}\leftarrow {\mathrm{f}}_{z_i}\left(r_{sys}\right)$ using Equation (13). Thus, the entire function composition for this equality becomes

$$\begin{array}{cc}\hfill {\mathrm{f}}_{r_1}\left({\mathrm{f}}_{z_1}\left(r_{sys}\right)\right)& ={\mathrm{f}}_{r_2}\left({\mathrm{f}}_{z_2}\left(r_{sys}\right)\right)\hfill \end{array}$$

(77)

4.2. Optimal Results

Applying the aforementioned constraints (Section 4.1) and using an iterative nonlinear optimization method such as one of the surveyed in [32], a bounded solution vector ${\mathsf{\theta }}^{*}$ converges to the the values shown in

Table 1. Optimal System Design Parameters.

Parameter	Big Rig	Small Rig
$b=max{\mathrm{f}}_b\left({\mathsf{\theta }}^{*}\right)$	131.61	108.92
$r_{sys}\left[\mathrm{mm}\right]$	37.0	28.0
$c_1\left[\mathrm{mm}\right]$	123.49	104.59
$c_2\left[\mathrm{mm}\right]$	241.80	204.34
$d\left[\mathrm{mm}\right]$	233.68	200.00
$k_1$	5.73	6.88
$k_2$	9.74	11.47

for two rig sizes.

Table 2. By-product Length Parameters.

Parameter	Big Rig	Small Rig
$r_{ref}\left[\mathrm{mm}\right]$	17.23	11.74
$r_{cam}\left[\mathrm{mm}\right]$	7	7
$h_{sys}\left[\mathrm{mm}\right]$	150.00	120.00

contains the by-product parameters corresponding to the dimensions listed in Table 1.

As Figure 3 illustrates, a realistic dimension for the radius of the camera hole, $r_{cam}$, must consider the maximum value between a physical micro-lens radius ($r_{lens}$) and the radius $r_{{\alpha }_{cam}{\mid }_{r_{sys}}}$ for an unoccluded field of view of the camera ${\alpha }_{cam}$ imaging the complete surface of mirror 1. Practically,

$$r_{cam}=max\left(r_{lens},r_{{\alpha }_{cam}{\mid }_{r_{sys}}}\right)$$

(78)

For both rigs, the expected vertical field of views are ${\alpha }_{sys}=75°-\left(-21°\right)\approx 96°$ according to Equation (54), and ${\alpha }_{SROI}=14°-\left(-14°\right)\approx 28°$ using Equation (55). Note that ${\theta }_{2,max}$ may be actually limited by the camera hole radius, so in reality ${\theta }_{cam}⇝59°$, and ${\alpha }_{sys}⇝80°$. For the big rig,

Table 3. Near Vertices of the SROI for the Big Rig.

Vertex	${}^{\left[C\right]}{\rho }_w$ [$\mathrm{mm}$]	${}^{\left[C\right]}{z}_w$ [$\mathrm{mm}$]
${\mathrm{P}}_{n{s}_{high}}$	93.5	144.4
${\mathrm{P}}_{n{s}_{mid}}$	65.2	98.4
${\mathrm{P}}_{n{s}_{low}}$	763.4	-170.3

shows the nearest vertices of the SROI that result from these angles (Figure 6).

4.2.1. Optimality of Parameters $k_1$ and $k_2$

Finally, we study the effect parameter $k_i$ has over the system radius $r_{sys}$ (Figure 9), the omnistereo baseline b (Figure 10), and the spatial resolution (Figure 11 and Figure 12). Figure 9 addresses the relation between $k_i$ and radius $r_{sys}$ (recall the rig size specified in Section 2.3). In Figure 11, it can be seen that for the same $r_{sys}$, realistic values for $k_1$ fall in the range $3<k_1<13$, and the vertical field of view ${\alpha }_1\to 0$ as $k\to 2$, which is expected according to the SVP property specified in Section 2.2. In fact, the left part of Figure 11 also demonstrates the necessary $r_{sys}$ to maintain ${\alpha }_{SROI}\approx 28$ for various values of $k_i$.

Figure 10 shows the inverse relationship between values of $k_1$ and the baseline, b, as we attempt to fit the view of a wider/narrower mirror profile (due to $k_1$) on the constant camera field of view, ${\alpha }_{cam}$. In order to make a fair comparison, let

$$k_1^{\prime }=k_1+{\epsilon }_k,\forall k_1>2,{\epsilon }_k>0$$

for which we find its new focal length $c_1^{\prime }$ while solving for the new $r_{sys}^{\prime }$ and $z_{max}^{\prime }$. Provided with a function such that $c_1\leftarrow {\mathrm{f}}_{c_1}\left(k_1\right)$, we perform the analysis for a given ${\alpha }_{SROI}$ and ${\alpha }_{cam}$ shown in Figure 10. Given the baseline function ${\mathrm{f}}_b$ defined in Equation (64), the following implication holds true:

$$\begin{array}{c}\hfill {\mathrm{f}}_b{\mid }_{c_1\leftarrow {\mathrm{f}}_{c_1}\left(k_1\right)}>{\mathrm{f}}_b{\mid }_{c_1^{\prime }\leftarrow {\mathrm{f}}_{c_1}(k_1+{\epsilon }_k)},\forall k_1>2,{\epsilon }_k>0\end{array}$$

(79)

Notice that $k_2$, $c_2$ and d are kept constant through this last analysis, and we ignore possible occlusions from the reflex mirror fixed at $d/2$.

4.2.2. Spatial Resolution Optimality

In this section, we compare the sensor’s spatial resolution, ${\eta }_i$, defined in Section 3.4 for the optimal parameters listed in Table 1 (for the big rig, only). In Figure 12, we verify how both resolutions ${\eta }_1$ and ${\eta }_2$ increase towards the equatorial region according to the spatial resolution theory presented in [29]. Indeed, the increase in spatial resolution within the SROI that covers the equatorial region (as indicated in Figure 6) justifies our model’s coaxial configuration intended for omnistereo applications.

In Figure 11, we compare the effect on ${\eta }_i$ for various mirror profiles, which depend directly on $k_i$. We illustrate the change in curvature due to parameters $k_1$ and $k_2$ and also show (in the legend) the respective $r_{sys}$ achieving a common vFOV of ${\alpha }_{SROI}\approx 28°$ as for the optimal parameters of the big rig. From this plot, we appreciate the compromise due to optimal parameters, $k_1^{(Opt.)}=5.7$ and $k_2^{(Opt.)}=9.7$, for a realistic system size due to $r_{sys}$ and a suitable range of spatial resolutions, ${\eta }_i$, within the SROI.

4.3. Prototypes

We validate our design with both synthetic and real-life models.

4.3.1. Synthetic Prototype (Simulation)

After converging to an optimal solution ${\mathsf{\theta }}^{*}$, we employ these parameters (Table 1) to describe synthetic models using POV-Ray, an open-source ray-tracer. We render 3D scenes via the camera of the synthetic omnistereo sensor like the example shown in Figure 2b. The simulation stage plays two important roles in our investigation:

(1)

to acquire ground-truth 3D-scene information in order to evaluate the computed range by the omnistereo system (as explained in Section 5); and

(2)

to provide an almost accurate geometrical representation of the model by discounting some real-life computer vision artifacts such as assembly misalignments, glare from the support tube (motivating the use of standoffs on the real prototype), as well as the camera’s shallow depth-of-field. All of these artifacts can affect the quality of the real-life results shown in Section 6.

4.3.2. Real-Life Prototypes

We have also produced two physical prototypes that can be installed on the Pelican quadrotor (made by Ascending Technologies [28]). Figure 13a shows the rig constructed with hyperboloidal mirrors of $r_{sys}\approx 37\mathrm{mm}$, and a Logitech^® HD Pro Webcam C910 camera capable of (2592 × 1944) pixel images at 15∼20 FPS. We decided to skip the use of the acrylic glass tube to separate the mirrors at the specified $h_{sys}$ distance, and instead we constructed a lighter 3-standoff mount in order to avoid glare and cross-reflections. This support was designed in 3D-CAD and printed for assembly. The three areas of occlusion due to the 3$\mathrm{mm}$-wide standoffs are non-invasive for the purpose of omnidirectional sensing and can be ignored with simple masks during image processing. In fact, we stamped fiducial markers to the vertical standoffs to aid with the panoramas generation (Section 5.1) and future calibration methods. To image the entire surface of mirror 1, we require a camera with a (minimum) field of view of ${\alpha }_{cam}>31°$, which is achieved by $r_{{\alpha }_{cam}}>1.4\mathrm{mm}$. In practice, as noted by Equation (78), microlenses measure around $r_{lens}\approx 7\mathrm{mm}$. Therefore, we set $r_{cam}>7\mathrm{mm}$, as a safe specification to fit a standard microlens through the opening of mirror 2 as shown in Figure 3.

Recall that $m_{sys}$ is limited by the maximum 650$\mathrm{g}$-payload that the AscTec Pelican quadrotor is capable of flying with (according to the manufacturer specifications [28]). The camera with lens weights approximately 25$\mathrm{g}$. A cylindrical tube made of acrylic has an average density ${\rho }_{tub}\approx 1.18$ g$·$cm${}^{-3}$, whereas the mirrors machined out of brass have a density ${\rho }_{mir}\approx 8.5$ g$·$cm${}^{-3}$. Empirically, we verify a close estimate of the entire system’s mass, such that $m_{sys}\approx 550\mathrm{g}$ for the big rig, and $m_{sys}\approx 150\mathrm{g}$ for the small rig.

5. 3D Sensing from Omnistereo Images

Stereo vision from point correspondences on images at distinct locations is a popular method for obtaining 3D range information via triangulation. Techniques for image point matching are generally divided between dense (area-based scanning [32]) and sparse (feature description [33]) approaches. Due to parallax, the disparity in point positions for objects close to the vision system must be larger than for objects that are farther away. As illustrated in Figure 6, the nearsightedness of the sensor is determined mainly by the common observable space (a.k.a. SROI) acquired by the limiting elevation angles of the mirrors (Section 3.3). In addition, we will see next (Section 5.2) that the baseline b also plays a major role in range computation.

Due to our model’s coaxial configuration, we could scan for pixel correspondences radially between a given pair of warped images $\left(\left[I_1\right],\left[I_2\right]\right)$ like in the approach taken by similar works such as [34]. However, it seems more convenient to work on a rectified image space, such as with panoramic images, where the search for correspondences can be performed using any of the various existing methods for perspective stereo views. Hence, we first demonstrate how these rectified panoramic images are produced (Section 5.1) and used for establishing point correspondences. Then, we proceed to study our triangulation method for the range computation from a given set of point correspondences (Section 5.2). Last, we show preliminary 3D point clouds as the outcome from such procedure.

5.1. Panoramic Images

Figure 14 illustrates how we form the respective panoramic image $\left[{\Xi }_1\right]$ out of its warped omnidirectional image $\left[I_1\right]$. As illustrated in Figure 7, $\left[I_i\right]$ is simply the region of interest out of the full image $\left[I\right]$ where projection occurs via mirror i. However, we can safely refer to $\left[I\right]$ because it will never be the case that projections via different mirrors overlap on the same pixel position ${}^{\left[I\right]}\mathit{m}$. In a few words, we obtain a panorama $\left[{\Xi }_i\right]$ by reverse-mapping each discretized 3D point ${\mathrm{P}}_{cy{l}_i}\in S_{cy{l}_i}$ to its projected pixel coordinates ${}^{\left[I\right]}\mathit{m}$ on $\left[I\right]$ according to Section 3.2.

More thoroughly, for $i=\{1,2\}$, $S_{cy{l}_i}$ is the set of all valid 3D points ${\mathrm{P}}_{cy{l}_i}$ that lie on an imaginary unit cylinder centered along the Z-axis and positioned with respect to the mirror’s primary focus ${\mathrm{F}}_i$. Recall that the radius of a unit cylinder is $r_{cyl}=1$, so its circumference becomes $w_{cyl}=2\pi r_{cyl}=2\pi$. Noticed that the imaging ratio, ${\chi }_{{\mathrm{I}}_{1:2}}=\frac{h_{I_1}}{h_{I_2}}$, illustrated in Figure 7 provides a way of inferring the scale between pairs of point correspondences. However, we achieve conforming scales among both panoramic representations by simply setting both cylinders to an equal height $h_{cyl}$, which is determined from the system’s elevation limits, $({\theta }_{sys,min},{\theta }_{sys,max})$, since they partake in the measurement of the system’s vertical field of view given by Equation (54). Hence, we obtain

$$\begin{array}{c}\hfill h_{cyl}=z_{cyl,max}-z_{cyl,min},\text{where}\left\{\begin{array}{cc}{z}_{cyl,max}\hfill & =tan\left({\theta }_{sys,max}\right)\hfill \\ z_{cyl,min}\hfill & =tan\left({\theta }_{sys,min}\right)\hfill \end{array}\right.\end{array}$$

(80)

Consequently, to achieve panoramic images $\left[{\Xi }_i\right]$ of the same dimensions by maintaining a true aspect ratio $w_{\Xi }:h_{\Xi }$, it suffices to indicate either the width (number of columns) $w_{\Xi }$ or the height (number of rows) $h_{\Xi }$ as number of pixels. Here, we propose a custom method for resolving the panoramic image dimensions by setting the equality for the length $l_{px}$ of an individual “square” pixel in the cylinder (behaving like a panoramic camera sensor):

$$\begin{array}{c}\hfill l_{px}=\frac{w_{cyl}}{w_{\Xi }}=\frac{h_{cyl}}{h_{\Xi }}\end{array}$$

(81)

For instance, if the width $w_{\Xi }$ is given, then the height is simply $h_{\Xi }=w_{\Xi }{h}_{cyl}/w_{cyl}$.

To increase the processing speed for each panoramic image $\left[{\Xi }_i\right]$, we fill up its corresponding look-up-table LUT${}_{{\Xi }_i}$ of size $w_{\Xi }\times h_{\Xi }$ that encodes the mapping for each panoramic pixel coordinates ${}^{\left[{\Xi }_i\right]}m={}^{\left[{\Xi }_i\right]}{\left[\begin{array}{cc}u,& v\end{array}\right]}^{\mathsf{T}}$ to its respective projection ${}^{\left[I_i\right]}\mathit{m}={}^{\left[I_i\right]}{\left[\begin{array}{cc}u,& v\end{array}\right]}^{\mathsf{T}}$ on the distorted image $\left[I_i\right]$. Each pixel ${}^{\left[{\Xi }_i\right]}\mathit{m}$ gets associated with its cylinder’s 3D point positioned at ${}^{\left[F_i\right]}{\mathbf{p}}_{cy{l}_i}$, which can inherently be indicated by its elevation ${}^{\left[F_i\right]}{\theta }_i$ and azimuth ${}^{\left[F_i\right]}{\psi }_i$ (relative to the mirror’s primary focus ${\mathrm{F}}_i$) as illustrated in Figure 4. Thus, the ray ${}^{\left[F_i\right]}{\mathbf{v}}_i$ of a particular 3D point directed about ${}^{\left[F_i\right]}\left({\psi }_i,{\theta }_i\right)$ must pass through ${\mathrm{P}}_{cy{l}_i}$ in order to get imaged as pixel ${}^{\left[I\right]}{\mathit{m}}_i$.

Since the circumference of the cylinder, $w_{cyl}$, is discretized with respect to the number of pixel columns or width $w_{\Xi }$, we use the pixel length $l_{px}$ as the factor to obtain the arc length $l_{{\psi }_i}$ spanned by the azimuth ${}^{\left[F_i\right]}{\psi }_i$ out of a given ${}^{\left[{\Xi }_i\right]}u$ coordinate on the panoramic image. Generally,

$$\begin{array}{c}\hfill {}^{\left[F_i\right]}{\psi }_i=\frac{l_{{\psi }_i}}{r_{cyl}}=\frac{w_{cyl}-{}^{\left[{\Xi }_i\right]}u{l}_{px}}{r_{cyl}}\end{array}$$

(82)

or simply ${}^{\left[F_i\right]}{\psi }_i=2\pi -{}^{\left[{\Xi }_i\right]}u{l}_{px}$ for the unit cylinder case.

An order reversal in the columns of the panorama is performed by Equation (82) because we account for the relative position between $S_{cy{l}_i}$ and the projection plane ${\pi }_{img}$. For $\left[{\Xi }_1\right]$, Figure 14 depicts the unrolling of the cylindrical panoramic image onto a planar panoramic image. However, note that ${\pi }_{img}$ is shown from above (or its back) in Figure 14, so the panorama visualization places the viewer inside the cylinder at ${\mathrm{F}}_1$.

Similarly, the elevation angle ${}^{\left[F_i\right]}{\theta }_i$ is inferred out the row or ${}^{\left[{\Xi }_i\right]}v$ coordinate, which is scaled to its cylindrical representation by $l_{px}$. Recall that both cylinders have the same height, $h_{cyl}$, computed by Equation (80). By taking into account any row offset from the maximum height position, ${}^{\left[F_i\right]}{z}_{cyl,max}$, of the cylinder, we get

$$\begin{array}{cc}\hfill {}^{\left[F_i\right]}{\theta }_i& =arctan\left({}^{\left[F_i\right]}{z}_{cyl,max}-{}^{\left[{\Xi }_i\right]}v{l}_{px}\right)\hfill \end{array}$$

(83)

Given these angles and assuming coaxial alignment, we evaluate the positon vector ${}^{\left[C\right]}{\mathbf{p}}_{cy{l}_i}$ for a point on the panoramic cylinder with respect to the camera frame $\left[C\right]$:

$$\begin{array}{cc}\hfill {}^{\left[C\right]}{\mathbf{p}}_{cy{l}_i}& =r_{\cancel{cyl}}\left[\begin{array}{c}cos\left({}^{\left[F_i\right]}{\psi }_i\right)\\ sin\left({}^{\left[F_i\right]}{\psi }_i\right)\\ tan\left({}^{\left[F_i\right]}{\theta }_i\right)\end{array}\right]+{}^{\left[C\right]}{\mathbf{f}}_i\hfill \end{array}$$

(84)

where $r_{cyl}$ cancels out for a unit cylinder. The direction equations Equations (82) and (83) leading to Equation (84) as a process: ${}^{\left[{\Xi }_i\right]}\mathit{m}\underset{\text{Equation}\left(83\right)}{\overset{\text{Equation}\left(82\right)}{\mapsto }}{}^{\left[F_i\right]}\left(\begin{array}{c}\psi \\ \theta \end{array}\right)\stackrel{\text{Equation}\left(84\right)}{\mapsto }{}^{\left[C\right]}{\mathbf{p}}_{cy{l}_i}$, which is eventually used as the input argument to Equation (36) in order to determine pixel ${}^{\left[I_i\right]}\mathit{m}$ via the mapping function ${\mathbf{h}}_{{\Xi }_i}:{\mathbb{R}}^2\mapsto {\mathbb{R}}^2$,

$$\begin{array}{ccc}\hfill {}^{\left[I_i\right]}\mathit{m}\leftarrow {\mathbf{h}}_{{\Xi }_i}\left({}^{\left[{\Xi }_i\right]}\mathit{m}\right)& :=& {\mathbf{f}}_{{\phi }_i}\left({\left.{}^{\left[C\right]}{\mathbf{p}}_{cy{l}_i}\right|}_{{}^{\left[{\Xi }_i\right]}\mathit{m}}\right)\hfill \end{array}$$

(85)

Stereo Matching on Panoramas

We understand that the algorithm chosen for finding matches is crucial to attain correct pixel disparity results. We refer the reader to [35] for a detailed survey of stereo correspondence methods. After comparing various block matching algorithms, we were able to obtain acceptable disparity maps with the semi-global block matching (SGBM) method introduced by [36], which can find subpixel matches in real time. As a result of this stereo block matcher among the pair of panoramic images $\left(\left[{\Xi }_1\right],\left[{\Xi }_2\right]\right)$, we get the dense disparity map $\left[{\Xi }_{\Delta m_{12}}\right]$ visualized as an image in Figure 15 and Figure 21a. Note that valid disparity values must be positive $({\left.\Delta m_{12}\right|}_{{}^{\left[{\Xi }_i\right]}{\mathit{m}}_1}>0)$ and they are given with respect to the reference image, in this case, $\left[{\Xi }_1\right]$. In addition, recall that no stereo matching algorithm (as far as we are aware) is totally immune to mismatches due to several well-known reasons in the literature such as ambiguity of cyclic patterns.

An advantage of the block (window) search for correspondences is that it can be narrowed along epipolar lines. Unlike the traditional horizontal stereo configuration, our system captures panoramic images whose views differ in a vertical fashion. As shown in [14], the unwrapped panoramas contain vertical, parallel epipolar lines that facilitate the pixel correlation search. Thus, given a pixel position ${}^{\left[{\Xi }_i\right]}{\mathit{m}}_1$ on the reference panorama $\left[{\Xi }_1\right]$ and its disparity value ${\left.\Delta {\mathit{m}}_{12}\right|}_{{}^{\left[{\Xi }_i\right]}{m}_1}$, we can resolve the correspondence ${}^{\left[{\Xi }_2\right]}{\mathit{m}}_2$ pixel coordinate on the target image, $\left[{\Xi }_2\right]$, by simply offsetting the v-coordinate with the disparity value:

$$\begin{array}{cc}\hfill {}^{\left[{\Xi }_2\right]}{\mathit{m}}_2& =\left[\begin{array}{c}{u}_1\\ v_1+{\left.\Delta m_{12}\right|}_{{}^{\left[{\Xi }_1\right]}{m}_1}\end{array}\right]\hfill \end{array}$$

(86)

5.2. Range from Triangulation

Recall the duality that states a point ${\mathrm{P}}_w$ as the intersection of a pair of lines. Regardless of the correspondence search technique employed, such as block stereo matching between panoramas $\left[{\Xi }_i\right]$ (Section 5.1.1) or feature detection directly on $\left[I\right]$, we can resolve for ${}^{\left[I\right]}\left({\mathit{m}}_1,{\mathit{m}}_2\right)$. From Equations (42) and (49), we obtain the respective pair of back-projected rays $\left({}^{\left[F_1\right]}{\mathbf{v}}_1,{}^{\left[F_2\right]}{\mathbf{v}}_2\right)$, emanating from their respective physical viewpoints, ${\mathrm{F}}_1$ and ${\mathrm{F}}_2$, which are separated by baseline b. We can compute elevation angles ${\theta }_1$ and ${\theta }_2$ using equations Equations (43) and (50). Then, we can triangulate the back-projected rays in order to calculate the horizontal range ${\rho }_w$ defined in Equation (22), as follows:

$${\rho }_w=\frac{bcos\left({\theta }_1\right)cos\left({\theta }_2\right)}{sin({\theta }_1-{\theta }_2)}$$

(87)

Finally, we obtain the 3D position of ${\mathrm{P}}_w$:

$${}^{\left[C\right]}{\mathbf{p}}_w=\left[\begin{array}{c}-{\rho }_{w}cos\left({\psi }_{12}\right)\\ -{\rho }_{w}sin\left({\psi }_{12}\right)\\ c_1-{\rho }_{w}tan\left({\theta }_1\right)\end{array}\right]$$

(88)

where ${\psi }_{12}$ is the common azimuthal angle (on the XY-plane) for coplanar rays, so it can be determined either by Equation (44) or Equation (51). Functionally, we define the “naive” intersection function that implements Equations (87) and (88) such that

$$\begin{array}{c}\hfill {}^{\left[C\right]}{\mathbf{p}}_w\leftarrow {\mathbf{f}}_{\Delta }(({\theta }_1,{\psi }_1),({\theta }_2,{\psi }_2),\mathsf{\theta })\end{array}$$

(89)

where θ is the model parameters vector defined in Equation (4) and can be omitted when calling this function because the model parameters should not change (ideally).

5.2.1. Common Perpendicular Midpoint Triangulation Method

Because the coplanarity of these rays cannot be guaranteed (skew rays case), a better triangulation approximation while considering coaxial misalignments is to find the midpoint of their common perpendicular line segment (as attempted in [23]). As illustrated in Figure 16, we define the common perpendicular line segment $\overline{{\mathrm{G}}_1{\mathrm{G}}_2}$ as the parametrized vector ${\mathbf{v}}_{1\perp 2}={\lambda }_{1\perp 2}{\widehat{\mathbf{v}}}_{1\perp 2}$, for the unit vector normal to the back-projected rays, ${\mathbf{v}}_1$ and ${\mathbf{v}}_2$, such that:

$$\begin{array}{c}\hfill {\widehat{\mathbf{v}}}_{1\perp 2}=\frac{{\mathbf{v}}_1\otimes {\mathbf{v}}_2}{||{\mathbf{v}}_1\otimes {\mathbf{v}}_2||}\end{array}$$

(90)

If the rays are not parallel ($||{\mathbf{v}}_1\otimes {\mathbf{v}}_2||\ne 0$), we can compute the “exact” solution, $\lambda ={\left[\begin{array}{ccc}{\lambda }_{G_1},& {\lambda }_{G_2},& {\lambda }_{1\perp 2}\end{array}\right]}^{\mathsf{T}}$, of the well-determined linear matrix equation

$$\begin{array}{cc}\hfill \mathbf{V}\lambda & =\mathbf{b},\text{where}\mathbf{V}=\left[\begin{array}{ccccc}{\mathbf{v}}_1,& & -{\mathbf{v}}_2,& & {\widehat{\mathbf{v}}}_{1\perp 2}\end{array}\right]\text{and}\mathbf{b}={}^{\left[C\right]}{\mathbf{f}}_2-{}^{\left[C\right]}{\mathbf{f}}_1\hfill \end{array}$$

(91)

It follows that the location of the midpoint ${\mathrm{P}}_{w_G}$ on the common perpendicular ${\mathbf{v}}_{1\perp 2}$ with respect to the common frame $\left[C\right]$ is

$$\begin{array}{cc}\hfill {}^{\left[C\right]}{\mathbf{p}}_{w_G}& ={}^{\left[C\right]}{\mathbf{f}}_1+{\lambda }_{G_1}{}^{\left[F_1\right]}{\mathbf{v}}_1+\frac{1}{2}{\lambda }_{1\perp 2}{}^{\left[G_1\right]}{\widehat{\mathbf{v}}}_{1\perp 2}\hfill \end{array}$$

(92)

5.2.2. Range Variation

Before we introduce an uncertainty model for triangulation (Section 5.3), we briefly analyze how range varies according to the possible combinations of pixel correspondences, ${}^{\left[I\right]}({\mathit{m}}_1,{\mathit{m}}_2)$ on the image $\left[I\right]$. Here, we demonstrate how a radial variation of discretized pixel disparities, $\Delta m_{12}$, affects the 3D position of a point obtained from triangulation (Section 5.2). Figure 17 demonstrates the nonlinear characteristics of the variation in horizontal range, $\Delta {\rho }_w$, from the discrete relation between pixel positions ${}^{\left[I\right]}{\mathit{m}}_i$ and their respective back-projected (direction) rays obtained from ${\mathbf{f}}_{{\beta }_i}$ and triangulated via function ${\mathbf{f}}_{\Delta }$ defined in Equation (89). It can be observed that the horizontal range variation, $\Delta {\rho }_w$, increases quadratically as $\Delta {\mathit{m}}_{12}\to 1\mathrm{px}$, which is the minimum discrete pixel disparity, which provides a maximum horizontal range ${\rho }_{w,max}\approx \left[18,28\right]$$\mathrm{m}$ (computed analytically). The main plot of Figure 17 shows the small disparity values in the interval $\overline{\Delta m_{12}}=[1,20]\mathrm{px}$, whereas the subplot is a zoomed-in extension of the large disparity cases in the interval $\overline{\Delta m_{12}}=[20,100]\mathrm{px}$.

The current analysis is an indicative that triangulation error (e.g., due to false pixel correspondences) may have a severe effect on range accuracy that increases quadratically with distance as it can be appreciated with the 8 $\mathrm{m}$ variation on the disparity interval $\overline{\Delta m_{12}}=[1,2]\mathrm{px}$. Also, observe the example of Figure 20 for a reconstructed point cloud, where this range sensing characteristic is more noticeable for faraway points. In fact, the following uncertainty model provides a probabilistic framework for the triangulation error (uncertainty) that agrees with the current numerical claims.

5.3. Triangulation Uncertainty Model

Let ${\mathbf{f}}_{{\mathrm{P}}_w}$ be the vector-valued function that computes the 3D coordinates of point ${\mathrm{P}}_{w_G}$ with respect to $\left[C\right]$ as the common perpendicular midpoint defined in Equation (92). We express this triangulation function component-wise as follows:

$$\begin{array}{c}\hfill {}^{\left[C\right]}{\mathbf{p}}_{w_G}\leftarrow {\mathbf{f}}_{{\mathrm{P}}_w}\left({\mathbf{m}}_{12}\right)\left[\begin{array}{c}{\mathrm{f}}_{x_w}\left({\mathbf{m}}_{12}\right)\\ {\mathrm{f}}_{y_w}\left({\mathbf{m}}_{12}\right)\\ {\mathrm{f}}_{z_w}\left({\mathbf{m}}_{12}\right)\end{array}\right]\end{array}$$

(93)

where ${\mathbf{m}}_{12}=[u_1,v_1,u_2,v_2]$ is composed by the pixel coordinates of the correspondence ${}^{\left[I\right]}({\mathit{m}}_1,{\mathit{m}}_2)$ upon which to base the triangulation (Section 5.2).

Without loss of generality, we model a multivariate Gaussian uncertainty model for triangulation, so that the position vector ${}^{\left[C\right]}{\mathbf{p}}_{w_G}$ of any world point is centered at its mean ${}^{\left[C\right]}{\mathsf{\mu }}_{{\mathbf{f}}_{{\mathrm{P}}_w}}$ with a $3\times 3$ covariance matrix ${\Sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}$:

$$\begin{array}{cc}\hfill {}^{\left[C\right]}{\mathsf{\mu }}_{{\mathbf{f}}_{{\mathrm{P}}_w}}=\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\end{array}\right],& {\Sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}=\left[\begin{array}{ccccc}{\sigma }_{{\mathrm{f}}_{x_w}}^2& & {\sigma }_{{\mathrm{f}}_{x_w}}{\sigma }_{{\mathrm{f}}_{y_w}}& & {\sigma }_{{\mathrm{f}}_{x_w}}{\sigma }_{{\mathrm{f}}_{z_w}}\\ {\sigma }_{{\mathrm{f}}_{x_w}}{\sigma }_{{\mathrm{f}}_{y_w}}& & {\sigma }_{{\mathrm{f}}_{y_w}}^2& & {\sigma }_{{\mathrm{f}}_{y_w}}{\sigma }_{{\mathrm{f}}_{z_w}}\\ {\sigma }_{{\mathrm{f}}_{x_w}}{\sigma }_{{\mathrm{f}}_{z_w}}& & {\sigma }_{{\mathrm{f}}_{y_w}}{\sigma }_{{\mathrm{f}}_{z_w}}& & {\sigma }_{{\mathrm{f}}_{z_w}}^2\end{array}\right]\hfill \end{array}$$

(94)

However, since ${\mathbf{f}}_{{\mathrm{P}}_w}$ is a non-linear vector-valued function, we linearize it by approximation to a first-order Taylor expansion and we use its Jacobian matrix to propagate the uncertainty (covariance) as in the linear case as follows:

$$\begin{array}{c}\hfill {\Sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}={\mathbf{J}}_{{\mathbf{f}}_{{\mathrm{P}}_w}}{\mathsf{\Omega }}_{{\mathbf{m}}_{12}}{{\mathbf{J}}_{{\mathbf{f}}_{{\mathrm{P}}_w}}}^{\mathsf{T}}\end{array}$$

(95)

where the $3\times 4$ Jacobian matrix for the triangulation function is

$$\begin{array}{c}\hfill {\mathbf{J}}_{{\mathbf{f}}_{{\mathrm{P}}_w}}=\left[\begin{array}{cccc}\frac{\partial {\mathrm{f}}_{x_w}}{\partial u_1}& \frac{\partial {\mathrm{f}}_{x_w}}{\partial v_1}& \frac{\partial {\mathrm{f}}_{x_w}}{\partial u_2}& \frac{\partial {\mathrm{f}}_{x_w}}{\partial v_2}\\ \frac{\partial {\mathrm{f}}_{y_w}}{\partial u_1}& \frac{\partial {\mathrm{f}}_{y_w}}{\partial v_1}& \frac{\partial {\mathrm{f}}_{y_w}}{\partial u_2}& \frac{\partial {\mathrm{f}}_{y_w}}{\partial v_2}\\ \frac{\partial {\mathrm{f}}_{z_w}}{\partial u_1}& \frac{\partial {\mathrm{f}}_{z_w}}{\partial v_1}& \frac{\partial {\mathrm{f}}_{z_w}}{\partial u_2}& \frac{\partial {\mathrm{f}}_{z_w}}{\partial v_2}\end{array}\right]\end{array}$$

(96)

and the $4\times 4$ covariance matrix of the pixel arguments being

$$\begin{array}{c}\hfill {\mathsf{\Omega }}_{{\mathbf{m}}_{12}}={\sigma }_{px}^2{\mathbf{I}}_4\end{array}$$

(97)

where we assume ${\sigma }_{px}=1\mathrm{px}$ for the standard deviation of each pixel coordinate in the discretized pixel space. The complete symbolic solution of ${\Sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}$ is too involved to appear in this manuscript. However, in Figure 18, we show the top-view of the covariance ellipsoid drawn at a three-${\sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}$ level for a point triangulated nearly around ${\rho }_w\approx 100\mathrm{mm}$. Figure 19 visualizes uncertainty ellipsoids drawn at a one-${\sigma }_{{\mathbf{f}}_{{\mathrm{P}}_w}}$ level for several triangulation ranges. We refer the reader to the end of Section 6.3 where we validate the safety of this 1 pixel deviation assumption through experimental results using subpixel precision.

6. Experiment Results

In this section, we demonstrate the capabilities of the omnistereo sensor to provide 3D information either as dense point clouds or as for the registration of sparse 2D features and 3D points. We also evaluate the precision of both projection and triangulation of a few detected corners from a chessboard whose various 3D poses are given as ground-truth.

6.1. Dense 3D Point Clouds

By implementing the process described in Section 5, we begin by visualizing the dense point-cloud obtained from the omnidirectional synthetic image given in Figure 2b, whose actual size is 1280 × 960 pixels. The associated panoramic images, $\left[{\Xi }_i\right]$, were obtained using function ${\mathbf{h}}_{{\Xi }_i}$ defined in Equation (85) and are shown in Figure 15. Pixel correspondences $({}^{\left[{\Xi }_1\right]}{\mathit{m}}_1,{}^{\left[{\Xi }_2\right]}{\mathit{m}}_2)$ on the panoramic representations are mapped via ${\mathbf{h}}_{{\Xi }_i}$ into their respective image positions ${}^{\left[I\right]}({\mathit{m}}_1,{\mathit{m}}_2)$. Then, these are triangulated with ${}^{\left[C\right]}{\mathbf{f}}_{{\mathrm{P}}_w}$ given in Equation (93), resulting in the set (cloud) of color 3D points $P_{\Delta }$ visualized in Figure 20. Here, the synthetic scene (Figure 2a) is for a room 5.0 $\mathrm{m}$ wide (along its $\mathrm{X}$-axis), 8.0 $\mathrm{m}$ long (along its $\mathrm{Y}$-axis), and 2.5 $\mathrm{m}$ high (along its $\mathrm{Z}$-axis). With respect to the scene center of coordinates, $\left[S\right]$, the catadioptric omnistereo sensor, $\left[C\right]$, is positioned at ${}_{\left[C\right]}^{\left[S\right]}\mathbf{t}={\left[\begin{array}{ccc}1.60,& -2.85,& 0.16\end{array}\right]}^{\mathsf{T}}$ in meters.

We also present results from a real experiment using the prototype described in Section 4.3.2 and shown in Figure 13a. The panoramic images and dense point cloud shown in Figure 21 are obtained by implementing the pertinent functions described throughout this manuscript and by holding the SVP assumption of an ideal configuration. We provide these qualitative results as preliminary proof of concept for the proposed sensor after employing a calibration procedure based on the generalized unified model proposed in [37].

6.2. Sparse 3D Points from Features

Using the SURF feature detector and descriptors [38], Figure 22 demonstrates 44 correct matches that are triangulated with Equation (93). Sparse 3D points can be useful for applications of visual odometry where the sensor changes poses and those registered point features can be matched against new images. Please, refer to [39] for a tutorial on visual odometry.

6.3. Triangulation Evaluation

6.3.1. Evaluation of Synthetic Rig

Due to the unstructured nature of the dense point clouds previously discussed, we proceed to triangulate sets of sparse 3D points whose positions with respect to the omnistereo sensor camera frame, $\left[C\right]$, are known in advance. We synthesize a calibration chessboard pattern $\left[G\right]$ containing $m\times n$ square cells for various predetermined poses ${}_{\left[G\right]}^{\left[C\right]}{\mathbf{T}}_h$. Since the sensor is assumed to be rotationally symmetric, it suffices to experiment with groups of $L=4$ chessboard patterns situated at a given horizontal range. A total of $Lmn$ 3D points are available for each range group. Each corner point’s position ${}^{\left[C\right]}{\mathbf{p}}_j$ is found with respect to $\left[C\right]$ via the frame transformation ${}^{\left[C\right]}{\mathbf{p}}_{j,g}={}_{\left[C_g\right]}^{\left[C\right]}{\mathbf{T}}_h{}^{\left[C_g\right]}{\mathbf{p}}_j$ for all indices $j\in \{1,\dots ,mn\},g\in \{1,\dots ,L\}$.

Figure 23 shows the set of detected corner points on the image from the group of patterns set to a range of ${}^{\left[C\right]}{\rho }_G=2\mathrm{m}$. We adjust the pattern’s cell sizes accordingly so its points can be safely discerned by an automated corner detector [35]. We systematically establish correspondences of pattern points on the omnidirectional image, and proceed to triangulate with Equation (93). For each range group of points, we compute the root-mean-square of the 3D position errors (RMSE) between the observed (triangulated) points ${}^{\left[C\right]}{\tilde{\mathbf{p}}}_j\leftarrow {\mathbf{f}}_{{\mathrm{P}}_w}({\tilde{\mathit{m}}}_1,{\tilde{\mathit{m}}}_2)$ and the true (known) points ${}^{\left[C\right]}{\mathbf{p}}_j$ that were used to describe the ray-traced image.

Table 4. Results of RMSE from Synthetic Triangulation Experiment.

${}^{\left[C\right]}{\rho }_G$ [$\mathrm{m}$]	RMSE [$\mathrm{mm}$]	SD [$\mathrm{mm}$]
0.25	0.46	0.31
0.50	1.20	0.71
1.0	4.62	2.55
2.0	14.85	9.06
4.0	57.67	31.34
8.0	219.09	129.92

compiles the RMSE results and the standard deviation (SD) for some group of patterns whose frames $\left[G_g\right]$, are located at specified horizontal ranges ${}^{\left[C\right]}{\rho }_G\in [0.25,8.0]$$\mathrm{m}$ away from $\left[C\right]$.

We notice that for all the 3D points in the synthetic patterns, we obtained an average error of $0.1\mathrm{px}$ with a standard deviation ${\tilde{\sigma }}_{px}=0.05\mathrm{px}$ for the subpixel detection of corners on the image versus their theoretical values obtained from ${\mathbf{f}}_{{\phi }_i}$ defined in Equation (36). This last experiment helps us validate the pessimistic choice of ${\sigma }_{px}=1\mathrm{px}$ for the discrete pixel space in the triangulation uncertainty model proposed in Section 5.3.

6.3.2. Evaluation of Real-Life Rig

The following experiment uses $L=5$ different poses of a real chessboard pattern with $5\times 8$ corner points where the square cell size is $24\mathrm{mm}$. As done in Section 6.3.1, the evaluated error is the Euclidean norms between the triangulated points and the ground-truth positions of the chessboard posses captured via a motion capture system. The RMSE for all projected points in this set of chessboard patterns is 2.5 $\mathrm{pixels}$ with a standard deviation of 1.5 $\mathrm{pixels}$. The RMSE for all triangulated points in this set is 3.5$\mathrm{mm}$ with a standard deviation of 1.4$\mathrm{mm}$. Figure 24 visually confirms the proximity of the triangulated chessboard poses against the ground-truth pose information.

7. Discussion and Future Work

The portable aspect of the proposed omnistereo sensor is one of its greatest advantages, as discussed in the introduction section. The total weight of the big rig using 37$\mathrm{mm}$-radius mirrors is about 550$\mathrm{g}$, so it can be carried by the AscTec Pelican quadrotor under its payload limitations of 650$\mathrm{g}$. The mirror profiles maximize the stereo baseline while obeying the various design constraints such as size and field of view. Currently, the mirrors are custom-manufactured out of brass using CNC machining. However, it is possible to reduce the system’s weight dramatically by employing lighter materials.

In reality, it is almost impossible to assemble a perfect imaging system that fulfills the SVP assumption and avoids the triangulation uncertainty studied in Section 5.3 on top of the error already introduced by any feature matching technique. The coaxial misalignment of the folded mirrors-camera system, defocus blur of the lens, and the inauspicious glare from the support tube are all practical caveats we need to overcome for better 3D sensing tasks. As described in the text for the real-life rig, we have avoided the traditional use of a support cylinder in order to workaround the cross-reflections and glare issues. Possible vibrations caused by the robot dynamics are reduced by vibration pads placed on the sensor-body interface. Details about our tentative calibration method for vertically-folded omnistereo systems has not been included in the current study since we would like the reader’s attention to be devoted to the sensor characteristics defended by this analysis.

Our ongoing research is also focusing on the development of efficient software algorithms for real-time 3D pose estimation from point clouds. Bear in mind that all the experimental results demonstrated in this manuscript rely upon a single camera snapshot. We understand that the narrow vertical field-of-view where stereo vision operates is a limiting factor for dense scene reconstruction from a single image, so we have also considered non-optimal geometries for the quadrotor’s view. In fact, increasing the region of interest for stereo (SROI) while maintaining the wide baseline implies an enlargement of each mirror’s radius. We believe that our omnidirectional system is more advantageous than forward-looking sensors because it can provide a robust pose estimation by extracting 3D point features from all around the scene at once. As in our past work [24], fusing multiple modalities (e.g., stereo and optical-flow) is a possibility in order to resolve the scale-factor problem inherent while performing structure from motion over the non-stereo regions of each mirror (near the poles).

In this work, we performed an extensive study of the proposed omnistereo sensor’s properties, such as its spatial resolution and triangulation uncertainty. We validated the projection accuracy of the synthetic model (the ideal case) where 3D points in the world are given exactly. In order to validate the precision of the real sensor, we require a perfectly constructed and assembled device so point projections can be accepted as the ultimate truth. This is hard to achieve at a low-cost prototyping stage. Although we acquired ground-truth 3D points via a position capture system alone, we deem this insufficient to validate the imaging accuracy of the real sensor because the precision of the calibration method is truly what is being accounted for. For reproducibility purposes, source code is available for the implementation of the theoretical omnistereo model, optimization, plots and figures presented in this analysis [40].