Rotation Estimation: A Closed-Form Solution Using Spherical Moments ^†

Abstract

Photometric moments are global descriptors of an image that can be used to recover motion information. This paper uses spherical photometric moments for a closed form estimation of 3D rotations from images. Since the used descriptors are global and not of the geometrical kind, they allow to avoid image processing as features extraction, matching, and tracking. The proposed scheme based on spherical projection can be used for the different vision sensors obeying the central unified model: conventional, fisheye, and catadioptric. Experimental results using both synthetic data and real images in different scenarios are provided to show the efficiency of the proposed method.

1. Introduction

Rotation estimation from images is important for many application as for motion estimation and registration in image processing [1], pattern recognition [2], localization and control of ground/aerial vehicles [3], and computer vision [4]. Feature based and direct (appearance based) are the two main approach categories proposed in the literature. The first method type uses Epipolar geometry applied to geometrical features as as points [5], lines [6], or contours [7] to estimate the fundamental matrix (uncalibrated vision sensors), the essential matrix, or homography matrix for the calibrated case. Examples of such methods using conventional cameras are Agrawal et al. [8] and Malis et al. [9] and using omnidirectional vision sensors is [10,11,12]. The direct approaches, on the other hand, avoid geometric features extraction by using grey level information of the whole image to define the descriptors to be used. Those methods are referred to in the literature as dense, direct, or global. Methods within this category can avoid the need of some higher-level image processing steps such as feature extraction, matching, and tracking. For instance, Reference [13] exploits optical flow estimation. However, a consequence of using optical flow in References [13,14] is to be limited to small camera displacements. To deal with larger motion, in Reference [4], Makadia and Daniilidis proposed a method using correlations between the two sets of spherical harmonic coefficients.

To address even larger camera displacements, motion estimation techniques based on global image representation is utilized in the literature. Makadia and Daniilidis [4] represented the spherical omnidirectional images in the frequency domain to recover rotation using the maximum correlation between the two sets of spherical harmonic coefficients. In Reference [15], the authors proposed to estimate the rotation based on 3D-mesh representation of spherical image intensities. However, these methods are not often computationally efficient due to the computation load of spherical harmonic coefficients.

The moments as descriptors provide useful characteristics such as the center of gravity of an object, orientation, disparity, and volume. As Fourier descriptors, they can be used as a compact representation of the image. In the case of 2D images, in addition to the classical geometrical moments, other kinds of moments have been defined in the literature as Zernike Moments [16] or complex moments [17]. Invariant characteristics to some transformations that an object can undergo, commonly called moment invariants, found many applications such as for pattern recognition. Thanks to their decoupling properties, they also have been used to define a minimal number of features for visual servoing and pose estimation as in References [18,19] and for visual servoing as in References [20,21].

Spherical projections can be obtained from images if the used camera obeys the unified central model. By using such projections, the authors in Reference [22] proposed a decoupled visual servoing scheme using spherical moments computed for a set of matched points. The proposed features allow to control the translational motions independently from the rotational ones. The current work exploits the spherical projection to estimate 3D rotations using spherical moments. The proposed scheme provides an analytical solution to rotation. An early version of this work was published in Reference [23]. In the next section, the central camera model and the definition of spherical moments are first recalled. The proposed approach is subsequently presented. In Section 3, experimental results of rotation estimation using synthetic data and real images acquired from a catadioptric camera mounted on a manipulator robot (Staubli) and a mobile robot (Pioneer) are presented to validate our approach.

2. Direct Estimation of Rotation Using Spherical Moments

2.1. Spherical Moments

The projection onto a unitary sphere t ${\mathbf{P}}_{\mathbf{s}}={\left[x_s{y}_s{z}_s\right]}^{\top }$ of a 3D point $\mathbf{P}$ is defined by the following:

$${\mathbf{P}}_{\mathbf{s}}={\left[x_s{y}_s{z}_s\right]}^{\top }=\frac{\mathbf{P}}{\Vert \mathbf{P}\Vert }.$$

(1)

Actually, the projection onto the unit sphere can be obtained using the unified central model for conventional, fisheye, or catadioptric [24]. This makes the proposed scheme in this work valid for a wide range of cameras.

From a spherical image (or Gaussian sphere [25]) of the environment, the photometric moments can be defined by the following:

$$m_{ijk}={\int \int }_S{x}_s^i{y}_s^j{z}_s^{k}I(x_s,y_s,z_s,t)ds,$$

(2)

where S is defined as the sphere surface. Let $\mathbf{R}$ be a rotation matrix and $\omega$ be the instantaneous rotational speed applied to the sensor frame. First, it can be easily shown that a rotational motion defined by a matrix $\mathbf{R}$ on 3D point $\mathbf{P}$ (${\mathbf{P}}^{\prime }=\mathbf{R}\mathbf{P}$, where ${\mathbf{P}}^{\prime }$ is the same 3D point after rotation) induces the same motion on its projection onto the unit sphere (${\mathbf{P}}_{\mathbf{s}}^{\prime }=\mathbf{R}{\mathbf{P}}_{\mathbf{s}}$). In tangent space, the effect of a rotation speed $\mathit{\omega }={\left[{\omega }_x{\omega }_y{\omega }_z\right]}^{\top }$ on a projected point on the sphere is given by its time derivative ${\dot{\mathbf{P}}}_{\mathbf{s}}$:

$${\dot{\mathbf{P}}}_{\mathbf{s}}={\left[{\mathbf{P}}_{\mathbf{s}}\right]}_{\times }\mathit{\omega }=-{\left[\mathit{\omega }\right]}_{\times }{\mathbf{P}}_{\mathbf{s}}={\mathbf{L}}_{{\mathbf{P}}_{\mathbf{s}}}\mathit{\omega }=\left[\begin{array}{ccc}0& -z_s& y_s\\ z_s& 0& -x_s\\ -y_s& x_s& 0\end{array}\right]\mathit{\omega },$$

(3)

where ${\mathbf{L}}_{{\mathbf{P}}_{\mathbf{s}}}$ is the interaction matrix and ${\left[\right]}_{\times }$ defines the skew-symmetric matrix of a vector. On the other side, by taking the time derivative of Equation (2), the time variation of the moment $m_{ijk}$ is linked to rotational speed $\mathit{\omega }$ by the following [26]:

$${\dot{m}}_{ijk}={\left[\begin{array}{c}j{m}_{i,j-1,k+1}-k{m}_{i,j+1,k-1}\\ k{m}_{i+1,j,k-1}-i{m}_{i-1j,k+1}\\ i{m}_{i-1,j+1,k}-j{m}_{i+1,j-1,k}\end{array}\right]}^{\top }\omega .$$

(4)

In the following, a scheme is proposed to define triples ${\mathbf{x}}_m={\left[x\left(m_{ijk}\right)y\left(m_{ijk}\right)z\left(m_{ijk}\right)\right]}^{\top }\in {\mathbb{R}}^{3\times 1}$ that behave as a projected point onto the sphere with respect to rotations.

2.2. Closed-Form Solution of Rotation Estimation

From moments of order 1 and by using Equation (4), the interaction matrix that links time derivatives of the triple ${\mathbf{x}}_{m_1}={\left[m_{100}{m}_{010}{m}_{001}\right]}^{\top }$ to the rotational velocities is obtained by the following:

$${\dot{\mathbf{x}}}_{m_1}=\left[\begin{array}{ccc}0& -m_{001}& m_{010}\\ m_{001}& 0& -m_{100}\\ -m_{010}& m_{100}& 0\end{array}\right]\mathit{\omega }={\left[{\mathbf{x}}_{m_1}\right]}_{\times }\mathit{\omega }.$$

(5)

As it can be seen, the interaction matrix given by Equation (5) has an identical form to the one obtained for a projected point onto the sphere. This implies a 3D rotation results in the same motion on ${\mathbf{x}}_{m_1}$. Let us now consider triples ${\mathbf{x}}_m$ defined from moments of orders larger than 1. In order to have ${\mathbf{x}}_m^{\prime }=R{x}_m$, the interaction matrix related to ${\mathbf{x}}_m={[x\left(m_{ijk}\right),y\left(m_{ijk}\right),z\left(m_{ijk}\right)]}^{\top }$ must have the same form as Equation (3), which means the following:

$$\begin{array}{cc}\hfill {\dot{\mathbf{x}}}_m& =\left[\begin{array}{ccc}0& -z\left(m_{ijk}\right)& y\left(m_{ijk}\right)\\ z\left(m_{ijk}\right)& 0& -x\left(m_{ijk}\right)\\ -y\left(m_{ijk}\right)& x\left(m_{ijk}\right)& 0\end{array}\right]\mathit{\omega }=-{\left[\mathit{\omega }\right]}_{\times }{\mathbf{x}}_m\hfill \end{array}$$

(6)

Under the assumption of constant speed (i.e., ${\left[\mathit{\omega }\right]}_{\times }$ is a constant matrix), solving for Equation (6) leads to the following:

$${\mathbf{x}}_m\left(t\right)=exp(-{\left[\mathit{\omega }\right]}_{\times }t){\mathbf{x}}_m\left(0\right)=\mathbf{R}{\mathbf{x}}_m\left(0\right)$$

(7)

where ${\mathbf{x}}_m\left(0\right)$ and ${\mathbf{x}}_m\left(t\right)$ are respectively the initial value and at the value time t of the triple (which defines the relation for a couple of images). Any rotational motion between two images can be obtained by applying a constant speed during time t. This means Equation (6) implies ${\mathbf{x}}_m\left(t\right)=\mathbf{R}{\mathbf{x}}_m\left(0\right)$ for any couple of images.

Let us now explain the procedure to obtain those triples using as an example: the moment vector ${\mathbf{w}}_{23}={\left[m_{300}{m}_{200}{m}_{300}{m}_{110}\cdots m_{003}{m}_{002}\right]}^{\top }$. The latter is composed of all the possible products between moments of order 2 and those of order 3. There are 6 possible moments of order equal to 2 and 10 of order equal to 3. This implies that ${\mathbf{w}}_{23}$ is of size equal to 60. Let us define triples ${\mathbf{x}}_{{\mathbf{w}}_{23}}={\left[{\mathit{\alpha }}_x^{\top }{\mathbf{w}}_{23}{\mathit{\alpha }}_y^{\top }{\mathbf{w}}_{23}{\mathit{\alpha }}_z^{\top }{\mathbf{w}}_{23}\right]}^{\top }$, where ${\mathit{\alpha }}_{\mathbf{x}}$, ${\mathit{\alpha }}_{\mathbf{y}}$, and ${\mathit{\alpha }}_{\mathbf{z}}$ are vectors of coefficients that define the triple coordinates as linear combinations of the entries of ${\mathbf{w}}_{23}$. Our goal is to find the conditions on those vectors of coefficients such that the time derivative ${\dot{\mathbf{x}}}_{{\mathbf{w}}_{\mathbf{23}}}$ is of the same form as Equation (6).

As an example, let us consider the time derivative of the moment product $m_{030}{m}_{200}$, which is a component of ${\mathbf{w}}_{23}$. Using Equation (4), we have the following:

$$\left\{\begin{array}{c}{\dot{m}}_{200}=0\times {\omega }_x-2m_{101}{\omega }_y+2m_{110}{\omega }_z\hfill \\ {\dot{m}}_{030}=3m_{021}{\omega }_x+0\times {\omega }_y-3m_{120}{\omega }_z\hfill \end{array}\right..$$

(8)

As it can be also concluded from Equation (4), the time variation of a moment is a function of moments of the same order and the rotational speed. From Equation (8), we obtain the time derivative of the product $m_{030}{m}_{200}$ by the following:

$$\begin{array}{cc}\hfill & \frac{d\left(m_{030}{m}_{200}\right)}{dt}=m_{200}{\dot{m}}_{030}+m_{030}{\dot{m}}_{200}\hfill \\ \hfill & =3m_{200}{m}_{021}{\omega }_x-2m_{101}{m}_{030}{\omega }_y+(2m_{110}{m}_{030}-3m_{200}{m}_{120}){\omega }_z.\hfill \end{array}$$

(9)

Note that the coefficients of ${\omega }_x$, ${\omega }_y$, and ${\omega }_z$ are nothing but linear combinations of products between moments of orders 2 and 3. Actually, the time derivative of any component of the vector ${\mathbf{w}}_{23}$ will be a linear combination of the other components. Based on this fact, ${\dot{\mathbf{w}}}_{23}$ can be written under the following form:

$${\dot{\mathbf{w}}}_{23}=\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}\right){\omega }_x+\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}\right){\omega }_y+\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}\right){\omega }_z,$$

(10)

where ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}$, ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}$ and ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}$ are $60\times 60$ matrices. Using Equation (10), the time derivative of the triple ${\mathbf{x}}_{{\mathbf{w}}_{23}}={\left[{\mathit{\alpha }}_x^{\top }{\mathbf{w}}_{23}{\mathit{\alpha }}_y^{\top }{\mathbf{w}}_{23}{\mathit{\alpha }}_z^{\top }{\mathbf{w}}_{23}\right]}^{\top }$ is obtained:

$${\dot{\mathbf{x}}}_{{\mathbf{w}}_{\mathbf{23}}}=\left[\begin{array}{ccc}{\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}& {\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}& {\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}\\ {\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}& {\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}& {\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}\\ {\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}& {\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}& {\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}\end{array}\right]\omega .$$

(11)

If Equation (11) is of the same form as Equation (6), this leads to the 9 following conditions:

$$\begin{array}{cc}\begin{array}{cc}\hfill & 1){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}=0;\hfill \\ \hfill & 2){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}=-{\mathit{\alpha }}_z^{\top }{\mathbf{w}}_{23};\hfill \\ \hfill & 3){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_{\mathbf{z}}}{\mathbf{w}}_{23}={\mathit{\alpha }}_y^{\top }{\mathbf{w}}_{23};\hfill \\ \hfill & 4){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}={\mathit{\alpha }}_z^{\top }{\mathbf{w}}_{23};\hfill \\ & 5){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}=0;\hfill \end{array}& \begin{array}{cc}\hfill & 6){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}=-{\mathit{\alpha }}_x^{\top }{\mathbf{w}}_{23};\hfill \\ \hfill & 7){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}{\mathbf{w}}_{23}=-{\mathit{\alpha }}_y^{\top }{\mathbf{w}}_{23};\hfill \\ \hfill & 8){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}{\mathbf{w}}_{23}={\mathit{\alpha }}_x^{\top }{\mathbf{w}}_{23};\hfill \\ \hfill & 9){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}{\mathbf{w}}_{23}=0.\hfill \end{array}\end{array}$$

(12)

The 9 constraints given by Equation (12) must be valid for any value of ${\mathbf{w}}_{23}$. Therefore, they can be simplified as follows:

$$\begin{array}{ccc}\begin{array}{cc}\hfill & 1){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}=\mathbf{0}\hfill \\ \hfill & 2){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}=-{\mathit{\alpha }}_z^{\top }\hfill \\ \hfill & 3){\mathit{\alpha }}_x^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_{\mathbf{z}}}={\mathit{\alpha }}_y^{\top }\hfill \end{array}& \begin{array}{cc}\hfill & 4){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}={\mathit{\alpha }}_z^{\top }\hfill \\ \hfill & 5){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}=\mathbf{0}\hfill \\ \hfill & 6){\mathit{\alpha }}_y^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}=-{\mathit{\alpha }}_x^{\top }\hfill \end{array}& \begin{array}{cc}\hfill & 7){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}=-{\mathit{\alpha }}_y^{\top }\hfill \\ \hfill & 8){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}={\mathit{\alpha }}_x^{\top }\hfill \\ \hfill & 9){\mathit{\alpha }}_z^{\top }{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}=\mathbf{0}.\hfill \end{array}\end{array}$$

(13)

The constraint set of Equation (13) can be written under the following form:

$$\left[\begin{array}{ccc}{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}^{\top }& \mathbf{0}& \mathbf{0}\\ {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}^{\top }& \mathbf{0}& \mathbf{I}\\ {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}^{\top }& -\mathbf{I}& \mathbf{0}\\ ⋮& ⋮& ⋮\\ \mathbf{0}& \mathbf{0}& {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}^{\top }\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_{\mathbf{x}}\\ {\mathit{\alpha }}_{\mathbf{y}}\\ {\mathit{\alpha }}_{\mathbf{z}}\end{array}\right]=\mathbf{0},$$

(14)

where $\mathbf{I}$ and $\mathbf{0}$ are respectively the identity and zero matrices of size $60\times 60$. In practice, ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}$, ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}$ and ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}$ are very sparse integer matrices. Solving Equation (14) for the case of the moment vector ${\mathbf{w}}_{\mathbf{23}}$ leads to three solutions for the triple ${\mathbf{x}}_{{\mathbf{w}}_{\mathbf{23}}}$ (refer to Equations (A1)–(A3) in the Appendix A). Such conditions can be solved using a symbolic computation software as Matlab Symbolic Toolbox.

The procedure used to obtain triples from ${\mathbf{w}}_{\mathbf{23}}$ can be extended straightforwardly to a different moment vector as follows:

• Define the moment vector $\mathbf{w}$ similarly as for ${\mathbf{w}}_{\mathbf{23}}$. The vector can be built by moment products of two different orders or more. For instance, products such as $m_{200}{m}_{220}{m}_{300}$ which combine moments of order 2, 3, and 4 can be also used. the vector $\mathbf{w}$ has to be built from moment products of the same nature and has to also include all of them.
• Using Equation (4), compute the matrices ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}$, ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}$, and ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}$ such that we obtain the following:

  $$\dot{\mathbf{w}}=\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}\mathbf{w}\right){\omega }_x+\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}\mathbf{w}\right){\omega }_y+\left({\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}\mathbf{w}\right){\omega }_z.$$

  (15)
• Solve the following system:

  $$\left[\begin{array}{ccc}{\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}^{\top }& \mathbf{0}& \mathbf{0}\\ {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}^{\top }& \mathbf{0}& \mathbf{I}\\ {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}^{\top }& -\mathbf{I}& \mathbf{0}\\ ⋮& ⋮& ⋮\\ \mathbf{0}& \mathbf{0}& {\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}^{\top }\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_{\mathbf{x}}\\ {\mathit{\alpha }}_{\mathbf{y}}\\ {\mathit{\alpha }}_{\mathbf{z}}\end{array}\right]=\mathbf{0},$$

  (16)

  where $\mathbf{I}$ and $\mathbf{0}$ are respectively the identity and zero matrices of the same size as ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_x}$, ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_y}$ and ${\mathbf{L}}_{{\mathbf{w}}_{\mathbf{23}}/{\omega }_z}$.

In practice, two different triples ${\mathbf{x}}_{\mathbf{w}}$ are enough to obtain a closed-form solution of the rotational motion since they can define an orthonormal basis. Let us consider, for instance, the two triples ${\mathbf{P}}_{\mathbf{1}}$ and ${\mathbf{P}}_{\mathbf{2}}$ computed from the moments of a first image using Equations (A1) and (A2) and ${\mathbf{P}}_{\mathbf{1}}^{\prime }$ and ${\mathbf{P}}_{\mathbf{2}}^{\prime }$, their corresponding ones, computed from the image after rotation. Therefore, we have ${\mathbf{P}}_{\mathbf{1}}^{\prime }=\mathbf{R}{\mathbf{P}}_{\mathbf{1}}$ and ${\mathbf{P}}_{\mathbf{2}}^{\prime }=\mathbf{R}{\mathbf{P}}_{\mathbf{2}}$, where $\mathbf{R}$ is the rotation matrix. Let us define the following base:

$${\mathbf{v}}_{\mathbf{1}}=\frac{{\mathbf{P}}_{\mathbf{n}\mathbf{1}}+{\mathbf{P}}_{\mathbf{n}\mathbf{2}}}{\Vert {\mathbf{P}}_{\mathbf{n}\mathbf{1}}+{\mathbf{P}}_{\mathbf{n}\mathbf{2}}\Vert },{\mathbf{v}}_{\mathbf{2}}=\frac{{\mathbf{P}}_{\mathbf{n}\mathbf{1}}-{\mathbf{P}}_{\mathbf{n}\mathbf{2}}}{\Vert {\mathbf{P}}_{\mathbf{n}\mathbf{1}}-{\mathbf{P}}_{\mathbf{n}\mathbf{2}}\Vert },{\mathbf{v}}_{\mathbf{3}}={\mathbf{v}}_{\mathbf{1}}\times {\mathbf{v}}_{\mathbf{2}}$$

(17)

where ${\mathbf{P}}_{\mathbf{n}\mathbf{1}}=\frac{{\mathbf{P}}_{\mathbf{1}}}{\Vert {\mathbf{P}}_{\mathbf{1}}\Vert }$ and ${\mathbf{P}}_{\mathbf{n}\mathbf{2}}=\frac{{\mathbf{P}}_{\mathbf{2}}}{\Vert {\mathbf{P}}_{\mathbf{2}}\Vert }$. First, it can be shown that the vectors ${\mathbf{v}}_{\mathbf{1}}$, ${\mathbf{v}}_{\mathbf{2}}$ and ${\mathbf{v}}_{\mathbf{3}}$ form an orthonormal base (i.e., $\Vert {\mathbf{v}}_{\mathbf{1}}\Vert =\Vert {\mathbf{v}}_{\mathbf{2}}\Vert =\Vert {\mathbf{v}}_{\mathbf{3}}\Vert =\mathbf{1}$ and ${\mathbf{v}}_{\mathbf{1}}.{\mathbf{v}}_{\mathbf{2}}={\mathbf{v}}_{\mathbf{1}}.{\mathbf{v}}_{\mathbf{3}}={\mathbf{v}}_{\mathbf{3}}.{\mathbf{v}}_{\mathbf{2}}=0$). Second, let us now consider a second base ${{\mathbf{v}}_{\mathbf{1}}}^{\prime }$, ${\mathbf{v}}_{\mathbf{2}}^{\prime }$ and ${\mathbf{v}}_{\mathbf{3}}^{\prime }$ computed from ${\mathbf{P}}_{\mathbf{1}}^{\prime }$ and ${\mathbf{P}}_{\mathbf{2}}^{\prime }$ using Equation (17). Since ${\mathbf{P}}_{\mathbf{1}}^{\prime }=\mathbf{R}{\mathbf{P}}_{\mathbf{1}}$ and ${\mathbf{P}}_{\mathbf{2}}^{\prime }=\mathbf{R}{\mathbf{P}}_{\mathbf{2}}$, it can be proven using Equation (17) that ${\mathbf{v}}_{\mathbf{1}}^{\prime }=\mathbf{R}{\mathbf{v}}_{\mathbf{1}}$, ${\mathbf{v}}_{\mathbf{2}}^{\prime }=\mathbf{R}{\mathbf{v}}_{\mathbf{2}}$ and ${\mathbf{v}}_{\mathbf{3}}^{\prime }=\mathbf{R}{\mathbf{v}}_{\mathbf{3}}$. The last equations can be written under the following form:

$$\mathbf{V}=\mathbf{R}{\mathbf{V}}^{\prime },$$

(18)

where $\mathbf{V}=\left[{\mathbf{v}}_{\mathbf{1}}{\mathbf{v}}_{\mathbf{2}}{\mathbf{v}}_{\mathbf{3}}\right]$ and ${\mathbf{V}}^{\prime }=\left[{\mathbf{v}}_{\mathbf{1}}^{\prime }{\mathbf{v}}_{\mathbf{2}}^{\prime }{\mathbf{v}}_{\mathbf{3}}^{\prime }\right]$. Finally, we obtain:

$$\mathbf{R}=\mathbf{V}{{\mathbf{V}}^{\prime }}^{\top },$$

(19)

where the rotation matrix can be estimated straightforwardly. Solving Equation (16) ensures that all possible triples are obtained from a given moment vector. Obtaining a mathematical proof on the existence and their number is out of the the scope of this contribution.

2.3. Rotation Estimation and Scene Symmetry

As explained in the previous paragraph, an orthonormal base can be built from the triples ${\mathbf{P}}_{\mathbf{1}}$ and ${\mathbf{P}}_{\mathbf{2}}$ to estimate the rotation matrix. In practice, if the scene has no symmetry, extensive tests (not included in the paper for brevity) show that using the triples ${\mathbf{P}}_{\mathbf{1}}$ and ${\mathbf{P}}_{\mathbf{2}}$ could be enough. If the scene has a symmetry with respect to a plane (for instance $xy$ plane), the z-coordinates of the triples ${\mathbf{P}}_{\mathbf{i}}$ are null. This means that all ${\mathbf{P}}_{\mathbf{i}}$ will belong to the $xy$ plane. If the scene is symmetrical with respect to $xy$ and $xz$ planes, for instance, then the triples ${\mathbf{P}}_{\mathbf{i}}$ will form a line in the direction of the x-axis.

3. Validation Results

In the following section, simulation results and real experiments are shown. For all the validation results, the spherical moments are computed directly using the pixel’s grey levels without warping the images to the sphere space. The used formulas are given in References [26,27] for the perspective and fisheye cases respectively:

$$m_{ijk}={\int \int }_{\mathbf{Image}}{x}_s^i{y}_s^j{z}_s^k\frac{{(\xi +z_s)}^3}{1+\xi z_z}I(x,y,t)dxdy,$$

(20)

where $\xi$ is the distortion parameter and $(x,y)$ are the coordinates of a point in the metric image. The formulas of $x_s$, $y_s$, and $z_s$ as functions of $(x,y)$ can be found in Reference [28]. Using Matlab with a computer equipped with a processor Intel(R) Core(TM) i5-7600 CPU @ 3.50 GHz 3.50 GHz and 16,0 Go RAM, the whole process including the computation of spherical moments of an image of size $480\times 640$ pixels until the rotation estimation takes around 20 ms. This leaves room for reducing the time cost using other programming tools than Matlab to few milliseconds.

3.1. Simulation Results

The objective of this part is threefold:

• To show the validity of the proposed method to cameras obeying unified model, two different kinds of camera model are used to compute the images. The first corresponds to a simulated fisheye camera with the parameters chosen as focal scaling factors $Fx=Fy=960$ pixels, coordinates of the principal point $u_x=240$ and $u_y=320$ pixels, and distortion parameter $\xi =1.6$. The second model corresponds a conventional camera with focal scaling factors $Fx=Fy=600$ pixels and coordinates of the principal point $u_x=240$ and $u_y=320$;
• To Validate our approach for large rotational motion;
• To test the effect of translational motion on the accuracy of the estimated rotations.

The Matlab code used for those simulations results can be downloaded online (http://aramis.iup.univ-evry.fr:8080/~hadj-abdelkader/Sensors/Sim_ForSensors.zip).

In the first results, a fisheye camera model is used to generate the images. More precisely, from the reference image shown on Figure 1a, three sets of 100 new images are computed based on randomly generated displacements. An example of the used images is shown on Figure 1b. All the three image sets were obtained with the same random rotations shown on Figure 2b but with translational motion of norms equal to 0 for the first set and 10 cm and 20 cm for the two others. For the two last sets, the same random translation directions are used but with different norms equal to 10 cm and 20 cm, respectively. To assess the estimated rotations, the norm of the vector composed by the errors on those Euler angles is used. The obtained accuracy results are shown on Figure 2a. It can be seen that, for the case where the camera displacement does not include translational motion, the rotation motion is well estimated (refer to the red plot of the Figure). The results obtained in the case where the camera displacement involves translational motion of norm 10 cm and 20 cm are shown respectively by the blue and the green plots of the Figure 2a. From the obtained plots, it can be seen that a maximum error of $4^{\circ }$ (respectively $8^{\circ }$) and an average error less then $3^{\circ }$ (respectively $5^{\circ }$) are obtained in the case of translation norm equal to 10 cm (respectively 20 cm).

Figure 1. Results using a fisheye camera model: (a) the reference image and (b) example of rotated images.

Figure 2. Simulation results using a fisheye camera model: (a) Estimation error of the rotation and (b) rotation ground truth expressed by Euler angles.

The second simulation results are based on the same previous setup, but this time, it is a conventional perspective camera that is used to generate the images. As for the first simulations, we consider three sets of 100 images corresponding to transnational motions of different norms (0 cm, 10 cm, and 20 cm). Figure 3a,b shows the reference image and an example of the used rotated image for which the rotation has to be estimated. The ground truth of the rotational motions is shown on Figure 4b, while the accuracy of the rotation estimation is depicted on Figure 4a. From this Figure, it can be seen that the rotations are well estimated in the case where pure rotational motions are considered (refer to the red plot). It can also be noticed from the same figure (the blue and the green plots) that maximum rotation errors around ${3.5}^{\circ }$ and ${7.5}^{\circ }$ are observed respectively for the cases where the camera displacement involves a translational motion of norms equal to 10 cm and 20 cm, respectively.

Figure 3. Results using conventional camera model: (a) the reference image and (b) example of the rotated images.

Figure 4. Simulation results using a perspective camera model: (a) Estimation error of the rotation and (b) rotation ground truth expressed.

3.2. Real Experiments

The following experimental results use the OVMIS (Omnidirectional Vision Dataset of the MIS laboratory) dataset (http://mis.u-picardie.fr/~g-caron/pub/data/OVMIS_dataset.zip). Based on the same database, the preliminary work [23] has provided several experimental results where the proposed method is compared to the SPHARM (SPherical HARmonic analyses) method [15] and to the phase correlation (COR) and photometric (PHO) methods [29]. In the following, the proposed method is validated on a larger set of scenarios from the dataset. Moreover, we compare the performance of the proposed method using the spherical moments computed from the image intensities and from its gradient. We also test the effect of introducing the weighting function on the spherical moment computation of image acquired by catadioptric camera. The used weighting is a function of the pixel coordinates $(x,y)$ of the following form:

$$W(x,y)=e^{-\frac{{(r-r_m)}^4}{{\sigma }^4}}$$

(21)

where $r=\sqrt{{(x-u_x)}^2+{(y-u_y)}^2}$ is the distance of the pixel to the principal point $(u_x,u_y)$ and $\sigma$ is a scalar that tunes the wide of the image zone to be used. Note that $W(x,y)$ has its maximum value 1 for $r=r_m$. Figure 5a shows the shape of the used weight function for this validation results. As can be seen from weighting function shape, the points close to the image border or to the blind zone in the mage center are given small weights to decrease the effect of appearance/disappearance of image parts. In all these experimental parts, the rotation motion is obtained by accumulating the estimated rotations between each two consecutive images.

Figure 5. (a)Weight function and (b) Scenario 3: Eight trajectories of the pioneer robot.

The first validation result shown in the video (Supplementary Materials) is based on Dataset 1 of OVMIS with the Staubli robot arm. The considered sequence composed of 144 images has been acquired by applying regular pure rotations around the optical axis by a catadioptric camera mounted on the end effector of a Staubli in an indoor environment which corresponds to a rotation of ${2.5}^{\circ }$ between two consecutive images. First, the proposed method is tested by considering all the image sequences. Then, one image from 10 and from 20 are considered, which corresponds to rotations of ${25}^{\circ }$ and ${50}^{\circ }$, respectively, between tow consecutive images. As it can be seen from the video that the obtained results show that the proposed method estimates accurately the rotations in the three considered cases.

The second validation result shown as well in the video (Supplementary Materials) is based on the Scenario 1 of the Pioneer database acquired using a catadioptric camera. A similar setup to the previous case is used: a pure rotational motion around the z-axis in an indoor environment. The proposed method is tested by considering all the image sequences, one image from 10 and one from 20. The obtained results show again that the proposed method estimates accurately the rotations in the three considered cases.

The last experiment tests the proposed method on the more challenging Scenario 3 of the OVMIS dataset of a pioneer robot moving according to the trajectory described in Figure 5b. The achieved path is of length 25.27 m, in the form of 8 motions that include both rotational and translation motions. It has been achieved by the mobile robot in 160 s, which corresponds to 0.187 m/s average speed. The ground truth is obtained by incorporating gyroscopic corrections into the odometry data based on wheel encoders. From the video (Supplementary Materials), it can be seen that the mobile robot has traversed uneven terrain, with changing illumination conditions due to moving cast shadows of the trees and with appearance/disappearance of some image parts.

Figure 6 compares the estimated rotation with/without using weighting function and based on moments computed from the image grey level or its gradient. From the obtained plots, it can be seen that using weighting function improves slightly the results obtained using the moment computed from the image intensities (compare the blue curve to the red one). On the other hand, it can be seen that the most accurate rotation estimation is obtained using the image gradient spherical moments and weighting function (refer to the green plot). The last experiments of the video (Supplementary Materials) describes the obtained results using the image gradient spherical moments by considering all the image sequences, one image from 10 and one from 20. From the obtained results, it can be seen that the rotation estimation is satisfactory for camera motion involving transnational and rotational motions.

Figure 6. Estimated rotation of the pioneer robot around the Z-axis (Scenario 3).

3.3. Discussion

The triples computed from moment vectors provide a closed-form estimation to rotational motion. Compared to the classical method based on the alignment of principal axis, the proposed solution has no ambiguities on rotation formula. Recall that the alignment of principal axis method usually uses moments of higher orders to remove the ambiguities. If the estimated rotation is used as a feature in visual servoing or in active vision, the closed-form formula has a second advantage. For instance, as it has been proposed in References [30,31], the rotation vector is computed from the rotation matrix and used to control the rotational speed of a camera. For accurate visual servoing and active vision, most strategies are based on the analytical form of the visual features dynamics. Independently from the used method to estimate the rotation matrix, the relation between the time variation of a rotation vector and the rotational speed keeps the same form. On the other hand, its dynamic with respect to the translational speed becomes complex to obtain without a closed-form formula. Using the proposed solution in this paper, the dynamics of the rotation vector with respect to the whole degree can be obtained from those of the spherical moments [26,30,31].

A natural question arises about which are the best two triples to be used for building an orthonormal base. For instance, a base can be built from the two triples from ${\mathbf{P}}_{\mathbf{1}}$ to ${\mathbf{P}}_{\mathbf{3}}$ as well as by using two others from ${\mathbf{P}}_{\mathbf{4}}$ to ${\mathbf{P}}_{\mathbf{7}}$. Although it is well accepted that the sensitivity to noise increases with the moment orders, there is not much difference between results obtained using moments of orders 2 and 3 and those obtained using moments of orders 4 and 3. Moreover, an orthonormal base can be obtained using more than only two triples as it is achieved in Reference [31]. Finally, recall also that the rotation matrix can be obtained using all the triples ${\mathbf{P}}_{\mathbf{i}}$ as a generalized solution of the orthogonal Procrustes problem [32].

4. Conclusions

In this paper, a new method to estimate rotational motions using photometric moments has been provided. The obtained theoretical results have been validated using simulations and real experiments using onboard omnidirectional cameras and two different robots (robot arm and ground mobile robot). The proposed method has been also tested in different scenarios with appearance/disappearance of some image parts and lightening changes. Future works will be devoted to using photometric spherical moments for the control of the 6 degrees of freedom of a camera.