FSD-BRIEF: A Distorted BRIEF Descriptor for Fisheye Image Based on Spherical Perspective Model

Abstract

Fisheye images with a far larger Field of View (FOV) have severe radial distortion, with the result that the associated image feature matching process cannot achieve the best performance if the traditional feature descriptors are used. To address this challenge, this paper reports a novel distorted Binary Robust Independent Elementary Feature (BRIEF) descriptor for fisheye images based on a spherical perspective model. Firstly, the 3D gray centroid of feature points is designed, and the position and direction of the feature points on the spherical image are described by a constructed feature point attitude matrix. Then, based on the attitude matrix of feature points, the coordinate mapping relationship between the BRIEF descriptor template and the fisheye image is established to realize the computation associated with the distorted BRIEF descriptor. Four experiments are provided to test and verify the invariance and matching performance of the proposed descriptor for a fisheye image. The experimental results show that the proposed descriptor works well for distortion invariance and can significantly improve the matching performance in fisheye images.

1. Introduction

For decades, feature detection and matching is one of the core areas of image processing in various applied fields, such as Visual based Simultaneously Localization and Mapping (V-SLAM), Structure from Motion (SfM), Augmented Reality (AR), general image retrieval, image mosaic, and image registration. Common features include Scale Invariant Feature Transform (SIFT) [1], Speed Up Robust Feature (SURF) [2], BRIEF [3] Oriented FAST and Rotated BRIEF (ORB) [4], KAZE [5], Binary Robust Invariant Scalable Keypoints (BRISK) [6], etc. and their derivations, such as Principle Component Analysis SIFT (PCA-SIFT) [7], Simplified-SIFT (SSIFT) [8], and Accelerated-KAZE (AKAZE) [9]. Neural network based features are also developed, such as L2-NET [10], HardNet [11], and AffNet [12]. These features are designed for pinhole images with little distortion and cannot achieve good performances for fisheye images with severe radial distortion.

Compared with a pinhole camera, a fisheye camera has a wide field of view (FoV), and the captured image contains more abundant information. This makes the fisheye camera extensively adopted in robot navigation, visual monitoring, virtual reality, visual measurement, and 3D reconstruction. However, due to the severe radial distortion of the fisheye image, adopting the common feature descriptors directly may lead to a significant reduction in matching performance.

In order to reduce the impact of distortion on the feature matching performance, we propose a novel distorted BRIEF descriptor based on the spherical perspective model, named Fisheye Spherical Distorted BRIEF (FSD-BRIEF). Firstly, we propose a method based on 3D gray centroid to determine the direction of each feature point in the spherical image. By constructing an attitude matrix of a feature point, the position and direction of the feature point in the spherical image can be described in a nonsingular form. In order to reduce the calculation error of the 3D gray centroid caused by uneven distribution of pixels in the spherical image, a pixel density function is designed to represent the degree of pixel density on the spherical surface by the size of the patch area mapped by each pixel in the fisheye image. We build an attitude coordinate system of each feature point and propose a coordinate mapping method to project the BRIEF descriptor template on the fisheye image. The distortion form of the projected BRIEF template is consistent with the image distortion near the feature point, which prevents the calculated BRIEF descriptor from the affection of the radial distortion in fisheye image. The main contributions of the paper include:

• A new pixel density function represented by the area of the spherical surface patch that each pixel of fisheye image occupies;
• A new method of determining the 3D gray centroid and the direction of feature points with pixel density function based on a spherical perspective model;
• A new feature point attitude matrix, providing a nonsingular description for both the position and the direction of the feature point in the spherical image surface;
• A novel descriptor template distortion method based on the spherical perspective model and the feature point attitude matrix.

The remaining of the paper is arranged as follows. In Section 2, the related work of the fisheye image point feature is presented. In Section 3, the notation of the perspective model is briefly introduced. Section 4 is about the method of determining and expressing the direction of feature point. Then the method of calculating the FSD-BRIEF descriptor is described. In Section 5, experimental results are provided and the performance of the proposed FSD-BRIEF is tested and verified. Section 6 briefly summarizes the work. In Section 7, the future work is stated.

2. Related Work

By virtue of its front lens protruding in a parabola shape, fisheye camera has a large FoV whose angle of view is close to or even more than 180°. Although this characteristic can maximize the angle of view, it brings severe radial distortion in its captured image, leading to different scale factors for pixels in different positions of the image. Thus, it could make the traditional feature descriptors designed for plane image fail to match in raw fisheye images [13,14].

Generally, the methods to extract descriptors in fisheye images can be divided into two main streams according to whether images are corrected or not: resampling and non-resampling approaches.

Resampling approaches [15,16,17] segment the FoV image into several sub-FoVs and correct them based on a plane perspective model, then feature descriptors can be extracted and matched on the corrected sub-FoV. Lin et al. [15] adopted a visual-inertial based UAV (Unmanned Aerial Vehicle) navigation system, where two sub-regions are sampled in the horizontal direction of the fisheye FoV to obtain two undistorted pinhole image fields, which cover 180° horizontal FoV, but they discarded the upper and lower parts of the vertical FoV. Miiller et al. [16] presented a robust visual inertial odometry and time-efficient omni-directional 3D mapping system, where the FoV of each fisheye camera is divided into two piecewise pinhole fields so as to overcome the distortion. However, some parts near the edge of the FoV are wasted. Wang et al. [17] proposed a new real-time feature-based simultaneous localization and mapping system, where a fisheye image is projected onto five surfaces of a cube, and then descriptors are extracted on the unfolded surfaces of the cube. However, the stretching distortion and seam distortion exist between surfaces, for example, a straight line will become a broken line. Thus, in the resampling approaches, the whole FoV of the fisheye image is hard to be fully utilized, and the continuity between sub-FoV cannot be guaranteed. In addition, due to the view geometry of the plane perspective model, there is a small stretching distortion in the edge of the sub-FoV.

Unlike the resampling approaches, which directly correct fisheye images to pinhole images, a non-resampling approach uses descriptors to describe features in fisheye images. For example, inspired by the planar SIFT framework [18,19,20], Arican et al. [21] designed a new scale invariant omni-directional SIFT feature based on Riemannian geometry. Lourenco et al. [22] proposed a Spherical Radial Distortion SIFT (sRD-SIFT) feature, where the extraction of the feature and the calculation of the descriptor was designed based on the spherical perspective model and the raw fisheye image without resampling. However, the improved algorithms based on SIFT are generally long time-consuming. Cruz-Mota et al. [23] and Hansen et al. [24] utilized spherical harmonic function as the basic function to study the spectral analysis of spherical panoramic images. Since Gaussian filtering on the sphere can be realized as a diffusion process through the spherical Fourier transform, spherical harmonic function is used to construct scale space on the sphere. In theory, the spherical harmonic function can be used to maintain the invariance of the descriptors to encounter the changes of the camera poses and positions. However, the spherical harmonic function usually needs a large amount of computation and has inherent bandwidth limitation. This greatly weakens the capability of dealing with large-scale matching problems and cannot meet the real-time requirements of many applications.

For improving the calculation speed, Qiang et al. [25] proposed Spherical ORB (SPHORB), a binary spherical feature based on the ORB feature, which is the first binary descriptor for a panoramic image based on hexagon geodesic grid. In essence, SPHORB is still a special resampling approach, which divides the spherical panoramic image into 20 regular triangle fields according to the shape of a regular icosahedron, and aligns the pixel of adjacent regular triangles seamlessly. However, in the hexagon geodesic grid, the image patches near the 12 vertices of the regular icosahedron are discarded due to the distortion of the pixel distribution pattern, resulting in 12 FoV holes occupying 1.4% of the total FoV.

Note that it can result in holes when resampling the fisheye image based on hexagon geodesic grid. To avoid this, Urban et al. [26] proposed a new distorted descriptor, called Masked Distorted BRIEF (mdBRIEF). Although this work distorts the descriptors to adapt to different image regions instead of correcting the distortion of the fisheye image, the direction angle of feature points is obtained in the raw fisheye image by calculating the gray centroid in a circle template, which is still affected by the fisheye image distortion. Furthermore, the descriptors are distorted excessively near the edge of the fisheye image since it is distorted based on the plane perspective model.

Most recently, Pourian et al. [27] proposed an end-to-end framework to enhance the precision of the descriptor matching between multiple wide-angle images. In their work, the global matching and the local matching of descriptors are combined in three stages. However, a new distortion in the edge of the corrected image is introduced when an equal rectangle image transformation is employed in the global matching stage, lowering the performance of the framework.

In summary, the binary descriptor that can make use of the whole FoV and keep invariance in each position of the fisheye image has not been proposed. In order to avoid the FoV holes caused by the resampling approaches, and reduce the excessive distortion of descriptors in large FoV images, in this paper, we design a novel Fisheye Spherical Distorted BRIEF (FSD-BRIEF) descriptor, which is a distorted binary feature descriptor based on the spherical perspective model for fisheye images.

3. Fisheye Camera Model

In this paper, in order to ensure that the FoV of the fisheye image can be fully utilized without losing the performance of the feature descriptor, a new descriptor FSD-BRIEF is designed based on spherical perspective model. Different from the plane perspective model, the projection surface is a unit sphere with the origin of camera coordinate system as center, so as to ensure that the scale factors of each position on the projection surface are consistent. The spherical perspective model and its perspective projection relationship are shown in Figure 1. We define the camera coordinate system as $O_c{X}_c{Y}_c{Z}_c$. The origin point $O_c$ is located at the optical center of the camera, the X-axis $O_c{X}_c$ points to the right along the long side of the imaging target surface, the Y-axis $O_c{Y}_c$ points downward along the wide edge direction of the imaging target surface, and the Z-axis $O_c{Z}_c$ points to the front of the camera along the optical axis direction. $P^{\prime }$ is the projection point on the spherical image surface of the space point P. $L^{\prime }$ is the projection large arc on the spherical image surface of the space Line L.

For a point P in a three-dimensional space, define its space coordinate in camera coordinate system as:

$${\mathbf{P}}_c={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^T$$

(1)

The projection point of P in the fisheye image is $\mathbf{p}$, and its pixel coordinates are expressed as follows:

$$\mathbf{p}={\left[\begin{array}{cc}u& v\end{array}\right]}^T$$

(2)

In this paper, Kannala-Brandt4 (KB4) [28] model is used as the fisheye camera model, its mathematical form is shown below:

$$\begin{array}{c}\theta =\arctan 2(\sqrt{x^2+y^2},z)\hfill \\ \phi =\arctan 2(y,x)\hfill \\ {\theta }_d=\theta (1+k_1{\theta }^2+k_2{\theta }^4+k_3{\theta }^6+k_4{\theta }^8)\hfill \\ u=f_x{\theta }_d\cos \phi +c_x\hfill \\ v=f_y{\theta }_d\sin \phi +c_y\hfill \end{array}$$

(3)

where $f_x$ and $f_y$ are the horizontal and vertical focal length of the camera, $c_x$ and $c_y$ are the coordinates of the principal points of the camera, and $k_1$, $k_2$, $k_3$, $k_4$ are the distortion coefficients. $\theta$ is the FoV latitude angle, which represents the angle between the $O_c{Z}_c$ axis and the vector $\overrightarrow{O_{c}P}$. $\phi$ is the FoV longitude angle, which denotes the angle between the $O_c{X}_c$ axis and the projection vector of $\overrightarrow{O_{c}P}$ on the $X_c{O}_c{Y}_c$ plane. ${\theta }_d$ is the angle $\theta$ as deflected by the fisheye lens. The arctan2 is the quadrant aware version of arctangent function.

Based on the spherical perspective model in Equation (3), $\mathsf{\Pi }$ represents the mapping function. The mapping from the point ${\mathbf{P}}_c$ to the pixel point $\mathbf{p}$ in fisheye image can be expressed as:

$$\mathbf{p}=\mathsf{\Pi }\left({\mathbf{P}}_c\right)$$

(4)

The inverse mapping function of $\mathsf{\Pi }$ is defined as ${\mathsf{\Pi }}^{-1}$, which indicates the mapping from the point $\mathbf{p}$ to the point $P^{\prime }$ on the spherical image surface as follows:

$${{\mathbf{P}}_c}^{\prime }={\mathsf{\Pi }}^{-1}\left(\mathbf{p}\right)$$

(5)

where ${\mathbf{P}}_c^{\prime }$ is the coordinate vector of point $P^{\prime }$ in the camera coordinate system. Notice that $\left|{{\mathbf{P}}_c}^{\prime }\right|=\sqrt{x^2+y^2+z^2}=1$.

4. FSD-BRIEF Descriptor

The procedure of extracting the FSD-BRIEF descriptor includes four steps, namely, pixel density function designing, 3D gray centroid calculation, feature point attitude matrix construction, and FSD-BRIEF descriptor extraction. In the spherical perspective model, the densities of pixels are distributed unevenly, lowering the effectiveness of descriptors. Thus, a pixel density function is proposed firstly to calculate the distribution compensation of each pixel so as to reduce the effect of uneven pixel distribution. Then, with the help of the pixel density function, a more accurate 3D gray centroid is designed to determine the direction of FSD-BRIEF descriptor and keep its rotation invariance in the spherical perspective model. Next, we further devise a nonsingular form, a feature point attitude matrix, to represent the position and the direction of a feature point. Finally, based on the feature point attitude matrix, an FSD-BRIEF descriptor is extracted by a constructed coordinate mapping relation between the BRIEF template and the raw fisheye image.

4.1. Pixel Density Function Designing

In this section, by defining the pixel density function, the distribution density of pixels on the unit sphere surface is expressed numerically.

Assuming that a pixel $\mathbf{p}$ in a fisheye image occupies a small patch $PIX\_PATCH\left(\mathbf{p}\right)$ of the corresponding unit sphere, the mathematical expression of $PIX\_PATCH\left(\mathbf{p}\right)$ is given by:

$$PIX\_PATCH\left(\mathbf{p}\right)=\left\{\left.{\mathsf{\Pi }}^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{p})\right|\mathbf{\Delta }\mathbf{p}={\left[\begin{array}{cc}\Delta u& \Delta v\end{array}\right]}^T,-\frac{1}{2}<\Delta u<\frac{1}{2},-\frac{1}{2}<\Delta v<\frac{1}{2}\right\}$$

(6)

where $\Delta u$ and $\Delta v$ are the coordinate offsets under the pixel coordinate system in the fisheye image. It is obvious that the area of the patch $PIX\_PATCH\left(\mathbf{p}\right)$ will be smaller if the distance between point $\mathbf{p}$ and its adjacent pixels is closer, which means that the pixel density of point $\mathbf{p}$ is denser.

Therefore, the pixel density function $m\left(\mathbf{p}\right)$ is defined as the area of the patch $PIX\_PATCH\left(\mathbf{p}\right)$. To simplify the computation of the curved surface area, we assume that the patch size is small enough to approximate as a parallelogram, so the pixel density function compensation $m\left(\mathbf{p}\right)$ can be computed by:

$$m\left(\mathbf{p}\right)=\left\{\begin{array}{c}\frac{1}{4}{∥\left[{\mathsf{\Pi }}^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{x})-{\mathsf{\Pi }}^{-1}(\mathbf{p}-\mathbf{\Delta }\mathbf{x})\right]\times \left[{\mathsf{\Pi }}^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{y})-{\mathsf{\Pi }}^{-1}(\mathbf{p}-\mathbf{\Delta }\mathbf{y})\right]∥}_2,\mathbf{p}\in I\hfill \\ 0,\mathbf{p}\notin I\hfill \end{array}\right.$$

(7)

where ${∥{\displaystyle \blacksquare }∥}_2$ means L2 norm operation, and $\mathbf{\Delta }\mathbf{x},\mathbf{\Delta }\mathbf{y}$ are the coordinate offsets as follows:

$$\begin{array}{c}\hfill \mathbf{\Delta }\mathbf{x}={\left[\begin{array}{cc}1& 0\end{array}\right]}^T\\ \hfill \mathbf{\Delta }\mathbf{y}={\left[\begin{array}{cc}0& 1\end{array}\right]}^T\end{array}$$

(8)

From Equation (7), the pixel density function $m\left(\mathbf{p}\right)$ of the whole FoV only depends on the mapping function $\mathsf{\Pi }$ of the spherical perspective model in Equation (4).

4.2. 3D Gray Centroid Calculation

To determine the direction of the FSD-BRIEF descriptor, we propose a 3D gray centroid. Compared with 2D gray centroid [13,14,26], the proposed 3D gray centroid is more accurate since it takes full advantage of the consistent scale factor on the spherical perspective model. The 3D gray centroid is calculated in a circle area on the unit spherical surface. Figure 2 illustrates the correspondence of the circle area between the unit spherical surface and the fisheye image plane. As shown in Figure 2, for a FAST (Features From Accelerated Segment Test) [29] feature point $\mathbf{p}$, its projection point on the unit spherical surface is $P^{\prime }$, and its 3D gray centroid calculation area is the circle area $PATCH\_3D\left(P^{\prime }\right)$ with $P^{\prime }$ as the center. $PATCH\left(\mathbf{p}\right)$ is the projection of the $PATCH\_3D\left(P^{\prime }\right)$ in the fisheye image plane $O_P{X}_P{Y}_P$. $\alpha$ is half of the apex angle of the cone formed by $PATCH\_3D\left(P^{\prime }\right)$ and the origin point $O_c$.

Note that the horizontal and vertical angular resolutions of fisheye cameras are approximately $f_x$ and $f_y$ (Pixels Per Radian) in KB4 model, and the values of $f_x$ and $f_y$ are often very close. In order to make the radius of the circular range cover about 15 pixel width while ensure the same mathematical status of $f_x$ and $f_y$, the value of $\alpha$ in radians is selected as 15 divided by the arithmetic mean of $f_x$ and $f_y$, that is,

$$\alpha =\frac{15}{\frac{f_x+f_y}{2}}=\frac{30}{f_x+f_y}$$

(9)

Define the projection area $PATCH\left(\mathbf{p}\right)$ as:

$$PATCH\left(\mathbf{p}\right)=\left\{\left.\left(\mathbf{p}+\mathbf{\Delta }\mathbf{p}\right)\right|{\Pi }^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{p})·{\Pi }^{-1}\left(\mathbf{p}\right)>\cos \alpha \right\}$$

(10)

where $\mathbf{\Delta }\mathbf{p}$ is the offset from the pixel $\mathbf{p}$ to the pixel in the area $PATCH\left(\mathbf{p}\right)$ in the fisheye image plane. ${\Pi }^{-1}\left(\mathbf{p}\right)$ is the position vector of $P^{\prime }$. $P^{\prime }$ is also the projection point of the pixel $\mathbf{p}$ on the unit sphere. ${\Pi }^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{p})$ represents the position vector of the projection point of the pixel $\mathbf{p}+\mathbf{\Delta }\mathbf{p}$ on the unit sphere. ${\Pi }^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{p})·{\Pi }^{-1}\left(\mathbf{p}\right)>\cos \alpha$ means that the angle between the two vectors ${\Pi }^{-1}(\mathbf{p}+\mathbf{\Delta }\mathbf{p})$ and ${\Pi }^{-1}\left(\mathbf{p}\right)$ is less than $\alpha$. The region $PATCH\left(\mathbf{p}\right)$ is actually the projection area of the region $PATCH\_3D\left(P^{\prime }\right)$ on the fisheye image.

The 3D gray centroid of the feature point $\mathbf{p}$ is defined as C. The symbol ${\mathbf{C}}_c$ denotes the coordinate vector of C in the camera coordinate system. The calculation formula of ${\mathbf{C}}_c$ is:

$${\mathbf{C}}_c=\frac{{\displaystyle \sum _{\stackrel{⌢}{\mathbf{p}}\in PATCH\left(\mathbf{p}\right)}}{\mathsf{\Pi }}^{-1}\left(\stackrel{⌢}{\mathbf{p}}\right)m\left(\stackrel{⌢}{\mathbf{p}}\right)I\left(\stackrel{⌢}{\mathbf{p}}\right)}{{\displaystyle \sum _{\stackrel{⌢}{\mathbf{p}}\in PATCH\left(\mathbf{p}\right)}}m\left(\stackrel{⌢}{\mathbf{p}}\right)I\left(\stackrel{⌢}{\mathbf{p}}\right)}$$

(11)

where $\stackrel{⌢}{\mathbf{p}}$ is a pixel in $PATCH\left(\mathbf{p}\right)$. $I\left({\mathbf{p}}_k\right)$ represents the gray value of the pixel $\stackrel{⌢}{\mathbf{p}}$ in $PATCH\left(\mathbf{p}\right)$, $m\left(\stackrel{⌢}{\mathbf{p}}\right)$ is the pixel density function value of $\stackrel{⌢}{\mathbf{p}}$, ${\mathsf{\Pi }}^{-1}\left(\stackrel{⌢}{\mathbf{p}}\right)$ indicates the 3D coordinate of the projection point on the unit sphere surface of $\stackrel{⌢}{\mathbf{p}}$.

4.3. Feature Point Attitude Matrix Construction

In order to avoid the singularity of direction expression of feature points on the poles of the unit spherical surface [25], we propose a feature point attitude matrix, a nonsingular expression, to represent the position and the direction of a feature point. The feature point attitude coordinate system $O_b{X}_b{Y}_b{Z}_b$ is shown in Figure 3. The origin point $O_b$ coincides with the origin point $O_c$ of the camera coordinate system. The Z-axis $O_b{Z}_b$ coincides with the vector ${\mathbf{P}}_c^{\prime }$. The Y-axis $O_b{Y}_b$ is consistent with the ${\mathbf{P}}_c^{\prime }\times {\mathbf{C}}_c$. The X-axis $O_b{X}_b$ direction is determined by right-hand rule. The X-axis is coplanar with the 3D gray centroid vector ${\mathbf{C}}_c$ and the position vector ${\mathbf{P}}_c^{\prime }$.

The coordinate transformation matrix ${\mathbf{R}}_{cb}$ from the feature point attitude coordinate system to the camera coordinate system can be obtained as follows:

$${\mathbf{R}}_{cb}=\left[\begin{array}{ccc}\frac{{\mathbf{C}}_{\mathrm{c}}-\frac{{\mathbf{C}}_{\mathrm{c}}·{{\mathbf{P}}_c}^{\prime }}{{\mathbf{P}}_c{{}^{\prime }}^2}{{\mathbf{P}}_c}^{\prime }}{\left|{\mathbf{C}}_{\mathrm{c}}-\frac{{\mathbf{C}}_{\mathrm{c}}·{{\mathbf{P}}_c}^{\prime }}{{\mathbf{P}}_c{{}^{\prime }}^2}{{\mathbf{P}}_c}^{\prime }\right|}& \frac{{{\mathbf{P}}_c}^{\prime }\times {\mathbf{C}}_{\mathrm{c}}}{\left|{{\mathbf{P}}_c}^{\prime }\times {\mathbf{C}}_{\mathrm{c}}\right|}& {{\mathbf{P}}_c}^{\prime }\end{array}\right]$$

(12)

The matrix ${\mathbf{R}}_{cb}$ is defined as feature point attitude matrix.

4.4. FSD-BRIEF Descriptor Extraction

In this section, to enhance the distortion invariance of the descriptor in the fisheye image, FSD-BRIEF will be extracted by distorting the BRIEF template based on the constructed feature point attitude matrix so that its template can fit the distortion form of the adjacent area of the feature point.

At first, for a feature point, we define its square neighborhood region as a BRIEF template with a coordinate system $O_B{X}_B{Y}_B$ whose origin point $O_B$ is located at the feature point and coordinate ranges from −15 to 15, as shown in Figure 4. The green lines are the selected 256 groups of pixel pairs on the template.

Then, the defined BRIEF template is scaled to a certain extent and placed at the feature point as shown in Figure 5. For doing so, the following three conditions must be satisfied:

• The center point $O_B$ of the descriptor template coincides with the projection point $P^{\prime }$ of the feature point $\mathbf{p}$ on the sphere. In other words, the coordinate of point $O_B$ in the feature point attitude coordinate system is ${\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^{\mathrm{T}}$.
• The directions of $O_B{X}_B$, $O_B{Y}_B$ axis of BRIEF template coordinate system are consistent with the directions of $O_b{X}_b$, $O_b{Y}_b$ axis of the feature point attitude coordinate system.
• There is a scale factor $\frac{\alpha }{15}$ between the coordinates in the BRIEF template coordinate system and the coordinates in the feature point attitude coordinate system.

Figure 6 shows a zoom-in of a local area along the direction of $O_b{Z}_b$ in Figure 5 at the feature point $P^{\prime }$. As shown in Figure 6, for a point $P^{″}$ on the BRIEF template, its homogeneous coordinate vector in $O_B{X}_B{Y}_B$ coordinate system is $\mathbf{s}$. The coordinate vector of point $P^{″}$ in the feature point attitude coordinate system is ${\mathbf{P}}_b^{″}$. Then, the ${\mathbf{P}}_b^{″}$ can be solved by:

$${\mathbf{P}}_b^{″}=\mathbf{Ds}$$

(13)

where

$$\begin{array}{c}\mathbf{D}=diag(\frac{\alpha }{15},\frac{\alpha }{15},1)\hfill \\ \mathbf{s}={\left[\begin{array}{ccc}{s}_x& s_y& 1\end{array}\right]}^T\hfill \end{array}$$

(14)

According to the law of 3D coordinate transformation and the ${\mathbf{P}}_b^{″}$, the coordinate ${\mathbf{P}}_c^{″}$ of point $P^{″}$ in the camera coordinate system can be calculated by:

$${\mathbf{P}}_c^{″}={\mathbf{R}}_{cb}{\mathbf{P}}_b^{″}$$

(15)

where ${\mathbf{R}}_{cb}$ is the feature point attitude matrix.

The projection point ${\mathbf{p}}^{″}$ of ${\mathbf{P}}_c^{″}$ in the fisheye image can be obtained by:

$${\mathbf{p}}^{″}=\mathsf{\Pi }\left({\mathbf{P}}_c^{″}\right)$$

(16)

To sum up, for a feature point whose attitude matrix is ${\mathbf{R}}_{cb}$, the coordinate mapping relationship between the point $\mathbf{s}$ in the BRIEF template and the projection point ${\mathbf{p}}^{″}$ in the fisheye image is:

$${\mathbf{p}}^{″}=\mathsf{\Pi }\left({\mathbf{R}}_{cb}\mathbf{Ds}\right)$$

(17)

According to Equation (17), the FSD-BRIEF of a feature point can be extracted by the calculated projection points of the FSD-BRIEF template in the fisheye image. Figure 7 shows the general view of the FSD-BRIEF descriptor. It is clear that the FSD-BRIEF template in the fisheye image changes with the position where the feature point is located, so as to ensure that the descriptor is adaptive to the different distortions in the fisheye image, and achieves a good performance on distortion invariance.

5. Experimental Evaluation

In this section, we present four experiments that were used for evaluating the performance of the proposed method. Experiment 1 was an ablation experiment carried out on a virtual dataset, which was used to verify the contribution of pixel density function towards improving the solution accuracy of FSD-BRIEF orientation. Experiment 2 was also conducted on the virtual dataset, aiming to prove the invariance of FSD-BRIEF compared with three BRIEF-based descriptors. Experiment 3 and Experiment 4 were performed to evaluate the matching performance of FSD-BRIEF under (1) different camera motions on a real dataset, and (2) distortion conditions on sRD-SIFT dataset [22], respectively. The results of these two experiments were compared with those produced by five state-of-the-art features.

5.1. Experiment 1: The Contribution Evaluation of the Pixel Density Function to the Accuracy of Feature Point Orientation

Dataset: In this experiment, we investigated the contribution of the pixel density function to the accuracy of feature point orientation. In order to have accurate ground truth of the direction of feature points, we produced a virtual dataset by simulating a projection of the first image of the Graffiti dataset [30]; this was used as a test image to two virtual fisheye cameras with different intrinsic parameters. At first, in the test image, $N_p$ feature points ${\mathbf{p}}_t^i(i=1,2,\dots ,N_p)$ were extracted. During the generation of the virtual dataset, the test image and a selected virtual fisheye camera were placed in the same virtual space. By placing the test image in different poses, we projected each feature point in the fisheye image on several selected positions with different longitude angle $\phi$ and latitude angle $\theta$. The relationship between the angle $\phi$, $\theta$ and the pose of the test image is shown in Appendix A. $\phi$ takes $N_{\phi }$ values and $\theta$ takes $N_{\theta }$ values. For each virtual fisheye camera, $N_{\mathbf{p}}\times N_{\phi }\times N_{\theta }$ test samples were generated. Each test sample consisted of a generated fisheye image $I(\phi ,\theta ,{\mathbf{p}}_t^i)$, a corresponding feature point position ${\mathbf{p}}_c^i(\phi ,\theta ,{\mathbf{p}}_t^i)$ in the fisheye image, and a ground truth feature point attitude matrix ${{\mathbf{R}}_{cb}^i}^{*}(\phi ,\theta ,{\mathbf{p}}_t^i)$. More details of the dataset are given in Appendix B.

Baseline: To verify the effectiveness of the pixel density function compensation proposed in this paper, we compared two algorithms, namely, the feature point attitude matrix computation part of FSD-BRIEF without the compensation (version 1) and with (version 2). In version 1, the 3D gray centroid was calculated without the pixel density compensation term $m\left(\stackrel{⌢}{\mathbf{p}}\right)$. That is, the gray centroid computation formula of version 1 is shown as Equation (18). In version 2, we used Equation (11) to calculate the 3D gray centroids of feature points.

$${\mathbf{C}}_c=\frac{{\displaystyle \sum _{\stackrel{⌢}{\mathbf{p}}\in PATCH\left(\mathbf{p}\right)}}{\mathsf{\Pi }}^{-1}\left(\stackrel{⌢}{\mathbf{p}}\right)I\left(\stackrel{⌢}{\mathbf{p}}\right)}{{\displaystyle \sum _{\stackrel{⌢}{\mathbf{p}}\in PATCH\left(\mathbf{p}\right)}}I\left(\stackrel{⌢}{\mathbf{p}}\right)}$$

(18)

Fisheye cameras: In order to verify the contribution of the pixel density function under different FoV cameras, two virtual cameras with different FoVs were selected for this experiment.

Table 1. The Intrinsic Parameters of 170° FoV and 210° FoV Camera.

Intrinsic Parameter	170° FoV Camera	210° FoV Camera
$f_x$	284.977	257.28
$f_y$	284.977	257.28
$c_x$	423.039	582.006
$c_y$	398.179	419.655
$k_1$	−0.00454	−0.0765
$k_2$	0.0396	0.00908
$k_3$	−0.0363	−0.0117
$k_4$	0.00584	0.00373

shows the intrinsic parameters of the two cameras.

Figure 8 shows the curve of the pixel density function of 170° FoV and 210° FoV cameras with $\theta$. From the curve, we can see that the curve of the pixel density function of 170° FoV cameras decreased in angle range 0–60°, and increased in angle range 60–80°. Another curve, which was for the pixel density function of 210° FoV camera, increased in the whole angle range of 0–90°.

Evaluation metrics: In the experimental verification process, the direction angle error of the feature point is used for quantitative evaluation. The direction angle error, denoted by $e(\phi ,\theta ,{\mathbf{p}}_t^i)$, is shown in Figure 9, where $P^i(\phi ,\theta ,{\mathbf{p}}_t^i)$ is the projection point of ${\mathbf{p}}_c^i(\phi ,\theta ,{\mathbf{p}}_t^i)$ on the unit sphere surface. The coordinate system $O_b^{*}{X}_b^{*}{Y}_b^{*}{Z}_b^{*}$ is the feature point attitude coordinate system corresponding to the ground truth feature point attitude matrix ${{\mathbf{R}}_{cb}^i}^{*}(\phi ,\theta ,{\mathbf{p}}_t^i)$, whilst $O_b{X}_b{Y}_b{Z}_b$ is the feature point attitude coordinate system corresponding to the calculated feature point attitude matrix ${\mathbf{R}}_{cb}^i(\phi ,\theta ,{\mathbf{p}}_t^i)$. Note that $\overrightarrow{O_b^{*}{X}_b^{*}}$ and $\overrightarrow{O_b{X}_b}$ are defined as the ground truth direction and the calculated direction of the feature point (see Section 4.3). The unit of $e(\phi ,\theta ,{\mathbf{p}}_t^i)$ is defined as degree (°). Let ${(\overrightarrow{O_b^{*}{X}_b^{*}})}_c$ and ${(\overrightarrow{O_b{X}_b})}_c$ be the coordinate of the unit direction vectors corresponding to $\overrightarrow{O_b^{*}{X}_b^{*}}$ and $\overrightarrow{O_b{X}_b}$ in the camera coordinate system, then:

$$\begin{array}{c}{(\overrightarrow{O_b^{*}{X}_b^{*}})}_c={{\mathbf{R}}_{cb}^i}^{*}\left(\phi ,\theta ,{\mathbf{p}}_t^i\right){\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^T\hfill \\ {(\overrightarrow{O_b{X}_b})}_c={\mathbf{R}}_{cb}^i(\phi ,\theta ,{\mathbf{p}}_t^i){\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^T\hfill \\ \cos \left(\frac{\pi }{180}e(\phi ,\theta ,{\mathbf{p}}_t^i)\right)=\overrightarrow{O_b{X}_b}·\overrightarrow{O_b^{*}{X}_b^{*}}={(\overrightarrow{O_b{X}_b})}_c^T{(\overrightarrow{O_b^{*}{X}_b^{*}})}_c\hfill \end{array}$$

(19)

From Equation (19), we can obtain the expression of $e(\phi ,\theta ,{\mathbf{p}}_t^i)$ as:

$$e\left(\phi ,\theta ,{\mathbf{p}}_t^i\right)=\frac{180}{\pi }\arccos \left(\left[\begin{array}{ccc}1& 0& 0\end{array}\right]{\mathbf{R}}_{cb}^i\left(\phi ,\theta ,{\mathbf{p}}_t^i\right){{\mathbf{R}}_{cb}^i}^{*}\left(\phi ,\theta ,{\mathbf{p}}_t^i\right){\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^T\right)$$

(20)

Note that values of $e(\phi ,\theta ,{\mathbf{p}}_t^i)$ could be calculated from experimental results indexed by $\phi$ (FoV longitude angle), $\theta$ (FoV Latitude Angle) and i (feature point index in test image). For an ideal method, $e(\phi ,\theta ,{\mathbf{p}}_t^i)$ is always zero, and the calculated direction of feature point is consistent with the real direction. In fact, due to the influence of noise, the angle error $e(\phi ,\theta ,{\mathbf{p}}_t^i)$ would not be zero. In this experiment, the smaller the value of $e(\phi ,\theta ,{\mathbf{p}}_t^i)$, the more accurate the calculated feature point direction.

In this study, the mean error $e_{mean}\left(\theta \right)$ and the standard deviation $e_{SD}\left(\theta \right)$ were used to evaluate the results of $e\left(\phi ,\theta ,{\mathbf{p}}_t^i\right)$. $e_{mean}\left(\theta \right)$ measures the average error of the feature point direction calculated by using all the points under the latitude angle $\theta$. $e_{SD}\left(\theta \right)$ measures the dispersion of the $e\left(\phi ,\theta ,{\mathbf{p}}_t^i\right)$ distribution under $\theta$. $e_{mean}\left(\theta \right)$ and $e_{SD}\left(\theta \right)$ are calculated as follows:

$$\begin{array}{c}{e}_{mean}\left(\theta \right)=\frac{{\displaystyle \sum _{\phi }}{\displaystyle \sum _i}e(\phi ,\theta ,{\mathbf{p}}_t^i)}{N_{\phi }{N}_{\mathbf{p}}}\hfill \\ e_{SD}\left(\theta \right)=\sqrt{\frac{{\displaystyle \sum _{\phi }}{{\displaystyle \sum _i}[e(\phi ,\theta ,{\mathbf{p}}_t^i)-e_{mean}\left(\theta \right)]}^2}{N_{\phi }{N}_p}}\hfill \end{array}$$

(21)

where, the $N_{\phi }$ and $N_{\mathbf{p}}$ are the number of $\phi$ and i values. The smaller the $e_{mean}\left(\theta \right)$ is, the more accurate the feature point direction is. The smaller the $e_{SD}\left(\theta \right)$ is, the more stable the result of feature point direction is.

Evaluations: In the 170° FoV camera, the range of $\theta$ is 10–80°. In the 210° FoV camera, the range of $\theta$ is 10–90°. The two statistics $e_{mean}\left(\theta \right)$ and $e_{SD}\left(\theta \right)$ are computed for both of the two cases. The comparison results are shown in

Table 2. The numerical results of direction angle error in 170° FoV camera.

θ (°)	Version 1 (Without Compensation)		Version 2 (With Compensation)		Error Reduction (%)
	Mean	SD	Mean	SD	Mean	SD
10	1.133	0.920	1.084	0.865	−4.306	−5.978
20	1.213	0.827	1.162	0.800	−4.140	−3.360
30	1.034	0.786	0.922	0.703	−10.895	−10.452
40	1.143	0.914	0.948	0.782	−17.111	−14.451
50	1.106	0.905	1.116	0.811	0.895	−10.367
60	1.030	0.796	0.947	0.668	−8.033	−16.065
70	1.756	1.251	0.849	0.656	−51.629	−47.526
80	5.185	3.326	1.342	1.011	−74.123	−69.592

and

Table 3. The numerical results of direction angle error in 210° FoV camera.

θ (°)	Version 1 (Without Compensation)		Version 2 (With Compensation)		Error Reduction (%)
	Mean	SD	Mean	SD	Mean	SD
10	0.697	0.521	0.684	0.504	−1.850	−3.322
20	0.800	0.620	0.781	0.574	−2.425	−7.339
30	1.134	0.840	0.980	0.720	−13.540	−14.277
40	2.052	1.402	1.518	1.080	−25.995	−22.989
50	1.974	1.357	1.218	0.932	−38.280	−31.344
60	1.474	1.163	0.837	0.594	−43.226	−48.942
70	2.085	1.415	0.920	0.717	−55.880	−49.322
80	2.310	1.591	0.899	0.703	−61.068	−55.838
90	2.373	1.605	0.929	0.725	−60.838	−54.859

. The error reduction of version 2 compared to version 1 are calculated as follows:

$$\eta =\frac{e_{v2}-e_{v1}}{e_{v1}}\times 100\%$$

(22)

where $\eta$ is the value of error reduction, $e_{v1}$ and $e_{v2}$ are the value of the direction angle error of version 1 and version 2 individually. Taking the horizontal axis as the $\theta$ value and the vertical axis as $e_{mean}\left(\theta \right)$ and $e_{SD}\left(\theta \right)$, the $e-\theta$ curves are also drawn in Figure 10.

For the 170° FoV camera, both of the two compensation schemes led to similarly stable results in the angle range of 10–60°. However, when the angle $\theta$ became large (especially in the range of 60–80°), the performance of Version 2 was obviously much better than that of Version 1. Both of the average angle error and the accuracy dispersion of the proposed method (version 2) were about 1° in the whole fisheye FoV of the dataset.

For the 210° FoV camera, the overall performance of Version 2 was continuously better than that of Version 1 throughout the range of 30–90°.

The experimental results showed that near the edge of FoV, especially in the FoV region where the pixel density function increased monotonously with the angle $\theta$, the pixel density compensation improved the accuracy and stability of feature point direction calculation significantly.

5.2. Experiment 2: Descriptor Invariance Evaluation of Fisheye Images in Different FoV Positions

Baselines: In this experiment, three typical BRIEF descriptors, including ORB, dBRIEF (Distorted BRIEF), and mdBRIEF, were selected as baselines. The descriptor of the feature point in each test sample in the virtual dataset generated in Experiment 1 was extracted by the tested features (FSD-BRIEF, ORB, dBRIEF, and mdBRIEF). In order to ensure a fair comparison of experimental results, all the binary descriptors were chosen to be 256 bits. dBRIEF is the version of mdBRIEF without on-line mask learning. For dBRIEF and mdBRIEF, we used the open source version provided in GitHub. For ORB, we used the functions provided in OpenCV and its default parameter settings.

Evaluation metrics: In this experiment, we define $D(\phi ,\theta ,{\mathbf{p}}_t^i)$ as the descriptor of the feature point ${\mathbf{p}}_c^i(\phi ,\theta ,{\mathbf{p}}_t^i)$. The associated Hamming distance error $\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)$ of the descriptor of the feature point was used to evaluate the invariance performance of algorithms. $\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)$ is calculated for each feature point test sample by each test feature as:

$$\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)=h(D(\phi ,\theta ,{\mathbf{p}}_t^i),D({\phi }_0,{\theta }_0,{\mathbf{p}}_t^i))$$

(23)

here we selected $D({\phi }_0,{\theta }_0,{\mathbf{p}}_t^i)$ as the reference standard descriptor to compute the Hamming distance error, where ${\phi }_0={45}^{°} {\theta }_0={10}^{°}$. For an ideal feature algorithm, for the same ${\mathbf{p}}_t^i$, no matter what values of $\phi$ and $\theta$ take, there is $\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)=0$. However, in practice, due to the resampling error of the fisheye camera, $\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)$ was not zero. Therefore, the smaller the calculated value of $\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)$, the stronger the invariance of the feature algorithm to radial distortion of the fisheye image.

Similar to Experiment 1, $\Delta D_{mean}\left(\theta \right)$ and $\Delta D_{SD}\left(\theta \right)$ were used as evaluation metrics. $\Delta D_{mean}\left(\theta \right)$ is the average value of the descriptor distance calculated by using all the points under the latitude angle $\theta$. $\Delta D_{SD}\left(\theta \right)$ is the dispersion of the $\Delta D\left(\phi ,\theta ,{\mathbf{p}}_t^i\right)$ distribution under $\theta$. The smaller the $\Delta D_{mean}\left(\theta \right)$ is, the stronger the invariance of feature algorithm to radial distortion of fisheye images. The smaller the $\Delta D_{SD}\left(\theta \right)$ is, the more stable the performance of the feature algorithm is. The computation formula of $\Delta D_{mean}\left(\theta \right)$ and $\Delta D_{SD}\left(\theta \right)$ was as follows:

$$\begin{array}{c}\Delta D_{mean}\left(\theta \right)=\frac{{\displaystyle \sum _{\phi }}{\displaystyle \sum _i}\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)}{N_{\phi }{N}_{\mathbf{p}}}\hfill \\ \Delta D_{SD}\left(\theta \right)=\sqrt{\frac{{\displaystyle \sum _{\phi }}{{\displaystyle \sum _i}[\Delta D(\phi ,\theta ,{\mathbf{p}}_t^i)-\Delta D_{mean}\left(\theta \right)]}^2}{N_{\phi }{N}_{\mathbf{p}}}}\hfill \end{array}$$

(24)

Evaluations: Since ${\theta }_0={10}^{°}$ was set for the reference standard descriptor $D({\phi }_0,{\theta }_0,{\mathbf{p}}_t^i)$, the ranges of $\theta$ were selected as 20–80° in 170° FoV camera, and 20–90° in 210° FoV camera respectively. The values of $\Delta D_{mean}\left(\theta \right)$ and $\Delta D_{SD}\left(\theta \right)$ of FSD-BRIEF, ORB, dBRIEF, and mdBRIEF were computed. The numerical results are shown in

Table 4. The numerical results of Hamming distance error in 170° FoV camera.

θ (°)	FSD-BRIEF		ORB		dBRIEF		mdBRIEF
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
20	25.100	7.033	25.692	8.005	20.458	6.520	3.458	1.779
30	20.658	6.284	22.767	7.700	18.833	5.974	3.192	1.583
40	21.825	6.994	25.867	8.073	23.850	7.237	4.667	2.413
50	21.300	7.209	30.017	8.798	31.917	9.516	7.083	3.635
60	23.325	7.407	37.050	10.598	44.217	11.882	11.633	5.680
70	26.533	6.904	52.175	12.154	60.742	13.563	19.883	8.170
80	33.850	10.045	87.233	17.572	92.792	15.495	43.083	13.704

and

Table 5. The numerical results of Hamming distance error in 210° FoV camera.

θ (°)	FSD-BRIEF		ORB		dBRIEF		mdBRIEF
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
20	20.892	5.639	21.158	6.309	17.600	4.467	2.775	1.345
30	22.608	6.125	25.467	7.242	20.000	5.804	3.208	1.460
40	25.767	7.475	30.925	8.592	23.708	7.439	3.792	1.788
50	25.875	7.996	41.442	11.538	31.083	9.718	5.058	1.881
60	28.867	7.978	57.508	15.822	42.558	13.937	8.058	4.101
70	30.317	8.176	70.775	16.628	58.758	14.927	17.892	8.579
80	36.250	10.375	84.217	16.844	95.292	16.522	44.975	14.635
90	45.000	14.170	97.450	21.129	-	-	-	-

. The corresponding curves of $\Delta D_{mean}\left(\theta \right)$ are shown in Figure 11, and the curves of $\Delta D_{SD}\left(\theta \right)$ are shown in Figure 12.

The experimental results of the two cameras showed that, in the angle range of 20–40°, FSD-BRIEF led to similarly stable descriptor errors as ORB and dBRIEF. However, in the angle range of 40–80°, the descriptor errors of ORB and dBRIEF tended to increase significantly, while the descriptor errors of FSD-BRIEF increased much less than that of ORB and dBRIEF. In the angle range of 75–80°, the descriptor error of FSD-BRIEF was smaller than that of mdBRIEF. However, the descriptor error of FSD-BRIEF was larger than that of mdBRIEF in the angle range of 20–60°; this is because an on-line mask learning scheme was performed in mdBRIEF, where the unstable binary bits were masked.

The standard deviations (SD) of FSD-BRIEF, ORB and dBRIEF were similar in the angle range of 20–40°. In the angle range up to 50°, the SD of FSD-BRIEF was significantly smaller than that of ORB and dBRIEF. In the angle range of 20–60°, the SD value of FSD-BRIEF was not as small as mdBRIEF, but smaller than mdBRIEF in the angle range of 70–80°.

Because dBRIEF and mdBRIEF distorted the descriptor template based on the plane perspective model, it could not extract the feature descriptor when $\theta$ was 90°, and there was no 90° effective value of the descriptor errors.

It can be observed from the results that, compared with other BRIEF based features, FSD-BRIEF could effectively adapt to the radial distortion of fisheye images and ensure the invariance of descriptors.

5.3. Experiment 3: Matching Performance Evaluation in Different Kind of Image Variance

Dataset: In order to verify the FoV edge distortion invariance, translation invariance, and scale invariance performances of the proposed FSD-BRIEF in image matching process, a dataset captured by a 210° FoV fisheye camera was made. The intrinsic parameter of the 210° FoV fisheye camera is shown in Table 1. There were three groups of images in this dataset, and each group contained 13 images. In the first group of images, through rotation of the camera, the test image fell on the edge of the camera’s FoV as close as possible, and the test image was distorted by the radial distortion of the fisheye camera to the greatest extent. In the second group of images, by moving and rotating the camera parallel to the test image plane, the test image fell in different positions of the camera FoV. In the third group of images, the camera moved forward and backward greatly relative to the test image, which made the projection of the test image in the fisheye image has a large-scale change.

Baselines: In this experiment, five state-of-the-art descriptors, AKAZE, BRISK, ORB, dBRIEF and mdBRIEF, were selected as baselines. For FSD-BRIEF, we used the FAST feature to extract feature points. For BRISK, ORB, and AKAZE, we used the functions provided in OpenCV with default parameter settings. For dBRIEF and mdBRIEF, we used the open source version provided in GitHub.

Evaluation metrics: In order to evaluate the matching performance of FSD-BRIEF proposed in this paper, according to [30], we conducted comparison experiments with state-of-the-art descriptors by calculating the PR (recall—“1-precision”) curve of the matching results. Designate $S^i$, $S^j$ to be a set of feature points detected in the image $I^i$ and $I^j$ respectively, then the set of ground truth matching points $G^{ij}$ can be given by:

$$G^{ij}=\{({\mathbf{p}}^i,{\mathbf{p}}^j)|\parallel {\mathbf{p}}^i-\mathsf{\Pi }\left({\mathbf{H}}^{ij}{\mathsf{\Pi }}^{-1}\left({\mathbf{p}}^j\right)\right)\parallel <\epsilon ,{\mathbf{p}}^i\in S^i,{\mathbf{p}}^j\in S^j\}$$

(25)

where $∥·∥$ refers to Euclidean distance between the ${\mathbf{p}}^i$ and the projecting point of ${\mathbf{p}}^j$ in image $I^i$, ${\mathbf{H}}^{ij}$ is the ground truth homography matrix from image $I^i$ to $I^j$, which was calculated by manually labeled corresponding points in the image sequence. The distance threshold $\epsilon$ was taken as 3 pixels. To evaluate the matching performance of test features, let $M^{ij}$ be the set of matching feature point pairs gained by the algorithm from the image $I^i$ and $I^j$, and $M^{ij}$ consisted of correct matches $M_{true}^{ij}$ and incorrect matches $M_{false}^{ij}$. Hence, as shown in Equation (26), the $recall\left({\epsilon }^{\prime }\right)$ presents the ability of the matching algorithm to find correct matches, and $1-precision\left({\epsilon }^{\prime }\right)$ indicates the algorithm’s capability of discarding unmatched points.

$$recall\left({\epsilon }^{\prime }\right)=\frac{{\displaystyle \sum _{1\le i<j\le n}}N(M_{true}^{ij}\left({\epsilon }^{\prime }\right))}{{\displaystyle \sum _{1\le i<j\le n}}N\left(G^{ij}\right)},1-precision\left({\epsilon }^{\prime }\right)=1-\frac{{\displaystyle \sum _{1\le i<j\le n}}N(M_{false}^{ij}\left({\epsilon }^{\prime }\right))}{{\displaystyle \sum _{1\le i<j\le n}}N\left(M^{ij}\left({\epsilon }^{\prime }\right)\right)}$$

(26)

where n is the number of images in the image sequence, $N(\ast )$ denotes the point pair number of a set, ${\epsilon }^{\prime }$ is a descriptor distance threshold that was used to obtain the correct matches whose Euclidean distance between their descriptors is below ${\epsilon }^{\prime }$. Each of the two measures yielded a so-called PR cure by increasing the threshold ${\epsilon }^{\prime }$ from zero gradually. That PR curve passed at a short distance of the ideal point (0, 1) meant the corresponding test feature was absolutely perfect which could make both the value of precision and recall rate 1. In practice, a good matching performance was achieved when the matching algorithm’s PR curve had the minimum distance to the point (0, 1), the highest recall, and the minimum 1-precision.

Evaluation: To test the matching performance in this dataset, we used the test features to extract and match features and drew PR curves. For each algorithm in each image, 300 strongest feature points were extracted. The PR curve results are shown in Figure 13.

From Figure 13a,b, the recall value at the end of the PR curve of FSD-BRIEF proposed in this paper was in the range of 0.75–0.8. For other features involved in the comparison, the recall value at the end of the PR curve was in the range of 0.3–0.6. The result showed that, compared with other features, FSD-BRIEF had significant FoV edge distortion invariance in the feature matching process of severely distorted images.

Figure 13c,d shows that the recall value at the end of the PR curve of FSD-BRIEF proposed in this paper was near 0.5. For other features involved in comparison, the recall value at the end of the PR curve was in the range of 0.25–0.5 and below FSD-BRIEF. The result showed that, compared with other features, FSD-BRIEF had better translation invariance in the feature matching process of fisheye images.

In Figure 13e,f, it can be observed that the recall value at the end of the PR curve of FSD-BRIEF proposed in this paper was in the range of 0.4–0.45. For AKAZE, BRISK, ORB, and dBRIEF, the recall value at the end of the PR curve was in the range of 0.25–0.4. The recall value of FSD-BRIEF was higher than mdBRIEF when $1-precision$ was in the range of 0.05–0.3. The results showed that FSD-BRIEF had better scale invariance in the feature matching process of fisheye images compared with most of the state-of-the-art features.

Using AKAZE, BRISK, ORB, dBRIEF and mdBRIEF as references, experimental results showed that FSD-BRIEF showed comparable performance in FoV edge distortion invariance, translation invariance, scale invariance, and matching performance in fisheye images.

5.4. Experiment 4: Matching Performance Evaluation in Different Distortion Images

Dataset: In order to verify the matching performance of FSD-BRIEF under different radial distortion, the sRD-SIFT dataset was used in this experiment. The sRD-SIFT datasets [22] were published with the work of sRD-SIFT. It consisted of three sets of images (FireWire, Dragonfly, and Fisheye), each set containing 13 images and captured by a camera with different radial distortion. The dataset contained significant scaling and rotation changes. Four images selected randomly for each dataset are shown in the right panels of Figure 14.

Fisheye cameras: The three sets of images were attached with the image of a checkerboard calibration board for camera calibration. Therefore, we calibrated each camera based on the KB4 fisheye camera model using the chessboard image provided. The calibration results are shown in

Table 6. The intrinsic parameters of the cameras in sRD-SIFT datasets.

Intrinsic Parameter	set1(FireWire)	set2(Dragonfly)	set3(Fisheye)
$f_x$	539.389	528.626	306.780
$f_y$	539.389	528.626	306.780
$c_x$	312.103	365.029	634.729
$c_y$	233.050	228.558	478.546
$k_1$	0.0537	−0.0994	−0.000788
$k_2$	0.0871	−0.0205	0.0181
$k_3$	0	0.00661	−0.0117
$k_4$	0	0.0150	0.00190

.

Evaluation: Similar to Experiment 3, to test the matching performance in the three groups of the sRD-SIFT dataset, we also employed the baseline descriptors (ORB, AKAZE, BRISK, dBRIEF and mdBRIEF) to extract and match 300 strongest keypoints for each image, then draw PR curves. The results are shown in Figure 14, where Figure 14a,b shows the results and the image group with the least distortion. Figure 14c,d shows the results and the image group with moderate distortion. Figure 14e,f shows the results of the image group with the most distortion captured by fisheye cameras.

Figure 14a,b shows that the PR curve of FSD-BRIEF almost coincided with that of ORB and AKAZE, and the performance of AKAZE was slightly better. The recall rate at the end of the curve of FSD-BRIEF, ORB, and AKAZE was in the range of 0.65–0.7, which was higher than that of BRISK and dBRIEF. From the result, we can see that the performance of FSD-BRIEF was equivalent to that of ORB in small distorted images.

Figure 14c,d shows that the PR curve of FSD-BRIEF almost coincided with that of ORB, and the recall at the end of the curve was around 0.6, which was higher than that of AKAZE, BRISK, and dBRIEF. From the result, we can see that the performance of FSD-BRIEF was equivalent to that of ORB in moderate distorted images and better than AKAZE, BRISK, and dBRIEF.

In Figure 14e,f, it can be observed that the recall value at the end of the PR curve of FSD-BRIEF was around 0.6, which was higher than that of ORB, AKAZE, BRISK, and dBRIEF, and almost the same as that of mdBRIEF. From the result, we can see that the performance of FSD-BRIEF was almost equivalent to that of mdBRIEF and better than ORB, AKAZE, BRISK, and dBRIEF in the most distorted images.

These experimental results show that the performance of FSD-BRIEF in large distortion image was better than most of the state-of-the-art features involved in the comparison. In small and moderate distorted images, the performance of FSD-BRIEF was similar to that of the ORB feature. That is because that the test image was close to the center of the FoV in this dataset, the radial distortion effect of the test image by the fisheye lens was limited compared with Experiment 3. Therefore, the performance of FSD-BRIEF in this paper on the sRD-SIFT dataset was not as prominent as the 210° FoV camera dataset in Experiment 3.

6. Conclusions

In this paper, to tackle the problem of the feature matching performance deterioration due to the impact of fisheye radial distortion, we proposed a novel distorted BRIEF descriptor, named FSD-BRIEF, for fisheye images based on the spherical projection model. First, for reducing the impact of the distortion on gray centroid calculation and the accuracy of feature point direction, we designed a pixel density function and evaluated its performance by comparing the feature point direction error results of the algorithms with and without using the function. The obtained results shown that the pixel density function can promote the precision of the feature point direction calculation. Second, the distortion invariance of the proposed FSD-BRIEF was verified and compared with other BRIEF based descriptors, and the associated results demonstrated that FSD-BRIEF works well for distortion invariance in different positions of fisheye images. In the matching experiments in 210° FoV camera datasets, FSD-BRIEF shown better performance for FoV edge distortion invariance, translation invariance, and scale invariance in large distortion fisheye images. In the sRD-SIFT dataset, the FSD-BRIEF descriptor can significantly improve the matching performance for large distortion images, and meanwhile can still produce excellent results for small distortion images.

7. Future Work

It is known that panoramic images have been widely used today. The proposed descriptor can be adapted and potentially applied to panoramic images, with some slight modifications of the camera model and the computation method of the pixel density function, respectively. Moreover, in the future work, we will design a distorted FAST detector based on the spherical perspective model for panoramic images to extract feature points at any position including the two Polar Regions.