Central Dioptric Line Image-Based Visual Servoing for Nonholonomic Mobile Robot Corridor-Following and Doorway-Passing

Abstract

Autonomous navigation in indoor environments demands reliable perception and control strategies for nonholonomic mobile robots operating under geometric constraints. While visual servoing offers a promising framework for such tasks, conventional approaches often rely on explicit 3D feature estimation or predefined reference trajectories, limiting their adaptability in dynamic scenarios. In this paper, we propose a novel nonholonomic mobile robot corridor-following and doorway-passing method based on image-based visual servoing (IBVS) by using a single dioptric camera. Based on the unifying central spherical projection model, we present the projection mechanism of 3D lines and properties of line images for two 3D parallel lines under different robot poses. In the normalized image plane, we define a triangle enclosed by two polar lines in relation to line image conic features, and adopt a polar representation for visual features, which will naturally become zero when the robot follows the corridor middle line. The IBVS control law for the corridor-following task does not need to pre-calculate expected visual features or estimate the 3D information of image features, and is extended to doorway-passing by simply introducing an upper door frame to modify visual features for the control law. Simulations including straight corridor-following, anti-noise performance, convergence of the control law, doorway-passing, and loop-closed corridor-following are conducted. We develop a ROS-based IBVS system on our real robot platform; the experimental results validate that the proposed method is suitable for the autonomous indoor visual navigation control task for a nonholonomic mobile robot equipped with a single dioptric camera.

1. Introduction

Navigating autonomously using only vision in urban or indoor environments is a challenging task for intelligent robots [1,2]. In the field of urban navigation, a typical Automatic Drive System for robots controls their low-level motion adaptively based on available visual information such as curves [3] or lanes [4]. For an unmanned aerial vehicle (UAV), visual information can be applied to control automatic landing based on ground lines [5,6]. In some smart home applications, researchers have developed smart wheelchairs for helping the elderly or disabled individuals, and proposed corresponding adaptive vision-guided corridor-following and doorway-passing methods as fundamental capabilities of nonholonomic mobile robots [7].

Vision-based control is also called visual servoing, which was originally developed for automatic object manipulation by using a robotic arm with an eye-in-hand or eye-to-hand configuration [8]. Generally, two types of control methods, position-based visual servoing (PBVS) and image-based visual servoing (IBVS) [9], can be used to design a mobile robot controller according to the feedback signals of the closed-loop systems. PBVS methods generally require estimating the robot’s pose or position from a vision system, and since the error and input signals are spatial pose, the controller is relatively simple and singular values can be avoided [10]. Trajectory tracking (TC) represents a compelling research direction, wherein a general framework based on position-based visual servoing (PBVS) is commonly adopted to follow dynamically evolving paths. The research [11] presents an online robust adaptive dynamic programming algorithm (RADPA) to control the robot with an omnidirectional vision system for the worst-case disturbance, and a neural network is used to approximate a zero-sum game in the learning scheme, which must be trained online. Based on an omnidirectional onboard camera, the research [12] proposed an adaptive PBVS method in which multiple sensors are used to estimate the robot’s pose. Another study [13] proposed the extended POSIT method to estimate robot pose and camera parameters online; however, this method is limited to environments where a camera must be pre-fixed to the ceiling. The IBVS method expresses the control objective and control law directly in the parameter domain of image features, and is more robust than PBVS when dealing with uncertainties and disturbances introduced by the models of the robot and the camera [14]. In the recent application [15], researchers employed the principle of virtual work to transform the propagation of errors in the image plane into the transmission of virtual forces, and proposed a novel image-based IBVS method designed for robotic manipulators. This approach enables a more intuitive formulation of physical constraints on the robot. It eliminates the need for Jacobian matrix inversion and effectively avoids singularities arising from matrix inversion. Another study [16] utilized both monocular and stereo cameras to propose a one-time alignment method that is independent of depth information for rapid-fire source alignment in fire scenarios. The IBVS method plays a significant role in further enhancing the alignment accuracy of the stereo camera. Therefore, some studies focus on robust control concerning the calibration issues of onboard monocular cameras [17] or overhead fixed cameras [18,19]. A common characteristic of these methods is their reliance on point features to facilitate the development of robust IBVS controllers.

As a practical extension of path-following in navigation control, vision-based corridor-following and doorway-passing for nonholonomic mobile robots are receiving increasing attention [3]. The research [20] proposed a novel doorway-passing method that steers a robot through the middle of a gate by defining the control laws according to different coordinates of door planar geometry. A wheelchair robot [1] is developed, which requires the identification of the vanishing point feature from corridor images for the corridor-following task. This method relies on using the projective properties of a unidirectional camera during robot ego-motion, and applies polar parameters to the IBVS controller, which has to be manually assisted. Studies [21,22] proposed IBVS navigation methods for Pepper indoor navigation by using appearance features with only line segments or both points and lines. The advantage is that a 3D reconstruction process is not required; however, the robot has to keep key images in a predefined database to initialize the control system. Chen [23] proposed a novel collision-free spatial path planning method for IBVS, demonstrating the existence of a unique path node (PN) on the line connecting the target and the camera, where the orientation error is zero. Based on this finding, intermediate points along the image path are generated as sub-goals for visual servo control, enabling perfect obstacle avoidance. Dong [24] developed a new Kalman-based depth estimation model and defined the state equations according to the number of state variables, thereby achieving adaptive tuning of the servo gain in image-based visual servo control. To further enable wheelchairs to pass through doorways safely [7], recently, deep learning frameworks were introduced into autonomous corridor-following. The research [25] developed an end-to-end CNN to recognize the entire corridor image and generate the control output. The proposed approach is claimed to be more robust compared with traditional methods. Building on this foundation, GRIEF-DE [26] enhances the robustness of IBVS under dynamic environmental changes by optimizing feature descriptors through evolutionary algorithms, effectively mitigating control failures caused by feature point drift. Additionally, some researchers [27] have integrated IBVS with YOLOv5, employing a gimbal camera-based IBVS servo system to improve UAV target tracking and navigation capabilities in GPS-denied environments. Tsapin [28] incorporated Squeeze-and-Excitation (SE) blocks into MobileNetV2, ResNet50, and DenseNet121 architectures for robust image processing in challenging environments. Their method achieved faster response times while maintaining the same level of accuracy as the traditional HOG-BoVW-BPNN approach. A more recent study [29] proposed a visual servoing algorithm that integrates deep learning, using a CNN to directly predict the required velocity commands for the robot. This approach eliminates the reliance on 3D pose estimation of the object, which is typically required in conventional PBVS methods. As a result, the method enhances the robustness of visual servoing tasks, particularly in environments lacking prominent features. Nevertheless, existing CNN-based approaches remain highly dependent on extensive training data and involve considerable computational overhead, thereby limiting their practicality for rapid deployment in varying environments.

The corridor-following methods mentioned above are mostly based on a single-directional camera and inevitably suffer from a restricted field of view (FOV). Therefore, a path planning approach can be employed to ensure that target features remain within the FOV during robot motion [30]. For resource-constrained mobile robots like Pepper, scholars have developed a topological navigation system [31] based on image memory and the teach-and-repeat paradigm. This system integrates two IBVS controllers to enhance navigation efficiency and accuracy. However, an alternative choice is to directly increase the FOV of the onboard camera by using omnidirectional vision. Based on the ground line, which is a common feature in man-made environments, previous research [32] proposed a catadioptric vision-based IBVS method for a nonholonomic robot. This method achieves robot control through a single 3D line, although it necessitates prior specification of the desired image feature based on the three-dimensional geometry of the target line. In the framework [33], the interaction matrix was further modified according to the quadratic form of a conic curve image projected from a target 3D line based on the single-viewpoint model [34]. Using omnidirectional vision, the investigation [35] proposed a real-time generalized Voronoi diagram to extract a free-space skeletonization path for nonholonomic robot IBVS navigation. This approach detects two points on the image skeletonization with a fixed distance to approximate a conic curve; therefore, a special camera configuration with respect to the robot has to be considered to satisfy the convergency of the control law.

In this paper, inspired by wheelchair IBVS research [7], we propose a novel 3D line-based IBVS method for both corridor-following and doorway-passing of nonholonomic robots equipped with a catadioptric camera. To our knowledge, purely applying a single catadioptric vision system to IBVS indoor navigation control has not been reported for both corridor-following and doorway-passing in the literature before. In our approach, we adopt the unified spherical projection model, which projects a 3D line into a conic curve in the image domain. By analyzing the properties of the intersection point of polar lines estimated under different robot poses, we design an image triangle with at least two polar lines for our polar-based control law, which can integrate two navigation control tasks together. Compared with the latest analogous research [7], the catadioptric camera we use has a much larger FOV than a general directional camera, which will reduce the possibility of losing the visual feature. In addition, we define the visual feature based on polar coordinates rather than Cartesian coordinates, and the polar feature representation naturally constrains the parameter space and overcomes the numerical issue in the framework [7]. Finally, it is unnecessary for our approach to estimate the spatial trajectory when performing the doorway-passing task. Instead, we integrate this task into our IBVS framework by using door frame image features without any estimation of spatial trajectory. Other similar IBVS methods based on omnidirectional vision are described in the frameworks [33,34], which depend on a single horizontal 3D line. However, these methods need to specify the desired feature image and are only suitable for the corridor-following task. To sum up, the novel contributions of this paper can be detailed as follows:

• A single dioptric vision-based indoor navigation control for nonholonomic robots is proposed, using line images of 3D environmental features for both corridor-following and doorway-passing. The approach is purely based on an IBVS scheme, which does not require the estimation of 3D geometric information in the control law.
• For the IBVS corridor-following task, based on the image features of two horizontal ground lines under a unifying spherical projection, we define a polar-line-based triangle in which a median line connecting the vertex and middle point on the triangle base is chosen to be the visual feature for the IBVS controller. The polar parameters of the median line naturally go to zero in the normalized image pace as the robot approaches the middle of the corridor, which relieves the pre-calculation of the expected image feature.
• Compared with the work that tries to establish several coordinates according to the geometry around the door based on a directional camera [20], our approach simply extends our IBVS scheme for corridor-following to the doorway-passing task by using two vertical and one upper door frame, and the polar line of the upper door frame line image is defined as the triangle base instead, which makes the robot approach the middle of the door more quickly than by using only two vertical door frames.

The paper is organized as follows. In Section 2, we investigate the image line properties of 3D parallel lines using the unified central projection model of a catadioptric camera. In Section 3, we introduce our nonholonomic mobile robot model, the polar IBVS control law, and the convergence analysis based on the proposed line image features. In Section 4, some simulation results are presented to validate the proposed method for nonholonomic mobile robot corridor-following and doorway-passing tasks. Finally, we conduct two types of experiments to further validate the proposed IBVS method on our real mobile robot platform.

2. Central Catadioptric Line Images

Figure 1 demonstrates the catadioptric camera and its corresponding images captured in a corridor environment. The image lines corresponding to 3D straight lines on the floor are seriously distorted by the camera fisheye lens. With a unified projection model [36], it is possible to approximate its imaging process and utilize some specific properties of the image line projected from the 3D line, which are desirable for robot IBVS-based corridor-following. Based on the conic fitting algorithm [37] and a modified RHT method on a sphere [38], the projective curves as well as their polar lines can be extracted, as shown in the right picture of Figure 1.

2.1. Line Image of Single 3D Line

As shown in Figure 2, a unified central projection model of a catadioptric camera has two frames, O and $O_c$, where O defines the center of a virtual unit sphere, and $O_c$ represents a new projection center with coordinates ${[0,0,-\xi ]}^T$ in relation to O, where $\xi$ represents the translation along the Z-axis between these two frames. According to the unified central projection model, a 3D straight line L will be sequentially projected into the unit sphere in relation to O, and then into a conic curve $\overline{\mathsf{\Omega }}$ in relation to $O_c$.

Let ${\mathcal{L}}_i$ be a pair of vectors defined as

$${\mathcal{L}}_i=\left\{{\mathbf{u}}_i,{\mathbf{n}}_i|\begin{array}{c}∥{\mathbf{u}}_i∥=1,\hfill \\ ∥{\mathbf{n}}_i∥=1,\hfill \\ {\mathbf{n}}_i·{\mathbf{u}}_i=0\hfill \end{array},i=1,\dots ,m\right\}$$

(1)

where ${\mathbf{u}}_i={\left(u_x,u_y,u_z\right)}^T$ denotes the normalized orientation of ${\mathcal{L}}_i$, and ${\mathbf{n}}_i={\left(n_x,n_y,n_z\right)}^T$ represents the unit normal vector of normalized plane ${\Pi }_i$, which is defined by ${\mathbf{u}}_i$ and center O, and ${\mathbf{n}}_i·{\mathbf{u}}_i=0$. Therefore, point $\mathbf{x}={\left(x,y,z\right)}^T$ lying on ${\mathcal{L}}_i$ is orthogonal to ${\mathbf{n}}_i$:

$$n_{x}x+n_{y}y+n_{z}z=0$$

(2)

Any line ${\mathcal{L}}_i$ on plane ${\Pi }_i$ projected to sphere center O will intersect the virtual unit sphere at arc $S_i$. It is then projected, in relation to $O_c$, into a conic curve ${\overline{\mathsf{\Omega }}}_i$ on the normalized plane ${\Pi }_{\infty }$, and any point $\overline{\mathbf{x}}$ of ${\overline{\mathsf{\Omega }}}_i$ verifies the following:

$${\overline{\mathrm{x}}}^T{\overline{\mathsf{\Omega }}}_i\overline{\mathrm{x}}=0$$

(3)

with

$${\overline{\mathsf{\Omega }}}_i=\left(\begin{array}{ccc}\alpha n_x^2-n_z^2{\xi }^2& \alpha n_x{n}_y& n_x{n}_z\\ \alpha n_x{n}_y& \alpha n_x^2-n_z^2{\xi }^2& n_y{n}_z\\ n_x{n}_z& n_y{n}_z& n_z^2\end{array}\right)$$

(4)

which describes the relationship between ${\overline{\mathsf{\Omega }}}_i$ and ${\mathbf{n}}_i$ of ${\mathcal{L}}_i$ given $\alpha =1-{\xi }^2$.

Finally, ${\overline{\mathsf{\Omega }}}_i$ is mapped into its image curve ${\widehat{\mathsf{\Omega }}}_i$ in the image plane by using affine transformation ${\mathbf{H}}_c$:

$${\widehat{\mathsf{\Omega }}}_i={\mathbf{H}}_c^{-T}{\overline{\mathsf{\Omega }}}_i{\mathbf{H}}_c^{-1}$$

(5)

where ${\mathbf{H}}_c$ is defined as follows:

$${\mathbf{H}}_c=\left(\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right)$$

(6)

by providing the parameters $f_x$, $f_y$, $c_x$, and $c_y$ of the camera, which can be estimated by using the camera calibration procedure [36].

2.2. Line Images of Two 3D Parallel Lines

Let $\widehat{\mathbf{O}}$ be the pole of ${\widehat{\mathsf{\Omega }}}_i$; then, its corresponding polar line ${\widehat{\mathsf{ß}}}_i$ can be defined as follows:

$${\widehat{\mathsf{ß}}}_i={\widehat{\mathsf{\Omega }}}_i·\widehat{\mathbf{O}}$$

(7)

Choose the projected center $\overline{\mathbf{O}}={\left(0,0,1\right)}^T$ in ${\Pi }_i$ as the pole of ${\overline{\mathsf{\Omega }}}_i$; from Equation (4), the polar line ${\overline{\mathsf{ß}}}_i$ of $\overline{\mathbf{O}}$ in relation to ${\overline{\mathsf{\Omega }}}_i$ is

$${\overline{\mathsf{ß}}}_i={\overline{\mathsf{\Omega }}}_i·\overline{\mathbf{O}}$$

(8)

which makes ${\overline{\mathsf{ß}}}_i={\left(n_{x,i},n_{y,i},n_{z,i}\right)}^T$ after some formula simplifications, so that point $\overline{\mathrm{x}}={\left(\overline{x},\overline{y},1\right)}^T$ on ${\overline{\mathsf{ß}}}_i$ satisfies the following equation:

$$n_{x,i}\overline{x}+n_{y,i}\overline{y}+n_{z,i}=0$$

(9)

The polar line ${\overline{\mathsf{ß}}}_i$ is actually the perspective projection of ${\mathcal{L}}_i$ in ${\Pi }_i$. In this way, it is able to represent the projection of a 3D line in a catadioptric camera system by a simple polar line. This projection property is used in our IBVS control law for robot corridor-following.

If two 3D parallel lines ${\mathcal{L}}_1$ and ${\mathcal{L}}_2$ are presented in the camera FOV, as shown in Figure 2, the intersection points $p_1$ and $p_2$ of two corresponding conic curves ${\widehat{\mathsf{\Omega }}}_1$ and ${\widehat{\mathsf{\Omega }}}_2$ and image center $\widehat{\mathbf{O}}$ should lie on the same line $\widehat{\mathbf{l}}$. Since ${\mathbf{H}}_c$ preserves incidence, three lines, ${\widehat{\mathsf{ß}}}_1$, ${\widehat{\mathsf{ß}}}_2$, and $\widehat{\mathbf{l}}$, will intersect at the same point $\widehat{\mathbf{D}}={\mathbf{H}}_c\overline{\mathbf{D}}$, and accordingly, ${\overline{\mathsf{ß}}}_1$, ${\overline{\mathsf{ß}}}_2$, and $\overline{\mathbf{l}}$ will also intersect at the same point $\overline{\mathbf{D}}$ in ${\Pi }_i$.

$\overline{\mathbf{D}}$ can be calculated by using Equation (9) for ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$:

$$\overline{\mathbf{D}}=\left(\begin{array}{c}{\overline{x}}_0\hfill \\ {\overline{y}}_0\hfill \end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{n_{z,2}{n}_{y,1}-n_{z,1}{n}_{y,2}}{n_{x,1}{n}_{y,2}-{n_{x,2}n}_{y,1}}}\hfill \\ {\displaystyle \frac{n_{z,2}{n}_{x,1}-n_{z,1}{n}_{x,2}}{n_{x,2}{n}_{y,1}-{n_{x,1}n}_{y,2}}}\hfill \end{array}\right)$$

(10)

Since ${\mathcal{L}}_1$ is parallel to ${\mathcal{L}}_2$, it can be proved that $n_{x,1}=n_{x,2}$, and

$${\overline{y}}_0={\displaystyle \frac{n_{z,2}-n_{z,1}}{n_{y,1}-n_{y,2}}}=tan\gamma$$

(11)

with the camera pitch angle $\gamma$ provided.

Observation reveals that ${\overline{y}}_0$ is only correlated with camera pitch angle $\gamma$ and is fixed if $\gamma$ is given. Therefore, it makes $\overline{\mathbf{D}}$ ($\widehat{\mathbf{D}}$) move along a horizontal line going through ${\overline{y}}_0$(${\widehat{y}}_0$). As shown in Figure 3, we obtain the following three properties:

• ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$ intersect at $\overline{\mathbf{D}}$, if $0<\gamma <\pi /2$.
• The polar lines ${\widehat{\mathsf{ß}}}_1$ and ${\widehat{\mathsf{ß}}}_2$ are two vertical parallel lines, and $\overline{\mathbf{D}}$ will be infinite, if $\gamma =\pi /2$ or $\gamma =-\pi /2$.
• ${\overline{\mathsf{\Omega }}}_1$ and ${\overline{\mathsf{\Omega }}}_1$ will degenerate into two straight lines so that $\overline{\mathbf{D}}=\overline{\mathbf{O}}$, if $\gamma =0$.

In Figure 4, we present the simulated line images and corresponding polar lines calculated from two horizontal 3D parallel lines ${\mathcal{L}}_1$ and ${\mathcal{L}}_2$ when the robot is placed in the corridor with different poses. The distance between ${\mathcal{L}}_1$ and ${\mathcal{L}}_2$ is L. The first three columns of Figure 4 demonstrate two projected line images represented by (blue and red) conic curves, (blue and red) polar lines in relation to pole $\widehat{\mathbf{O}}$ of two curves, and intersection $\widehat{\mathbf{D}}$ of two polar lines in the normalized image plane when the robot rotates in three directions ${\theta }_w\in \left(-{30}^{\circ },0^{\circ },{30}^{\circ }\right)$ at different positions in the corridor. In the top row, the robot is situated in the middle of the corridor; as it rotates from $-{30}^{\circ }$ to ${30}^{\circ }$, $\widehat{\mathbf{D}}$ translates from left to right, with y of $\widehat{\mathbf{D}}$ remaining unchanged, so that each image in the bottom row also has a similar observation. This proves the previous motion result of $\widehat{\mathbf{D}}$. Another point is when the robot is situated in the middle of ${\mathcal{L}}_1$ and ${\mathcal{L}}_2$, with ${\theta }_w=0^{\circ }$; the left and right conics and polar lines will be symmetric with respect to $\widehat{\mathbf{O}}\widehat{\mathbf{D}}$, respectively, while this is not the case in the bottom row when ${\theta }_w=0^{\circ }$ since there is a slight offset $0.2L$ from the middle point of corridor.

In Figure 5, a simulated robot with different poses is placed in front of two vertical parallel lines ${\mathcal{L}}_1$ and ${\mathcal{L}}_2$, which often happens during robot indoor navigation when it is going through a doorway. In this case, $\overline{\mathbf{D}}={\left(0,tan\gamma ,1\right)}^T$, which means that $\widehat{\mathbf{D}}$ will not change as the robot pose varies. However, the slope of the polar line and angle between two polar lines may change accordingly. A similar symmetry phenomenon can be observed when the robot’s orientation is ${\theta }_w=0^{\circ }$.

3. IBVS for Corridor-Following

The IBVS control approach is applied to a nonholonomic mobile robot equipped with an omnidirectional camera, and the involved frames, relevant variables, and constants are illustrated in Figure 6. Our IBVS method is used to control the robot to move towards the middle of the corridor and hold an orientation parallel to the side wall by only using the image features from two corresponding horizontal 3D lines, and additionally guide the robot to pass through the doorway with a little visual feature modification within a unified control scheme.

3.1. Modeling of Mobile Robot

The camera pitch angle $0<\gamma <\pi /2$. The camera center is ${(w,0,-l)}^T$ with respect to the robot center. The width of the corridor is L. The height of the camera is h. The transformation matrix $\mathbf{T}$ between the camera and robot frame is calculated by

$$\mathbf{T}=\left(\begin{array}{cccc}0& -1& 0& w\\ -sin\gamma & 0& -cos\gamma & 0\\ cos\gamma & 0& -sin\gamma & -l\end{array}\right)$$

(12)

The robot pose is represented by ${\mathbf{X}}_w={\left(x_w,y_w,{\theta }_w\right)}^T$, where ${\left(x_w,y_w\right)}^T$ depicts robot location, and ${\theta }_w\in (-\pi ,\pi ]$ represents robot orientation.

Let the control variable $\mathbf{u}={\left(v,\omega \right)}^T$; then, the nonholonomic robot kinematic can be modeled as

$${\dot{\mathbf{X}}}_w=\left(\begin{array}{c}{\dot{x}}_w\\ {\dot{y}}_w\\ {\dot{\theta }}_w\end{array}\right)=\left(\begin{array}{c}{sin\theta }_w\\ {cos\theta }_w\\ 0\end{array}\right)v+\left(\begin{array}{c}0\\ 0\\ 1\end{array}\right)\omega$$

(13)

3.2. Visual Features for Corridor Horizon Lines

$\overline{\mathbf{D}}$ of two polar lines ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$ is closely related to robot pose. As shown in Figure 7, ${\overline{\mathsf{ß}}}_1$, ${\overline{\mathsf{ß}}}_2$, and a horizon (red) line segment $\overline{l}$ passing through $\overline{\mathbf{O}}$ are chosen as three edges of a triangle in ${\Pi }_i$. Let ${\overline{\mathsf{ß}}}_0$ be the line segment $\overline{\mathbf{D}}{\mathbf{X}}_f$ passing through $\overline{\mathbf{D}}$, and ${\mathbf{X}}_f={(x_f,y_f)}^T$ be the median point of $\overline{l}$.

Rewrite Equation (9) for each ${\overline{\mathsf{ß}}}_i$ in a polar coordinate form:

$$cos{\theta }_i\overline{x}+sin{\theta }_i\overline{y}+{\rho }_i=0$$

(14)

where $i\in \{1,2\}$.

We then obtain the interior angles ${\theta }_l$ and ${\theta }_r$ of the triangle enclosed by two polar lines ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$. The polar angle ${\theta }_m$ can be calculated as follows:

$${\theta }_m=arctan\left({\displaystyle \frac{1}{2}}\left(cot{\theta }_l-cot{\theta }_r\right)\right)$$

(15)

Define the visual feature for IBVS as $\mathbf{s}={\left({\theta }_m,{\rho }_m\right)}^T$. $x_f$ can be calculated by the intersection with ${\overline{\mathsf{ß}}}_0$ since $n_{x,1}=n_{x,2}$,

$$x_f=-{\displaystyle \frac{n_{z,2}+n_{z,1}}{2n_{x,1}}}$$

(16)

The polar radius ${\rho }_m$ for $\overline{\mathbf{D}}{\mathbf{X}}_f$ can be represented as

$${\rho }_m=x_{f}cos{\theta }_m+y_{f}sin{\theta }_m$$

(17)

3.3. Visual Features for Doorway Vertical Lines

In Figure 5, $\overline{\mathbf{D}}$ is constant regardless of robot pose variation, and the chosen feature $\mathbf{s}={\left({\theta }_m,{\rho }_m\right)}^T$ will degenerate into one feature.

$$x_f={\overline{\mathbf{D}}}_{x}tan{\theta }_m=tan\gamma tan{\theta }_m$$

(18)

Therefore, as shown in Figure 8, besides two polar lines ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$ of two vertical door frames, we introduce a third image conic ${\overline{\Omega }}_3$ projected from the upper door frame, and let middle point $X_f$ lie on the polar line ${\overline{\mathsf{ß}}}_3$ of ${\overline{\Omega }}_3$; then, the polar radius ${\rho }_m$ for ${\overline{\mathsf{ß}}}_3$ can be calculated by

$${\rho }_m=x_{f}sin{\theta }_m+y_{f}cos{\theta }_m$$

(19)

3.4. IBVS Control Law

Given median line $\overline{\mathbf{D}}{\mathbf{X}}_f$ and polar line ${\overline{\mathsf{ß}}}_3$ in ${\Pi }_i$, interaction matrix $\mathbf{L}$ is theoretically feasible and can be simplified as

$$\mathbf{L}=\left(\begin{array}{cc}\mathbf{A}\hfill & \mathbf{B}\hfill \end{array}\right)$$

(20)

where

$$\mathbf{A}={\displaystyle \frac{1}{h}}\left(\begin{array}{ccc}{\lambda }_{\theta }cos{\theta }_m\hfill & {\lambda }_{\theta }sin{\theta }_m\hfill & {-\lambda }_{\theta }{\rho }_m\hfill \\ {\lambda }_{\rho }cos{\theta }_m\hfill & {\lambda }_{\rho }sin{\theta }_m\hfill & {-\lambda }_{\rho }{\rho }_m\hfill \end{array}\right)$$

$$\mathbf{B}=\left(\begin{array}{ccc}-{\rho }_{m}cos{\theta }_m& -{\rho }_{m}sin{\theta }_m& -1\\ \left({\rho }_m^2+1\right)sin{\theta }_m& -\left({\rho }_m^2+1\right)cos{\theta }_m& 0\end{array}\right)$$

with ${\lambda }_{\theta }=\sqrt{{\rho }_m^2+1}/{\rho }_m·sin{\theta }_{m}sin\gamma$ and ${\lambda }_{\rho }=-\sqrt{{\rho }_m^2+1}cos{\theta }_{m}sin\gamma$ [32].

In this way, $\mathbf{L}$ can be employed universally for horizontal or vertical lines.

The error function $\mathbf{e}$ based on visual features can be defined as follows:

$$\mathbf{e}=\left(\mathbf{s}-{\mathbf{s}}^{*}\right)$$

(21)

where ${\mathbf{s}}^{*}$ is the desire value of visual feature $\mathbf{s}$.

Since the target 3D lines remain static as the camera moves, we may define the visual feature dynamics as

$$\dot{\mathbf{s}}=\mathbf{L}{\mathsf{\varnothing }}_c$$

(22)

where $\mathbf{L}$ links the variation in visual feature $\mathbf{s}$ and camera velocity ${\mathsf{\varnothing }}_c={\left(v_c,{\omega }_c\right)}^T$. The actual robot velocity $\mathsf{\varnothing }$ in the camera frame can be expressed as

$${\mathsf{\varnothing }}_c={\mathbf{T}}_R^C\mathsf{\varnothing }$$

(23)

where ${\mathbf{T}}_R^C$ represents transformation from robot frame to camera frame

$${\mathbf{T}}_R^C={\left(\begin{array}{cccccc}0& -sin\gamma & cos\gamma & 0& 0& 0\\ -l& 0& -w& 0& -cos\gamma & -sin\gamma \end{array}\right)}^T$$

(24)

By injecting Equations (23) and (24) into Equation (22), we obtain

$$\stackrel{.}{\mathbf{s}}={\mathbf{J}}_{v}v+{\mathbf{J}}_{\omega }\omega$$

(25)

where

$${\mathbf{J}}_v=-{\displaystyle \frac{1}{h}}\left(\begin{array}{c}{\lambda }_{\theta }sin{\theta }_{m}sin\gamma +{\lambda }_{\theta }{\rho }_{m}cos\gamma \hfill \\ {\lambda }_{\rho }sin{\theta }_{m}sin\gamma +{\lambda }_{\rho }{\rho }_{m}cos\gamma \hfill \end{array}\right)$$

and

$${\mathbf{J}}_{\omega }=\left(\begin{array}{c}-{\displaystyle \frac{{\lambda }_{\theta }cos{\theta }_{m}l}{h}}+{\displaystyle \frac{{\lambda }_{\theta }{\rho }_{m}w}{h}}+{\rho }_{m}sin{\theta }_{m}cos\gamma +sin\gamma \hfill \\ -{\displaystyle \frac{{\lambda }_{\rho }cos{\theta }_{m}l}{h}}+{\displaystyle \frac{{\lambda }_{\rho }{\rho }_{m}w}{h}}+\left({\rho }_m^2+1\right)cos{\theta }_{m}cos\gamma \hfill \end{array}\right)$$

(26)

If an exponential decay of $\mathbf{e}$ is required based on $\dot{\mathbf{e}}=\lambda \mathbf{e}$, according to Equation (25), Equation (25) can be rewritten as

$$-\lambda \mathbf{e}={\mathbf{J}}_{v}v+{\mathbf{J}}_{\omega }\omega$$

(27)

Combining Equations (21) and (27), assuming the robot moves at a constant driving speed $v=v^{*}$, we then obtain the IBVS control law for both corridor-following and doorway approaching based on monocular omnidirectional vision

$$\omega =-{{\mathbf{J}}_{\omega }}^{+}\left(\lambda \mathbf{e}+{\mathbf{J}}_v{v}^{*}\right)$$

(28)

3.5. Convergency Analysis

In this section, we present the stability analysis of the closed-loop visual servo systems. Consider the convergence of the control law for corridor-following; we define the Lyapunov function $\mathcal{L}={\displaystyle \frac{1}{2}}{\parallel \mathbf{e}\left(t\right)\parallel }^2$, and its corresponding derivative is

$$\dot{\mathcal{L}}={\mathbf{e}}^T\dot{\mathbf{s}}$$

(29)

From Equations (25) and (28), Equation (29) is rewritten as

$$\dot{\mathcal{L}}={\mathbf{e}}^T\left({\mathbf{J}}_v{v}^{*}-{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}(\lambda \mathbf{e}+{\mathbf{J}}_v{v}^{*})\right)$$

(30)

Simplifying Equation (30), we have

$$\dot{\mathcal{L}}=-\lambda {\mathbf{e}}^T{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}\mathbf{e}+{\mathbf{e}}^T({\mathbf{J}}_v-{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}{\mathbf{J}}_v)v^{*}$$

(31)

Therefore, the sufficient condition for the local asymptotic stability of such a nonlinear system is $\dot{\mathcal{L}}<0$ when $\mathbf{s}\to {\mathbf{s}}^{*}$,

$${\mathbf{e}}^T({\mathbf{J}}_v-{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}{\mathbf{J}}_v)v^{*}<\lambda {\mathbf{e}}^T{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}\mathbf{e}$$

(32)

3.5.1. Corridor-Following Framework

For a corridor-following task, as the robot approaches the corridor center line, $\mathbf{s}\to {\mathbf{s}}^{*}$, then $x_f\to 0,{\theta }_m\to 0$, and ${\rho }_m\to 0$, and according to the Law of Sines and Equation (11), ${lim}_{{\rho }_m\to 0,{\theta }_m\to 0}{\displaystyle \frac{sin{\theta }_m}{{\rho }_m}}=cot\gamma$, we then obtain

$$\underset{\mathbf{s}\to \mathbf{s}*}{lim}{\lambda }_{\theta }=cos\gamma$$

(33)

and

$$\underset{\mathbf{s}\to \mathbf{s}*}{lim}{\lambda }_{\rho }=-sin\gamma$$

(34)

When $\mathbf{s}\to {\mathbf{s}}^{*}$, ${\widehat{\mathbf{J}}}_{\omega }$ can be estimated by

$$\begin{array}{c}\hfill \underset{\mathbf{s}\to \mathbf{s}*}{lim}{\widehat{\mathbf{J}}}_{\omega }={\mathcal{K}}_1·\mathrm{Y}\end{array}$$

(35)

where ${\mathcal{K}}_1=\left(\begin{array}{cc}1& -l/h\\ l/h& 1\end{array}\right)$ and $\mathrm{Y}={(sin\gamma ,cos\gamma )}^T$

Since ${\mathcal{K}}_1>0$ and $Ker{\mathbf{J}}_{\omega }^T=Ker{\mathbf{J}}_{\omega }^{+}$, and suppose $\mathbf{e}={(\delta v,\delta \omega )}^T\ne 0$, then

$$\underset{\mathbf{s}\to {\mathbf{s}}^{*}}{lim}{\mathbf{J}}_{\omega }^T\mathbf{e}=\parallel \mathbf{e}\parallel sin(\gamma -arctan\left({\displaystyle \frac{l}{h}}\right)+artan\left({\displaystyle \frac{\delta \omega }{\delta v}}\right))$$

(36)

Suppose that the robot motion curvature $c\left(t\right)={\displaystyle \frac{\omega }{v}}$ does not change much as it approaches the corridor centre line; this leads to ${lim}_{\mathbf{s}\to {\mathbf{s}}^{*}}\dot{c}\left(t\right)=0$, and subsequently ${\displaystyle \frac{\delta \omega }{\delta v}}={\displaystyle \frac{\omega }{v}}$.

Therefore, $\exists \mathbf{e}\ne 0$, and ${lim}_{\mathbf{s}\to {\mathbf{s}}^{*}}{\mathbf{J}}_{\omega }^T\mathbf{e}\ne 0$ equals the constrain of $c\left(t\right)$, which should satisfy the following inequality:

$$c\left(t\right)={\displaystyle \frac{\omega }{v}}\ne {\displaystyle \frac{l-htan\left(\gamma \right)}{h+ltan\left(\gamma \right)}}$$

(37)

3.5.2. Doorway-Passing

For a doorway approaching task, as the robot approaches doorway center, $\mathbf{s}\to {\mathbf{s}}^{*}$, then $x_f\to 0,{\theta }_m\to 0$ and ${\rho }_m\to \parallel y_f\parallel$, which makes

$$\underset{\mathbf{s}\to \mathbf{s}*}{lim}{\lambda }_{\theta }=0$$

(38)

and

$$\underset{\mathbf{s}\to \mathbf{s}*}{lim}{\lambda }_{\rho }=-\sqrt{{\rho }_m^2+1}sin\gamma$$

(39)

When $\mathbf{s}\to {\mathbf{s}}^{*}$, ${\widehat{\mathbf{J}}}_{\omega }$ can be estimated by

$$\begin{array}{c}\hfill \underset{\mathbf{s}\to \mathbf{s}*}{lim}{\widehat{\mathbf{J}}}_{\omega }={\mathcal{K}}_2·\mathrm{Y}\end{array}$$

(40)

where ${\mathcal{K}}_2=\left(\begin{array}{cc}1& 0\\ \sqrt{{\rho }_m^2+1}(l-{\rho }_{m}w)/h& 1+{\rho }^2\end{array}\right)$ and $\mathrm{Y}={(sin\gamma ,cos\gamma )}^T$.

Since ${\mathcal{K}}_2>0$ and $Ker{\mathbf{J}}_{\omega }^T=Ker{\mathbf{J}}_{\omega }^{+}$, and suppose $\mathbf{e}={(\delta v,\delta \omega )}^T\ne 0$, then

$$\underset{\mathbf{s}\to {\mathbf{s}}^{*}}{lim}{\mathbf{J}}_{\omega }^T\mathbf{e}=(1+\kappa )sin\gamma \delta v+\iota cos\gamma \delta \omega$$

(41)

where $\kappa =\sqrt{1+{\rho }_m^2}(l-{\rho }_{m}w)/h$, and $\iota =1+{\rho }_m^2$.

Therefore, $\exists \mathbf{e}\ne 0$, and ${lim}_{\mathbf{s}\to {\mathbf{s}}^{*}}{\mathbf{J}}_{\omega }^T\mathbf{e}\ne 0$ equals the constrain of $c\left(t\right)$, which should satisfy the following inequality:

$$c\left(t\right)={\displaystyle \frac{\omega }{v}}\ne -tan\gamma {\displaystyle \frac{1+\kappa }{\iota }}$$

(42)

To sum up, for our robot system, if these preconditions are kept during robot motion, it is then reasonable to assume that $\mathbf{e}\notin Ker{\mathbf{J}}_{\omega }^{+}$, which means ${\mathbf{J}}_{\omega }^{+}\mathbf{e}$ is always non-null and ${\mathbf{J}}_{\omega }^T\mathbf{e}\ne 0$.

Therefore, we have $e^T{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}\mathbf{e}={\left({\mathbf{J}}_{\omega }^T\mathbf{e}\right)}^2/\left({\mathbf{J}}_{\omega }^T{\mathbf{J}}_{\omega }\right)>0$.

Since $v^{*}>0$, and ${\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}\ne \mathbf{I}$, Equation (31) can be rewritten as

$${\displaystyle \frac{{\mathbf{e}}^T({\mathbf{J}}_v-{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}{\mathbf{J}}_v)}{{\mathbf{e}}^T{\mathbf{J}}_{\omega }{\mathbf{J}}_{\omega }^{+}\mathbf{e}}}<{\displaystyle \frac{\lambda }{v^{*}}}$$

(43)

Equation (43) provides a sufficient condition for the convergence of the closed-loop system at $\mathbf{e}=0$. It can be seen that parameters may affect the convergence of the control law and this lower bound is necessary for the stability of this system. We will present some simulation results on $\lambda$ under a corridor environment in the next section.

4. Simulation Results

In Figure 9, a block diagram of our IBVS system is presented. The system is implemented under the ROS framework with two functional nodes. The first is the Image Processing Node, which receives environmental images captured by a panoramic camera. It extracts image features through a sequence of operations, including edge detection, line fitting, and an improved stochastic Randomized Hough Transform (sRHT). More implementation details are provided in the experimental section. The second is the IBVS Control Node, which takes the extracted image features as input to a predefined IBVS control law. This law computes the robot’s control commands (e.g., motion velocity or steering angle), which are then transmitted to the underlying mobile robot for execution.

This method follows the classical two-stage visual servoing pipeline: feature detection followed by image-feature-error-based control. Compared with recent learning-based end-to-end CNN approaches, the proposed method has limited adaptability to nonholonomic constraints and dynamic environmental disturbances. However, under typical task accuracy requirements, it offers an intuitive and computationally efficient solution with controllable computational cost, making it well suited for rapid deployment.

4.1. Simulation for Corridor-Following

In the first virtual scenario, the robot is placed inside a corridor (width $L=4$ m). The floor lines are represented by two parallel 3D lines: $\left\{x=-2,y=t,z=0\right\}$ and $\left\{x=2,y=t,z=0\right\}$. These lines are projected into the image plane through a simulated catadioptric camera. The image size is $1280\times 1024$ pixels, with an image center at $c_x=683,c_y=511$ and focal lengths $f_x=f_y=400$. Other involved constants are as follows: $h=1$ m, $w=0.1$ m, $l=0.2$ m, and $\gamma ={45}^{\circ }$. The robot’s linear velocity is set to $v=0.2$ m/s. The robot’s initial pose is $y_w=1$ m, ${\theta }_w=0^{\circ }$. The constant $\lambda$ in Equation (27) is set to 1.

The navigation task is to guide the robot to move along the middle of the corridor and keep it heading along the corridor based on our control law. To validate the stability of the proposed control law, an intrinsic matrix is calibrated with errors ($\pm 10\%$ on $f_x$ and $f_y$; $\pm 5$ pixels on $c_x$ and $c_y$), and visual features are computed through image measurements; we assume that the conic curves are detected and tracked using the method proposed with added white Gaussian noise (standard deviation $\sigma =2$).

Figure 10a shows the robot trajectory based on our visual servoing method. It can be seen that the robot finally reaches the middle of the corridor, and its heading direction is parallel to the reference 3D lines. Figure 10b,c present the evolution of two visual features, $x_f$ and ${\theta }_m$, respectively. Figure 10d–f show the evolution of the robot trajectory parameters, such as lateral distance $y_w$, heading angle ${\theta }_{\mathrm{w}}$, and angular velocity w, respectively. From these figures, angle and lateral deviations are well regulated to zero; although there is a kind of low-frequency oscillation caused by the errors of parameters, it has little influence on the convergence result, which confirms the validity of the control law.

Figure 11 presents some visual features’ trajectories simulated during corridor-following in Figure 10, such as conic features in image (Figure 11a,b), polar lines (Figure 11c,d), the median line (Figure 11e), and the trajectories of polar features in the polar coordinates (Figure 11f), with noise added for clarity. The red curves, points, and lines represent the initial features, while the blue curves, points, and lines are the features when the robot arrives at a steady state, and the green ones are intermediate transitions at every control frame. It can be seen from these figures that conic and polar curves tend to converge towards the final target curve. In Figure 11f, the feature point $\left({\rho }_m,{\theta }_m\right)$ trajectory of ${\overline{\mathsf{ß}}}_1$, ${\overline{\mathsf{ß}}}_2$ and ${\overline{\mathsf{ß}}}_0$ in the parameter space are present, the feature point trajectory of the median line ${\overline{\mathsf{ß}}}_0$ moves to the final steady state $\left(0,0\right)$, while ${\overline{\mathsf{ß}}}_1$ and ${\overline{\mathsf{ß}}}_2$ move to the final steady state $\left(0.81,-0.62\right)$ and $\left(0.81,0.62\right)$, respectively. This reveals the effectiveness of our method, since the feature value of the steady state for ${\overline{\mathsf{ß}}}_1$ or ${\overline{\mathsf{ß}}}_2$ is not as easy to compute as for ${\overline{\mathsf{ß}}}_0$.

4.2. Anti-Noise Capability

In this simulation, different levels of Gaussian noise are added to the image to verify the validity of our control law. The initial robot poses are shown in

Table 1. Initial robot poses in Figure 12.

State	①	②	③	④	⑤	⑥
${\theta }_w$ (degree)	0	30	60	80	60	30
$y_w$ (m)	−1.5	−1	−0.5	0	0.75	1.5

.

Figure 12 shows trajectories that start from different initial poses with or without different levels of image white Gaussian noise. In this simulation, $\sigma =5,10,15$ is applied to image feature detection, respectively. The solid color lines refer to the cases without noise, while the corresponding dotted lines refer to cases with Gaussian noise.

It can be seen that the trajectories eventually converge in all situations. As the deviation of noise level increases, the trajectory shows a slight deviation from the corresponding real one. Let $\epsilon$ be the absolute error around the middle line; we may obtain the MSE of the relative position distance $y_0$ along the median line of the corridor, as shown in

Table 2. Convergence error of relative position $y_0$ in the corridor under different levels of image noise.

Noise Level $\mathit{\sigma }$ (Pixel)	MSE $\mathit{\epsilon }$ of Steady State (cm)
5	$-0.0308\pm 0.0003$
10	$0.2741\pm 0.1068$
15	$-1.3441\pm 0.3639$
20	$1.0792\pm 1.2669$

. The results reveal that the means and standard deviations of $\epsilon$ generally increase as the image noise rises, except the means when $\sigma =10,15$. However, the standard deviation of $\epsilon$ when $\sigma =15$ is larger than $\sigma =10$, and the mean value is much more uncertain when $\sigma =15$.

4.3. Convergence of Control Law

In our IBVS method, the lower bound of $\lambda$ allows the convergence of our control law. To test the influence of parameter $\lambda$ on the control law in Equation (28), as shown in Figure 13, we present a simulation in which different values of parameter $\lambda \in \{0.1,0.2,0.5,1,2,5\}$. We keep the robot with the same external conditions, including initial pose, robot velocity, camera parameters, and image noise, and then the convergence speed only depends on $\lambda$. The robot trajectory with different $\lambda$ reveals that the increase in $\lambda$ makes it tend to steady state more smoothly and quickly. It can be seen that $\lambda <0.5$ may lead to the overshoot of the system, $\lambda =5$ and $\lambda =2$ nearly coincide, and the rise time is a bit longer than $\lambda =1$. Therefore, in our IBVS system, $\lambda =1$ is good enough for our corridor-following task.

4.4. Simulation for Doorway Approaching

In this part, simulations for doorway approaching based on our IBVS method are presented. This situation often occurs when the robot plans to pass a door anonymously by using door frame features. Similar to the corridor-following task, a simulated robot is place in front of a doorway with a random initial pose, and our IBVS is used to guide the robot to approach the middle point of the doorway as far as possible.

As shown in Figure 14a, the doorway in our simulation is represented by three door frames, two vertical 3D parallel lines (red points) and an upper horizontal door frame that connects these two points; the parameter equations of the vertical lines are $\left\{x=0,y=0.5,z=1\right\}$ and $\left\{x=0,y=-0.5,z=1\right\}$, respectively. Other parameters for the IBVS algorithm are the same as for the previous corridor-following task, and some initial robot states are shown in

Table 3. Initial robot state and final average error for each trajectory in Figure 14a.

State	①	②	③	④	⑤	⑥
${\theta }_w$ (degree)	0	30	60	80	60	30
$y_w$ (m)	−1.5	−1	−0.5	0	0.75	1.5
$x_w$ (m)	1	0.5	−0.25	−0.5	−0.5	−1
Error (cm)	4.02	0.70	0.11	0.19	0.23	1.89

. By using these three lines, it is possible for a robot with different given initial poses to approach the target door center. The final error with respect to the middle of the door for each trajectory is presented in the last row of Table 3, and the deviation may be closely related to the given initial $y_w$ since the errors from ① or ⑥ are comparatively larger than those from other initial states (②, ③, ④, or ⑤).

To analyze whether or not a third door frame is necessary in addition to two vertical frames, in Figure 14b, we present two IBVS-based robot motion trajectories by using $X_f$ from two polar lines (Figure 8) and three polar lines (Figure 8), respectively. It can be seen that the former trajectory converges much faster and stops much closer to door center than the latter, which reveals the feasibility of employing a third horizontal door frame. In Figure 14c, we also present the change in three polar lines gathered from the trajectory in Figure 14; feature $X_f$ goes from the initial position to the final target, and the initial acute triangle formed by the three polar lines becomes an obtuse triangle, and almost degenerates to a line segment eventually. Notice that, in front of the door in Figure 14b, the corresponding triangle obtained at position P on this trajectory is marked in red. We will explain it later in the experimental part.

4.5. Simulation in Closed-Loop Corridor

To further validate the effectiveness of our method, we present a closed-loop square corridor environment, which involves several parts of the corridor, including the corners. As shown in Figure 15a, the robot starts from the initial pose and follows four different parts of the corridor, including the straight corridor and corner areas, to complete all the navigation tasks by using our IBVS method. The constants and parameters of the robot are the same as in the above simulations.

Figure 15b shows a magnified part of the region overlapped with a dashed box in Figure 15a. It demonstrates how the robot moves from the initial pose to the middle of the corridor and realizes a smooth transition between two adjacent pairs of corridor lines. As the robot approaches the corner from a certain distance, two parallel lines on the floor will provide less information for IBVS of the robot; in this case, the vertical boundary lines in the corridor are selected as the new candidate target 3D lines for our IBVS when the mobile robot is approaching the corner, as shown in Figure 15b. The robot trajectories are labeled with different colors, and between the two green ones, we demonstrate a smooth transition trajectory generated by using our IBVS method with two vertical 3D lines. Until the robot leaves the corner and two vertical boundary lines get out of the field of view, it selects two floor horizontal lines in the subsequent corridor again. This simulation result validates that the control law may also perform well in a closed-loop environment; even when different features are adopted, the path generated by our IBVS framework satisfies the corridor-following navigation task.

5. Experimental Results

In this section, we demonstrate two experimental results by using our IBVS algorithm on a real robot platform. We introduce the robot platform and the corresponding image processing procedure. Then, a corridor-following experiment and doorway approaching experiment are presented to show the effectiveness of our approach.

5.1. Mobile Robot Platform

We choose a low-cost mobile robot, TurtleBot2, as our experimental platform to verify the proposed IBVS approach in two indoor environments, corridor and doorway, respectively, as shown in Figure 6. The robot is equipped with a single Dalsa^® Genie^® camera CR-GM00-H1402 with resolution $1400\times 1024$, and we use Fujinon^® FE185C057HA-1 with focus length 1.8 mm, which is a super wide-angle lens with a ${185}^{\circ }$ FOV. The image center is $c_x=699.8,c_y=512.5,f_x=390,f_y=389.7$. The onboard computer is ThinkPad P50 (20ENCTO1WW, Lenovo, Made in China) with Intel^® Xeon^® E3-1505M v5 2.80 GHz CPU and 16 G memory.

5.2. IBVS System

We developed our IBVS system under a ROS (Robot Operating System) by using Visual Studio Code on Ubuntu 16.04. The system integrates two additional nodes, the image acquisition node and IBVS node, respectively. As shown in Figure 16a, the former node grabs the catadioptric image from camera. The latter will subsequently detect the edge pixels by using a Canny edge detector (Figure 16b), and then adopt a modified line image detection algorithm based on [5] to generate the control value for robot motion. In this approach, all the edge points are inversely projected onto a unit sphere, and the target projected line image pixels will lie along the corresponding two great circles on the sphere, as shown in Figure 16c; we then use a Randomized Hough Transform (RHT) to cluster the great circles and generate the candidate great circle (red one) for each target spatial line, as shown in Figure 16d. In Figure 16e, we present two detected line images overlapped by red curve segments for two reference lines on the ground. As shown in Figure 16f, these two segments are fitted into two conics based on the research [37], and the corresponding polar lines are calculated for point $X_f$, which are then used to control robot motion; the whole control cycle of our IBVS system is about 10 Hz.

5.3. Experimental Results for Corridor-Following

In this experiment, we present the corridor-following performance of our IBVS method by using two horizontal parallel lines. The width of the corridor is about 3.7 m, and a surveillance camera is placed on the roof of the corridor to record the robot trajectory, as shown in Figure 16. A Genie^® camera is fixed on the aluminum profile, which is installed at the center of the robot top tray; it makes $\gamma =45$, and the parameters related to the configuration of the camera are $h=1.0$ m, $w=0.001$ m, and $l=0.32$ m.

As shown in Figure 17, we choose two-sided black lines of the corridor as reference parallel lines to control robot motion. The robot is firstly placed at point (a) with initial pose $y_w=1$ m, ${\theta }_w=0$, and then it is gradually guided towards the middle line of the corridor by our IBVS algorithm. We demonstrate this trajectory marked by a red dotted line, and the translucent robots overlapping on this path from (b) to (e) are some recorded robot images for clarity. It can be seen that this trajectory is similar to the previous simulation result; the robot approximately reaches the middle line and the heading direction is nearly parallel to the reference line.

As shown in Figure 18, we present visual feature curves $X_f$ for the IBVS controller and the corresponding controller angular velocity output $\omega$. The two curves of the whole control procedure are accompanied by a small range of noise, and $\omega$ experiences a damped oscillation procedure; however, as a whole, $X_f$ and $\omega$ go towards 0 gradually.

In Figure 19, we illustrate the processed images with detected curve segments (red curves) at positions from (a) to (f) in Figure 17. Observations of detected curves reveal that these two curves may change from initial deflection to approximate symmetry with respect to $x=c_x$ in the image. We also demonstrate the changes in left and right conics for the reference corridor lines (in Figure 19g,h) and their corresponding polar lines (in Figure 19i,j) in the image. The red curves in each figure are initial image features at (a) in Figure 17, the blue curves represent the image features when robot arrives at (f) in Figure 17, and the green ones correspond to the feature transactions from the initial state to final state. It can be seen that, in final state (f), the left and right image features tend to be symmetrical with respect to $x=c_x$ as the robot moves to the middle of the corridor.

Our IBVS system is based on the ROS framework, which is not a real-time system, and we do not optimize the image processing part of the IBVS node to deal with environmental disturbance, so that observation noise and a processing delay will inevitably be introduced. Therefore, in practice, it can be seen from Figure 18 that curves corresponding to $X_f$ and $\omega$ are accompanied by obvious disturbance, and the convergence speed is slow. However, our IBVS node still generates a damped oscillation output $\omega$ and manages to guide the robot approximately to the middle of the corridor.

5.4. Experimental Results for Doorway-Passing

In this part, we conduct doorway approaching experiment, as shown in Figure 20. To further validate the simulation, we choose a door with the same height and width as the previous simulation setup. Hence, the target 3D lines include two vertical door frames and an upper door frame. We also use a surveillance camera to record the robot motion from the top, as shown in Figure 20a; the robot motion trajectory is presented, like in the previous experiment of corridor-following, and the initial robot state marked by (1) is close to ⑥ in Figure 14a, and the stop point (4) is close to point P in Figure 14a. From Figure 20b–e, we demonstrate three door frames detected in the image grabbed from positions (1) to (4), respectively. The triangle containing the three polar lines of the door frame conics is shown in Figure 20c. It can be seen that the shape change of the triangle is similar to Figure 14c, from an initial acute triangle to an obtuse one. In this experiment, the shape of the final triangle is also similar to the triangle shape at P in Figure 14b; in addition, there are some outlier points around the trajectory of feature $X_f$ because of the instability of image feature extraction. However, it seems not to affect the convergence and further verifies the extensibility of our IBVS method from the corridor-following task to doorway approaching task to some extent.

5.5. Experimental Results for Contrast

To demonstrate the superior performance of our proposed IBVS control method, we conduct comparative experiments using a recently published study [39] as a reference. This reference introduces a state-of-the-art panoramic vision-based IBVS path-tracking approach for mobile robots, which plans a desired trajectory in the image plane and executes visual servo control accordingly. Both methods are designed for IBVS control using panoramic cameras and therefore share common characteristics.

To quantitatively compare the path-tracking performance of the two methods, we implement simulation experiments using a unicycle-type differential-drive model under nonholonomic constraints. The setup is defined as follows: (1) The model state is denoted as $(x,y,\theta )$, where $x,y$ represent the planar position and $\theta$ denotes the heading angle. (2) The robot is required to follow a straight-line path defined by $y=0$. (3) The initial state includes both lateral and angular deviations: the robot starts 2 m off the desired path with an initial heading error of ${30}^{\circ }$. Both methods are assigned the same maximum forward velocity and initialized under identical conditions. We record the lateral deviation (path error) over time and the convergence time required for the model to achieve stable tracking, which serve as metrics to evaluate control accuracy and responsiveness.

As shown in Figure 21a, the error curves for both methods under the same path-tracking task reveal that our method reduces the error at a faster rate, reaching approximately one-quarter of the initial error within 5 s. Moreover, the overall error remains consistently lower than that of the reference method throughout the experiment. This indicates that our IBVS controller yields a smaller path deviation and higher tracking precision during convergence. The corresponding motion trajectories in the plane (top-down view) are illustrated in Figure 21b. Compared to the reference method, our approach executes a sharper turning maneuver, allowing the robot to merge back onto the target line within a shorter distance. Combining the error curves and trajectory plots, we conclude that our proposed method achieves smaller path deviations and faster convergence while ensuring stability, thus exhibiting superior control performance.

6. Conclusions

In this paper, we propose a novel IBVS-based indoor navigation control for a nonholonomic mobile robot using a single omnidirectional camera (Lens FUJINON FE185C057HA-1, camera Dalsa Genie HM1400). Based on the unified spherical projection properties, we define the line images of 3D parallel lines. The corresponding visual features in normalized space will naturally be zero when the robot is following the middle line of the corridor, which is desirable for our IBVS system since the target image features are provided implicitly. Therefore, the proposed method is applicable to the corridor-following task in an indoor environment. We present the control law, stability analysis, and further extension to the doorway approaching task by introducing an additional door frame feature. Several simulations on corridor-following, stabilization, and the convergence of the control law under different conditions are used to validate the proposed method. Our approach is also simulated for a doorway approaching task and a closed-loop corridor environment. In addition, we develop a ROS-based IBVS system on a real nonholonomic mobile robot platform, and conduct corridor-following and doorway approaching experiments. All the simulation and experimental results verify the effectiveness and practicability of our proposed IBVS method. The proposed approach is not only suitable for indoor environment navigation, but it has a great significance for intelligent visual control of robot autonomous rescue in indoor fire scenarios.

It is worth noting that the present study focuses solely on goal-directed tasks and does not address the issue of obstacle avoidance. Moreover, the simulation environment we constructed is relatively idealized. In practical applications, the system is subject to significant environmental disturbances, particularly in terms of scene texture variability and lighting conditions. These factors can adversely affect the stability and accuracy of critical tasks such as image line extraction of 3D parallel lines and edge detection. In scenarios involving directional failure, the reliability of robot trajectory data and motion stability also remains to be further improved. In future work, we aim to enhance the generalizability, real-time performance, convergence speed, and overall stability of the proposed IBVS approach under various types of noise and even in the presence of obstacles. This will involve further efforts in several key areas, including robustness against environmental disturbances, advanced image processing and dynamic feature capture, motion control optimization, and disturbance rejection.