Mobile Robot Navigation Based on Embedded Computer Vision

Abstract

The current computational advance allows the development of technological solutions using tools, such as mobile robots and programmable electronic systems. We present a design that integrates the Khepera IV mobile robot with an NVIDIA Jetson Xavier NX board. This system executes an algorithm for navigation control based on computer vision and the use of a model for object detection. Among the functionalities that this integration adds to the Khepera IV in generating guided driving are trajectory tracking for safe navigation and the detection of traffic signs for decision-making. We built a robotic platform to test the system in real time. We also compared it with a digital model of the Khepera IV in the CoppeliaSim simulator. The navigation control results show significant improvements over previous works. This is evident in both the maximum navigation speed and the hit rate of the traffic sign detection system. We also analyzed the navigation control, which achieved an average success rate of $93\%$. The architecture allows testing new control techniques or algorithms based on Python, facilitating future improvements.

1. Introduction

Process automation is made possible by technological tools, such as robots and embedded systems. Robots are automated machines capable of performing tasks with a certain degree of autonomy or under human supervision. Embedded systems, on the other hand, are electronic systems programmed to perform one or more specific tasks, adding functionality to other systems. Both tools contribute to the creation of technological solutions in important sectors, such as agriculture, manufacturing, and domestic assistance [1,2,3,4]. In this sense, they seek to replace human intervention in repetitive or heavy tasks, increasing the productivity (tasks solved in less time) and quality (less error in the expected results) of the processes. The same occurs in bio-inspired mobile robots, where the use of artificial neural networks and centralized patterns is applicable [5,6].

There are different devices for the locomotion of a mobile robot; among these are the wheels, the tracks, and the articulated legs [7,8,9]. The differential drive mobile robot (DDMR) is the easiest to implement and control; it includes two motorized wheels (one on each side of its base) and a freewheel as a support point (in the front part of its base). This configuration causes the robot’s movement and orientation to change due to the contribution of the speeds of both wheels, allowing it to move on smooth surfaces with good energy efficiency [10,11]. Its applications extend from academia, for the development and testing of navigation algorithms, to industrial environments for raw materials transport. The above requires that the internal controller of the DDMR has computational resources for the following functions: perception (sensor reading), planning (use of algorithms), and execution (actuators control) [12,13].

Navigation control is necessary for planning [14]. It is an algorithm executed by the controller in charge of manipulating the actuators (in real time) for the safe movement of the mobile robot. It can use intelligent control techniques to balance the response speed and computational cost [15]. The navigation control intelligence determines the mobile robot autonomy to explore scenarios, either previously established (with walls or lines on the ground) or unknown (in open field). In addition, for the application of the DDMR control, functions such as following trajectories, avoiding obstacles, or moving objects may be required [16]. Therefore, we study these topics: (a) configurations that enable the controller to interpret its environment from sensor information, (b) algorithms that enhance controller intelligence in decision making, and (c) technologies that add functionality and reduce controller workload [17,18,19].

Several controllers used for autonomous navigation are mentioned in the literature, among which PID control, model predictive control, and fuzzy logic control stand out. The selection of the controller will depend on the type of robot, the environment where it is operated, and the task to be carried out [20,21,22]. Due to computational advances, it is also possible to integrate Machine Learning (ML) models into navigation controls. These models allow a computational system to learn to solve a task without the need to receive explicit programming on how to do it, for example, classification (separate objects in two or more classes), clustering (collecting objects with similar characteristics), and regression (establishing the relationship between two or more variables). An ML-based controller can be trained to learn to navigate in specific conditions or to adapt to dynamic environments from training. This consists of feeding a large dataset into a model and indicating the expected results (case study) to find characteristic patterns that allow it to make predictions when new data are involved, with a probability of being right (inference) [23,24,25].

Electronic development boards and microcomputers can be used to prototype technological solutions and carry out proofs of concept, that is, to verify the technical feasibility of a proposal by experiments [26,27]. Among these, Arduino UNO and Raspberry Pi stand out because they started the concept of open-source hardware (open-source hardware). This refers to all technological projects whose design documentation is publicly accessible so that they can be studied, freely modified, and redistributed without the need to purchase a license or subscription [28]. This has led manufacturers of electronic devices to create electronic development boards such as ESP32-DevKit and microcomputers such as Jetson Nano, which have functionalities to implement embedded systems with applications from the Internet of Things (IoT) to computer vision. ML models usually require a lot of computational resources and memory, but with advances in techniques to optimize them, an embedded system might use these ML models to make inferences [29].

At the Pontificia Universidad Católica de Valparaíso (PUCV), Chile, work has been carried out to provide greater intelligence to the Khepera IV mobile robot from the company K-TEAM. For example, [30] shows the development of a platform that obtains the absolute position of one or more Khepera within a given workspace. This solution requires a PC that captures images of the base of the platform using a camera and Swistrack v4 (OpenCV-based software) that detects and locates the robots present in the images. Finally, for all the robots to communicate with the PC, Easy Java Simulations (EJS) software is used, which allows each robot to consult, through TCP Socket, its absolute position and the error so that each one can correct its speeds and orientation. In [31], the methodology to calibrate the internal sensors of the Khepera using neural networks is presented, and in [32], the Q-learning technique is used, where the model learns by exploring an unknown environment to determine the actions to execute by trial and error. Both works require repeating a series of steps many times to adjust their models. In order not to wear out the useful life of the Khepera, they use the CoppeliaSim simulator, and from a digital model of the Khepera, developed at PUCV [33,34,35], they train their models in a controlled environment, as required by each application. This opens up two possibilities: performing repetitive tests, depending on the application, using the Khepera digital model, and obtaining reliable results before testing on a real Khepera, extending its useful life.

The Khepera is a DDRM with two spherical support wheels, with a diameter of 14 cm and a height of 5.8 cm. It has a remarkable performance in both simulation and real life, based on a sufficiently powerful device to read its sensors and control its actuators and navigate in real time. However, it has limited computational power to carry out the computational vision (not fitted with a GPU). To take advantage of the potential mentioned above of the Khepera, an embedded system can be integrated that adds the functionality to detect objects using ML models or even Deep Learning (DL), for example [36].

After analyzing the state of the art of embedded systems, research was carried out on NVIDIA development cards to carry out AI solutions. Due to the workload that image processing and inference using a model can carry, the Jetson Xavier NX was selected. It is compatible with model creation environments, such as TensorFlow [37], PyTorch [38], and Caffe [39]. In addition, it has several interfaces, because it also features a CPUr, including HDMI, GPIO, USB Ports, and Ethernet, which opens possibilities for other types of projects. In addition, its dimensions are 7 cm × 4.5 cm, making it suitable for placing on top of the Khepera.

The motivation of this work is to integrate both systems to create an embedded system that allows the following trajectories using computer vision techniques and object detection in real time. To achieve this, a TCP Socket-type connection is first established between both systems through WiFi, and then an algorithm is developed that has the functionality to segment and follow trajectories and the functionality to make decisions when detecting objects using ML or DL models. This navigation algorithm was tested in three experiments, both in real time and in simulation.

The main contributions of this work are the following:

• The development of an embedded system that integrates the Khepera and Jetson Xavier NX, both systems exchanging information by TCP Socket. To the best of our knowledge, our work is the first solution that links the Khepera and the Jetson Xavier NX to implement a navigation DDRM based on trajectory tracking and object detection.
• The construction of a robotic platform for testing algorithms based on image processing for trajectory tracking and real-time object detection models.
• The verification of the technical feasibility of using the YOLOv5 architecture and its potential to generate models using transfer learning so that they can be used by an embedded system with GPU, managing to create robust intelligent systems in a shorter amount of time.
• The proposed system presents an improvement with respect to previous works, as our approach increases the speed of the robot by improving the success rate of object recognition.

Carrying out this solution becomes a challenge due to the complexity involved in establishing communication between two systems so that they interact as a single system, the developing of navigation algorithms, and the generation of object detection models and integrating them into navigation algorithms. The experiments carried out both in simulation and in real time allowed us to verify the potential of the solution.

This article is organized as follows: Section 2 defines the problem and the solution approach. Section 3 shows the final implementation of the system and the experiments carried out to verify the scope of the system. Section 4 presents conclusions and possible future improvements.

2. Problem Statement and Proposed Solution

An autonomous DDRM requires sensors that deliver essential information about its environment so that it can navigate safely. One of the most popular in the industry is the LiDAR sensor [40] which, based on lasers, can measure the distance to various points in the environment and create a map with objects and surfaces, making it very accurate but expensive; in addition, the controller requires more computational resources to read the map. The next option that requires less expensive sensors, but still uses high computational resources, is the use of computer vision for guided path tracking and real-time object detection for decision making, all of which are based on the use of a camera, segmentation algorithms, and a detection model.

The work presented in [41] uses the platform of [30] to test navigation control based on trajectory tracking and traffic sign detection, where the Khepera sends an image of the road to the server, which has the algorithm to process it. This included both trajectory tracking using OpenCV and signal detection, which uses Haar Cascade, and a basic ML technique used to detect whether or not something in an image belongs to a specific class. After processing these images, the server uses EJS to send the new linear and angular velocities commands to the Khepera. This algorithm obtained an average success rate of $88.8\%$ for the traffic sign detection model and a maximum speed of $2.7$ cm/s for the trajectory tracking, probably due to the use of Haar Cascade models to perform the entire object detection task. Moreover, due to the number of separate components and programs it comprises, it becomes less flexible to modify and add more functionalities.

The trajectory tracking and traffic sign detection functionalities, using a single model, can be integrated into an embedded system, reducing the complexity of the whole system significantly and allowing the solution to scale to future improvements as required by the programmer, tasks that can be executed with the Jetson Xavier NX.

2.1. Kinematic Model

The kinematic model of a differential robot is the mathematical modeling of its movement without considering the external forces involved. It can estimate with high precision the position and orientation of the mobile robot, considering the following:

• The robot moves on a flat surface.
• The wheel slip must be negligible.
• The robot is rigid with no flexible parts.
• There are internal control loops to achieve the commanded speeds.

These conditions are met by the Khepera IV, which makes it possible to use the model to describe its motion through space, seeking to control its linear velocity (V) and angular velocity ($\omega$). The equations that define the differential kinematic model are

$$V=\frac{({\omega }_r+{\omega }_l)R}{2},$$

(1)

$$\omega =\frac{({\omega }_r-{\omega }_l)R}{2L},$$

(2)

where ${\omega }_r$ and ${\omega }_l$ are the angular velocities of each robot motor, R is the radius of the wheels, and L is the distance between the wheels. Solving for ${\omega }_r$ and ${\omega }_l$, we obtain

$${\omega }_r=\frac{(2V+\omega L)}{2R},$$

(3)

$${\omega }_l=\frac{(2V-\omega L)}{2R}.$$

(4)

The proposed navigation control will use the kinematic model to make estimates and send ${\omega }_r$ and ${\omega }_l$ to the Khepera IV so that it adjusts them in real time. For the Khepera IV to move from a starting point to an endpoint, the controller must synchronously vary the angular velocities of its wheels to form an efficient trajectory, as shown in Figure 1.

2.2. Path Tracking

With the Jetson Xavier NX, it is possible to create scripts with Python, which is an open-source programming language that is well-supported in libraries and functionalities. Among the available libraries is OpenCV, which is dedicated to image processing and computer vision. A camera is installed for path tracking, pointing forward at the front of the Khepera IV. It has a viewing angle of ${60}^{\circ }$, causing it to capture more than just the sector of the path to follow so that, before performing the algorithm, the image is cropped to display the way in front of the robot and only a part of it (sample the path). The controller then performs the following steps using OpenCV:

• Filter the areas that are not the color of the road because they will not be analyzed.
• Select the area of the road color with the largest area.
• Identify the center of that region, and its value is taken in x (it is where the variation can be given by error).
• Calculate the error between the location of the center of the road and the center of the camera (e); this is normalized to take values between 0 and 1.
• Estimate the new angular speed ($\omega$) needed to adjust the direction of the Khepera, as well as the linear speed (V), needed for its advance, using the following equations:

  $${\omega }_n=\omega ·e·k_p$$

  (5)

  $$v_n=v·(1-|e\left|\right)$$

  (6)

where ${\omega }_n$ and $v_n$ are the updated values of the angular and linear velocities, respectively. Equation (5) implements a P controller responsible for adjusting the angular speed of the Khepera according to the e value. The gain $k_p$ is adjusted to $0.9$ after several experiments. Equation (6) is a way of relating the linear velocity with the angular one but in the following way: if the calculated error e is very large, the linear velocity will tend to be 0; otherwise, it will tend to its nominal value. This ensures that the orientation adjustment has priority over the forward motion, which is important in path tracking. Figure 2 illustrates the route tracking conceptual model and the results obtained by implementing it in the controller.

2.3. Object Detection

When using ML or DL models, there are techniques for creating models from scratch; however, certain architectures (neural network layer structure) are usually used that achieve good results with a reduced use of resources and these are continuously studied and improved, creating new versions. Among these architectures are MobileNet SSD, Faster R-CNN, and YOLO. The YOLOv5 ([42,43]) was selected for its scope in object detection in real environments and its functionality in generating models using transfer learning, which helps reduce the model training time because the architecture of the model is not trained from scratch. This architecture is based on PyTorch, which is a framework compatible with Python and the Jetson Xavier NX.

Transfer learning (TL) is used to create new models, which consists of taking a previously trained model (with a large dataset) and re-training it with a custom (and possibly smaller) dataset. This is conducted in such a way that the ability to recognize patterns or features is obtained from the first model but is now performed exclusively with a custom dataset to be more efficient than training the model from scratch. In the case of YOLOv5, its models are pre-trained with the COCO dataset, which has more than 330,000 images labeled with 80 different categories. Another advantage of the architecture is that it has different versions to select according to the available computational resources. For microcomputers, YOLOv5s is recommended.

The prototype of the robotic platform was created using CoppeliaSim. This allowed repetitive tests to verify the trajectory tracking and adjust the controller before testing it in the Khepera IV. Later, within the same environment, the traffic signal models to be detected were created. They are shown in Figure 3.

Table 1. Traffic Signs Descriptions.

ID	Action	Traffic Signal
v120	maximum speed
left	turn left
v60	half maximum speed
stop	stop tour for 3 s
right	turn right
pucv	arrival point

shows the list of traffic signs to be detected. The first column has the assigned name, the second column describes the task that the navigation system will perform when detected, and the last column shows its corresponding view.

Then, the robotic platform was used to generate the dataset of the traffic signs to be detected (custom dataset). To do this, groups of 4 transit signals were placed on it at strategic points, which varied randomly in the following experiment (3 were carried out). Finally, the Khepera, with the navigation control for path tracking, traveled the platform taking captures as the one shown in Figure 4. A total of 3 videos were obtained at 6 fps with a duration of 30 s each. When breaking them down into images, 540 were reached. From these, the best 135 were manually selected and labeled (indicating where the traffic signals were located and what type they were) in the YOLOv5 format, using the RoboFlow online tool.

By having the personalized dataset, we proceeded to perform TF with the YOLOv5 model, having been trained with the COCO dataset. This model has in-depth knowledge of pattern recognition and feature detection in images. To apply TF, part of the architecture of the model was taken. Specifically, the convolutional layers, which is where said knowledge is encoded, and the weights of the output layers were restarted to train them with the dataset of traffic signs. This generates a model whose task is the detection of 6 kinds of traffic signs. This process is illustrated in Figure 5.

For this training, it was necessary to use a server with the following characteristics:

• CPU: Intel Core i9-10900X.
• lGPU: RTX2080, comes with 8 GB of GDDR6 memory.
• RAM: 64 GB.
• Storage: 1 TB.

At the end of the training and validation, an object detection model was obtained with a confusion matrix where each class had $100\%$ success. Figure 6 shows examples of how it works in real time with the traffic sign detection model.

3. Final Implementation of Navigation Control and Experimentation

First, the conceptual model of the components that make up the solution was established. This is shown in Figure 7a while Figure 7b shows its physical implementation.

Then, all the algorithms proposed in the solution were integrated into a single navigation control algorithm. The algorithm shown in Figure 8a was generated in such a way that the Jetson Xavier NX establishes the connection via TCP Socket with the Khepera IV and then the navigation control remains working, which is based on the following trajectories until detecting traffic signs to perform actions. These traffic signals must be at a distance of 20 cm (or less) and must be located on the right side of the trajectory; the estimation of the distance between the mobile robot and a traffic sign is made from the area that uses the bounding box of its detection.

In parallel, the main Khepera program is executed. This is shown in Figure 8b, which, after establishing the connection via TCP Socket, keeps asking if there are data available to update the angular velocities of its wheels.

A robotic platform was built whose workspace is $1.5\times 1.2$ m${}^2$ in order to be able to place traffic signs to create different scenarios. Figure 9a shows a scenario within the CoppeliaSim simulator while Figure 9b shows its implementation on the real platform.

3.1. Scene 1

Scene 1 is shown in Figure 10. It consists of the robot following the trajectory, and every time it detects a traffic signal 20 cm away, the system stops for 2 s, but it does not execute the sign action. This is carried out to see the number of hits of the robot navigation control. For the 10 attempts that were made both in simulation and in real life, 100% success was obtained. A light blue line is drawn to indicate the trajectory that the robot must follow, and the $xy$ axes are also placed to see the reference of the robot within its workspace. The above applies also to the next test scenes.

Figure 11 shows the behavior of the speeds according to the detected signal, with Figure 11a showing the behavior of the navigation control in simulation and Figure 11b for the real test. Both cases are very similar and demonstrate that the vehicle navigates at a maximum linear velocity of 10 cm/s. Figure 12a shows the trajectory generated until the “pucv” signal is detected, which indicates the end of the route. Figure 12b shows the distance between the mobile robot and the point of arrival, the traffic signal “pucv”, while navigating the route for 60 s.

3.2. Scene 2

Scene 2 is shown in Figure 13. It consists of the robot taking the action corresponding to the traffic signal detected at 20 cm while it is traversing the trajectory. For the 10 attempts in this scenario, a $100\%$ success rate was obtained.

Figure 14a shows the speed changes according to the traffic signal detected, at 20 cm, within the CoppeliaSim simulation. Figure 14b shows the same but for the real implementation. Figure 15a shows the trajectory generated until reaching the end of the route when detecting the “pucv” signal while Figure 15b shows the distance between the mobile robot and the point of arrival, the traffic signal “pucv”, while navigating the route for 30 s.

3.3. Scene 3

Scenario 3 is shown in Figure 16. Now, the robot follows the trajectory, and when a traffic signal is found, it carries out the corresponding action. For the 10 attempts of this experiment, $80\%$ success was obtained, because on 2 occasions certain turns left the robot out of the center of the road, and while it adjusted to the center, it did not adequately detect one of the traffic signs.

Figure 17a shows the speed changes according to the traffic signal detected at 20 cm within the simulation in CoppeliaSim, and Figure 17b shows the same but for the real implementation. Figure 18a shows the trajectory generated until reaching the end of the route when detecting the “pucv” signal while Figure 18b shows the distance between the mobile robot and the point of arrival, the traffic signal “pucv”, while navigating the route for 45 s.

Looking at the results of the experiments, it can be observed that the maximum linear velocity was maintained at 10 cm/s, the average hit rate for object detection was $100\%$, and the average hit rate for navigation control was $93\%$. This is illustrated in

Table 2. Experiment Results.

	Maximum Linear Velocity (cm/s)	Object Detection Hit Rate	Navigation Control Hit Rate
Scene 1	10	$100\%$	$100\%$
Scene 2	10	$100\%$	$100\%$
Scene 3	10	$100\%$	$80\%$
Average	10	$100\%$	$93\%$
Results from [41]	$2.7$	$88.8\%$	no data

. These results show an improvement to those obtained in [41].

Table 3. Path tracking error and total time.

	Scene 1		Scene 2		Scene 3
	Accumulated Error (${\mathit{m}}^{1/2}$)	Time (s)	Accumulated Error (${\mathit{m}}^{1/2}$)	Time (s)	Accumulated Error (${\mathit{m}}^{1/2}$)	Time (s)
Ideal	0	53	0	25	0	41
Simulation	3.7	55.5	10.6	27	12.3	44.4
Real	4.0	54.7	10.4	26.6	12.3	42.6

includes the overall performance of both the simulation and real tests in comparison with an ideal motion. This optimal motion consists of following the same trajectories in terms of paths and traffic signs but considering perfect detection, identification, and control. Each scene includes a column with the accumulated absolute error (AAE) with respect to the ideal motion and a column with the total time counted from the start to the reaching of the target. The absolute AAE is calculated as follows:

$$AAE=\sqrt{\sum _{k=1}^N|x\left(k\right)-x_T\left(k\right)|+|y\left(k\right)-y_T\left(k\right)|}$$

(7)

where $(x_T,y_T)$ is the ideal trajectory, $(x,y)$ is the trajectory to be studied, and N is the number of sample points.

On the one hand, ideal behavior in all scenarios takes less time, which is to be expected. On the other hand, the comparison of the AAE between the simulated and real tests demonstrates the similarity of both test outcomes, which is also predictable because of the good quality of the simulator engine.

To complement the previous results, a more complex scenario is included. This is tested in simulation, where more traffic signs and more paths are taken. No real test is performed as the previous figures and results, summarized in Table 3, indicate a close similarity between the simulation and real outcomes.

Figure 19 shows the arena where several different traffic signs and paths are used for this case. This scenario was designed based on [41], replicating the one shown in Figure 14. The results of this test are shown in Figure 20.

Figure 20a shows the behavior of the linear and angular speeds for the robot navigating throughout the scenario. It reaches a maximum displacement speed of 10 cm/s and a low error in the trajectory tracking which is illustrated in Figure 20b, taking 120 s to complete the journey.

4. Conclusions

In this work, the development of a system that integrates a mobile robot (Khepera IV) and an embedded system (Jetson Xavier NX) was presented. The latter adds functionalities to the Khepera IV in such a way that its navigation potential is fully used and improved with the computational resources of the Jetson Xavier NX board. This solution benefits from the use of the Python programming language, which is supported by computer vision libraries and frameworks for the use of object detection models.

To verify the operation of the system, a robotic platform was built, and the CoppeliaSim simulator was used as a complement. Both were used to carry out experiments and analyze the response of the navigation control using the Khepera IV, in terms of the maximum displacement speed, hit rate in object detection, and hit rate in the navigation control response. The obtained results surpass previous works, giving a contribution to the state of the art in the use of embedded systems to provide functionalities to mobile robots.

The comparison between the operation of the navigation control within the simulator concerning the real environment showed very similar results that allowed to establish a high confidence for the use of the simulator to carry out tests and modifications to any algorithm before testing it in the real mobile robot, thus extending its useful life. The differences observed in the comparison are due to the fact that within the real environment, there are imperfections both in the base of the platform and in the structure of the Khepera IV, causing friction when navigating and not captured by the simulator.

The experimental tests allowed us to observe the strengths and limitations of the algorithms of the system, making it necessary to improve what refers to navigation control, specifically in some turns where the robot moved away from the center of the road, ignoring traffic signs. For future work, the use of advanced control techniques is recommended, allowing the Khepera IV to move at higher speeds while following the trajectory with lower overall error, minimizing the error in detecting the traffic signals.