A Safe Navigation Algorithm for Differential-Drive Mobile Robots by Using Fuzzy Logic Reward Function-Based Deep Reinforcement Learning

Abstract

Researchers are actively exploring advanced algorithms to enhance robots’ ability to navigate complex environments while avoiding obstacles. Four different environments were designed in the Webots simulator, including a mobile robot, a goal, a static obstacle, and one or two dynamic obstacles. The robot’s state vector was determined based on its position, the goal, and sensor variables, with all elements randomly placed in each learning and test step. A multi-layer perceptron (MLP) agent was trained for 1000 episodes in these environments using classical and fuzzy logic-based reward functions. After the training process was completed, the agents trained with the fuzzy logic-based reward function were tested for each environment. As a result of the test, while the robot’s arrival rate was 100% in the first three environments, it was measured as 91% in the fourth environment. In the last environment, the rate of crashing into a wall or dynamic obstacle was observed to be 7%. In addition, the agent trained in the fourth environment was found to successfully reach the target in multi-robot environments. The agent trained fuzzy logic-based reward function obtained the best result for four different environments. Based on these results, a fuzzy logic-based reward function was proposed to address the tuning problem of the classical reward function. It was demonstrated that a robust fuzzy logic-based reward function was successfully designed. This study contributed to the literature by presenting a reinforcement learning-based safe navigation algorithm incorporating a fuzzy logic-based reward function.

1. Introduction

Mobile robots are robotic systems capable of moving within a specific environment and performing various tasks autonomously or via tele-operated control. The robots can have wheeled or legged structures. Mobile robots have a wide range of applications such as factory automation, cleaning tasks, personal assistance, and security operations [1,2,3]. They can effectively navigate dynamic environments by using LiDAR, cameras, ultrasonic sensors, and artificial intelligence-based algorithms to perceive their surroundings and make decisions.

Artificial intelligence learning methods can be separated into three sub-fields such as supervised, unsupervised, and reinforcement learning (RL) methods. Many sectors such as automotive [4], control [5], and finance [6] use RL methods. RL is a frequently used tool for steering mobile robots without hitting dynamic or static obstacles [7,8]. RL consists of two main components: an agent and an environment. In this process, the agent is trained based on the reward information received from the environment. Reward information is typically defined as a function.

Fuzzy logic (FL), a calculation method that can deal with uncertainty and imprecise data, is used in many different fields such as image processing [9], metallurgy industry [10], robotics [11], etc. FL is a valuable tool in robotics, offering solutions to control and navigation challenges by managing uncertainties and nonlinearities.

This study aims to use an RL-based method to enable a mobile robot to reach a predefined target without colliding with static or dynamic obstacles. One of the main challenges in RL is determining the reward function. A FL-based reward function is designed to overcome this challenge in this study. Additionally, the goal is to reduce the size of the sensors on the robot, thereby making the artificial neural network architecture more compact. As a result, the training time of the neural network will be shortened, and the robot’s response time while running this network will be reduced.

Related Works

Mobile robots need to provide safe travel without hitting obstacles in the environment. Therefore, path planning, collision avoidance, and safe navigation in mobile robot applications are being studied by many researchers to overcome these problems. For example, the pure pursuit algorithm has been used so that the mobile robot could follow the determined path point [12]. A new approach that is based on particle-swarm optimization and Bezier curve has been developed to plan a path [13]. A path-planning process for mobile robots has been realized by using an enhanced ant-colony algorithm [14]. Adaptive ant-colony-algorithm-based path-planning optimization for mobile robots methods has been achieved [15]. Similarly, path planning and optimization have been realized to navigate a mobile robot in multi-obstacle environments [16]. In another study, a collision-less navigation algorithm that depends on a grid environment has been developed [17]. The collision avoidance problem of mobile robots has been solved by virtual agents and a differential game [18]. In other work, a sensor network has been designed so as to help mobile robots safely navigate industrial environments [19]. In this study, a reinforcement learning-based algorithm was developed in addition to existing path-planning methods. This algorithm not only plans the path but also controls the robot’s motors. Furthermore, since it is trained directly with the robot model, the planned path incorporates the robot’s dynamic information.

RL is a more suitable learning method to control robots because it does not require any dataset. Also, RL generates robust control output according to unpredictable environmental changes. For example, RL-based welding operation has been realized by using an industrial robot [20]. The inverse kinematics of a 5-DoF endoscopic instrument has been solved by utilizing RL [21]. In another study, an autonomous navigation algorithm for a multi-robot system has been developed by using RL [22]. RL could be integrated into value- and policy learning. There are many value- and policy-learning methods, like deep q network (DQN), deep deterministic policy gradient (DDPG) and proximal policy optimization (PPO). The PPO algorithm has been used in many studies. An assistant laparoscopy robot-training system has been realized utilizing generative adversarial imitation learning and PPO algorithms [23]. A human-motion capture system that is based on the PPO algorithm has been achieved using aerial robots [24]. In another study, a PPO-based guided constrained control algorithm for quadrupedal robot locomotion was developed [25]. PPO is preferred for its stable convergence and strong performance in continuous action spaces, particularly in complex navigation tasks. Studies show that PPO outperforms DDPG in mobile robot navigation due to its enhanced stability and optimization [26]. PPO stands out for its fast convergence, efficient sample utilization, and strong algorithmic stability. Compared to asynchronous advantage actor–critic (A3C), it offers a more stable training process while maintaining adaptability. While soft actor–critic (SAC) excels in high-dimensional path planning with robust parameter adjustments, PPO remains competitive due to its balanced efficiency and reliability. However, PPO requires careful parameter selection and sufficient training samples, making it crucial to optimize these factors for best performance. Overall, its stability and efficiency make PPO a strong candidate for reinforcement learning tasks [27]. PPO and RL with human feedback (RLHF) are important approaches in artificial intelligence and machine learning. PPO is a policy-optimization algorithm that makes the learning process of agents more stable and efficient. RLHF, on the other hand, guides the agent’s learning process with human feedback. Thus, it helps the agent to understand its environment faster and more accurately and to make decisions. RHLF enables successful performance on complex tasks such as large language models [28,29]. The fuzzy logic-based reward function approach presented in this paper can improve the performance of these methods. Thus, PPO, the RL algorithm, was used in this study. The PPO algorithm was trained with a fuzzy logic-based reward function instead of classical reward functions. It has been shown that the fuzzy logic-based reward function makes the parameter-adjustment process easier compared to classical reward functions. Additionally, this function not only simplifies parameter tuning but also makes the training process more effective and robust.

The combination of FL with other intelligent systems, such as genetic algorithms and particle-swarm optimization, has further enhanced robotic performance. In educational settings, fuzzy logic also aids in teaching control systems [30]. Its versatility and integration with other techniques make it an essential component in advancing robotic autonomy and intelligence. Particularly, when FL was examined in mobile robot control, a mobile robot was navigated to a target in an environment featuring static obstacles by FL [31]. In another study, a FL-based mobile robot controller that goes to the target without hitting the obstacles has been designed [32]. A FL-based controller has been designed to control the tilting angle of a two-wheeled mobile robot and track trajectory [33]. A mobile robot was controlled by a neuro-fuzzy system in Mishra et al.’s study [11]. In another study combining different disciplines, a mobile robot was controlled by a combination of FL and a kinematic model [34]. The integration of FL and RL has gained prominence due to their complementary strengths in handling uncertainty and decision making in complex environments. Recent studies highlighted their combined effectiveness in various applications. Karaduman et al. demonstrated how fuzzy logic improves adaptability in dynamic cyber–physical systems and IoT by addressing uncertainties [35]. Developed fuzzy reinforcement learning (FRL) method was accelerated decision making, particularly in challenging environments [36]. There are also robotic studies using these two methods in the literature [37,38,39,40,41,42]. For example, the chattering effects of the 3-rigid-link robotic manipulator control have been solved by Q-learning-based fuzzy logic inference [37]. In another study, a fuzzy logic-based actor has been used in RL methods so as to control a nonlinear system [38]. The trajectory tracking problem of rotary steerable systems has been solved utilizing FL and RL algorithms [39]. FL and RL algorithms have been used in order to vehicular fog computing [40]. The autonomous driving issue has been studied utilizing FL-based DQN [41]. The developed probabilistic FL-based RL method tested on different systems has proven to be more successful than classical methods [42]. When examining the methods combining FL and RL in the literature, it has been observed that FL has not been used as a reward mechanism in the studies conducted. FL has generally been used as a mechanism that generates policies or produces outputs in response to state inputs. In this study, as a contribution to the literature, a fuzzy logic-based reward function is proposed, considering that adjusting the classical reward function is a challenging task.

There are many studies in the literature that use the classical reward function [43,44,45,46,47,48,49,50]. For example, Chen et al. developed a single-legged robot based on beaver-like inspiration and the robot has been controlled RL using classical reward function [43]. In another study, the mobile robot used in this study was trained with the classical reward function-based RL algorithm to avoid collisions with obstacles [45]. The policy learning-based RL algorithm combined with kinematic analysis and immune optimization enables mobile robots to perform path planning. In this study, the classical coefficient reward function was used [47]. A reconfigurable policy-based RL algorithm with a classical coefficient reward function was used to move the mobile robot in an unknown environment [48]. The current work is about a mobile robot controlled by a deep RL-based neural network. In this study, it is aimed to reach a determined target point without obstacle collision by using neural networks for differential-drive mobile robots. There are some studies that are similar to the current study in the literature. For example, a Turtlebot mobile robot simulator has been formed and recursive neural network-based pulse with modulation controller has been designed by using this simulator. Linear and curvilinear trajectories have been tracked utilizing the recursive neural network by the Turtlebot mobile robot [51]. Collision-free trajectory generation for mobile robots has been realized by using value learning that is a method of RL [52]. In another study, a new RL algorithm which rely on value learning has been formed by experience replay after a mobile robot navigator has been developed [53]. In similar other work, an optimal path-planning algorithm has been progressed by developing value learning [54]. Collision-free harvesting robot software that has been formed by using DDPG [55]. In other works, genetic network programming and RL-based mobile robot software has been developed, meaning the robot can continue to move even if its sensors fail [56]. The skidding and slipping problem of mobile robots has been solved using adaptive neural network-trained gradient descent RL methods [57]. There are three main contributions of the presented study to the field of robotics and RL.

A new fuzzy logic-based reward function was designed for the mobile robot to navigate safely. The problem of adjusting the coefficients in the classical reward functions has been resolved with this designed reward function. By leveraging fuzzy logic, the system can dynamically adjust rewards based on environmental conditions, improving adaptability and robustness. The size of the network and the training time have been minimized by reducing the sensor size, leading to a more efficient learning process. This approach not only optimizes computational resources but also enhances the robot’s real-time decision-making capabilities, making it more suitable for practical applications in dynamic environments. Furthermore, the proposed reward function could be generalizable for other mobile robot navigation tasks.

2. Methodology

In RL, federal learning processes all the data in one place, making the learning process more consistent and controlled. Federal learning maintains data integrity, eliminating synchronization problems and communication delays. Federal learning allows for the direct control and refinement of discovery policies, resulting in coherent and efficient discovery, which can be difficult to achieve in distributed learning. Due to these advantages, this study uses federal learning fed from a single robot. RL consists of two main components, namely agent and environment. These components consist of state ($S\left(k\right)$), reward ($R\left(k\right)$), and action ($A\left(k\right)$) which are used to communicate, learn, and control. The block diagram of the RL and proposed system is given in Figure 1.

In this study, multi-layer perceptron (MLP) was used as an agent. TurtleBot3 Burger mobile robots were used as the environment of the RL by using Webots robot simulator, which can provide realistic simulation environments for robotic applications.

2.1. Environment Structure

An environment was created by using the Webots simulator, as shown in Figure 2. Simulator parameters were chosen as default and these parameters were given in

Table 1. Webots robot simulator parameters.

Parameter	Value
Gravity	9.81
Time Step	128
Constraint Force Mixing	10−5
Error-Reduction Parameter	0.2
Optimal Thread Count	1
Physics Disable Time	1
Physics Disable Linear Threshold	0.01
Physics Disable Angular Threshold	0.01
Line Scale	0.1
Drag Force Scale	30
Drag Torque Scale	5

. The simulation of the Turtlebot3 Burger robot, a differential-drive mobile robot, on the Webots simulator has been used. The simulator includes a 360-degree Laser Imaging Detection and Ranging (LiDAR) distance sensor and two motors that provide wheel movement. The work has been carried out solely in a simulation environment, but the training scenarios have been designed to reflect challenges that might be encountered in the real world. The robot reaches the target after encountering dynamic or static obstacles during all test and training stages. This demonstrates the robustness and resilience of the proposed system. Cylinder obstacles with dimensions of 0.2 × 0.4 m (radius × height) were placed on the 8 × 8 m rectangle area (5). The red obstacle was a static obstacle (3). Also, the dynamic obstacle (4) was illustrated in a green color. In Figure 2, the robot (1), goal (2), static obstacle (3), dynamic obstacle (4), and workspace (5) were labeled from 1 to 5, respectively. The robot and the target were placed randomly in the environment each learning and test steps. The static obstacle was placed in the middle of the target with the robot. The dynamic obstacle followed the defined circular trajectory centered on the static obstacle or goal position in Environments 2–4. In Environment 2, the dynamic obstacle moves by centering on the static obstacle position, while in Environment 3 it uses the target position as the center. In Environment 4, there are dynamic obstacles around both static obstacles and the target. The designed trajectory formulation of dynamic obstacle was given in Equations (1) and (2).

$$x_{dy}\left(k\right)=x_{st}+r_{1}cos\left(\pi {\displaystyle \frac{k-128}{128}}\right)$$

(1)

$$y_{dy}\left(k\right)=y_{st}+sign\left(r2-0.5\right)r_{1}sin\left(\pi {\displaystyle \frac{k-128}{128}}\right)$$

(2)

where $(x_{dy}, y_{dy})$ and $(x_{st}, y_{st})$ were typified positions of the dynamic obstacle and the center of the circle on the x-y axes, respectively. Also, $r_1=[0.5, 1]$ and $r_2=[0, 1]$ were random numbers. The discrete-time index was illustrated as k. The total sample time step was determined as 512 for each episode.

$S\left(k\right)$ and $R\left(k\right)$ are the output signals of the environment block. The determined $S\left(k\right)=\left[Position\right(k), Velocity(k), Sensor(k\left)\right]$ consists of the position, velocity, and the sensor vectors of the robot as given in Equations (3)–(5).

$$Position\left(k\right)=[\Delta E_x\left(k\right), \Delta E_y\left(k\right), \Delta E\left(k\right), {\Theta }_1\left(k\right), {\Theta }_2\left(k\right)]$$

(3)

$$Velocity\left(k\right)=[V_x\left(k\right), W_z\left(k\right)]$$

(4)

$$Sensor\left(k\right)=[Senso{r}_1\left(k\right), Senso{r}_2\left(k\right), Senso{r}_3\left(k\right), Senso{r}_4\left(k\right)]$$

(5)

The position vector given in Equation (3) includes the locations of the robot and the target relative to each other. Components of the position vector are calculated by using Equations (6)–(8). $x_{target}\left(k\right)$, $y_{target}\left(k\right)$, $x_{robot}\left(k\right)$, and $y_{robot}\left(k\right)$ denote the target and robot position on the x-y axes. ${\Theta }_2\left(k\right), V_x\left(k\right), W_z\left(k\right)$ were represented as the orientation angle on the z-axis, linear velocity on the x-axis, and angular velocity on the z-axis of the robot, respectively. The velocity vector was composed utilizing $V_x\left(k\right)$ and $W_z\left(k\right)$. There is a LiDAR sensor mounted on the TurtleBot3 Burger mobile robot. This sensor produces 360 data which have a resolution of $1°$ in the range of 120–3500 mm. Sending all the sensor data to the neural network will cause the neural network to expand. Thus, this situation will result in high hardware requirements and long training time. For these reasons, the sensor data were first saturated at 500 mm. This value is used as the maximum value that the robot detects how close it is to the obstacle. This value should be increased if the robot needs to detect at a greater distance from the obstacle, and vice versa. Then, 360 sensor data were reduced to 4 sensor data, by calculating the minimum of each 90 sequential data.

$$\Delta E_x\left(k\right)=x_{target}\left(k\right)-x_{robot}\left(k\right), \Delta E_y\left(k\right)=y_{target}\left(k\right)-y_{robot}\left(k\right)$$

(6)

$$\Delta E\left(k\right)=\sqrt{{\Delta E_x\left(k\right)}^2+{\Delta E_y\left(k\right)}^2}$$

(7)

$${\Theta }_1\left(k\right)=ta{n}^{-1}(\Delta E_y\left(k\right)/\Delta E_x\left(k\right))$$

(8)

Another output of the environment is $R\left(k\right)$. Two different $R\left(k\right)$, classical ($R_{Classic}\left(k\right)$) and fuzzy logic ($R_{Proposed}\left(k\right)$), were used in this study.

2.1.1. Classical Reward Function

Classical reward function ($CR\left(k\right)$) was given in Equation (9).

$$R_{Classic}\left(k\right)=\left\{\begin{array}{cc}1500\hfill & :\mathrm{if}\Delta E\left(k\right)<0.2\hfill \\ -1000\hfill & :\mathrm{if}\gamma \left(k\right)>0.85\hfill \\ CR\left(k\right)=-{\alpha }_1\parallel \Delta E\left(k\right)\parallel -{\displaystyle \frac{\parallel \gamma \left(k\right)\parallel }{{\alpha }_2}}-{\alpha }_3\parallel \theta \left(k\right)\parallel \hfill & :\mathrm{otherwise}\hfill \end{array}\right.$$

(9)

Here, the first condition symbolizes that the robot arrives at to target. The second condition represents the robot colliding with any obstacle or wall. $CR\left(k\right)$ was given as the last term. $CR\left(k\right)$ consists of three terms. The distance between the robot and the target is shown $\parallel \Delta E\left(k\right)\parallel$. This expression allows the robot to approach the target. Another term is $\parallel \gamma \left(k\right)\parallel$, involving the distance between the robot and obstacle or wall. This term enables the robot to run away from the obstacle or wall. The orientation term is calculated by utilizing $\theta \left(k\right)=\mid {\Theta }_2\left(k\right)-{\Theta }_1\left(k\right)\mid$. This term enables the robot to go directly (without turning) to the target. $\gamma \left(k\right)$ is obtained from Equation (10).

$$\gamma \left(k\right)=\left\{\begin{array}{cc}1-Senso{r}_2\hfill & :\mathrm{if}{V}_x\left(k\right)>0 and W_z\left(k\right)>0\hfill \\ 1-Senso{r}_3\hfill & :\mathrm{if}{V}_x\left(k\right)>0 and W_z\left(k\right)\le 0\hfill \\ 1-Senso{r}_4\hfill & :\mathrm{if}{V}_x\left(k\right)\le 0 and W_z\left(k\right)>0\hfill \\ 1-Senso{r}_1\hfill & :\mathrm{if}{V}_x\left(k\right)\le 0 and W_z\left(k\right)\le 0\hfill \end{array}\right.$$

(10)

Here, the sensor values were subtracted from 1 to reduce the reward as the robot approached obstacles or walls. The value that is important according to the sensor placement and robot movement figures out the $\gamma \left(k\right)$. Lastly, ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ in Equation (9) represents weight coefficients.

2.1.2. Fuzzy Logic Reward Function

The reward function that was added the fuzzy logic term is given in Equation (11).

$$R_{Proposed}\left(k\right)=\left\{\begin{array}{cc}1500\hfill & :\mathrm{if}\Delta E\left(k\right)<0.2\hfill \\ -1000\hfill & :\mathrm{if}\gamma \left(k\right)>0.85\hfill \\ FuzzyLogic\left(de\right(k), d\gamma (k), d\theta (k\left)\right)\hfill & :\mathrm{otherwise}\hfill \end{array}\right.$$

(11)

where $\Delta E\left(k\right)<0.2$ and $\gamma \left(k\right)>0.85$ expressed arrival at the target and obstacles/walls with which the robot has collided. The arrival and collision ratios were determined as fixed coefficients (1500, −1000) after numerous trial-and-error processes. Tests showed that when the arrival value was set too low (close to the fuzzy logic output), the robot prioritized reducing its velocity and moved in minimal steps rather than reaching the target. On the other hand, when the collision weight was equal to or greater than the arrival weight, the robot prioritized avoiding collisions instead of reaching the target. Therefore, the arrival weight was set to be greater than the collision weight. However, when the arrival weight was excessively increased, the robot ignored obstacles and moved directly toward the target. Fuzzy logic input terms were shown as $de\left(k\right)=switch(\Delta E(k)-\Delta E(k-1\left)\right)$ $d\gamma \left(k\right)=switch\left(\gamma \right(k)-\gamma (k-1\left)\right)$ and $d\theta \left(k\right)=switch\left(\theta \right(k)-\theta (k-1\left)\right)$. The switch function was created to investigate whether the data obtained as a result of the robot’s movements in two sequential time intervals are suitable for the desired purpose. The function was given in (12). Thus, the accuracy of the motion performed by the robot is given as an input to the reward function.

$$switch\left(input\right)=\left\{\begin{array}{cc}99\hfill & :\mathrm{if}input>0\hfill \\ -99\hfill & :\mathrm{otherwise}\hfill \end{array}\right.$$

(12)

The output of this process is provided to the fuzzy logic function, which evaluates the robot’s proximity to the target, nearby obstacles, and whether its orientation has been corrected. The $FuzzyLogic\left(\right)$ function was designed by using the Mamdani method. Input membership functions are defined for the outputs of the switch functions, and output membership functions are designed symmetrically, as shown in Figure 3. A key consideration in the output functions is creating a balanced scale (e.g., good–bad or perfect–terrible). If negative values (bad–terrible) are prioritized, the robot will focus more on avoiding obstacles than reaching the target. Conversely, if positive values (good–perfect) are prioritized, reaching the target becomes more important than avoiding obstacles. In the first case, the robot may remain stationary, while in the second, it may repeatedly collide with obstacles while heading toward the target. This method consists of the fuzzification of inputs, rule evaluation, aggregation, and defuzzification. Hence, the inputs and output membership functions were given in Figure 3. After this step, the rules were determined, as shown in

Table 2. Rule table of fuzzy logic-based reward function.

Rules		$\mathit{de}$	$\mathit{d}\mathit{\gamma }$	$\mathit{d}\mathit{\theta }$		Reward
1	if	Good	Good	Good	then	Perfect
2	if	Good	Good	Bad	then	Good
3	if	Good	Bad	Good	then	Good
4	if	Good	Bad	Bad	then	Bad
5	if	Bad	Good	Good	then	Good
6	if	Bad	Good	Bad	then	Bad
7	if	Bad	Bad	Good	then	Bad
8	if	Bad	Bad	Bad	then	Terrible

. The rule table was created with a focus on the robot’s approach to the target, its movement away from obstacles, and the correction of its orientation.

The aggregation process was carried out after membership functions and the rule table were determined. The result of the aggregation process was evaluated by defuzzification steps and the $FuzzyLogic\left(\right)$ function was formed.

2.2. Agent Structure

The agent making decisions based on rewards is another block of RL. A MLP structure is shown in Figure 4 was used as an agent in the current study.

The MLP structure includes input, hidden, and output cells. The size of input cells equals the size of $S\left(k\right)$ vector. In the structure, four hidden layers were used and each of hidden layer cells was determined as 2048. Also, rectified linear units (ReLU) were utilized as activation function. These hyper-parameters were chosen according to previous studies [58,59]. Outputs of MLP structure were $A\left(k\right)=[\Delta V_x\left(k\right), \Delta W_z\left(k\right)]$. $\Delta V_x\left(k\right)$ and $\Delta W_z\left(k\right)$ were symbolized as the change in linear and angular velocities. Also, $\Delta V_x\left(k\right)$ and $\Delta W_z\left(k\right)$ were limited to the range of −0.05 and 0.05. $V_x\left(k\right)$ and $W_z\left(k\right)$ were obtained by using these outputs and Equations (13) and (14).

$$V_x\left(k\right)=V_x(k-1)+\Delta V_x\left(k\right)$$

(13)

$$W_z\left(k\right)=W_z(k-1)+\Delta W_z\left(k\right)$$

(14)

After the linear and angular velocities of a robot were obtained, the angular velocity of left ($w_l$) and right ($w_r$) wheels was calculated by using the kinematic equations given in Equation (15).

$$\left[\begin{array}{c}{V}_x\\ W_z\end{array}\right]=\left[\begin{array}{cc}{\displaystyle \frac{D}{2}}& {\displaystyle \frac{D}{2}}\\ {\displaystyle \frac{D}{L}}& -{\displaystyle \frac{D}{L}}\end{array}\right]\left[\begin{array}{c}{w}_l\\ w_r\end{array}\right]$$

(15)

D and L indicate the wheel diameter and wheelbase of a differential-drive mobile robot. D and L parameters of TurtleBot3 Burger robots, that were used in this study, are $D_{TurtleBot}=0.066$ m and $L_{TurtleBot}=0.160$ m.

RL could be separated into two sub-fields, namely value (Q) and policy ($\pi$) learning. Policy could be described as $\pi :\mathcal{S}\to \mathcal{P}\left(\mathcal{A}\right)$ function. The main purpose of this function is to generate Markov decision process between state-space ($\mathcal{S}$) and action-space ($\mathcal{A}$). This process is formed by using current-state ($s_t$), action ($a_t$), and reward ($r_t$) values. These values are used in Equation (16) which shows the discounted reward ($R_t$) equation.   

$$R_t=\sum _{i=t}^T{\varsigma }^{i-t}r\left(s_i, a_i\right), \varsigma \in \left[0, 1\right]$$

(16)

where $\varsigma$ represents discount factor. The main purpose of policy-learning RL is that maximizes initial expected distribution $J=\mathbb{E}\left[R_1\right]$. Bellman equation is often used in RL as seen in Equation (17).

$$Q^{\pi }(s_t, a_t)=\mathbb{E}\left[r_t+\varsigma \mathbb{E}\left[Q^{\pi }\left(s_{t+1}, a_{t+1}\right)\right]\right]$$

(17)

Target policy can be described as $\mu :\mathcal{S}\leftarrow \mathcal{A}$ and $Q^{\mu }(s_t, a_t)$ was given in Equation (18).

$$Q^{\mu }(s_t, a_t)=\mathbb{E}\left[r_t+\varsigma \mathbb{E}\left[Q^{\mu }\left(s_{t+1}, \mu \left(s_{t+1}\right)\right)\right]\right]$$

(18)

Loss function depends on Q-learning [60] and can be defined as in Equation (19).

$$L\left({\theta }^Q\right)=\mathbb{E}\left[{\left(Q\left(s_t, a_t\mid {\theta }^Q\right)-y_t\right)}^2\right]$$

(19)

$$y_t=r_t+\varsigma Q\left(s_{t+1}, \mu \left(s_{t+1}\right)\mid {\theta }^Q\right)$$

(20)

Function parameters represent ${\theta }^Q$ in Equations (19) and (20). PPO, that is, policy-learning methods, were used in this study.

Proximal Policy Optimization Algorithm (PPO)

The PPO algorithm is formed by improving trust-region policy optimization and only first-order optimization is used in this improvement process. The optimization process is realized by utilizing the surrogate function ($\mathcal{L}$). $\mathcal{L}$ is given in Equation (21).

$$\mathcal{L}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min\left({\mathcal{R}}_t\left(\theta \right){\widehat{A}}_T, clip\left({\mathcal{R}}_t\left(\theta \right), 1-ϵ, 1+ϵ\right){\widehat{A}}_T\right)\right]$$

(21)

$${\mathcal{R}}_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t{s}_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t{s}_t\right)}}$$

(22)

The probability ratio is shown as ${\mathcal{R}}_t$ in Equation (22). ${\widehat{A}}_{1,\dots ,T}$ denotes the advantage estimates. The PPO algorithm is shown in Algorithm 1 for actor–critic network and detailed information about the algorithm can be found in [61].

Algorithm 1 Proximal Policy Optimization Algorithm.

for episode: 1 → M do        Initialize $Batch$        Reset environment and obtain $s_1$        for iteration: 1 → T do             $a_t=\pi (s_t\mid {\theta }^{\pi })+{\mathcal{N}}_t$             Get $r_t$ $s_{t+1}$ according to $a_t$             $Batch\leftarrow [s_t, a_t, r_t, s_{t+1}]$             Compute ${\widehat{A}}_1, \dots , {\widehat{A}}_T$        end for        Compute new weight by optimizing surrogate $\mathcal{L}$        Update networks  end for

In Algorithm 1, the batch and environment restart. The actor network generates an action output according to the state and the noise parameter is added to the action output. This action output is applied to the environment then the next state and reward are calculated. State, action, next state, and reward are stored batch. The advantage ${\widehat{A}}_{1,\dots ,T}$ is calculated according to reward and prediction values. The proposed reward function based on fuzzy logic was calculated in response to the action taken by the agent in Algorithm 1 and included in the process as a reward function.Then networks are optimized according to $\mathcal{L}$, which is given in Equation (21).

3. Results and Discussion

In the current study, the first-proposed MLP architecture was trained by using classic and fuzzy logic reward functions using different environments. The determined neural networks were trained during ${10}^3$ episodes and the training process performance of the algorithms can be seen in

Table 3. Comparisons of the training results of classical and fuzzy logic-based reward functions.

	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	${\mathit{\alpha }}_3$	${\mathit{C}}_{\mathbf{ALL}}$	C($\overline{\mathit{X}}$ ± SD)	${\mathit{A}}_{\mathbf{ALL}}$	A($\overline{\mathit{X}}$ ± SD)	${\mathit{S}}_{\mathbf{ALL}}$	S($\overline{\mathit{X}}$ ± SD)	${\mathit{C}}_{\mathit{L}100}$	${\mathit{A}}_{\mathit{L}100}$	${\mathit{S}}_{\mathit{L}100}$	${\mathit{C}}_{\mathbf{Best}}$	${\mathit{A}}_{\mathbf{Best}}$	${\mathit{S}}_{\mathbf{Best}}$
Environment 1	6	20	15	249	24.0 ± 9.54	614	63.5 ± 17.70	365	39.4 ± 26.25	13	73	60	7	93	86
	10	20	15	253	24.1 ± 18.28	672	70.3 ± 19.19	419	46.3 ± 37.35	17	75	58	2	94	92
	8	15	15	244	25.0 ± 15.57	591	60.3 ± 17.45	347	35.34 ± 31.08	2	88	86	2	94	92
	8	25	15	327	32.2 ± 17.42	406	40.6 ± 35.71	79	8.4 ± 51.87	10	81	71	7	88	81
	8	20	12	183	17.5 ± 6.25	682	70.4 ± 15.29	499	52.9 ± 20.33	10	89	79	7	92	85
	8	20	18	198	19.0 ± 16.63	717	74.3 ± 25.28	519	55.2 ± 41.60	11	89	78	2	94	92
	8	20	15	214	19.6 ± 11.00	658	68.7 ± 19.37	444	49.1 ± 29.55	14	80	66	5	92	87
	$R_{Proposed}\left(k\right)$			110	10.4 ± 7.87	748	76.0 ± 17.92	638	65.6 ± 25.57	1	99	98	0	100	100
Environment 2	6	20	15	466	46.5 ± 10.80	378	38.9 ± 13.07	−88	−7.6 ± 23.37	37	48	11	29	61	32
	10	20	15	440	43.7 ± 12.75	384	39.3 ± 22.75	−56	−4.5 ± 34.72	33	58	25	18	79	61
	8	15	15	462	47.8 ± 15.42	411	40.7 ± 21.49	−51	−7.0 ± 35.92	16	80	64	16	80	64
	8	25	15	500	50.4 ± 5.10	354	36.3 ± 12.80	−146	−14.1 ± 14.81	44	53	9	44	54	10
	8	20	12	395	38.1 ± 15.98	467	48.4 ± 21.92	72	10.3 ± 37.65	30	54	24	20	78	58
	8	20	18	478	49.3 ± 13.45	430	43.6 ± 18.30	−48	−5.7 ± 30.81	30	67	37	24	75	51
	8	20	15	278	25.4 ± 14.93	604	64.1 ± 21.92	326	38.6 ± 36.74	30	62	32	10	87	77
	$R_{Proposed}\left(k\right)$			109	9.27 ± 14.31	773	79.6 ± 28.57	644	70.4 ± 42.67	3	96	93	0	100	100
Environment 3	6	20	15	243	22.7 ± 15.42	667	69.3 ± 19.50	424	46.5 ± 34.84	13	82	69	4	91	87
	10	20	15	222	20.3 ± 13.30	651	67.6 ± 19.79	429	47.3 ± 32.74	10	83	73	9	89	80
	8	15	15	445	45.5 ± 14.25	399	39.6 ± 18.03	−46	−5.9 ± 31.35	16	77	61	16	78	62
	8	25	15	737	76.4 ± 34.43	246	21.9 ± 3266	−491	−54.4 ± 67.08	8	87	79	8	88	80
	8	20	12	288	27.9 ± 9.71	584	60.6 ± 17.18	296	32.7 ± 26.30	24	64	40	11	87	76
	8	20	18	435	42.8 ± 7.58	488	51.0 ± 12.27	53	8.2 ± 19.63	43	54	11	25	73	48
	8	20	15	273	27.0 ± 13.83	581	59.2 ± 20.78	308	32.1 ± 34.43	10	86	76	9	88	79
	$R_{Proposed}\left(k\right)$			112	9.7 ± 11.74	849	87.6 ± 17.24	737	77.9 ± 28.93	2	98	96	0	100	100
Environment 4	6	20	15	522	52.1 ± 15.93	386	39.6 ± 21.10	−136	−12.5 ± 36.50	38	59	21	29	67	38
	10	20	15	494	49.6 ± 12.77	419	42.6 ± 13.58	−75	−6.9 ± 26.02	42	54	12	31	63	32
	8	15	15	576	57.8 ± 11.29	348	35.8 ± 11.92	−228	−22.0 ± 22.68	49	46	−3	36	63	27
	8	25	15	444	43.7 ± 13.97	484	49.6 ± 17.50	40	5.9 ± 30.90	38	59	21	22	73	51
	8	20	12	481	47.4 ± 16.27	432	44.8 ± 16.67	−49	−2.5 ± 32.77	42	56	14	25	69	44
	8	20	18	514	51.4 ± 16.68	407	40.9 ± 17.35	−107	−10.5 ± 33.81	33	66	33	21	70	49
	8	20	15	470	45.8 ± 18.99	444	47.2 ± 22.77	−26	1.4 ± 41.50	51	40	−11	15	79	64
	$R_{Proposed}\left(k\right)$			299	29.7 ± 16.13	598	60.5 ± 24.06	299	30.8 ± 40.18	14	83	69	15	84	69

. In this table, the ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ were given in Equation (9). $C_x$, $A_x$, and $S_x(A_x-C_x)$ were shown to be the robot colliding with an obstacle/wall, the robot arriving at the target during training processes, and the score of this sequence, respectively. $All$, $L100$, and $Best$ represent the entire training process, the last 100 episodes of the training process, and the best 100 episodes of the training process, respectively. In Table 3, ${\alpha }_1=8$, ${\alpha }_2=20$, and ${\alpha }_3=15$ values were found by the trial-and-error method. We tested whether there were better values than these values by increasing or decreasing these values by 25%. These results of the increasing or decreasing process are given in Table 3. The $R_{Proposed}\left(k\right)$ function based on fuzzy logic generated a better score than the classic reward function. Also, Table 3 presents the mean ($\overline{X}$) and standard deviation ($SD$) calculated over 100-step windows for the last 900 steps, respectively.

When the proposed fuzzy logic-based reward function was ignored in Table 3, it was seen that there were three highest scoring coefficients for Environment 1. For Environments 2 and 4, the best scores were achieved using coefficients 8, 20, and 15. In Environment 3, the highest score was obtained with coefficients 6, 20, and 15. However, across all four environments, the fuzzy logic-based reward function outperformed the coefficient-based reward functions in generating the best scores. Fuzzy logic-based reward functions have the ability to produce more precise and fine-tuned decisions than classical reward functions. This ability comes from the fact that the reward function, especially the objective function used in reinforcement learning, is more focused on achieving the global maximum value of the reward function. That is, the fuzzy logic approach occurs when the reward function more precisely eliminates not only the global optimum, but also the local optimum. With classical reward functions, the risk of reinforcement learning algorithms getting stuck in local minima is quite high. This can prevent the system from reaching the global optimum and lead to less efficient solutions. However, since fuzzy logic-based methods design the decision-making process to be more flexible and tolerant of uncertainties, such problems are less common. As seen in Table 3, the problem identified using fuzzy logic overcomes the local minimization problem faced by classical approaches. Thus, more precise and optimal decision outputs are obtained. The training performance of the fuzzy logic-based agent is illustrated in Figure 5, Figure 6, Figure 7 and Figure 8. Figure 5 presents the average episode reward, calculated based on the reward values from the last 100 episodes. The robot’s success rate in reaching the target and its collision rate with obstacles or walls is depicted in Figure 6 and Figure 7, respectively, both derived from the last 100 episodes. Finally, Figure 8 displays the training score, determined by subtracting the collision rate from the success rate.

After completing the training process, agents trained with the fuzzy logic-based reward function were tested over 100 steps. The test results were shown in

Table 4. The testing of an agent trained using the $R_{Proposed}\left(k\right)$ reward function during 100 episodes.

				Arrival Time (s)			Distance Traveled (m)
Env	Crashed	Arrived	Score	Min	Max	Median	Min	Max	Median
1	0	100	100	15.872	40.32	23.808	3.122	8.02	4.705
2	0	100	100	18.944	34.176	24.064	3.716	6.774	4.747
3	0	100	100	16.00	34.04	22.144	3.133	7.748	4.362
4	7	91	84	17.152	32.00	24.064	3.203	6.252	4.637

, where ‘env’ represents the environment. As seen in Table 4, the robot achieved a 100% target-reaching rate in Environments 1–3, while in Environment 4, this rate dropped to 91%. Additionally, the robot collided with an obstacle or wall 7 times during testing in Environment 4. However, when the training period was extended to 4000 steps to address this issue, the robot’s target-reaching rate improved to 100%. The measured reward values during the training of artificial neural network architecture for 4000 episodes were presented in Figure 9. The arrival time was measured from the moment the robot started moving until it successfully reached the target. Additionally, the total distance traveled was recorded as the path taken from the starting position to the target.

Agents trained in different environments were tested over 100 episodes. Then, in order to examine the proposed method more clearly, it was tested as presented in Figure 10. Parts a and d of Figure 10 present the starting and ending moments of the experiment. Parts b and c depict the moments when the robot encounters the first and second dynamic obstacles, respectively. The signals produced by the sensor model when the robot encounters the first dynamic obstacle were represented by the yellow-colored area in Figure 11. Similarly, when the robot encounters the second dynamic obstacle, the signals generated by the sensor model were represented by the cyan-colored area in Figure 11. The output data of the trained agent corresponding to these colored areas were presented in the $\Delta V_x$ and $\Delta W_z$ graphs in Figure 12. The linear ($V_x$) and angular velocity ($W_z$) of the robot were obtained by aggregating the $\Delta V_x$ and $\Delta W_z$ values, as in Figure 12. When Figure 10, Figure 11 and Figure 12 were examined together, the robot escaped by changing its angular velocity when it encountered the first obstacle (which can be seen in the yellow-colored areas in Figure 10b, Figure 11 and Figure 12). When it encountered the second obstacle, it decreased its linear speed to negative and moved in the opposite direction. This is indicated by the linear velocity decreasing from 0.2 m/s to −0.2 m/s. During this movement, it also reduces angular velocity from 0.2 rad/s to approximately 0 rad/s. This is because it wants to avoid the obstacle directly, without using its angular velocity. It reached the target after passing the obstacle (which can be seen in the cyan-colored areas of Figure 10c, Figure 11 and Figure 12).

After the training and general testing processes were completed, two robots, with agents trained for Environment 4, were tested in the same environment as presented in Figure 13. Parts a and h of Figure 13 present the starting and ending moments of the experiment. During this test, the paths followed by robot 1 and robot 2 are shown with black and red lines, respectively. As seen in Figure 14, robot 1 detects dynamic obstacles at moments b, d, and f of Figure 13, respectively. These moments were shown in Figure 14 with yellow-, cyan-, and magenta-colored areas. The outputs produced ($\Delta V_x$ and $\Delta W_z$) by the agent in response to these fields and the calculated robot velocities ($V_x$ and $W_z$) are presented in Figure 15. It was observed that robot 1 avoided the first two dynamic obstacles by changing its angular velocity, while it avoided the third obstacle by changing both its angular and linear velocity.

When moments c, d, and e of Figure 13 were examined, robot 2 detected the dynamic obstacle and robot 1, respectively, as can be seen in Figure 16. These moments were shown in Figure 16 with yellow- and cyan-colored areas. The outputs produced ($\Delta V_x$ and $\Delta W_z$) by the agent for these environmental effects and the velocity information ($V_x$ and $W_z$) of robot 2 were visualized in Figure 17. The reaching the target moment of robot 2 was illustrated in Figure 13g. It was observed that robot 2 avoids both the dynamic obstacle and robot 1 by changing the angular velocity.

The advantages of this study are that it proposes a fuzzy logic-based reward function that is more effective than the classical reward function and reduces the size of the artificial neural network architecture by reducing the sensor size. However, the current work has some limitations, such as the dependence on the robot’s position and orientation, the ability to generate responses up to a specific speed of dynamic obstacles, and the wheels can only move up to a maximum speed of 0.2 m/s or rad/s. In general, when Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 are examined, it is observed that when robots encounter an obstacle, they reduce their maximum linear speed from 0.2 m/s to a stop (approximately 0 m/s) or even decrease it to −0.2 m/s in an attempt to evade the obstacle. This speed adjustment applies not only to linear speed but also to angular speed. During the obstacle-avoidance process, the angular speed, initially at 0.2 rad/s, gradually decreases to around 0 rad/s, assisting in the escape maneuver. The primary factor triggering this avoidance behavior is the absence of a sensor input of 1, meaning the detection of an obstacle.

In this study, a total of four different environments were used, progressing from simple to complex. The first environment contains only one static obstacle, while the second and third environments each include one static and one dynamic obstacle. The fourth and most complex environment consists of one static and two dynamic obstacles. The training process was conducted with a clipping ratio of 0.2 and a discount factor of 0.99, as in the original study [61]. Additionally, the learning rate was set to ${10}^{-4}$. The training process was carried out using these parameters in all environments. For similar studies, these training parameters are likely to lead to a viable solution. Additionally, the artificial neural network trained in the fourth environment was used for the multi-robot system presented in this study. The successful results obtained have demonstrated the generalization of the training process.

This study has certain limitations. The first limitation is the dependency of the proposed system on the LiDAR sensor. The robot detects obstacles in its environment using the LiDAR sensor. For robots without a LiDAR sensor, alternative distance-measurement sensors capable of perceiving the entire environment can be used. Another limitation is that the study is designed for differential-drive mobile robots. If the developed algorithm is to be applied to Ackerman mobile robots, an appropriate kinematic transformation must be performed.

4. Conclusions

The current study aimed to present a new approach depending on fuzzy logic reward function in order to improve the classical reward function of the RL. For this purpose, an agent that is a major part of RL has formed actor–critical MLP structures. MLP structures were trained to utilize PPO algorithms. The environment, another important part of RL, was designed using the Webots robot simulator program. A state vector containing the robot, obstacle, and goal parameters was determined. The sensor output data were shrunk to reduce the processing load and the neural network size while creating these parameters. Afterwards, two different reward functions were designed as classic and fuzzy logic. The classic reward function was formed by adding or subtracting conventional system parameters. In order to determine the importance of these parameters relative to each other, some coefficients were required when adding or subtracting. In this study, these coefficients were determined by trial-and-error techniques. It was investigated whether there was a better result in their environment by increasing and decreasing established coefficients, as could be seen in Table 3. Then, the fuzzy logic reward function ($R_{Proposed}\left(k\right)$) was designed in order to eliminate the problem of searching for the most suitable coefficients. The designed $R_{Proposed}\left(k\right)$ function generated the best score in all environments according to Table 3. Then, the fuzzy logic-based trained agents were run through 100 tests in each environment. According to the test results, it was observed that the agents trained in the first three environments had a target arrival rate of 100% in their own environment. For the agent trained in Environment 4, the target arrival rate was measured to be 91% and the rate of hitting an obstacle or wall was 7%. When the training time of the agent was extended up to 4000 episodes, the goal-reaching rate was found to be 100%. Following the general tests, the agent was tested individually and its responses were observed. In individual tests, it was found to effectively avoid dynamic obstacles. The same agent was also tested in another environment with multiple robots. It was observed that the robots in this environment reached the specified targets without hitting any obstacle or robot. According to these results, the shrunk sensor output data performed the requested operation and an effective fuzzy logic-based reward function was successfully created. In future studies, the robust response capability of the system will be improved by applying these methods to reinforcement learning-based systems involving multi-robot systems.