Micro-Platform Verification for LiDAR SLAM-Based Navigation of Mecanum-Wheeled Robot in Warehouse Environment

Abstract

Path navigation for mobile robots critically determines the operational efficiency of warehouse logistics systems. However, the current QR (Quick Response) code path navigation for warehouses suffers from low operational efficiency and poor dynamic adaptability in complex dynamic environments. This paper introduces a deep reinforcement learning and hybrid-algorithm SLAM (Simultaneous Localization and Mapping) path navigation method for Mecanum-wheeled robots, validated with an emphasis on dynamic adaptability and real-time performance. Based on the Gazebo warehouse simulation environment, the TD3 (Twin Deep Deterministic Policy Gradient) path planning method was established for offline training. Then, the Astar-Time Elastic Band (TEB) hybrid path planning algorithm was used to conduct experimental verification in static and dynamic real-world scenarios. Finally, experiments show that the TD3-based path planning for mobile robots makes effective decisions during offline training in the simulation environment, while Astar-TEB accurately completes path planning and navigates around both static and dynamic obstacles in real-world scenarios. Therefore, this verifies the feasibility and effectiveness of the proposed SLAM path navigation for Mecanum-wheeled mobile robots on a miniature warehouse platform.

1. Introduction

As e-commerce grows at a fast pace, the warehousing and logistics sector is confronted with twofold challenges: the surge in orders and the increase in operational complexity [1]. However, the traditional manual handling mode is inefficient and error-prone, hindering the achievement of modern warehousing and logistics’ high efficiency demands. In contrast, mobile robots can maintain high efficiency over a long period, which not only saves labor costs but also significantly improves the accuracy rate. Thus, they have become staples in smart factories, manufacturing hubs, and warehouses for goods handling [2]. As the core technology of mobile robots, path navigation includes environment perception, autonomous positioning, and path planning [3].

Within the domains of environment perception and autonomous positioning, the Quick Response (QR) code path navigation [4] is widely used in warehousing scenarios due to its relatively simple deployment and controllable cost. However, modernized warehousing environments face challenges such as high-frequency personnel movement, goods temporarily blocking the aisles, and dynamically changing operating aisles, making the operating environment complex and variable. Because traditional QR code navigation lacks effective real-time perception, it struggles to adjust paths flexibly in dynamic scenarios, reducing system efficiency. In contrast, laser SLAM (Simultaneous Localization and Mapping) boasts simple equipment requirements, broad applicability, and exceptional positioning precision [5]. It breaks through the dependence of QR code navigation on preset physical markers, enabling mobile robots to autonomously perceive complex dynamic environments. This allows for accurate map construction and simultaneous self-localization in unknown environments [6], thus adapting to complex warehouses and facilitating collision-free, efficient path planning.

In warehouse settings, path planning for mobile robots stands as a core technology enabling autonomous navigation during logistics operations [7]. The majority of traditional path-planning studies are categorized under algebraic, optimization, and AI approaches [8] and have been widely applied to warehouse robot path planning. These methods primarily rely on sensors to perceive the surrounding environment, determine the robot’s position and orientation, and employ planning algorithms to compute the optimal path from start to end. In essence, algebraic strategies employ algebraic equations to tackle exhaustive path planning issues [9], whereas optimization strategies generally resort to heuristic algorithms to optimize the robot’s route [10]. This includes intelligent planning algorithms like the Swarm algorithm [11], the ant colony method [12], and the genetic approach [13], along with the search-centric Astar algorithm [14]. However, these methods generally rely on static, known environments with certain limitations in practical applications. Hybrid algorithms can combine the strengths of multiple algorithms, making them suitable for dynamic environments. Wang et al. proposed a hybrid approach combining an improved Astar algorithm and an optimized Dynamic Window Approach (DWA), enhancing global path optimization efficiency and obstacle avoidance capabilities. It demonstrates improved stability and reliability when traversing both static and dynamic obstacles [15]. Zhang et al. proposed a hybrid algorithm of the Modified Golden Jackal Optimization (MGJO) and the Improved Dynamic Window Approach (IDWA). The problem of being stuck in local optima during global path planning has been resolved, as validated in feasibility experiments [16]. Compared to previous methods, this paper proposes a hybrid path planning algorithm combining Astar and Time Elastic Band (TEB), the feasibility of which has been validated in a warehouse micro-platform.

Lately, Deep Reinforcement Learning (DRL) rooted in AI has found extensive use in robot path planning [17]. Given the complex and dynamic characteristics of warehouse environments, a single path planning algorithm struggles to meet the demands for efficiency and precision in operational processes [18]. DRL is incorporated into mobile robot path navigation, with no need to depend on data or models. Instead, it gathers its own state and environmental data through sensors, and following algorithmic computation, selects the next action based on the reward size. During this process, the system keeps learning and optimizes strategies to pursue the highest reward [19], thereby achieving stable and efficient path planning. Meanwhile, DRL is capable of handling robot path planning tasks with dynamic changes, along with efficiently processing high-dimensional environmental information [20]. Abdalmanan et al. used Deep Q Network (DQN) navigation in open environments to avoid collisions and complete path planning [21]. Since DQN cannot handle continuous action control, Shripad V et al. built upon this foundation and adopted the Deep Deterministic Policy Gradient (DDPG). Solving decision problems under uncertainty and dynamics in a two-dimensional environment has improved the success rate of reaching targets [22]. Choi et al. proposed a hierarchical DRL architecture that mainly uses the Soft Actor–Critic (SAC) algorithm for path planning in both static and dynamic surroundings. An integrated path planner was used to improve performance [23]. Fujimoto et al. proposed Twin Deep Deterministic Policy Gradient (TD3), which introduces a twin Q-network and a delayed update strategy [24]. It enables handling of continuous control tasks and resolves the Q-value overestimation issue in DDPG. Additionally, compared to SAC, the TD3 lightweight design enables higher computational efficiency, allowing it to rapidly converge to high-precision control strategies [25].

Despite extensive research in navigation path planning, most existing studies focus on simple heuristic algorithms applied to two-dimensional maps. These approaches suffer from oversimplifying real-world warehouse environments and poor dynamic adaptability. While a few have proposed DRL for path navigation, they lack effective experimental validation in warehouse settings.

This paper aims to bridge these gaps by making the following key contributions:

(1) A Gazebo-based 3D warehouse environment for the TD3-based path planning algorithm is established. The offline training results confirm that TD3 applies to deep reinforcement learning-based path planning for mobile robots in logistics and warehousing environments.

(2) A Mecanum-wheeled mobile robot platform was developed to experimentally validate the feasibility of the hybrid Astra-TEB path planning approach in real-world scenarios. Experiments confirm that robots can achieve efficient obstacle avoidance in both static and dynamic environments.

This study proceeds as follows: Section 2 constructs the Mecanum-wheeled robot and warehouse environment. Section 3 presents the TD3 path planning method. Section 4 demonstrates warehousing micro-platform verification for navigation. Lastly, the conclusion is presented in Section 5.

2. Mecanum-Wheeled Robot and Warehouse Environment

2.1. Modeling for Mecanum-Wheeled Robots Based on SLAM

Figure 1 shows the integral module for the SLAM mobile robot, which includes a control module, a sensor module, an actuation module, and a monitoring module. Among them, the sensor module consists of LiDAR, a binocular camera, an IMU, an odometer, etc., gathering data on both the environment and the autonomous robot. The control module is tasked with processing and computing sensor data, which consists of the main control module (Jeston Nano) and the underlying controller (STM32). The bottom controller first receives and fuses the sensor information and then transmits the processed data to the master control module, which utilizes SLAM to realize the positioning and map construction of the robot. The actuator module drives the robot according to the instructions of the control module. The monitoring module monitors the real-time data and movement of the mobile robot using a PC.

Figure 1. Integral module for mobile robot.

Among them, the wheel part of the SLAM robot is modeled as a Mecanum wheel. The Mecanum wheel is equipped with three degrees of freedom, including rotation around the wheel axis, movement in the direction of the roller axis, and rotation about the wheel–ground contact point [26]. The Mecanum wheel is based on the chiral symmetry of the left-rotating and right-rotating wheels, which not only increases the stability of the mechanism but also improves the load capacity while increasing the control convenience. Figure 2 shows the Mecanum-wheeled robot’s kinematic structure.

Figure 2. Kinematic modeling of a mobile robot based on the Mecanum wheel.

The relationship between the translational velocities $v_{cx}$,$v_{cy}$ of the C-wheel centroid and the overall velocities $v_x$, $v_y$, $v_{z }$ is as follows:

$$v_{cx}=v_x+v_z\times \sqrt{{\left({\displaystyle \frac{H}{2}}\right)}^2+{\left({\displaystyle \frac{W}{2}}\right)}^2}\times \sin {\alpha }_C$$

(1)

$$v_{c_y}=v_y+v_z\times \sqrt{{\left({\displaystyle \frac{H}{2}}\right)}^2+{\left({\displaystyle \frac{W}{2}}\right)}^2}\times \cos {\alpha }_C$$

(2)

where $W$ represents the wheelbase between the left and right Mecanum wheels; H represents the wheelbase between the front and rear Mecanum wheels; $v_x$ and $v_y$ are the x and y axis translational velocities; $v_z$ is the speed of rotation of the robot around the z axis; and ${\alpha }_C$ denotes the angle between the wheel’s center of mass and the line from the origin.

The moving velocities $v_{cx}$, $v_{cy}$ of the C-wheel’s center of mass are related to the wheel velocity $v_{cw}$ and the roller linear velocity $v_{cr}$ as follows:

$$v_{c_x}=v_{cw}+v_{cr} \times \sin \beta$$

(3)

$$v_{c_y}=-v_{cr}\times \cos \beta$$

(4)

where $\beta$ denotes the angle between the wheel axis and the roller axis.

Therefore, the C-wheel roller line velocity is as follows:

$$v_{cw}+v_{cr}\times \sin \beta =v_x+v_z\times \sqrt{{\left({\displaystyle \frac{H}{2}}\right)}^2+{\left({\displaystyle \frac{W}{2}}\right)}^2}\times \sin {\alpha }_C$$

(5)

$$v_{cr}=-{\displaystyle \frac{v_y+v_z\times \sqrt{{\left(\frac{H}{2}\right)}^2+{\left(\frac{W}{2}\right)}^2}\times \cos {\alpha }_C}{\cos \beta }}$$

(6)

Based on Equations (3) and (6), define $l=\sqrt{{\left({\displaystyle \frac{H}{2}}\right)}^2+{\left({\displaystyle \frac{W}{2}}\right)}^2}$.

$$v_{cw}-v_y-v_z\times l\times \cos {\alpha }_C\times \tan \beta =v_x+v_z\times l\times \sin {\alpha }_C$$

(7)

$$v_{cw}-v_y-v_z\times l\times \cos {\alpha }_C\times \tan \beta =v_x+v_z\times l\times \sin {\alpha }_C$$

(8)

Given that $l\ast \sin {\alpha }_C={\displaystyle \frac{H}{2}} \mathrm{a}\mathrm{n}\mathrm{d} l\ast \cos {\alpha }_C={\displaystyle \frac{W}{2}}$, the velocity of the C-wheel is calculated as follows:

$$v_{cw}=v_x+v_y+v_z\times \left({\displaystyle \frac{H}{2}}+{\displaystyle \frac{W}{2}}\right)$$

(9)

A-wheel, B-wheel, and D-wheel center of mass velocities are related to roller linear velocities as follows:

$$\left\{\begin{array}{c}{v}_{a_y}=-v_{ar}\times \mathit{cos}\beta \\ v_{b_y}=v_{br}\times \cos \beta \\ v_{d_y}=v_{dr}\times \cos \beta \end{array}\right.$$

(10)

Finally, based on Equation (7), we solve the overall velocities $v_x,v_y, \mathrm{a}\mathrm{n}\mathrm{d} v_z$ as follows:

$$v_{aw}=v_x+v_y-v_z\times l\times \left(\sin {\alpha }_C+\cos {\alpha }_C\right)$$

(11)

$$v_{bw}=v_x-v_y-v_z\times l\times \left(\sin {\alpha }_C+\cos {\alpha }_C\right)$$

(12)

$$v_{cw}=v_x+v_y+v_z\times l\times \left(\sin {\alpha }_C+\cos {\alpha }_C\right)$$

(13)

$$v_{dw}=v_x-v_y+v_z\times l\times \left(\sin {\alpha }_C+\cos {\alpha }_C\right)$$

(14)

$$v_x={\displaystyle \frac{v_{aw}+v_{bw}+v_{cw}+v_{dw}}{4}}$$

(15)

$$v_y={\displaystyle \frac{v_{aw}-v_{bw}+v_{cw}-v_{dw}}{4}}$$

(16)

$$v_z={\displaystyle \frac{-v_{aw}-v_{bw}+v_{cw}+v_{dw}}{2H+2W}}$$

(17)

Derived from the above formula, the Mecanum-wheeled robot’s kinematic model yields its overall velocities in x, y, and z directions and four wheel speeds.

2.2. Gazebo-Based Warehouse Environment Construction

The construction of the warehouse simulation environment is based on the floor plan of an intelligent warehouse, as shown in Figure 3. It includes seven areas: receiving area, AR/RS warehouse area, distribution area, picking area, pallet storage area, packing area, and shipping area. The basic workflow is as follows:

Figure 3. CAD intelligent warehouse floor plan.

(1)

Preparation for goods receipt: Palletized-unit goods are first unloaded to the receiving area in preparation for warehousing. After operations including quantity counting, inspection, label printing, and system entry are completed, forklifts transport the goods to the entrance of the AR/RS warehouse area for storage.

(2)

Order processing: Based on order requirements, goods are retrieved from the AR/RS warehouse area. AGVs then transport the goods to the distribution area for buffering, after which they are sent to the picking area for order picking operations.

(3)

Pallet handling: Empty pallets are conveyed to the pallet storage area, while nonempty pallets are returned to the distribution area shelf for storage.

(4)

Goods sorting and distribution: The picked goods are transferred via conveyor belts to the packing area for packaging and subsequently loaded onto trucks in the shipping area for distribution.

Aiming at the function of the robot in the distribution area, design a warehouse environment simulation scenario. It picks up the goods from the distribution area and sends them to the picking area for picking, then sends the empty pallet to the pallet storage area, and the nonempty pallet is sent back to the distribution area for storage.

By the design of the aforementioned training scenarios, a 3D warehouse model is constructed using Gazebo 11, forming the platform basis for subsequent TD3-based mobile robot path planning training. The specific parameters for warehouse environment modeling in the Gazebo environment are presented in Table 1. The 3D training scenario is built in Gazebo simulation software, as shown in Figure 4.

Table 1. Gazebo modeling parameters.

Warehouse Area	x-Coordinate (m)	y-Coordinate (m)
AR/RS	0	(6, 9)
Distribution	(−1, 8)	(−9, 6)
Pallet storage	(−7, 9)	(−9, 6)
Picking	(−3, 5)	(−9, 6)
Overall storage	(−8, 8)	(−9, 6)

Figure 4. Construction of the Gazebo Training Scenario.

3. TD3 Deep Reinforcement Learning-Based Path Planning

3.1. DRL Problem Modeling

3.1.1. State Space

Mobile robots must gather environmental and their own positional information so that they can make optimal decisions after training with deep reinforcement learning algorithms. This information constitutes the “state space.” In this paper, the state space S for path planning encompasses two aspects: first, the positional relationship between the robot and obstacles, along with the distance from the robot to the destination; second, its own posture data and movement state [27].

$${ d}_o=[{ d}_{o1},{ d}_{o2},...{ d}_{on}]$$

(18)

where ${ d}_o$ represents the distance of obstacles in n directions obtained by laser radar.

$${ d}_t=\sqrt{{{(X}_t-X)}^2+{{(Y}_t-Y)}^2}$$

(19)

$${ \varnothing }_t=arctan2 (Y-Y_t, X-X_t)-{ \theta }_t$$

(20)

where ${ d}_t$ represents the distance of the robot from the end point at time t; ${ \varnothing }_t$ denotes the azimuth;${ (X}_t$, $Y_t$) is the coordinate of the robot at time t; $(X, Y$) is the coordinate of the end point; and ${ \theta }_t$ denotes the current yaw angle derived from the mobile robot’s IMU.

We employ the Boolean variable ‘done’ as a state variable: it is True if a collision occurs or the end is reached at the end of training, and False otherwise, as shown in Equation (18).

$$done=\left\{\begin{array}{c}True, d_o<0.35 \mathrm{m} \mathrm{or} d_t<0.3 \mathrm{m}\\ \\ False, \mathrm{otherwise}\end{array}\right.$$

(21)

Therefore, the state space S of this paper is shown in Equation (19).

$$S=\left[{ d_o, d}_t,{\varnothing }_t, X_t,Y_t\right]$$

(22)

3.1.2. Action Space

Continuous movement results in a relatively smooth path for the robot, without frequent or abrupt starts and stops, matching operational traits of the real world. Therefore, the robot action space includes specific linear and angular speeds [28].

$$A=[v, w]$$

(23)

where $v$ and $w$ are the linear speed and rotational angular speed of the robot, respectively.

3.1.3. Reward Function

To obtain higher positive rewards for continuous movement and thereby facilitate forward exploration, the mobile robot is designed with a reward function. During normal operation, it should complete path planning smoothly and safely, with an immediate reward ${\displaystyle \frac{\mathrm{v}}{2}}-\left|{\displaystyle \frac{\mathrm{w}}{2}}\right|-\mathrm{r}3{\displaystyle \frac{d_{\mathrm{t}}}{2}}$ assigned per timestep. This formulation encourages rapid task completion through positive velocity reward v, penalizes aggressive turning through the negative angular velocity term |w|, and enforces obstacle avoidance through safety distance penalty $r3$.

$$r3=\left\{\begin{array}{c}0,{ d}_t>1\\ 1-d_t, d_t<1\end{array}\right.$$

(24)

$$R_1={\displaystyle \frac{\mathrm{v}}{2}}-\left|{\displaystyle \frac{\mathrm{w}}{2}}\right|-\mathrm{r}3{\displaystyle \frac{{\mathrm{d}}_{\mathrm{t}}}{2}}$$

(25)

where r3 represents the weight for penalizing the distance between the mobile robot and an obstacle, and $R_1$ represents the movement state reward.

$$R_2=\left\{\begin{array}{c}{ 5\ast (d}_t-{ d}_{t-1}{), d}_t>d_{t-1}\\ {-10\ast (d}_t-{ d}_{t-1}{), d}_t<{ d}_{t-1}\end{array}\right.$$

(26)

where $R_2$ is a proximity reward function. It establishes the reward for each step according to the distance of the mobile robot to the destination.

$$R_3=\left\{\begin{array}{c}100,d<0.03 \mathrm{m}\\ -100, d_t<0.35 \mathrm{m}\end{array}\right.$$

(27)

where $R_3$ represents collision penalties and rewards for reaching end points.

$$R=R_1+R_2+R_3$$

(28)

where R represents the total reward. By using $R_1$, $R_2$, and $R_3,$ a comprehensive reward function is constructed, encompassing three components: movement state reward, end-approaching reward, and end-reaching reward. This incentivizes the robot to seek higher reward values for optimal path exploration continuously.

3.2. TD3 Algorithm Design

The TD3 is an improved version of DDPG, which refines a structure comprising one Actor and two Critic Networks. Firstly, TD3 employs twin Critics to calculate the corresponding Q-value to actions and selects the network yielding the minimum Q-value, thereby addressing the Q-value overestimation issue inherent in DDPG [29]. Secondly, it incorporates delayed policy updates for the Actor, whereby the Critics updates multiple times until attaining a relatively stable state, after which the Actor performs an update to more precisely determine the max Q-value estimate. Finally, it introduces target policy smoothing by adding noise to allow TD3 to avoid over-fitting in the policy network [30]. Therefore, TD3 requires the construction of an Actor, which takes the environmental data as input and outputs the mobile robot actions, along with two Critics that take both the environmental state and actions from the Actor as inputs and output the Q (state, action).

In the intelligent warehouse environment, the TD3 algorithm uses the Actor to ${\pi }_{v^{\prime } (s_{t + 1})}$ outputs a deterministic action for the current state S. To encourage exploration, exploration noise $\delta$ is added.

$$a_t={\pi }_{v^{\prime }(s_{t+1})}+\delta$$

(29)

Use two target Critics $Q_{w_i\prime } (s_{t + 1}, a_t)$ and take the minimum Q-value.

$$y=r+\mathsf{\gamma }{\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{i}=\mathrm{1,2}}{Q}_{w_i\prime }(s_{t+1},a_t)$$

(30)

where $r$ is the immediate reward, and $\mathsf{\gamma }$ (with $\gamma \in (0,1)$) is a discount factor determining the priority of long-term rewards.

In order to avoid frequent updates of the Actor leading to policy oscillations or unstable training, TD3 updates the Actor only after a certain number of Critic updates. Loss functions for the Critic and Actor are outlined below:

$$L(w_i)=E_s~D\left[{(Q_{w_i\prime }(\mathrm{s},a)-y)}^2\right]$$

(31)

$$L(v)=-E_s~D[Q_{w_i\prime }(\mathrm{s},{\pi }_{v(s)})]$$

(32)

The Critic loss $L(w_i)$ is trained to minimize the squared error between its predicted Q-value $Q_{w_i\prime }(\mathrm{s},a)$ and the target Q-value y. Additionally, the Actor $L(v)$ maximizes the $Q_{w_i\prime }(\mathrm{s},{\pi }_{v(s)}$) for the actions ${\pi }_{v(s)}$ it outputs.

Update the actor by the deterministic policy gradient to maximize expected return via gradient ascent.

$${\nabla }_{v}J(v )={\displaystyle \frac{1}{N}}\sum _{\mathrm{s}\in \mathsf{\beta }}{\nabla }_{\mathrm{a}}{Q}_{w1}(s, a){|}_{a ={\pi }_{v(s)}}{\nabla }_v{\pi }_{v(s)}$$

(33)

where $\beta$ denotes a randomly sampled batch of states from the experience replay buffer, with batch size N; ${\mathrm{Q}}_{\mathrm{w}1 }(\mathrm{s},\mathrm{a}$) represents the gradient of the output of the first Critic network with respect to the action input $a_t$; and ${\nabla }_v{\mathsf{\pi }}_{v(\mathrm{s})}$ is the gradient of the output of the Actor network ${ \mathsf{\pi }}_v$ at state s with respect to its own parameters v.

The $\tau$ soft update coefficients are set by the soft update method to update the target network parameters in the direction of the current network parameters in small increments to avoid training oscillations due to parameter updates that are too fast.

$$\left\{\begin{array}{c}{w_i}^{\prime }\leftarrow \tau w_{\mathrm{i}}+(1-\tau ){w_i}^{\prime }\\ v^{\prime }\leftarrow \tau v+(1-\tau )v^{\prime }\end{array}\right.$$

(34)

Building upon the above-described TD3 algorithm design, a process flow for the offline training of the TD3 path planning algorithm in an intelligent warehousing scenario is established, as shown in Algorithm 1.

Algorithm 1: TD3 Path Planning Strategy

Initialize $\mathrm{Critic} \mathrm{Networks} {\mathrm{Q}}_{\mathrm{w}1}, {\mathrm{Q}}_{\mathrm{w}2}$$, \mathrm{and} \mathrm{Actor} \mathrm{Network}{ \mathsf{\pi }}_{\mathrm{v}}$ with random weights; $\mathrm{target} \mathrm{networks} {{\mathrm{w}}_1}^{\prime }\leftarrow {\mathrm{w}}_1, {\mathrm{w}}_2^{\prime }\leftarrow {\mathrm{w}}_2, v^{\prime }\leftarrow v$; $\mathrm{replay} \mathrm{buffer} \mathrm{D}$$; \mathrm{exploration} \mathrm{noise} \mathsf{\delta }$; random near obstacle triggers = true. for t = 1 to T do  $\mathrm{observe} \mathrm{reward} \mathrm{R} \mathrm{and} \mathrm{new} \mathrm{state} {\mathrm{s}}_{\mathrm{t} + 1}$  $\mathrm{Store} \mathrm{transition} ({\mathrm{s}}_{\mathrm{t}}$$, {\mathrm{a}}_{\mathrm{t}}$$, \mathrm{R}, {\mathrm{s}}_{\mathrm{t} + 1}$$) \mathrm{in} \mathrm{D}$  $\mathrm{Randomly} \mathrm{sample} \mathrm{a} \mathrm{batch} \mathsf{\beta }=\left\{(\mathrm{s}, \mathrm{a}, \mathrm{r}, {\mathrm{s}}^{\prime }) \right\}$ of size N from D  $\mathrm{Clip} {\mathrm{a}}_{\mathrm{t}}$ to [−1, 1];  Select action:  $\mathrm{If} { \mathrm{d}}_{\mathrm{o}}$ < 0.35  random near obstacle triggers=true  ${\mathrm{a}}_{\mathrm{t}}$= pre-generated random action(8-15steps)  $\mathrm{V} ={\mathrm{v}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ w = [−3.14, 3.14] rad/s  random steps counter = random steps counter− 1  $\mathrm{Else} {\mathrm{a}}_{\mathrm{t}} = {\mathsf{\pi }}_{{\mathrm{v}}^{\prime }({\mathrm{s}}^{\prime })}$$+\mathsf{\delta }$$, \mathsf{\delta } ~\mathrm{clip} (\mathrm{N} (0,\mathsf{\sigma }),\text{-}\mathrm{k}, \mathrm{k})$  $\mathrm{Compute} \mathrm{Target} \mathrm{Value}: \mathrm{y} = \mathrm{r} + \mathsf{\gamma } {\min }_{\mathrm{i} = 1,2}{\mathrm{Q}}_{{{\mathrm{w}}_{\mathrm{i}}}^{\prime }}({\mathrm{s}}_{\mathrm{t} + 1},{\mathrm{a}}_{\mathrm{t}})$  $\mathrm{Update} \mathrm{Critics} {\mathrm{w}}_{\mathrm{i}} = {\displaystyle \frac{1}{\mathrm{N}}}{\min }_{{\mathrm{w}}_{\mathrm{i}=1,2}}\sum _{(\mathrm{s}, \mathrm{a}, \mathrm{r}, {\mathrm{s}}^{\prime })\in \mathsf{\beta }}{(\mathrm{y}\text{-}{\mathrm{Q}}_{\mathrm{wi}}(\mathrm{s}, \mathrm{a}))}^2$  if t mod d = 0:   Update actor by the deterministic policy gradient   Update target networks  end if end for

3.3. TD3 Algorithm Training

3.3.1. Setup of Training Environment

The experiments were carried out using the following software: Ubuntu 20.04, Python 3.8.10, and PyTorch1.10. A computer with a Ryzen 7 5800H, 16 Gb RAM, and an NVIDIA GeForce RTX 4050 graphics card with 6 GB GDDR6 graphics memory was used. The total training time is 16.5 h.

In the Gazebo simulation environment, a mobile robot is simulated using Pioneer P3DX, which is equipped with LiDAR sensors based on the principle of calculating the distance between objects according to the time-of-flight method. The laser data are transmitted to the TD3 network, and the output angular and linear velocities are used to control the movement of the P3DX, as shown in Figure 5. The P3DX picks up the goods from the distribution area and delivers them to the picking area. Additionally, the P3DX sends the empty pallets from the picking point to the pallet storage area, and the nonempty pallets are sent back to the storage point in the distribution area. Training parameters are configured as illustrated in Table 2.

Figure 5. Gazebo simulation training scene.

Table 2. Settings of parameters for the TD3 algorithm.

Training Parameter	Parameter Value
Max_steps	500
Soft Update Coefficient	0.005
Batch Size	40
Picking	2
Discount Factor	0.99
Replay Buffer Size	1 × 10−6
Noise	0.5

The TD3 training of P3DX is realized by setting up the training environment, starting the training cycle, single-step execution process, reward calculation, evaluation, and reset, and the training process is outlined in Figure 6.

Figure 6. Approximate training process of P3DX.

3.3.2. Training Results and Analysis

The TD3 algorithm underwent 480 training rounds in the Gazebo simulation environment, and the variation of max and average reward with training round counts is shown in Figure 7. Based on the training performance of the TD3 algorithm, the learning process can be divided into the following three distinct phases:

Figure 7. TD3 reward value. (a) Max reward; (b) average reward.

In the first phase (Episodes 0–100), the maximum reward exhibits oscillatory behavior with a gradual upward trend, while the average reward declines to its lowest level. This indicates that the P3-DX is in an intensive exploration stage, frequently colliding with obstacles in the warehouse environment.

In the second phase (Episodes 100–250), the maximum reward experiences significant fluctuations after a peak value, reflecting the agent’s attempt to explore high-risk, high-reward maneuvers. Meanwhile, the average reward demonstrates a rapid increase, suggesting that the algorithm begins to identify potentially effective navigation policies and progressively improves its performance.

In the final phase (after Episode 250), both the maximum and average rewards stabilize at high values, indicating convergence of the TD3 algorithm. The agent has successfully learned a robust and efficient policy, enabling consistent and effective path planning within the intelligent warehouse environment.

4. Warehousing Micro-Platform Verification for Navigation

The TD3 algorithm undergoes offline training of mobile robots in a large-scale warehouse simulation [31], and the strategy is continuously adjusted according to the environmental feedback to keep pace with the live dynamic variations in the complex and changing environment. However, in practical application scenarios, TD3 training relies on a large number of samples, which is insufficient in real-time, and at the same time puts high requirements on the arithmetic power of the hardware platform. In existing platform applications, the Astar-TEB hybrid algorithm can realize low-cost and rapid deployment of applications; the Astar algorithm can efficiently plan global paths based on known maps [32]; and the TEB algorithm is a local planning algorithm, which can make up for the shortcomings of Astar that is too dependent on static environments and carry out local tiny dynamic adjustments of paths. The advantages of Astar’s fast path planning and TEB’s local dynamic adjustment are utilized as the refinement and supplementation of the TD3 algorithm’s bottom-layer execution in real scenarios, and in parallel, the real-time feedback from path planning results is used to guide the training of the TD3 algorithm and accelerate its convergence so as to guarantee the performance of real-time path planning.

4.1. Astar-TEB Algorithm Experiments

To confirm the effectiveness of SLAM mobile robot path planning in real environments, a physically simulated warehouse environment is constructed for simulations to assess the path planning performance of the Astar-TEB hybrid algorithm.

According to Figure 8, drawing on Gazebo’s warehouse environment simulation to build the actual training scene, it is divided into a pallet storage area, a picking area, a distribution area of three regions, and a mobile robot, and based on the ROS platform, using the cartographer [33] algorithm to build the map.

Figure 8. Actual training scenarios.

The mobile robots were equipped with the Raysens N10 LiDAR and N200WP odometer, with the same computer hardware as the TD3 experiment. The LiDAR adopts the TOF (Time of Flight), capable of 360° 2D scanning and detection of the surrounding environment. Its designed detection accuracy is ±3 cm, and the maximum range is 25 m. Testing was conducted in a 5×2.5 m environment with a total of 10 repeated experiments, achieving an average of 34 s per full task cycle with an output latency of 0.1 s.

For path planning on the warehousing micro-platform, a hybrid approach merging the global Astar algorithm and local TEB algorithm is employed [34]. In the RViz map, 0, 1, 2, and 3 target points are set, with both the starting point and end point located at position 0 in the distribution area. The live training process, traveling trajectory map, and real-time speed and corner diagrams are employed to demonstrate the operational state of the mobile robot during logistics tasks.

In the first stage, the robot departs from the distribution area at point 0 and arrives at point 1, as shown in Figure 9, which simulates that the robot carries the goods from the distribution area to the picking area for picking operations. Next, proceed straight from the picking area. It decelerates and turns from point 2 (purple circle), depicted in Figure 10, to simulate the path of completing a pallet storage operation. Finally, it completes one warehousing operation task from point 3 to reach the end of point 0, as shown in Figure 11.

Figure 9. Distribution area start to picking area.

Figure 10. Picking area to pallet storage area.

Figure 11. Pallet storage area to the distribution area end point.

The overall operation trajectory, real-time speed, and corner state in the micro-platform experiment are shown in Figure 12, Figure 13, Figure 14 and Figure 15. Ultimately, the robot successfully navigated around obstacles at every turning point, completing the autonomous path planning task without incident.

Figure 12. Traveling trajectory, real-time speed, and angle in the first stage.

Figure 13. Traveling trajectory, real-time speed, and angle in the second stage.

Figure 14. Traveling trajectory, real-time speed, and angle in the third stage.

Figure 15. The total traveling trajectory, real-time speed, and angle.

(1)

In the Distribution to Picking Area, the robot traveled for 10 s with an average speed of 0.42 m/s. The speed variation reached 0.5 m/s, and the steering angle variation remained moderate at 20°, reflecting adaptation to the wide aisles and sparse obstacles in this segment.

(2)

In the Picking to Pallet Storage, the travel time increased to 18 s. The average speed decreased to 0.30 m/s to ensure safety during turning maneuvers. Steering variation reached 200°, reflecting multiple directional adjustments.

(3)

In the Pallet Storage to Distribution Area, the robot operated for 9 s. It maintained a low speed of 0.15 m/s, with minimal speed fluctuation. This motion profile emphasizes precise positioning and stability during the final approach to the target. These patterns align with expected behavior in structured warehouse environments.

In summary, the micro-platform experiment simulates the mobile robot to realize the logistics handling and sorting operation in the warehouse environment, thus verifying the effectiveness of the path planning of the Astar-TEB hybrid algorithm.

4.2. Dynamic Scene Testing of Mobile Robot

To verify the robots’ path planning ability for logistics warehouse operations under unexpected emergency conditions, two dynamic scenarios involving obstacles blocking the road and dynamic obstacle avoidance are setup to conduct a performance test and verify the dynamic obstacle avoidance ability of the Astar-TEB hybrid algorithm.

4.2.1. Obstacle Blocking Road Environment Test

Scattered obstacles are added to the preset path of the actual training scenario, and the mobile robot detects the obstacles ahead, avoids the obstacles through the Astar-TEB hybrid algorithm, replans the path, and reaches the destination, as shown in Figure 16.

Figure 16. Obstacles blocking the road experiment.

4.2.2. Dynamic Obstacle Avoidance Environment Test

In the actual training scenario, the mobile robot detects the vehicle ahead, then decelerates and performs obstacle avoidance through the Astar-TEB hybrid algorithm to ensure that it has a certain safety distance from the vehicle, after which it replans the path to reach the destination, as shown in Figure 17.

Figure 17. Dynamic obstacle avoidance scenario experiment.

Results of 15 repeated obstacle avoidance experiments show the robot performs better in static obstacle scenarios, as shown in Table 3 below. In dynamic obstacle scenarios, the robot’s performance declines, but it still successfully completes the task in most cases. Therefore, the Astar-TEB hybrid algorithm can continuously obtain real-time environmental information and rectify or reset the path to complete the path planning in the warehouse environment.

Table 3. Results of the evaluation performance in static and dynamic environments.

Stage	Re-Planning Time(s)	Success Rate
Static Obstacle	0.2	93%
Dynamic Obstacle	2.3	80%

5. Conclusions

In pursuit of mobile robots’ dynamic adaptability and responsiveness in logistics warehousing settings, this paper explores and verifies a path planning strategy based on the TD3 and Astar-TEB hybrid algorithm. Key findings are elaborated below.

(1)

Based on the TD3 deep reinforcement learning path planning method, offline training and verification were performed via a Gazebo-simulated warehouse model. The simulation results indicate that mobile robots can make continuous, effective decisions and complete path planning tasks in warehouse environments.

(2)

A laser SLAM-equipped Mecanum-wheeled mobile robot based on the Astar-TEB hybrid algorithm. Experiments demonstrate the successful completion of path planning tasks and obstacle avoidance in both static and dynamic real-world scenarios.

In the future, the above TD3 deep reinforcement learning offline training results will be deployed to Mecanum-wheeled mobile robots for real-scenario vehicle verification. Next, modifications to the state space design and reward function will be made to accommodate platform-specific constraints and task requirements. Finally, for larger warehouse maps, the system can be extended to hybrid SLAM full-scale automated guided vehicles (AGVs) or autonomous mobile robots (AMRs), accounting for additional state space dimensions such as shelf height and lighting conditions. Such measures will allow the proposed path planning strategy to be effectively applied in actual logistics and warehousing scenarios.