A Reinforcement Learning-Based Double Layer Controller for Mobile Robot in Human-Shared Environments

Abstract

Various approaches have been explored to address the path planning problem for mobile robots. However, it remains a significant challenge, particularly in environments where a multi-tasking mobile robot operates alongside stochastically moving humans. This paper focuses on path planning for a mobile robot executing multiple pickup and delivery tasks in an environment shared with humans. To plan a safe path and achieve high task success rate, a Reinforcement Learning (RL)-based double layer controller is proposed in which a double-layer learning algorithm is developed. The high-level layer integrates a Finite-State Automaton (FSA) with RL to perform global strategy learning and task-level decision-making. The low-level layer handles local path planning by incorporating a Markov Decision Process (MDP) that accounts for environmental uncertainties. We verify the proposed double layer algorithm under different configurations and evaluate its performance based on several metrics, including task success rate, reward, etc. The proposed method outperforms conventional RL in terms of reward (+63.1%) and task success rate (+113.0%). The simulation results demonstrate the effectiveness of the proposed algorithm in solving path planning problem with stochastic human uncertainties.

1. Introduction

In recent years, autonomous mobile robots have been widely employed in automatic factories [1], logistic centers [2], warehouses [3], and manufacturing systems [4], playing an increasingly important role in transportation, pickup, and delivery systems. Today, the demand for autonomous mobile robots that can collaborate with humans is rapidly increasing. Consequently, planning an optimal safe path for a mobile robot performing multiple pickup and delivery operations in human-shared environments has become a critical and urgent challenge.

Path planning is widely recognized as one of the fundamental problems in robotic control. For a mobile robot performing multiple tasks, the control problem can generally be divided into two subproblems: the task scheduling problem, which determines the order in which tasks are executed, and the local path planning problem, which computes an optimal path to accomplish each individual task. The task scheduling problem determines the execution order of tasks, aiming to find an optimal sequence for visiting multiple goal locations. This is commonly formulated as the Vehicle Routing Problem (VRP) [5] or the Traveling Salesman Problem (TSP) [6], which have been extensively studied in static environments where the path cost between locations is fixed and known. However, the path cost in dynamic environments is uncertain and often time-varying. Local path planning focuses on computing a conflict-free path from a start location to a goal location while satisfying specific performance criteria such as minimizing travel distance, commonly referred to as the Shortest Path Planning (SPP) problem [7,8]. These two subproblems are deeply interdependent: the outcome of task scheduling directly influences local planning decisions; in turn, local path feasibility can affect task scheduling results. Moreover, both subproblems are significantly affected by the presence of stochastically moving humans in human-shared environments, which introduces uncertainty and increases the complexity of planning.

As is well known, path planning in static environments has already been studied in considerable depth and various methods have been developed, including D* [9] and A* [10]. However, the path planning problem for a mobile robot designed with multiple pickup and delivery tasks is NP-hard and remains a significant challenge, particularly in dynamic environments. The primary difficulties arise from (1) controlling multiple tasks, and (2) dealing with environmental uncertainties, including dynamic obstacles and stochastic human behavior.

To date, various methods have been developed to generate a global path for a mobile robot performing multiple tasks. To manage task sequences, syntactically co-safe Linear Temporal Logic (scLTL)—a fragment of Linear Temporal Logic (LTL) [11]—has been widely applied for specifying and controlling multiple tasks formulated as scLTL expressions [12]. For example, Mi et al. [13] employed scLTL to represent task specifications and combined it with the A* pathfinding algorithm. It has been proved that scLTL is efficient in task descriptions and control. In terms of path planning, real-time algorithms [14,15] are commonly used to ensure safe navigation by avoiding moving obstacles in dynamic environments. These methods typically adopt a hierarchical path planning framework; instead, a global planner provides guidance toward the goal, while a local planner is responsible for obstacle avoidance. Unfortunately, real-time path planning methods incur high computational costs due to continuous re-planning, and are unable to explicitly address environmental uncertainties at the planning level. Although real-time re-planning at each time step is widely regarded as the safest strategy for handling environmental uncertainties [16], it is often impractical in real-world scenarios such as warehouses, logistics centers, and automated factories. This is primarily because mobile robots in these environments are required to complete tasks within strict time constraints, making constant re-planning computationally infeasible.

Reinforcement Learning (RL) [17,18] has been widely adopted for modeling complex environmental dynamics and learning optimal control policies through interaction. Numerous Deep Reinforcement Learning (DRL) methods [19,20] have been developed for planning in dynamic environments. Chai et al. [21] proposed a hierarchical Deep Learning (DL)-based control framework to plan optimal maneuver trajectories and guide the robot with uncertainties. However, their proposed motion planning model was trained offline and the learning cost was expensive. In our previous work [22], we proposed an FSA-MDP controller by integrating scLTL with a Markov Decision Process (MDP) to plan a global path for the mobile robot considering the uncertainties of jamming attackers. However, this approach still relies on real-time re-planning mechanisms when environmental changes occur.

Recent studies have demonstrated that addressing uncertainties at the planning level is not only effective but also more practical for real-world deployment [13]. On the one hand, computational costs are significantly reduced, as continuous re-planning is no longer required. On the other, it helps to prevent the robot from becoming trapped in local optima, improving the overall task success rate. In human-shared environments, ensuring safe operation of mobile robots is both fundamental and critical. At the same time, achieving a high task success rate is essential for improving the efficiency of systems such as warehouses and automated factories. Motivated by these considerations and inspired by the principle of addressing uncertainties at the planning level, we propose a mobile robot controller solving the path planning and task controlling problems in human-shared environments. An RL-based double layer algorithm is proposed to plan a global safe path for a robot performing multiple pickup and delivery tasks in an environment with uncertainties. The key contributions of this study are summarized as follows:

1.

We estimate environmental uncertainties through stochastic simulations and propose a hierarchical mobile robot controller that integrates scLTL and RL. Unlike conventional approaches that rely on real-time obstacle avoidance, our method explicitly handles environmental uncertainties at the planning level.

2.

A novel RL-based double-layer algorithm is developed. Unlike traditional product- or encoder-type algorithms that use scLTL and RL, the proposed algorithm separates the learning problem into high-level and low-level layers. This decomposition enables the robot to compute an optimal path in human-shared environments with significantly reduced learning costs.

3.

At the higher level, an FSA-RL learner is proposed on the basis of FSA transitions to find an optimal global strategy for multiple tasks. Notably, we extend the deterministic state transitions of the FSA to a stochastic formulation, allowing the controller to better adapt to dynamic environments.

4.

At the lower level, we construct an MDP learner that improves upon the classic MDP by redefining the reward function to explicitly account for environmental uncertainties at the planning level. This enhancement enables the generation of an optimal local path under dynamic and uncertain conditions.

The rest of this paper is organized as follows: Section 2 presents a review of related works; Section 3 briefly reviews the scLTL specifications; Section 4 introduces the problem formulation; the proposed safe planner is presented in Section 5, followed by the simulation results and discussions in Section 6; finally, a brief conclusion is provided in Section 7.

2. Related Work

In path planning, obstacle avoidance has traditionally been the primary requirement for mobile robots. Numerous efficient algorithms have been proposed for path planning and task assignment, including A* [10,23], D* [9,24], Reinforcement Learning (RL) [17,25], and Monte Carlo Tree Search (MCTS) [26,27]. However, as is widely known, although the A* algorithm is efficient for finding the shortest path in static environments, it struggles to address uncertainties in dynamic environments, with D* facing the same limitation. While the RL and MCTS methods are effective in both static and dynamic settings, unfortunately these approaches typically require extensive learning costs to derive an optimal policy by repeating trials. What is more, even in static environments they cannot provide any guarantees for the learned policy.

These classic algorithms do not perform well when addressing uncertain factors, especially unpredictable obstacles such as human operators. When a human suddenly appears in front of a mobile robot, the robot has to stop proactively to ensure human safety. Thus, more robust methods [28,29,30,31] have been presented for environments requiring human–robot coexistence. Cheng et al. [28] proposed a safe RL controller with control barrier functions and utilized Gaussian Processes (GPs) to model the system dynamics and uncertainties. However, Gaussian functions encounter difficulties when modeling uncertainties with multiple peaks. Mohamed et al. [29] proposed a framework that uses DRL as an enabling technology to enhance the intelligence and safety of robots in human–robot collaboration scenarios. Liu et al. [30] presented a DRL approach to realize the real-time collision-free motion planning of an industrial robot for human–robot collaboration. Shao et al. [31] proposed a Reachability-based Trajectory Safeguard (RTS) controller which leverages reachability analysis to ensure safety during training and operation. While these methods are efficient in human–robot collaboration, they are depended on online learning and cannot solve the issue of collisions at the planning level. A novel approach to collision avoidance based on human–robot interaction was proposed in [32], where the robot communicates its presence to nearby humans through voice messages. Upon approaching a certain distance, the human is required to provide feedback in order to complete the interaction process. Zeng et al. [33] presented a virtual force field-based mobile robot navigation algorithm in which both active and critical regions are used to deal with the uncertainty of human motion. The region sizes for the robot’s navigation are calculated based on worst-case avoidance conditions. Although  the above-mentioned methods have shown promising results, they all consider humans as fixed subjects in the physical workspace.

Hybrid methods combining Global Path Planning (GPP) and Local Path Planning (LPP) [34] are widely applied in navigation, where LPP is used for real-time obstacle avoidance. For instance, this scheme is widely utilized in the field of autonomous driving [35]. Strictly speaking, this is a real-time planning system in which the re-planning mechanism is invoked when an unexpected obstacle is encountered. The main advantage is that it provides high safety assurance. However, this approach does not consider the future risk or uncertainties at the planning level. In an automated factory scenario where mobile robots work together with humans, a mobile robot usually needs to accomplish its tasks within a given time budget. Thus, such real-time planning systems encounter high cost while also easily becoming trapped in continuous re-planning, leading to low task success rates.

Recently, numerous path planning methods that consider uncertain human actions have been proposed to handle human–robot interactions. Chan et al. [36] presented a conflict-free speed alteration strategy for human safety. Although the superiority of this method was shown by comparing it with two existing methods, this method plans one-dimensional moving paths, which is unsuitable for an autonomous system. Similar works have made progress in the probabilistic prediction of human motion [37,38] and robust sequential trajectory planning for multi-robot systems [39,40]. However, fusing these techniques into a real-time planning system remains a challenge because of the difficulty encountered in joint planning and prediction for multiple robots and humans. Bajcsy’s research [41] introduced a scalable framework for robot navigation that accounts for high-order system dynamics and maintains safety in the presence of external disturbances, other robots, and nondeterministic intentional agents. However, this framework decomposes multi-agent pathfinding into a sequence of instances in which it is necessary to re-plan the paths at every time step for all agents [42,43]. While such a framework maintains human safety, it does not account for the time budget of the robot or its task success rate. Consequently, this framework is difficult to realize in real-world applications.

Table 1. Comparison of different path planning methods.

Methods	Environment		Risk Foresight
	Static	Dynamic
A*	✓	–	–
D*	✓	–	–
RRT*	✓	–	–
RL	✓	✓	✓
FSA-MDP	✓	✓	✓
MCTS	✓	✓	✓
Real-time solution	✓	✓	–

summarizes and compares the related path planning methods discussed above. As described previously, shortest path planning methods such as A*, D*, and RRT* are not suitable for dynamic environments in which humans move randomly. Methods such as RL, FSA-MDP, and MCTS adopt learning schemes to obtain an optimal control policy, and are efficient in both static and dynamic environments. Moreover, these learning schemes include the possibility of addressing future risk in the learning phase through techniques such as stochastic simulations. Overall, RL is a good scheme, while MCTS suffers expensive computation cost. It is well known that real-time solutions are the safest approach to path planning which combines the two key components of global path planning and real-time obstacle avoidance. The real-time method is widely applied in navigation; however, it lacks risk foresight ability and cannot solve conflicts at the planning level. Above all, addressing risks at the planning level rather than relying on real-time methods is crucial for ensuring the safety of mobile robots working alongside humans while achieving a high task success rate.

It has been proven that syntactically co-safe Linear Temporal Logic (scLTL), which is one fragment of linear temporal logic (LTL) [11], is efficient in terms of both task descriptions and control. Multiple tasks can be formulated using an scLTL formula [44,45], offering efficient methodologies for managing multi-task scenarios and seamlessly integrating with path planning methods. For example, in [22] the authors investigated an RL approach merging MDP with scLTL and proposed an FSA-MDP algorithm to generate optimal paths against jamming attacks. Building upon this foundation, our primary aim is to generate safe and efficient plans that satisfy scLTL tasks for environments such as warehouses and automated factories where humans and robots must coexist or cooperate.

Table 2. Comparison of different methods for task sequencing.

Methods	Handling of Environmental Uncertainty	Human- Interaction	Task Sequencing
Product MDP [46]	–	–	✓
Feb-MDP [22]	real-time way	–	✓
DRL [47]	–	–	✓
Plan-Seq-Learn [48]	–	–	✓
G2RL [15]	real-time way	–	–
Co-HAN [49]	real-time way	✓	–
Ours	at the planning level	✓	✓

shows a comparison of the different methods. Ushio et al. [46] proposed a product MDP learner based on LTL for robots performing multiple tasks. However, their method only considers uncertainties in state transitions and does not address environmental uncertainties. The FSA-encoder-based MDP (Feb-MDP) [22] integrates scLTL and reinforcement learning, handling environmental uncertainties through real-time re-planning. However, this approach was designed for jamming attack scenarios, which differ significantly from human–robot interaction settings. DRL [47] and Plan-Seq-Learn [48] primarily focus on task sequencing for robotic manipulators, but lack mechanisms for handling environmental uncertainties. Furthermore, path planning for mobile robots in human-shared environments is fundamentally different from motion planning for manipulators. Teja et al. [49] proposed a human-aware navigation planner called Cooperative Human-Aware Navigation (Co-HAN), which effectively handles various human-robot interaction contexts; however, their approach does not consider the task sequencing problem. As summarized in Table 2, our goal is to develop a path planner for mobile robots that meets the following two requirements: (1) addressing environmental uncertainties explicitly at the planning level, and (2) solving the task sequencing problem and generating an optimal global path for performing multiple tasks. In contrast to traditional approaches, the proposed algorithm resolves potential conflicts during the planning phase, enabling the robot to complete multiple tasks in human-shared environments through one-time planning without the need for continuous re-planning.

3. Preliminaries

scLTL is one fragment of LTL. An scLTL formula commonly consists of a set of atomic propositions, Boolean operators, and temporal operators. Let $PrP$ be the set of atomic propositions. An scLTL formula over the atomic propositions $PrP$ is recursively defined as

$$\phi ::\top \left|prp\right|\neg prp|{\phi }_1\wedge {\phi }_2|{\phi }_1\vee {\phi }_2\left|\mathbf{F}\phi \right|\mathbf{X}\phi |{\phi }_1\mathbf{U}{\phi }_2.$$

(1)

Note that:

• $\perp =\neg \top$ (False) is an scLTL formula,
• $\mathbf{F}\phi =\top \mathbf{U}\phi$ is an scLTL formula, and
• given an scLTL formula $\phi$, it is written in a positive normal form where ¬ is only written in front of an atomic proposition $prp$.

In addition, ¬ (negation), ∧ (conjunction), and ∨ (disconjuction) are Boolean operators, while X (next), F (in the future), and U (until) are temporal operators. By combining the Boolean operators and temporal operators, complex and rich mission specifications can be described. An scLTL formula is interpreted by an infinite word $w\in {\left(2^{PrP}\right)}^{\omega }$, where $w=w\left[0\right]w\left[1\right]w\left[2\right]\dots$. Let $w\left(i\right)=w\left[i\right]w[i+1]\dots$; then, the semantics formula $\phi$ over word w is defined recursively in Equation (2) as follows:

$$\begin{array}{cc}\hfill & w\left(i\right)\vDash \top \hfill \\ \hfill & w\left(i\right)\vDash c\iff c\in w\hfill \\ \hfill & w\left(i\right)\vDash {\phi }_1\wedge {\phi }_2\iff w\left(i\right)\vDash {\phi }_1\wedge w\left(i\right)\vDash {\phi }_2\hfill \\ \hfill & w\left(i\right)\vDash {\phi }_1\vee {\phi }_2\iff w\left(i\right)\vDash {\phi }_1\vee w\left(i\right)\vDash {\phi }_2\hfill \\ \hfill & w\left(i\right)\vDash \mathbf{X}\phi \iff w(i+1)\vDash \phi \hfill \\ \hfill & w\left(i\right)\vDash {\phi }_1\mathbf{U}{\phi }_2\iff \exists k\ge i,\forall j(i\le j<k),w\left(k\right)\vDash {\phi }_2\wedge w\left(j\right)\vDash {\phi }_1.\hfill \end{array}$$

(2)

For any scLTL formula $\phi$, there exists an FSA ${\mathcal{A}}_{\phi }=<Q,PrP,\delta ,q_0,F>$ that accepts all infinite words satisfying the formula $\phi$, where Q is the finite set of states, $PrP$ is the set of atomic propositions, $\delta$: $Q\times 2^{PrP}\to Q$ is the state transition function, $q_0\in Q$ is the initial state, and $F\subseteq Q$ is the set of the accepting states. Several off-the-shelf tools for the computation of ${\mathcal{A}}_{\phi }$ have been developed [50,51]. (The tool used in this paper, please see https://spot.lre.epita.fr/app/; accessed on 7 July 2025)

4. Problem Formulation

We consider the online pickup and delivery problem of an agent under environmental uncertainties. An agent is designed to collect workpieces from $I(I\in {\mathbb{N}}_{+})$ specific workstations and transport them to different destinations. Let $o_i$ be the position of the workstation $i(1\le i\le I)$ and $g_i$ the destination of the workpieces from workstation i.

For each workstation i, the agent:

• Visits $o_i$ and collects workpieces, which is called the collection task
• Delivers the collected workpieces to its destination $g_i$, named the delivery task
• $T_i^1,T_i^2$ are the budgets for the collection and delivery tasks, respectively.

Therefore, I workstations define $2I$ tasks. We denote the task state at time step t as ${\tau }^t=\{{\tau }_1,\dots ,{\tau }_i,\dots ,{\tau }_I,{\tau }_{I+1},\dots ,{\tau }_{I+i},\dots ,{\tau }_{2I}\}$, where ${\tau }_i$, ${\tau }_{I+i}$ represent the state of the collection and delivery tasks of workstation i, respectively. Here, $\forall {\tau }_i,{\tau }_{I+i}$, ${\tau }_i,{\tau }_{I+i}\in \{False,True\}$, where:

• $False$ shows a task that is not achieved
• $True$ represents a task that is achieved.

Thus, the initial task state ${\tau }^0=\{{\tau }_1,\dots ,{\tau }_i,\dots ,{\tau }_{2I}\}=\{False,\dots ,False,\dots ,False\}$. After ${\tau }_i$ is achieved at time step t, ${\tau }^t=\{False,\dots ,True,\dots ,False\}$. After all tasks are completed, this becomes $\{True,\dots ,True,\dots ,True\}$.

Definition 1.

If a task is achieved within its given budget, we say that it is online; otherwise, it is offline.

Definition 2.

If all tasks are online completed, we say that it is a success; otherwise, we consider it as a failure.

Environmental uncertainties arise from the states of the doors. There are $N(N\in {\mathbb{N}}_{+})$ doors in the environment that the agent has to pass through. Each door $n(1\le n\le N)$ can be in one of two states, open or closed. The state of each door is determined by the door control system. Let $d_n^t$ define the state of door n at time step t; then: 

$$d_n^t=\left\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{o}\mathrm{r}\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{d}\mathrm{o}\mathrm{p}\mathrm{e}\mathrm{n},\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(3)

When the agent needs to pass through a door, it sends an open request to the door control system. However, the request may be denied if the door is not available when the outside is dangerous, such as in a scenario where other vehicles are passing through.

Based on the $2I$ tasks of the agent, we define a set of atomic propositions $PrP={\cup }_{i=1}^{2I}Pr{P}_i$ and $Pr{P}_i=\{nul{l}_i,pic{k}_i,delive{r}_i\}(1\le i\le I)$, where:

• $nul{l}_i$: The agent does not accomplish the collection task at workstation $o_i$
• $pic{k}_i$: The agent successfully completes the online collection task at workstation $o_i$
• $delive{r}_i$: The agent successfully completes the online delivery task of workpieces from workstation $o_i$.

All tasks can be formulated into an scLTL formula $\phi$, as follows:

$$\phi =\underset{i=1}{\overset{I}{\bigwedge }}(\mathbf{F}pic{k}_i\wedge \mathbf{F}delive{r}_i\wedge (\neg delive{r}_i\mathbf{U}pic{k}_i))$$

(4)

where:

• $\mathbf{F}pic{k}_i$: The agent will achieve online collection task ${\tau }_i$ in future
• $\mathbf{F}delive{r}_i$: The agent will achieve online delivery task ${\tau }_{I+i}$ in future
• $\neg delive{r}_i\mathbf{U}pic{k}_i$: The delivery task ${\tau }_{I+i}$ will not be achieved before the collection task ${\tau }_i$ is completed.

In this paper, we aim to generate an optimal solution for the online pickup and delivery problem under environment uncertainties.

5. Proposals

The architecture of the proposed planner is illustrated in Figure 1. We first model the environmental uncertainties based on the observations. Then, an RL-based double-layer learning algorithm is introduced to learn an optimal policy while accounting for these uncertainties. The lower-level layer consists of an MDP learner which takes a starting position and a goal, then searches for a local optimal safe path under uncertainties. The higher-level layer employs an FSA-RL learner to achieve global optimization across all tasks. The details of the proposed planner are presented below.

5.1. Model Environment Uncertainties

The uncertainties are primarily caused by human-driven vehicles such as forklifts. Let $Z_t=\{{\mathbf{z}}_t^1,\dots ,{\mathbf{z}}_t^K\}$ be the observations of all forklifts at time step t by the door control system and let ${\mathbf{z}}_t^k=<z_t^k,dir{e}^k>$ be the state of forklift k. Here, $z_t^k$ is the observed position and $dir{e}^k$ is the observed direction, where $dir{e}^k\in \{\leftarrow ,\to \}$. Each forklift can take two actions, $forward$ and $wait$. The direction of a forklift remains fixed throughout its operation. The $wait$ action is reserved for emergency situations only. The probabilistic model is illustrated in Figure 2. Each forklift is human-operated, and the driver controls its speed within a specified range. Suppose that a forklift is observed at position v at time step t, as shown in Figure 2, where $z_t^k=v$ and $dir{e}^k$ is ←. It moves to $z_{t+1}^k$ with $forward$ action under $p_f$. The state transition is written as $(z_t^k,forward,z_{t+1}^k)$. Note that the transition is not deterministic.. Due to speed variations, the forklift may move forward by one or two steps, i.e., $z_{t+1}^k=v_1$ or $v_2$.

Each door n is associated with a designated barrier area $B_n$, and the door control system evaluates the risk level of $B_n$ to determine whether to grant or deny an open request. As illustrated in Figure 3, when the agent attempts to pass through a corridor via door $n_1$ to reach another workshop, it sends an open request to the door controller upon approaching the door. The door controller evaluates the risk level of $B_{n_1}$; if it confirms safety, it responds with a confirmation message “Y” and opens door $n_1$. In contrast, if a forklift is detected approaching the corridor near door $n_2$, then the area is considered unsafe. As a result, the controller denies the request and responds with a refusal message “N”, keeping the door closed.

In the following, we show the details of the risk evaluation for barrier area $B_n$ of each door n. Here, $B_n=\{v_1^n,v_2^n,\dots \}$ is the set of vertices around the door n. Suppose that a forklift k enters the corridor at $t_0$ and that the observation is ${\mathbf{z}}_{t_0}^k=<s_{t_0}^k,dir{e}^k>$. We simulate its path until it leaves the corridor and calculate the conditional probability of each vertex $v\in B_n$ occupied by forklifts:

$$p_{k,t}\left(v\right|Z_{t_0})={\displaystyle \frac{\mathit{visited} \mathit{times} \mathit{of} v}{\mathit{simulation} \mathit{times}}}$$

(5)

where $t_0<t<T_k$, $v\in B_n$, and $T_k$ is the time step at which forklift k leaves the corridor. The risk level of $B_n$ at time step t is evaluated by

$$r(B_n,t)=\sum _{v\in B_n}\sum _{k=1}^K{p}_{k,t}\left(v\right|Z_{t_0}).$$

(6)

Thus, we obtain the risk distribution $R_{isk}=\{\{r(B_1,t_0+1),\dots ,r(B_1,T^{*})\},\dots ,\{r(B_N,T^{*})$, $\dots ,r(B_N,T^{*})\left\}\right\}$, where $T^{*}$ is the time step at which the last forklift leaves the corridor.

5.2. High-Level: FSA-RL Learner

The high level is a global learner based on FSA. All tasks are first written in formula $\phi$, as shown in Equation (4). Afterwards, an FSA ${\mathcal{A}}_{\phi }$ is transformed from formula $\phi$ using off-the-shelf tools, where ${\mathcal{A}}_{\phi }=<Q,PrP,\delta ,q_0,F>$. Based on ${\mathcal{A}}_{\phi }$, we construct the high-level learner ${\mathcal{M}}_h=<S_h,A_h,P_h,{\delta }_h,R_h,q_0,q_F,PrP,L>$ where:

• $S_h=Q$ is the state set
• $A_h=\{move,stay\}$ is the action set
• $P_h:S_h\times A_h\times S_h\to [0,1]$ is the state transition probability
• ${\delta }_h:S_h\times PrP\times L\left(S_l\right)\to S_h$ is the state transition function (with $S_l$ being the low-level state set)
• $L:S_l\to 2^{PrP}$ is the label function
• $PrP$ is the atomic proposition set
• $R_h$ is the reward function, which is determined by the low-level MDP learner
• $q_0$ is the initial state and $q_F\in F$ is the accept state.

The agent chooses an action $a_h\in A_h$ for state transition from $s_h$ to $s_h^{\prime }$ with probability $P\left(s_h^{\prime }\right|s_h,a_h)$. We claim that in the case of $a_h=move$, the agent moves to next state $s_h^{\prime }(s_h^{\prime }\ne s_h)$ with probability $p\left(s_h^{\prime }\right|s_h,a_h)$ and may remain at $s_h$ with probability $1-p\left(s_h^{\prime }\right|s_h,a_h)$. An example is shown in Figure 4 (left), where the agent chooses to move to $q^{\prime }$ from q. However, it may stay at q with probability $1-p\left(q\right|q,a_h)$. The agent stays at q when $a_h=stay$ such that $p\left(q\right|q,a_h=stay)=1$, as shown in Figure 4 (right).

In detail, the state transition is defined by ${\delta }_h(·)$ as in Equation (7):

$$s_h^{\prime }={\delta }_h(s_h,prp,L\left(s_l\right))$$

(7)

where $prp\in Pr{P}_i$ is an atomic proposition which represents a task and $L\left(s_l\right)$ is the label. Note that the action set A is redefined as

$$A_h=\{pic{k}_i,delive{r}_i,nul{l}_i\},$$

(8)

where $move$ action is replaced by $\{pic{k}_i,delive{r}_i\}$ and $stay$ is replaced by $nul{l}_i$. Here, ${\delta }_h(·)$ is different from the deterministic state transition $\delta (·)$ in ${\mathcal{A}}_{\phi }$. Given a state $s_h=q\in Q$, let $neighbours$ be the set of all possible states to which q may transit. $\forall q^{\prime }\in neighbours$, it has a corresponding atomic proposition $prp$ that leads the agent to move. As described previously, each $prp$ represents an online collection task ${\tau }_i$ or delivery task ${\tau }_{I+i}$. Let $f(·)$ define the mapping from $prp$ to a position of the corresponding task, defined as follows:

$$s_l=f\left(prp\right)$$

(9)

where $s_l\in \{o_1,\dots ,o_I,g_1,\dots ,g_I\}$. After the agent chooses a neighbor to visit, it invokes the lower level to obtain a label $L\left(s_l\right)$, where:

$$L\left(s_l\right)=\left\{\begin{array}{cc}True,\hfill & \mathrm{i}\mathrm{f}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{i}\mathrm{s}\mathrm{o}\mathrm{n}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{d},\hfill \\ False,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(10)

If task ${\tau }_i$ or ${\tau }_{I+i}$ is achieved online, that is, if the agent visits $s_l$ within the given budget in the lower level, then $L\left(s_l\right)$ would return $True$. Then, the agent transits from $s_h=q$ to $s_h^{\prime }=q^{\prime }$, written as ${\delta }_h(q,prp,L\left(s_l\right)=True)\to q^{\prime }$. Otherwise, $L\left(s_l\right)=False$, and ${\delta }_h(q,prp,L\left(s_l\right)=False)\to q$. Figure Figure 5 shows an example in which an agent chooses to move to $q_1^{\prime }$ with $pr{p}_i=pic{k}_i$ from state q. Here, $\tilde{q}$ is the last state and the last transition is ${\delta }_h(\tilde{q},\widetilde{prp},L\left(\widetilde{s_l}\right)=True)\to q$, where $\widetilde{prp}$ is the chosen atomic proposition at $\tilde{q}$ and $\widetilde{s_l}=f\left(\widetilde{prp}\right)$. Note that there is a special case when $q=q_0$. As shown in Figure 5 (right), we define its last state transition as ${\delta }_h(q_0,nul{l}_0,L\left(\widetilde{s_l}\right)=True\to q_0$. At this point it should be claimed that $s_l=\widetilde{s_l}$ if $prp=nul{l}_i$, formulated as

$$s_l=f\left(prp\right)=f\left(\widetilde{prp}\right),\mathrm{if}prp=nul{l}_i.$$

(11)

In the transition of ${\delta }_h(q_0,nul{l}_0,L\left(\widetilde{s_l}\right)=True)\to q_0$, we define $s_0=f\left(nul{l}_0\right)$.

The high level invokes the low-level learner to learn a local control policy. The low-level learner returns a label $L\left(s_l\right)$ indicating the local outcome. Meanwhile, if the corresponding task is accomplished online, a local path $x_{q,q’}$ is also returned.

The scLTL formula $\phi$ defines the global reward condition. The agent receives a global reward if and only if $\phi$ is satisfied, where

$$R_h(s_h,a_h,s_h^{\prime })=E,if{s}_h^{\prime }=q_F.$$

(12)

In addition, the immediate reward function is also designed to accelerate the learning process, defined by

$$R_h(s_h,a_h,s_h^{\prime })=\left\{\begin{array}{cc}rE+|x_{s_h,s_h^{\prime }}|\ast step\_cost,\hfill & \mathrm{i}\mathrm{f}{s}_h\ne s_h^{\prime },\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}\right.$$

(13)

where $|x_{s_h,s_h^{\prime }}|$ is the length of the path from $s_h$ to $s_h^{\prime }$ returned by the lower level and $step\_cost$ is the cost for each movement.

5.3. Low-Level: MDP Learner

The lower level consists of a local MDP learner for learning an optimal control policy and generating an optimal local path. We model the environment as ${\mathcal{M}}_l$, defined by ${\mathcal{M}}_l=<S_l,A_l,P_l,R_l,s_{l_0},s_{l_g},R_{isk}>$, where:

• $S_l=S$ is the state set
• $A_l=\{forward,backward,right,left,wait\}$ is the action set
• $P_l:S_l\times A_l\times S_l\to [0,1]$ is the state transition probability
• $R_l$ is the reward function with $R_l(s_l,a_l,s_l^{\prime })\to \mathbb{R}$, where $s_l,s_l^{\prime }\in S_l$ and $a_l\in A_l$
• $s_{l_0}$ is the initial state and $s_{l_g}$ is the goal state, both of which are provided by the higher level
• $R_{isk}$ is the risk set of the environment.

The environment of the lower level is a 2D grid world. The state set $S_l$ contains all vertices of the environment. Starting from the given initial state $s_{l_0}$, the agent chooses an action $a_l\in A_l$ and explores the environment to gain experience in the form of a Q-table. The reward function $R_l$ is designed as follows:

$$R_l(s_l,a,s_l^{\prime })=\left\{\begin{array}{cc}goal\_reward,\hfill & \mathrm{i}\mathrm{f}{s}_l^{\prime }=s_{l_g},\hfill \\ step\_cost,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(14)

To enhance the efficiency of learning with uncertainties, we modify the reward function $R_l$ (Equation (14)) as follows:

$$R_l(s_l,a,s_l^{\prime })=\left\{\begin{array}{cc}goal\_reward,\hfill & \mathrm{i}\mathrm{f}{s}_l^{\prime }=s_{l_g},\hfill \\ step\_cost+u(s_l^{\prime },t^{\prime }),\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}\right.$$

(15)

where $t^{\prime }$ is the time step at which the agent transits to $s_l^{\prime }$ and $u(·)$ calculates the reward from environmental uncertainties. When an agent approaches a door, it sends a request and the door control system may return an a message “Y” to open the door or “N” to refuse. However, whether or not a door will be available in the future is unknown. Thus, we consider the future uncertainties $R_{isk}$ to predict whether or not a door could open. For each door n,

$$d_n^t=\left\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}r(B_n,t)<\theta ,\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}\right.$$

(16)

where $\theta$ is a risk threshold. As shown in Equation (16), when the risk of door n is less than $\theta$, we consider that the door could be passed though with a higher probability. When $r(B_n,t)\ge \theta$, this means that passing through the door will pose a significant collision risk which is unacceptable; in this case, the door refuses to open, and we define $u(·)$ as

$$u(s_l,t)=\left\{\begin{array}{cc}Penalty,\hfill & \mathrm{i}\mathrm{f}{d}_n^t=0,\hfill \\ r(B_n,t)\ast c\_cost,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}\right.$$

(17)

where $Penalty$ is a negative constant value and $c\_cost$ is the collision penalty. Here, $u(s_l,t)$ captures the influence of environmental uncertainties. Specifically, the robot receives a penalty if it is refused passage through a door when it is indicated that proceeding may pose a collision risk with a forklift. It is important to note that a collision may still occur even if the door system grants access (i.e., opens the door), as the door-opening decision is based on a threshold condition $r(B_n,t)<\theta$. To reflect this residual risk, the robot receives a reward proportional to the predicted risk value, provided by $r(B_n,t)·c_{c}ost$. This design enables the local learner to utilize the predicted risk distribution $R_{isk}$ to learn a safer path.

Both the high- and low-level learners adopt a Q-learning mechanism to learn the optimal policy. The learning structures of these learners are shown in Algorithms 1 and 2, respectively. Figure 6 illustrates the interaction between the high-level and low-level learners in learning the optimal global path for executing multiple tasks.

Algorithm 1: FSA-RL learner

Algorithm 2: Low-level MDP learner

Figure 6. High- and low-level interaction for learning the optimal global path.

Figure 6. High- and low-level interaction for learning the optimal global path.

As described above, the high-level learner is responsible for searching the global task strategy, while the low-level focuses on learning local paths. This hierarchical decomposition effectively transforms a complex global learning problem into several simpler local learning problems. For a single-layer RL learner, the size of the search space is $|S_l|\times 2^{2I}$. In contrast, our hierarchical approach reduces the search space to $\left|Q\right|$ for the high-level learner and $|S_l|$ for low-level learner, where $\left|Q\right|$ denotes the number of automaton states derived from scLTL. As is well-known, learning an optimal policy in a small search space is much easier than in a large search space. By decomposing the complex global learning problem into two levels, high-level for task sequencing and low-level for local path planning, the learning cost is reduced. Moreover, the proposed method reduces redundant learning cost, as it eliminates the need to re-learn a local path from $s_{l_0}$ to $s_{l_g}$ after it has already been successfully learned. In contrast, a single-layer RL learner would need to continue learning until finding an optimal global path, resulting in higher learning costs and slower convergence.

6. Results and Discussion

6.1. Reward Design and Environment Setting

The details of the rewards for the high- and low-level learners are designed as shown in

Table 3. Reward design in the high- and low-level learners.

Types of Reward	Value	Learner
E	$1.0$	high-level
$rE$	$0.0$	high-level
step_cost	$-0.1$	high-/low-level
goal_reward	$1.0$	low-level
Penalty	$-1.0$	low-level
c_cost	$-1.0$	low-level

. The other parameters are shown in

Table 4. Parameter settings.

Parameter	Value
$budget$	100	Section 4
$p_f$	$1.0$ (randomly move one step or two steps)	Section 5.1
$p_w$	≈$0.0$
$simulationtimes$	2000
$\alpha$	$0.5$	high-/low-level
$\gamma$	0.9
$ϵ$	$0.2$
cimputation_budget	1000
$\theta$	$0.1$	low-level

. The agent should finish all tasks within the given $budget=100$ time steps. The learning parameters in Q-learning are set as $\alpha =0.5,\gamma =0.9$, and $ϵ=0.2$. Figure 7 shows how $\alpha ,\gamma$, and $ϵ$ affect the learning process. As shown in Figure 7a, the learning rate $\alpha$ influences the convergence speed but does not affect the final convergence reward. In this paper, we set $\alpha =0.5$ as a tradeoff between learning stability and speed. The discount factor $\gamma$ has a significant impact on the final convergence reward, as shown in Figure 7b; here, we set $\gamma =0.9$, as it achieves the highest reward. The exploration rate $ϵ$ in the $ϵ$-greedy policy is typically set to 0.2; Figure 7c confirms that this value provides a favorable balance between exploration and exploitation. The risk threshold $\theta =0.1$.

We verify the proposed FSA-RL algorithm in dynamic environments, as shown by the different configurations in Figure 8. In Figure 8a, three workstations, six tasks, and two forklifts are considered. Six doors are set in the environment. To increase the environmental uncertainties, we then increase the number of tasks to twelve, with six workstations and four forklifts. To model the environment uncertainties, 2000 simulations are performed and the risk distribution $R_{isk}$ is calculated. During the simulation, $p_f=1.0$ and $p_w\approx 0.0$ for the motion model of the forklifts. Note that the uncertainties are caused by a forklift moving one step or two steps ahead, which we randomly simulate. The start time of the agent is set as $t_0=0$, while the forklifts start moving from time step $t=30$.

We conduct 100 simulations and evaluate the proposals by the average conflict number, average task success rate, average reward, and average length of generated paths. Moreover, we compare the results with the RL method.

6.2. Results on Scenario 1: Workstation Number $I=3$ and Forklift Number $K=2$

Scenario 1 verifies the proposed algorithm under the configuration as shown in Figure 8a. We first construct FSA ${\mathcal{A}}_{\phi }$ from the scLTL formula $\phi$ (Equation (4)) with $I=3$. Figure 9 demonstrates the state transitions of ${\mathcal{A}}_{\phi }$. Based on FSA ${\mathcal{A}}_{\phi }$, we construct the state space of the high-level learner. As shown in Figure 9, it contains 27 nodes, where $q_0=1$ is the initial state and $q_F=0$ is the accept state.

From $q_0=1$, the agent randomly chooses an atomic proposition from $\{pic{k}_1,pic{k}_2,pic{k}_3\}$ to explore the state space of the high-level learner. Here, we take $pic{k}_1$ as an example. The algorithm maps $pic{k}_1$ to its task position with Equation (9), where $s_l=o_1=f\left(pic{k}_1\right)$ and ${\overline{s}}_l=s_0=f\left(nul{l}_i\right)$. Then, the agent sends $(s_0,o_1)$ to the low-level learner and evokes it to obtain $L\left(s_l\right)$ and the corresponding local path x. After $L\left(s_l\right)=True$, that is, when x satisfies the requirement, the agent moves to state 6, written as ${\delta }_h(s_h=1,pic{k}_1,L\left(s_l\right)=True)\to s_h^{\prime }=6$. Otherwise, if $L\left(s_l\right)=False$, the generated path x does not satisfy the requirement and ${\delta }_h(s_h=1,pic{k}_1,L\left(s_l\right)=False)\to s_h=1$.

The reward of the learning process is shown in Figure 10. The blue curve shows the learning curve of the classic RL method, which is characterized by substantial oscillatory behavior. The green curve shows the learning curve of the proposed FSA-RL; its reward converges at around −1.0, much larger than classic RL method, which converges at around −5.0.

Figure 11 shows the details of the simulation results from the perspective of refused times, average task success rate, average reward, and average length of the generated paths. Note that “refused times” refers to the number of times the robot fails to pass through a door along the generated path due to the door rejecting the open request. As shown in Figure 11a, when using the classic RL method in 100 simulation runs, the agent was refused 37 times by the door system and failed to pass the door, achieving an average task success rate 0.63 in Figure 11b. Meanwhile, the proposed method enabled the agent to pass through the door and successfully deliver all workpieces in 100 simulation runs; thus, the number of refusals in Figure 11a is 0, and the average task success rate in Figure 11b is 1.0.

Figure 11c,d shows the average reward and average length of the generated paths. There is little difference in the average length between the RL and the proposed FSA-RL methods, with the average length being 79 for RL and 80 for the proposed method. The difference lies in the average reward; as Figure 11c shows, the average reward of the proposed FSA-RL method is −0.92, much larger than −1.94 with the RL method. The reason for this is that the door system refuses to allow the agent to pass through when the agent follows the path generated using RL.

Figure 12 shows an example of the generated paths. Figure 12a shows the path generated with RL. It is clear that the agent visits workstations $o_1\to o_2\to o_3$ first, then passes through doors $\#3$ and $\#6$ before proceeding to the other workshop for delivery tasks, in the order $g_2\to g_3\to g_1$. Unfortunately, when the agent requests to pass through door $\#3$ at time step $t=36$, the door system tends to refuse due to the conflict risk from forklifts in the corridor. The risk values of door $\#3$ at time steps 37 and 38 are $r(B_3,37)=0.5$, $r(B_3,38)=0.15$, respectively (

Table 5. Scenario 1: Risk level of each door when the agent passes through.

Time-Step t	37	38	39	40	41	42	43	44	45	46
door $\#1$	0.010	0.150	0.480	0.650	0.460	0.190	0.050	0.010	0.000	0.000
door $\#2$	0.500	0.150	0.110	0.050	0.010	0.000	0.000	0.000	0.000	0.000
door $\#3$	0.500	0.150	0.060	0.020	0.010	0.000	0.000	0.000	0.000	0.000

). Apparently, it is not a safe policy to choose door $\#3$.

The proposed FSA-RL method generates a different policy for moving through the corridor, as shown in Figure 12b. The pickup policy is same as with the RL method, visiting $o_1\to o_2\to o_3$ sequentially. The agent then chooses to pass through doors $\#1$ and $\#4$. Moreover, the delivery policy is also different. In the path generated with FSA-RL, the agent performs delivery tasks following the order $g_1\to g_2\to g_3$. The agent requests to open door $\#1$ at time step $t=44$ with $r(B_1,45)=0.0$, $r(B_1,46)=0.0$. Clearly, door $\#1$ is safe for the agent to pass through.

6.3. Results on Scenario 2: Increase Forklift Number to $K=4$

In this scenario, we increase the number of forklifts to $K=4$ and reset the start position of the agent in order to further investigate performance under a complex environment. The four forklifts begin moving at $t=30$. The results are shown in Figure 13 and Figure 14.

As shown in Figure 13, the number of refusals with the RL method is 54, while the number with the proposed FSA-RL method is 2. Compared with the RL method, the proposed method reduces the refusals by about 96.3%. Compared with the results in Figure 11, it can be seen that the number of refusals increases. Consequently, the average task success rate of RL is 0.46 and that of FSA-RL is 0.98 (Figure 13b), for an improvement of about 113.0%.

As the number of forklifts increases, the refusals increase and the task success rate decreases. For the generated paths, the average reward decreases for both the RL and FSA-RL methods. The average reward of FSA-RL is improved by 63.1% compared to that of the RL method. Figure 13c shows that the average reward for the RL method drops to −2.60, while that of the proposed FSA-RL decreases only slightly to −0.96 in comparison to Figure 11c. Meanwhile, the average length also increases, as shown in Figure 13d.

Figure 14 shows examples of the generated paths.

Table 6. Scenario 2: Risk level of each door when the agent passes through.

Time-Step t	36	37	38	39	40	41	42	43	44	45	46
door $\#1$	0.336	0.714	0.730	0.727	0.739	0.500	0.197	0.048	0.006	0.000	0.000
door $\#2$	0.807	0.500	0.147	0.111	0.020	0.010	0.000	0.000	0.000	0.000	0.000
door $\#3$	1.178	0.728	0.208	0.024	0.020	0.010	0.000	0.000	0.000	0.000	0.000

shows the risk level of each door from time step 36 to 46. In Figure 14a, the agent visits workstations $o_2\to o_1\to o_3$ to perform collection tasks. Then, it passes through the door $\#2$ and visits $g_4\to g_2\to g_1$ for delivery tasks. In detail, the agent requests to open door $\#2$ at time step $t=37$ and enters the corridor at time step $t=38$. The risk values of door $\#2$ are $r(B_2,38)=0.147$ and $r(B_2,39)=0.111$. Thus, the agent encounters a greater risk of being refused. The refusal results in Figure 13a also verify this point, with the agent being refused by the door 54 times. Figure 14b shows the generated path. The visit order for collection tasks is the same as with the RL method, $o_2\to o_1\to o_3$. After finishing the collection tasks, the agent moves to the other workshop through door $\#1$. The agent enters the corridor at $t=44$, where the risk value is $r(B_1,44)=0.006$. Thus, the agent is only refused twice in 100 simulation runs.

The simulation results show that the proposed RL-based double-layer controller is efficient in finding an optimal path for a mobile robot working in human-shared environments. The number of refusals is greatly reduced compared to the classic RL method. The quality of the generated path is improved from the viewpoint of both task success rate and path reward. Simulation results demonstrate that the proposed method outperforms traditional RL approaches. One advantage comes from our algorithm’s ability to decompose the complex path planning problem into several smaller subproblems with reduced search spaces, making them easier to solve. In traditional RL approaches, the size of the search space is $\left|S\right|\times 2^{2I}$ due to the need to handle all tasks and states. In contrast, our hierarchical method significantly reduces the complexity; the high-level planner operates over a smaller abstract state space of size $\left|Q\right|$, while the low-level planner operates over $\left|S\right|$. For example, the size of the search space for the RL algorithm is $\left|S\right|\times 2^{2I}=20\times 22\times 2^6=28160$. In our proposed algorithm, the size of high-level search space is $\left|Q\right|=26$, while the size of low-level search space is $\left|S\right|=22\times 20=440$. Each subproblem needs to be learned only once, eliminating the need for repeated learning. In contrast, traditional RL must continuously learn optimal policies across multiple tasks. This leads to unavoidable redundant learning, as the learning process is not terminated until all tasks are achieved. The decomposition of the complex path planning problem also enhances the flexibility of our algorithm. Another advantage of our algorithm is its risk foresight ability thanks to modeling environmental uncertainties. As Table 5 and Table 6 show, the risk distribution $R_{isk}$ over a certain time horizon can be obtained using Monte Carlo simulations. Based on $R_{isk}$, we design the reward function in Equation (17) for the low-level learner in order to handle environmental uncertainties.

In the proposed double-layer learning scheme, the high-level learner is applied to learn the global policy for multiple tasks, while the low-level learner is applied for local path planning while considering environmental risks. Thus, the proposed double-layer algorithm is more efficient at handling environment uncertainties for a mobile robot designed to perform multiple tasks. The shortcomings of the proposed method are the same as other reinforcement learning algorithms, namely, that it cannot provide guarantees and that the search space increases with the number of tasks.

7. Conclusions

This paper addresses the path planning problem of a mobile robot performing multiple pickup and delivery tasks in human-shared environments. We propose a reinforcement learning-based double-layer control algorithm to learn an optimal path in automated factory settings while considering the stochastic movements of human-operated forklifts. The proposed algorithm decomposes the complex global path planning problem into several smaller subproblems that are easier to solve. The high-level layer focuses on global strategy optimization using scLTL and reinforcement learning, while the low-level layer handles local path planning under environmental uncertainties. Compared with conventional RL methods, our approach reduces the number of refusals by approximately 96.3%, improves the average reward by 63.1%, and increases the task success rate by 113.0%. In the future, we plan to: (1) investigate Lyapunov-based stability analysis to provide safety guarantees, (2) develop more robust path planning strategies under uncertainty, and (3) apply the proposed algorithm to real-world robotic systems.