An Online Human-Aware Behavior Planning Method for Nondeterministic UAV System Under Probabilistic Model Checking

Highlights

What are the main findings?

• The Markov decision process is used to construct the global probabilistic behavior model of the UAV offline, but a finite state automaton in finite horizon is dynamically constructed online.
• A value iterative algorithm is introduced to solve the optimal behavior plan online within the finite horizon, then an infinite horizon planning and execution algorithm is formed.

What is the implication of the main finding?

• This paper proposes an online human-aware behavior planning method to enable a UAV dynamically satisfy the high-level LTL task description from the human collaborators, which has the potential to be applied to human–UAV collaboration.

Abstract

This paper proposes an online human-aware behavior planning method to enable Unmanned Aerial Vehicles (UAVs) to dynamically satisfy high-level linear temporal logic (LTL) task descriptions from human collaborators. The proposed method has the potential to be applied to the industrial human–UAV collaboration. The specific process is as follows. Firstly, the global high-level task of the UAV is described using the formal language of LTL, which can usually be issued by human collaborators in natural language. Secondly, the task description is transformed into a deterministic Rabin automaton, and then the state values of the automaton are given by the distance from the accepted states. Thirdly, the Markov decision process is used to construct the probabilistic behavior model of the UAV offline. Based on this model, a finite state automaton in the finite horizon is dynamically constructed online, and the Cartesian product system within the horizon is constructed, in which the expected arrival states within the horizon are set by the minimum state values. Finally, aiming at maximizing the reachability probability of the expected states, the value iterative algorithm is introduced to solve the optimal behavior plan online within the finite horizon. After it, an infinite horizon planning and execution algorithm is formed. Experiments and analysis show that the results of the online behavior planning are consistent with the task logic at the semantic level, indicating the correctness of the proposed method. Moreover, the proposed method can effectively alleviate the state explosion caused by the global probabilistic model checking and improve the efficiency of plan generation.

1. Introduction

Nowadays, plentiful practical applications on the UAV technologies have been investigated and discussed [1,2]. And recently, formal languages have been proposed to specify complex tasks for UAVs, such as linear temporal logic formulas, as an intuitive yet powerful way to describe both spatial and temporal requirements for UAV behaviors. In this work, the UAV is allowed to operate along with human collaborators in close proximity to perform the desired tasks to achieve human–UAV shared production goals [3,4]. Similar to human–human collaboration, the UAV should be able to infer the intention of its human collaborators and generate behavior to avoid collisions or improve fluency during human–UAV collaboration, which can be referred to as human-aware behavior planning [5]. Some research studies [6,7] indicate that the accomplishment of human-aware behavior planning can garner positive effects during human–UAV collaboration. According to the experiments of different settings, more behavior-level UAV adaptation of the human’s behavior can better obtain team fluency and efficiency, as well as human satisfaction and comfort. To incorporate human intervention into behavior planning, shared autonomy has been studied by numerous research works. The domain of shared autonomy aims to study approaches to bring human into the loop of UAV control while maintaining a degree of autonomy [8,9]. In particular, the method’s need to design and verify autonomous systems and provide better feedback in human–UAV collaboration systems has been stressed. Model checking a formal method traditionally used in the design and verification of complex behaviors for software systems [10,11] has the potential to address these needs. Under model checking theory, considerable attention has been devoted to the research on the automatic synthesis of a controller for UAV [12,13], which is used to perform complex and high-level tasks described as temporal logic formulas, such as ‘periodically visiting the locations A, B and C, while avoiding the location D’. Research on this kind of problem mainly relies on a three-step hierarchical program [14,15,16]. First, the dynamics of the UAVs are abstracted into a finite discrete transition system based on sampling [17] or element decomposition. Second, using formal verification techniques, a discrete behavior plan is generated to complete the task. Third, the discrete behavior plan is transformed into a continuous controller of the original system. In these hierarchical steps, the generation of discrete behavior plans first requires combining a UAV system model and an automaton that derives the temporal logic task into a large-scale product system, resulting in high computational complexity in the process of generation, which is not suitable for online human–UAV collaboration scenarios. Many scholars seek to introduce model predictive control or receding horizon control technology into the generation process of discrete behavior plans.

1.1. Related Work

At present, the human-aware behavior planning methods for UAVs aiming at dynamically satisfying the linear temporal logic description of high-level tasks designed for deterministic transition systems (DTSs) mainly focus on two system models. One is a simple deterministic transition system, the other is an affine system based on a deterministic transition system. For deterministic transition systems, Ding et al. [18] consider a receding horizon control of finite deterministic systems, which must satisfy the high-level task descriptions represented by linear temporal logic formulas. They propose a controller synthesis framework inspired by model predictive control, in which the rewards of the states are locally optimized at each time step in the finite horizon, and real-time optimal control is used. By applying appropriate constraints, the infinite trajectory generated by the controller meets the desired temporal logic formula. Ding et al. [19] also consider a scenario in which UAVs can monitor in areas with multiple targets and meet a set of high-level and rich LTL task descriptions. They use a receding horizon controller to compute the optimal path of the UAV in a subset of task space, and provide a framework to ensure that the overall strategy of the system meets the expected LTL description, and the control decision is based on the local information obtained online. Cai et al. [20] study the online optimal behavior planning problem of autonomous UAVs satisfying linear temporal logic constraints. The hard and soft LTL constraints are considered, in which the hard constraint enforces the security requirements, while the soft constraint indicates that it can be relaxed to avoid adherence to user-specified tasks. A receding horizon control is synthesized to maximize the cumulative utility in a finite horizon, while ensuring that the security constraints and most of the soft constraints are fully satisfied. Ulusoy et al. [21] propose a receding horizon method to control the autopilot UAV to meet the rich task descriptions. These tasks come from the service requests in different areas to the autonomous driving UAVs. The global task description consists of a set of static and pre-defined temporal logic representations of requests, a set of regular expressions of dynamic requests that can only be locally perceived, and the service priority order of dynamic requests. Tumova et al. [22] study the synthesis of strategies for multi-robot systems to achieve the complex, high-level and long-term goals assigned to each robot. The purpose is to reduce the computational complexity by decomposing the strategy synthesis problem into a series of short horizon programming problems that can be solved iteratively at the runtime of the robots. The correctness of the method is discussed, and some assumptions are found. For deterministic affine systems, Wongpiromsarn et al. [23] propose a receding horizon framework to meet a class of linear temporal logic specifications, which can describe a wide range of attributes, including safety, stability, response, and guarantee. It is proved that the composition of the target generator, trajectory planner, continuous controller, and the corresponding receding horizon framework can ensure the correctness of the system. Nenchev et al. [24] solve the control problem of mobile UAVs that must complete limited tasks in a partially unknown static environment with the shortest time. The task is expressed as a syntax safe linear temporal logic (scltl) formula. Instead of abstracting the hybrid system of behavior in the modeling environment, they propose a method based on the parameterization of continuous behavior, and introduce a violation measurement for forced satisfaction description. Then, a parametric optimal control problem (OCP) is described, which can be solved in receding horizon only after previously unknown environment attributes are detected. Wongpiromsarn et al. [25] describe a Python-based (Python 2.7) software toolbox Tulip for embedded control software synthesis, which is proved to be correct relative to the expression subset of linear temporal logic specification. Tulip integrates finite state abstraction of the control system, digital synthesis from LTL description, and receding horizon planning. Tulip uses the receding horizon framework to decompose the synthesis problem into a series of smaller problems so as to ensure the correctness and reduce the computational complexity of the synthesis program. Shaffer et al. [26,27] propose the hierarchical integration of reactive synthesis, which assures desired system design traits, and dynamic allocation, used for making heuristic-based decisions, to manage UAVs fighting a wildfire. Yoo et al. [28] present an algorithm for automatically synthesizing a continuous non-linear flight controller according to a complex temporal logic task specification that can include contingency planning rules. Zhang et al. [17] propose a randomized sampling-based motion planning algorithm with probabilistic guarantees of completeness and optimality which greatly reduce the computation time. Li et al. [29] propose an optimal policy based on model-checking techniques to satisfy the complex task. Obviously, the above methods are designed for deterministic systems. As far as we know, an unresolved issue that has not received sufficient attention is online strategy synthesizing for nondeterministic systems under probabilistic model checking [30,31,32], which aims to dynamically complete probabilistic behavior modeling, formal verification, and behavior planning.

1.2. Contribution

The specific process of the method proposed in this paper is as follows. First, the global high-level task of the UAV is described using the formal language of linear temporal logic, which can usually be issued by human collaborators in natural language. Second, the task description is transformed into a deterministic Rabin automaton through the calculation of the ltl2dstar tool [33,34], and then the state values of the automaton are given based on the distance from the accepted states of the automaton. Third, the Markov decision process is used to construct the global probabilistic behavior model of the UAV offline, where the transitions triggered by the actions from a state are probabilistic and not deterministic. Based on this model, a finite state automaton in finite horizon is dynamically constructed online, and the product system within the horizon is constructed by Cartesian product, in which the expected arrival states within the horizon are set by the minimum state values. Fourth, aiming at maximizing the reachability probability of the expected states, a value iterative algorithm is introduced to solve the optimal behavior plan online within the finite horizon. After it, an infinite horizon planning and execution algorithm is formed. Through some simulation experiments, the results and analysis show that the strategies of the online behavior planning are consistent with the task logic at the semantic level, indicating the correctness of the proposed method. Moreover, the proposed method can effectively alleviate the state explosion caused by the global probabilistic model checking and improve the efficiency of plan generation. Finally, based on the Robot Operating System (ROS) and lidar, a smart UAV that can simultaneously build and navigate is built, and the speech recognition software and the algorithm of this paper are integrated to complete the autonomous navigation of the smart UAV under the voice commands of the human collaborators, which verifies the effectiveness of the method.

The organizational structure of the paper is as follows. In Section 2, some preliminaries of linear temporal logic and probabilistic behavioral models are given. In Section 3, a shared autonomy framework for industrial human–UAV collaboration is proposed. In Section 4.1, firstly the global task description for a UAV is specified; secondly the probabilistic behavior model of the UAV is established offline and then the Cartesian product system is constructed in the finite time domain. Online behavior planning within the finite time domain and the execution across the infinite time domain are presented in Section 4.2 and Section 4.3, respectively. Simulations are conducted in Section 5 and subsequently applied to the online human-aware behaviors planning for a smart car in Section 6. Finally, a brief summary and prospects are given in Section 7.

2. Preliminaries

2.1. Linear Temporal Logic

Linear temporal logic (LTL) is employed to describe the high-level task specifications of UAVs. Roughly, an LTL formula is composed of a set of atomic propositions Π, standard Boolean operators ¬ (negative), ∨ (disjunctive), ∧ (conjunction), → (implication), and temporal operators $\mathcal{X}$ (next), $\mathcal{U}$ (until), □ (always), ◊ (eventually). The syntax of an LTL formula is as follows:

$$\phi \stackrel{\Delta }{=}\mathrm{T}\left|p\left|{\phi }_1\wedge {\phi }_2\left|\neg \phi \right.\right.\right.\left|\mathcal{X}\phi \left|{\phi }_1\right.\right.\mathcal{U}{\phi }_2$$

(1)

where $\mathrm{T}\stackrel{\Delta }{=}\mathrm{True}$ and $p\in \mathrm{AP}$. The semantics of LTL are given as a set of infinite words over the alphabet $2^{\Pi }$. A word satisfies an LTL formula $\varphi$ if $\varphi$ is true at the first position of the word; $\square \phi$ means that $\phi$ is true at all positions of the word; $◊\phi$ means that $\phi$ eventually becomes true in the word; $\mathcal{X}\phi$ …; ${\phi }_1\mathcal{U}{\phi }_2$ means that ${\phi }_1$ has to hold at least until ${\phi }_2$ is true. A detailed description of the syntax and semantics of LTL can be found in [9].

To apply these property patterns to the UAV missions, let atomic propositions indicate the most recent waypoint visited by the UAV and again consider the patterns in Example 1. Based on these property patterns, a number of useful mission specifications could be developed by an operator.

Example 1.

For instance, an operator might surveil a particular set of ordered waypoints according to a specification formed by a simple ‘sequencing’ property, e.g.,

$$◊\left(w_1\wedge ◊\left(w_2\wedge ◊w_3\right)\right)$$

(2)

Or, the operator might want to surveil one of the waypoints once using a ‘coverage’ property and establish a repeating patrol route for a set of ordered waypoints using a ‘recurrent sequencing’ property, e.g., using the specification

$$◊w_2\wedge \square ◊\left(w_3\wedge ◊\left(w_1\wedge ◊w_5\right)\right)$$

(3)

UAV never surveils a particular waypoint, e.g.,

$$\square \neg w_1$$

(4)

The set of words satisfying LTL descriptions over the atomic proposition set can be represented by a deterministic Rabin automaton (DRA).

Definition 1

(DRA [10]). A DRA is defined as a tuple $\mathcal{R}=\left(Q_{\mathcal{R}},q_{0,\mathcal{R}},{\Sigma }_{\mathcal{R}},{\delta }_{\mathcal{R}},\mathrm{Ac}{\mathrm{c}}_{\mathcal{R}}\right)$, where $Q_{\mathcal{R}}$ is a finite set of states; ${\Sigma }_{\mathcal{R}}$ is an alphabet; ${\delta }_{\mathcal{R}}\subseteq Q_{\mathcal{R}}\times {\Sigma }_{\mathcal{R}}\times Q_{\mathcal{R}}$ is a transition function; $q_{0,\mathcal{R}}\in Q_{\mathcal{R}}$ is the initial state; $\mathrm{Ac}{\mathrm{c}}_{\mathcal{R}}\subseteq 2^{Q_{\mathcal{R}}}\times 2^{Q_{\mathcal{R}}}$ is a set of accepted states pairs; and $\mathrm{Ac}{\mathrm{c}}_{\mathcal{R}}=\left\{\left(L^1,K^1\right),\left(L^2,K^2\right),\dots ,\left(L^N,K^N\right)\right\}$, $L^i,K^i\subseteq Q_{\mathcal{R}},\forall i=1,2,\dots ,N$. An infinite run $q_0{q}_1{q}_2\cdots$ of $\mathcal{A}$ is accepted if there are some state pairs $\left(L^i,K^i\right)\in \mathrm{Ac}{\mathrm{c}}_{\mathcal{R}}$ such that $\forall m\ge n,q_m\notin L^i$, for $\exists n\ge 0$ and $\stackrel{\infty }{\exists }n\ge 0$, $q_n\in K^i$, where $\stackrel{\infty }{\exists }$ means that there are infinite times. In other words, the run should intersect with $L^i$ only finitely many times and $K^i$ infinitely many times.

2.2. Probabilistic Behavior Model

The Markov decision process is used to model the global behavior of UAVs, which is mainly used to describe the probability and uncertain behavior of UAVs.

Definition 2

(MDP [10]). MDP is defined as a tuple $\mathcal{M}=\left(S,\overline{s},{\alpha }_{\mathcal{M}},{\delta }_{\mathcal{M}},L\right)$, where S is a finite set of states; $\overline{s}\in S$ is the initial state; ${\alpha }_{\mathcal{M}}$ is a set of behaviors; ${\delta }_{\mathcal{M}}:S\times {\alpha }_{\mathcal{M}}\to Dist\left(S\right)$ is a probabilistic transition function with $Dist\left(S\right)$ as a set of distribution over S; and $L:S\to 2^{\mathrm{AP}}$ is a labeling function, which maps each state to a set of atomic propositions.

The transition between states in MDP includes two steps. First, one or more behaviors are selected from the behavior set ${\alpha }_{\mathcal{M}}$. The set of valid behaviors at the state is given by $A\left(s\right)\stackrel{\Delta }{=}\left\{a\in {\alpha }_{\mathcal{M}}\left|{\delta }_{\mathcal{M}}\left(s,a\right)>0\right.\right\}$. The transition generated by selecting the behavior $a\in A\left(s\right)$ is uncertain. Secondly, a subsequent state $s^{\prime }$ is randomly selected according to the probability distribution ${\delta }_{\mathcal{M}}\left(s,a\right)$, that is, the probability of transition to the state $s^{\prime }$ is equal to ${\delta }_{\mathcal{M}}\left(s,a\right)\left(s^{\prime }\right)$. In deterministic models, the transitions triggered by the behaviors are valid.

An infinite state path of MDP is a state behavior sequence $\pi =s_0{a}_0{s}_1{a}_1\cdots$, in which, for all $i\in \mathbb{N}$, $s_i\in S$, $a_i\in A\left(s_i\right)$ and ${\delta }_{\mathcal{M}}\left(s_i,a_i\right)\left(s_{i+1}\right)>0$. A finite state path $\rho =s_0{a}_0{s}_1{a}_1\cdots a_{n-1}{s}_n$ is the prefix of an infinite state path that terminates at the state $s_n$. $FPat{h}_{\mathcal{M},s}$ and $IPat{h}_{\mathcal{M},s}$ are respectively defined as the path set containing finite states and that containing infinite states starting from state s in $\mathcal{M}$. For a finite state path, $\left|\rho \right|=n$ means its length, and $last\left(\rho \right)=s_n$ means its final state. For an infinite state path $\pi =s_0{a}_0{s}_1{a}_1\cdots$, its $\left(i+1\right)$-th state $s_i$ is recorded as $\pi \left(i\right)$, and its trace $tr\left(\pi \right)$ is a behavior sequence $a_0{a}_1\cdots$.

Definition 3

(Plan [12]). The plan of MDP can be obtained by a function $\sigma :FPat{h}_{\mathcal{M}}\to Dist\left({\alpha }_{\mathcal{M}}\right)$, only if $a\in A\left(last\left(\rho \right)\right)$, $\sigma \left(\rho \right)\left(a\right)>0$.

The plan is used to select a behavior in each state of MDP so as to eliminate the uncertainty of the behaviors. The set of all plans for $\mathcal{M}$ is denoted as $Ad{v}_{\mathcal{M}}$.

3. Proposed Framework

This section presents the proposed shared autonomy framework into the human–UAV collaboration, which is shown in Figure 1 and divided into several phases, including task description, task interpretation, probabilistic behavior modeling, product system modeling, plan generation, task execution, and task monitoring.

(1)

Task description aims to describe the high-level task instructions input by human in natural language as the LTL formula.

(2)

Task interpretation is responsible for converting the LTL formula into an automaton, which can be understood by the UAV.

(3)

Probabilistic behavior modeling is used to model the probabilistic behavior of the UAV.

(4)

The probabilistic behavior model is synthesized with a task automaton in the form of a Cartesian product system.

(5)

In the plan generation, the behavior plan is generated with a specific strategy generation algorithm.

(6)

The corresponding task achievement probability is given to the task monitor.

Therefore, the human collaborators can monitor whether the UAV completes the task according to the requirements through task monitoring so as to dynamically adjust the high-level task instruction.

4. Methodology

4.1. Product System Based on Finite State Automaton

4.1.1. Finite State Automaton Within h

Based on the state values obtained from the DRA (transformed from the global task LTL), a finite state automaton is constructed online by graph search, and the accepted states within the horizon are determined.

Definition 4

(FSAh).The finite state automaton ${\mathcal{A}}^h$ within the time domain h is defined as ${\mathcal{A}}^h=\left(Q_{\mathcal{A}},q_{init,\mathcal{A}},{\Sigma }_{\mathcal{A}},{\delta }_{\mathcal{A}},F_{\mathcal{A}}\right)$, where

• $Q_{\mathcal{A}}\subset Q_{\mathcal{R}}$ is a finite set of states;
• $q_{init,\mathcal{A}}$ is the initial state within the time domain;
• ${\Sigma }_{\mathcal{A}}={\Sigma }_{\mathcal{R}}$;
• $Q_{\mathcal{A}}^0=\left\{q_{init,\mathcal{A}}\right\}$;
• For all $1\le j\le h$, if and only if $q\in Q^{j-1}$, $q^{\prime }\in Q_{\mathcal{A}}^j$ and $\left(q,\sigma ,q^{\prime }\right)\in {\delta }_{\mathcal{A}}^j$;
• Finally, $Q_{\mathcal{A}}={\bigcup }_{0\le j\le h}{Q}_{\mathcal{A}}^j$ and ${\delta }_{\mathcal{A}}={\bigcup }_{1\le j\le h}{\delta }_{\mathcal{A}}^j$;
• $q^{max}\in Q_{\mathcal{A}}\setminus \left\{q_{init,\mathcal{A}}\right\}$, $q{\to }^{*}{q}_{acc}$, and $\left|{\delta }_{q,q_{acc}}\right|$ is the minimum.

FSA h is not a DRA because it does not have an infinite state path, which is an automaton that reads a finite number of words and can be regarded as a directed graph.

By calculating the shortest distance from the state $q\in \mathcal{R}$ to the accepted states, the following distance to be completed can be defined for the DRA.

Definition 5

(Distance to be completed $d_{\phi }$). If $\forall i\in \left\{1,\cdots ,N\right\}$, $q\notin K^i$ and $\exists q_{\mathrm{Acc}}$, such that $q{\to }^{*}{q}_{\mathrm{Ac}{\mathrm{c}}_{\mathcal{R}}}$, ${\to }^{*}$ means that there are connectivity paths, then

$$d_{\phi }\left(q\right)=\underset{q^{\prime }\in Su{c}_q}{min}\left\{d_{\phi }\left(q^{\prime }\right)+{\left|{\delta }_{q,q^{\prime }}\right|}^{-1}\right\}$$

(5)

where $\left|{\delta }_{q,q^{\prime }}\right|$ means the transition distance from the state q to the state $q^{\prime }$. If $\exists i\in \left\{1,\dots ,N\right\}$, $q\in K^i$, then $d_{\phi }\left(q\right)=0$. For the other state q, $d_{\phi }\left(q\right)=\left|Q_{\mathcal{R}}\right|$, where $\left|Q_{\mathcal{R}}\right|$ represents the total number of $\mathcal{R}$.

As can be seen from the above definition, $d_{\phi }$ represents the minimum transition distance to an accepted state set $K^i$. If an accepted state set $K^i$ of $Ac{c}_{\mathcal{R}}$ can be reached from q, then $d_{\phi }\left(q\right)\le \left|Q_{\mathcal{R}}\right|-1$. If q is an accepted state, $d_{\phi }\left(q\right)=0$. If there is no transition from q to an accepted state set $K^i$ in the $Ac{c}_{\mathcal{R}}$, then $d_{\phi }\left(q\right)=\left|Q_{\mathcal{R}}\right|$. The value of $d_{\phi }$ can be calculated by the fixed point iteration method as follows:

$$d_{\phi }^{k+1}\left(q\right)=min\left\{d_{\phi }^k\left(q\right),{min}_{q^{\prime }\in Su{c}_q}\left\{d_{\phi }^k\left(q^{\prime }\right)+{\left|{\delta }_{q,q^{\prime }}\right|}^{-1}\right\}\right\}$$

(6)

where if $\forall i\in \left\{1,\dots ,N\right\}$, $q\in K^i$, then $d_{\phi }^0\left(q\right)=0$. For the other state q, $d_{\phi }^0\left(q\right)=\left|Q_{\mathcal{R}}\right|$.

4.1.2. Product System Within H

The finite state automaton and its accepted states allow us to define tasks completed in time and order, which can make UAV transits complete the global tasks. We complete this process by defining a product system, which can capture the allowed behavior (state–behavior pair) of the UAV within the time domain.

Definition 6

(Product system within H). Within time domain H, the product system ${\mathcal{P}}^H$ of the MDP $\mathcal{M}$ and the finite state automaton ${\mathcal{A}}^h$ is defined as ${\mathcal{P}}^H=\left(Q_{\mathcal{P}},q_{init,\mathcal{P}},{\Sigma }_{\mathcal{P}},{\delta }_{\mathcal{P}}\right)$, where we have the following:

• $Q_{\mathcal{P}}\subset S\times Q_{\mathcal{A}}$ is a finite set of states;
• $q_{init,\mathcal{P}}=\left(s_{init},q_{init,\mathcal{A}}\right)$, $s_{init}$ is the initial state of the MDP within H;
• ${\Sigma }_{\mathcal{P}}={\Sigma }_{\mathcal{A}}$;
• $Q_{\mathcal{P}}^0=\left\{q_{init,\mathcal{P}}\right\}$;
• For all $1\le j\le H$, if and only if $A\left(s\right)\in {\alpha }_{\mathcal{M}}$, $\sigma \subset L\left(s\right)$ and $\left(q,\sigma ,q^{\prime }\right)\in {\delta }_{\mathcal{A}}$, $\left(s^{\prime },q^{\prime }\right)\in Q_{\mathcal{P}}^j$, $\left(\left(s,q\right),\sigma ,\left(s^{\prime },q^{\prime }\right)\right)\in {\delta }_{\mathcal{P}}^j$;
• Finally, $Q_{\mathcal{P}}={\bigcup }_{0\le j\le H}{Q}_{\mathcal{P}}^j$ and ${\delta }_{\mathcal{P}}={\bigcup }_{1\le j\le H}{\delta }_{\mathcal{P}}^j$.

The tuple ${\mathcal{P}}^H$ is a product MDP and can be regarded as a directed graph. The state value of $Q_{\mathcal{P}}$ is inherited from the local automaton:

$$d_{\phi }\left(Q_{\mathcal{P}}\right)=\left\{\begin{array}{c}{d}_{\phi }\left(Q_{\mathcal{A}}\right),\\ 0,\end{array}\right.\begin{array}{c}\mathrm{if}{Q}_{\mathcal{A}}\subseteq Q_{\mathcal{P}}\\ \mathrm{otherwise}\end{array}$$

(7)

4.1.3. Determining the Target State Within H

As this paper constructs the product system based on a local automaton, there may be no accepted states within the current horizon. Therefore, the following method is proposed to determine the local target state for the product system ${\mathcal{P}}^H$ within the finite horizon. As in Definition 6, each state in the product system ${\mathcal{P}}^H$ has a distance $d_{\phi }\left(Q_{\mathcal{P}}\right)$ from the accepted states. The smaller the distance is, the closer the UAV reaches the state to meet the task specification. Therefore, it is necessary to obtain the states with the minimum distance from the state set of the Cartesian product system within the current horizon $Q_{\mathcal{P}}^M={\cup }_{Q_{\mathcal{P}}}arg{min}_{Q_{\mathcal{P}}}{d}_{\phi }\left(Q_{\mathcal{P}}\right)$ to determine the local target state. Because the product system ${\mathcal{P}}^H$ is obtained by Cartesian product of the MDP in the horizon H and the automaton in the horizon h, multiple states may have the same minimum distance from the accepted states, but not all these states can be regarded as the local target state due to the acceptance condition of the automaton. Thus, the classical Dijkstra algorithm is introduced to calculate the local target state $Q_{\mathcal{P}}^F$ in the set $Q_{\mathcal{P}}^M$ closest to the initial state of ${\mathcal{P}}^H$. Finally, a plan ${\sigma }_{\mathcal{P}}^{*}=arg{max}_{{\sigma }_{\mathcal{P}}}{Pr}_{{\mathcal{P}}^H,q_{init,\mathcal{P}}}^{max}\left(Q_{\mathcal{P}}\right),Q_{\mathcal{P}}\in Q_{\mathcal{P}}^F$ is generated for the UAV to arrive at the state $Q_{\mathcal{P}}^F$ with the maximum probability from the initial state.

Example 2.

An example of determining the local target state within the current horizon is shown in Figure 2. The MDP model of the UAV is illustrated in Figure 2a. The LTL task specification is $\phi \stackrel{\Delta }{=}\neg A\mathrm{U}B\wedge \mathcal{X}\mathrm{F}C$, which means that the UAV cannot pass through A in the process of going to B, and then go to C after arriving at B. The DRA transformed from the task specification of the UAV is shown in Figure 2b, and the initial state is 2. When $h=1$, the DRA can reach states 1 and 3 from the initial state 2. When reaching state 1, it means that the task specification is violated. Due to the previously defined state values, UAV will not reach A, and the reachable states in the DRA are limited to states 2 and 3. Therefore, the constructed finite horizon product system is shown in Figure 2c. According to the defined distance metric, state 3 of DRA is closer to the accepted state 0 than state 2, so the distance of state 3 is less than state 2. Then, in the constructed product system, the states in the same color region have the same distance, in which the distance in the yellow region is less than that in the green region, that is, it is necessary to determine the local target state within the current horizon in the yellow region. The initial state of the current product system is (1, 2), while the state closest to the initial state in the yellow region is (3, 3). Thus, state (3, 3) is determined as the target state of the current product system, indicating that within the current horizon, the UAV can meet some of the task specification φ and reach B without going through A.

When $h=2$, states 1, 3, and 0 can be reached from the initial state 2 in the DRA. The same condition for reaching 1 is violated. Due to the previously defined state values from which the UAV will not reach A, the reachable states in the DRA are 2, 3, and 0. Therefore, the constructed finite horizon product system is shown as Figure 2d, in which the states in the blue region have the minimum distance, that is, the target state within the current horizon is determined in the blue region. The state closest to the initial state (1, 2) is (4, 0), so the target state within the current horizon is (4, 0). It means that the UAV can meet the task specification φ in the current horizon.

4.2. Behavior Planning Within the Time Domain H

The value iteration algorithm is used to obtain an optimal plan ${\sigma }^{max}$ of the product system ${\mathcal{P}}^H$ to maximize the cumulative probability of reaching the desired target state. The specific formula is as follows:

$${\sigma }^{H,max}\left(s\right)\stackrel{\Delta }{=}arg\underset{a\in {\alpha }_{\mathcal{M}}}{max}\left({\sum }_{s^{\prime }\in S}{\delta }_{\mathcal{M}}\left(s,a\right)\left(s^{\prime }\right)·P{r}_{s^{\prime }}^{max}\left(reach\left(q^{max}\right)\right)\right)$$

(8)

The reverse value iteration and the plan generation process can be seen in ref. [5]. Algorithm 1 shows the procedure of the online behavior planning within finite time domain.

Algorithm 1 Procedure of $short\_horizon\_plan$.

Input:  The MDP $\mathcal{M}$ of the UAV, the deterministic Rabin automaton $\mathcal{R}$, the current state of the UAV S, the current state of the automaton q, the finite horizon $h,H\in \mathbb{N}$. Output:  The optimal behavior plan ${\sigma }^{max}$ to maximize the probability of task satisfiability within the horizon. 1: Initialize $q_{init,\mathcal{P}}=\left(s,q\right)$; 2: Construct the finite state automaton ${\mathcal{A}}^h$, calculate the local target state $q^{max}$ (Definition 4); 3: Construct the product system ${\mathcal{P}}^H$ (Definition 6); 4: Generate the optimal behavior plan ${\sigma }^{H,max}$ within the horizon based on value iteration.

4.3. Tasks Execution Across Infinite Time

In order to meet the global task description, after the UAV obtains the optimal behavior plan within the horizon H, it executes according to the behavior plan. When it reaches the expected state within the horizon, it takes the reached expected state as the initial state in the new horizon to replan the optimal behavior plan for the next cycle. The process is repeated until the UAV meets the global task description. Algorithm 2 shows the execution process of infinite horizon iteration.

Algorithm 2 Infinite horizon planning and execution.

Input:  State set of the UAV S, behavior set ${\alpha }_{\mathcal{M}}$, distribution $Dist\left(S\right)$, alphabet $\mathrm{AP}$, label function L, task description $\phi$. Output:  Null. 1: Initialize horizon $h,H\in \mathbb{N}$; 2: Use the ltl2dstar tool to transform $\phi$ into the deterministic Rabin automaton $\mathcal{R}$, and calculate state values (Definition 1); 3: Construct the product system ${\mathcal{P}}^H$ (Definition 6); 4: Generate the optimal behavior plan ${\sigma }^{H,max}$ within the horizon based on value iteration.

5. Simulation and Analysis

5.1. Settings

The correctness and effectiveness of the proposed method are verified by simulation. A UAV is required to operate in a search-and-rescue scenario according to the instructions of human collaborators in the grid mountainous areas. Due to the complex environment and task, it is assumed that the UAV has actuator deviations. The UAV automatically leaves the initial location, successively goes to positions $A,B,C$ for search tasks. Then, it delivers supplies to the target area. Simulations below are all carried out on a computer with core i7-1065G7.

5.2. Correctness and Effectiveness

Before completing the whole simulation, we verify the correctness and efficiency of the proposed method through some simple LTL operators. The test case is shown in Figure 3.

Firstly, the LTL task specification of the UAV is set as ${\phi }_1\stackrel{\Delta }{=}\neg A\mathcal{U}C$, which means that ‘the UAV avoids reaching the location A until reaching the location C’, and the generated DRA using the ltl2dstar tool is shown in Figure 4. As can be seen from the figure, there are three states in the automaton, of which the initial state is state 0 and the accepted state is state 2. All states can be reached from the initial state in one step, i.e., $h=1$, and the simulation parameters set for ${\phi }_1$ are shown in

Table 1. The simulation parameters for ${\phi }_1\stackrel{\Delta }{=}\neg A\mathcal{U}C$.

Group 1	Group 2	Group 3
$h=1$	$h=1$	$h=\infty$
$H=2$	$H=3$	$H=\infty$

.

The deterministic plans generated within different time domains are shown in Figure 5.

The related time consumption and the size of the generated state space are shown in Figure 6. It can be seen from the figures that the proposed method can generate the correct plans for the UAV, and can effectively improve the timeliness of plan generation compared with the classical probabilistic model checking method.

Secondly, the LTL task specification of the UAV is set as ${\phi }_2\stackrel{\Delta }{=}\mathrm{G}\left(A\to \mathcal{X}\neg A\right)$, which means that ‘the UAV always visits the location A and next avoids reaching the same location’, and the generated DRA is shown in Figure 7.

The deterministic plans generated within different horizons are shown in Figure 8, and the simulation parameters set for ${\phi }_2$ are shown in

Table 2. The simulation parameters for ${\phi }_2\stackrel{\Delta }{=}\mathrm{G}\left(A\to \mathcal{X}\neg A\right)$.

Group 1	Group 2	Group 3
$h=1$	$h=2$	$h=\infty$
$H=1$	$H=2$	$H=\infty$

.

The related time consumption and the size of the generated state space are shown in Figure 9. It can be seen that the proposed method can generate the correct plan for the UAV, and can effectively improve the efficiency of plan generation compared with the global calculation method.

Thirdly, the LTL task specification of the UAV is set as ${\phi }_3\stackrel{\Delta }{=}\mathrm{G}\neg A\wedge \mathrm{GF}C$, which means that ‘the UAV always avoids reaching the location A and repeats reaching the location C’, and the generated DRA is shown in Figure 10.

The deterministic plans generated within different time domains are shown in Figure 11, and the simulation parameters set for ${\phi }_3$ are shown in

Table 3. The simulation parameters for ${\phi }_3\stackrel{\Delta }{=}\mathrm{G}\neg A\wedge \mathrm{GF}C$.

Group 1	Group 2	Group 3
$h=1$	$h=2$	$h=\infty$
$H=2$	$H=4$	$H=\infty$

.

The related time consumption and the size of the generated state space are shown in Figure 12. It can be seen from the figure that the same conclusion as above can be made.

5.3. Result Analysis

In this subsection, according to the simulation settings in Section 5.1, the top-level task specification of the UAV is described as the following LTL formula:

$${\phi }_1\stackrel{\Delta }{=}\mathrm{F}A\wedge \mathcal{X}\mathrm{F}B\wedge \mathcal{X}\mathrm{F}C\wedge \mathcal{X}\mathrm{F}D\wedge \mathcal{X}\mathrm{F}E$$

(9)

Then, it can be transformed into a deterministic Rabin automaton. Subsequently, the probabilistic behavior model of the UAV is constructed. The set of behaviors that the UAV can choose to perform at each grid of the mountainous area is ${\alpha }_{\mathcal{M}}=\left\{\to ,\downarrow ,\leftarrow ,\to \right\}$. It is assumed that these behaviors can be realized by the underlying controller of the UAV, so that the UAV can move between adjacent grids. In addition, we also assume that there is an actuator bias, for example, when the UAV performs behavior ‘←’ at the state $s9$, there is a probability arrival state $s10$ of 0.8, and there are probability arrival states $s6$ and $s15$ of 0.1, respectively. When the path between two states is blocked, or there is an impassable state in cross grid, the execution probability of the current behavior of the UAV is 1.0 and 0.9, respectively. Moreover, atomic labels A, B, C, D, and E are set at the states $s16$, $s25$, $s19$, $s5$, and $s4$, respectively.

Then, Algorithm 1 is used to generate the behavior plans within the finite time domain. Here, we set the time domain of the automaton as $h=1$ and the time domain of the MDP as $H=3$. Successively the finite state automatons within the time domain and the product systems within the time domain are constructed. Figure 13 shows the directed graph representation of the product system in the first planning cycle, of which the corresponding starting node is $q_{init,\mathcal{P}}=\left(\prime s{9}^{\prime },6\right)$, and the value of the expected state within the time domain is 0.5.

Based on the product system constructed within the finite time domain, the value iterative algorithm is used to obtain the optimal plan to maximize the cumulative probability of reaching the desired states, where the iterative convergence threshold is set to $\epsilon \stackrel{\Delta }{=}{10}^{-6}$, and the generated behavior plans within the finite time domain are shown in Figure 14. In the figure, the states of the MDP and the optimal state–behavior pairs corresponding to the product system within the time domain $H=3$ are marked with a red polygon box and red arrow, respectively. Therefore, the optimal plan for the UAV to reach the desired state within the time domain $h=1$ planned from the initial state $s9$ is ${\sigma }^{max}=s9\leftarrow s10\downarrow$, in which the corresponding optimal behavior values at $s9$ and $s10$ are about 0.494327 and 0.494450, respectively. By intuitively understanding the temporal logic of the LTL Formula (9), the primary task performed by the UAV is ‘$FA$’ (that is, starting from the initial state $s9$ to reach A). It can be seen that the results of the online behavior planning are consistent with the task logic at the semantic level, indicating the correctness of the finite time domain planning method.

Finally, assuming that there are no environmental disturbances and the actuator deviation of the UAV, we use Algorithm 2 to plan the global behaviors of the UAV. Figure 15 shows the results of dynamic modeling, planning and unbiased execution of the UAV according to the LTL Formula (9) from the initial state s9, and marks the corresponding behavior return value, in which the blue letter-number-arrow marks indicate the planning results of the initial points within the horizon, and the red letter-number-arrow marks indicate the planning results of the other states within the horizon. Similarly, through the visual analysis of the temporal relationship of the tasks, it can be seen that the results of dynamic modeling and planning are consistent with the temporal logic of LTL formulas, and the planned behaviors within a single horizon are optimal compared with other behaviors. To sum up, the proposed method under the framework of probabilistic model checking can effectively realize the dynamic modeling of UAVs and online behavior planning.

To further demonstrate the efficiency improvement of the proposed method, the time consumption of plan generation and the size of generated state space within different horizons are analyzed and compared with the infinite horizon plan and the sampling-based method [17]. The results, presented in Figure 16 and Figure 17, show that the proposed method can effectively alleviate the state explosion caused by probabilistic model checking and improve the efficiency of plan generation compared with the infinite time domain plan, of which the computation time is a little longer than the sampling-based method.

To further verify the correctness of the proposed method, in addition to the task specification (9), some complex task specifications with different structures are considered as follows.

For the reason of space, the corresponding DRAs of task specification ${\phi }_2$ and ${\phi }_3$ are not provided here:

$${\phi }_2\stackrel{\Delta }{=}\mathrm{F}A\wedge \mathcal{X}\neg B\mathrm{U}C\wedge \mathcal{X}\mathrm{F}C\wedge \mathcal{X}\mathrm{F}D$$

(10)

$${\phi }_3\stackrel{\Delta }{=}\mathrm{F}(A\wedge \mathrm{F}B\wedge \mathrm{F}C)\wedge \neg B\mathrm{U}C$$

(11)

Then the determinations of goal states for different task specifications containing different types of operators within different horizons are tested, i.e., the strategies are synthesized under different H and h, compared with global strategy synthesis. For the task specification ${\phi }_2$, the product automatons are constructed in 0.00914 s and 0.01283 s, respectively, under $H=3$, $h=1$, and $H=6$, $h=2$, and the times of constructing the product system are 4.1822 s and 6.3167 s, which is faster than the global synthesis (0.02732 s and 11.6418 s). And the time of synthesizing the strategy are 2.0745 s, 4.2838 s and 10.6219 s, respectively. The strategy synthesized with the maximum probability of satisfying it under $H=3$, $h=1$, and $H=6$, $h=2$ is shown in Figure 18a and Figure 18b, respectively, while the global strategy synthesized is shown in Figure 18c.

As can be seen from the above figures, the strategies synthesized by the proposed method are correct under different time horizons, as they are making the UAV move towards satisfying task specification ${\phi }_2$, which is the same as the global synthesis. And for the task specification ${\phi }_3$, the product automatons are constructed in 0.01223 s and 0.01351 s, respectively, under $H=4$, $h=1$ and $H=6$, $h=2$, and the time of constructing the product system is 6.3641 s and 9.1905 s, which is much shorter than that of global synthesis, where the time of constructing the product automaton and product system is 0.17683 s and 13.75924 s. And the time of synthesizing the strategy is 5.6602 s, 8.0493 s, and 12.4981 s, respectively. The strategy synthesized with the maximum probability of satisfying it under $H=4$, $h=1$, and $H=6$, $h=2$ is shown in Figure 19a and Figure 19b, respectively, while the global strategy synthesized is shown in Figure 19c.

As can be seen from the above figures, the strategy synthesized under $H=4$, $h=1$ is not optimal due to its short horizon. And the strategy synthesized under $H=6$, $h=2$ is the same as that of global synthesis. While all the above strategies synthesized are correct, we guide the UAV to move towards satisfying the task specification ${\phi }_3$.

6. Experiments

In this section, the proposed method is applied to the developed ROS-based hardware-in-the-loop simulation system shown in Figure 20 and Figure 21, where the environment is constructed in Unreal Engine with the task areas, and the dynamics of the UAV is real. The human collaborator injects the top-level task instruction in the ground control station, and a behavior plan is generated online. Based on this, the collaborator can adjust the task at any time, thus playing the role of real-time supervision and intervention. We use the remote control PC to log in to the onboard MiniPC via the router (here, Rockchip 3588 is used), use the voice assistant toolkit (xfei-asr) to receive and translate the collaborator’s voice commands, and then send the generated LTL-form commands to the on-board MiniPC. The onboard MiniPC as the host computer is equipped with ROS. Its main functions include (i) using the package task-generator to convert the LTL form instructions into a task automaton that the UAV can understand in the finite horizon; (ii) according to the current task progress and the expected task goal in the horizon, using the package plan generator to calculate the behavior plan of the discretized workspace in the horizon; (iii) after receiving the odometer data and sensor data fed back by the lower computer, according to the current pose of the UAV, combined with the current state–behavior mapping in the behavior plan, the package local planner is used to calculate the next path planning result; and (iv) mapping the planning result to the speed and angular velocity of the control, and feeding it back to the lower computer (the Pixhawk is used here). After receiving the speed and angular velocity of the upper computer, the lower computer converts it into the PWM value of the final drive motor, and at the same time, it calculates the encoder information into the speed and angular velocity of the control point and sends it to the upper computer.

For the hardware-in-the-loop experiment, we adopted the task ${\varphi }_1$ and scenario as above, where the UAV is mainly consisted of three parts: the mission computing board operating ROS, the flight control board setting as HIL mode with a GPS module, and the real dynamic models in simulation environment. The Rockchip 3588 establishes serial communication with the Pixhawk autopilot to send the target positions through the Marvos package, and the Pixhawk autopilot connects to the workstation by Universal Serial Bus to send the motor drive signals of the UAV computed from the target positions to simulation environment. The UAV communicates wirelessly to the computer via ROS. Under different horizons, the task plans are generated according to the ${\varphi }_1$ given by the human collaborator in the ground control station and then deployed to the UAV. Then, the UAV starts to execute the task according to the plans. The process of task execution is shown in Figure 22, and as can be seen from the figure, under the horizon $H=3$, $h=1$, and $H=4$, $h=2$, the UAV both completes the task ${\varphi }_1$ correctly, which reaches the corresponding task areas in time.

7. Conclusions

In this paper, through introducing the receding horizon mechanism into the probabilistic model checking, an online human-aware behavior planning method for nondeterministic system is proposed to make the UAV dynamically satisfy the top-level LTL task description from the human collaborators. Experiments and analysis show that the results of the online behavior planning are consistent with the task logic at the semantic level. Moreover, the proposed method can effectively alleviate the state explosion caused by the global probabilistic model checking.

In the follow-up, the theoretical research work will be further refined, and the improvement efficiency of computational performance will be deeply analyzed. In addition, the proposed method can be applied to some broader real industrial human–UAV collaboration fields.