Event-Triggered Intervention Framework for UAV-UGV Coordination Systems

Abstract

Air-ground coordination systems are usually composed of unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV). In such a system, UAVs can utilize their much more perceptive information to plan the path for UGVs. However, the correctness and accuracy of the planned route are often not guaranteed, and the communication and computation burdens increase with more sophisticated algorithms. This paper proposes a new type of air-ground coordination framework to enable UAVs intervention into UGVs tasks. An event-triggered mechanism in the null space behavior control (NSBC) framework is proposed to decide if an intervention is necessary and the timing of the intervention. Then, the problem of whether to accept the intervention is formulated as an integer programming problem and is solved using model predictive control (MPC). Simulation results show that the UAV can intervene in UGVs accurately and on time, and the UGVs can effectively decide whether to accept the intervention to get rid of troubles, thereby improving the intelligence of the air-ground coordination system.

1. Introduction

With the rapid development of science and technology, robot, as an advanced tool that integrates a number of advanced technologies, are having more and more impacts on human society. For complex and dynamic tasks and environments, a multi-robot system has the advantages of lower operating costs, fewer system requirements, stronger adaptability, and flexible scalability, when compared to a single robot [1].

Among all kinds of robots, unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs) have been widely used in civilian and military fields. UAVs can provide a global and accurate view of the environment by making use of their fast moving speed and their being less prone to GPS signal interruption. UGVs have high load capacity and can endure long-term task requirements. Air-ground coordination systems, formed by combining the functional characteristics of UAVs and UGVs, can not only effectively avoid weaknesses such as the short flight time of UAVs and poor perception of UGVs, but also provide breakthrough ideas for multi-robot coordination systems with huge heterogeneity and functional complementarity [2,3]. Air-ground coordination systems have been widely used in many scenarios, such as agriculture [4], rescue [5], exploration [6], and surveillance [7].

UAV-UGV coordination systems can be classified into eight different settings, from the perspective of the functional roles that UGVs and UAVs have in a system [8]. The functional roles of UAVs and UGVs can be divided into: sensors, actuators, decision makers, and auxiliary facilities. UAV-UGV coordination systems can be expressed as <X|Y>, where X is the functional role of UAVs and Y is the functional role of UGVs. Most air-ground coordination systems can be classified as different types using different combinations of UAV and UGV roles [9,10,11,12]. One typical class is written as <S,D|A>, where UAVs act as sensors and decision makers, and UGVs act as actuators. This class of systems is of particular interest because there are tasks that UGVs cannot complete by their own intelligence. For example, a UGV may fall into a local minimum when avoiding multiple obstacles (e.g., stops in the middle between two obstacles). This necessitates UAV intervention for UGV on-line in the process of task execution. In this case, UAVs perceive the environment from the air and make decisions for UGVs. This type of system traditionally collects information through the manual control of UAVs, which help path planning for UGVs offline [13]. Such a process has difficulty in coping with dynamic environments. Many researchers propose to achieve path planning and control on-line [14,15]. However, this is limited in cases when UGVs do not need UAVs to make perception decisions when performing their basic tasks. In addition, on-line decision making and path planning bring communication and computation burdens. Therefore, key problems arise as how to build a decision-making model for UAV intervention and how to solve the problem of task conflicts between external interventions and the UGVs own tasks.

To solve the potential decision-making/planning/control conflicts, the null space behavior control (NSBC) method is one of the potential solutions, where different basic tasks for UAVs and UGVs are defined as behaviors with assigned priority. This method ensures that, under the premise of the complete execution of high-priority tasks, the partial execution of secondary tasks is possible [16]. An air-ground cooperative formation method based on the NSBC has been proposed to ensure that the formation shape can be maintained when UGVs or UAVs encountered obstacles [17,18,19]. A new type of human decision-making behavior model has been proposed in the framework of the NSBC using an event-triggered mechanism [20]. The human intervention task, with the highest priority, is triggered only when defined decision variables reach a threshold. It reduces the burden of communication and computation by intervening robots in an event-triggered way and enables human assistance to complete tasks beyond robots’ own capabilities. However, the existing methods cannot avoid risks brought on by bad or malignant interventions from either humans or UAVs.

Motivated by the above discussions, in this paper, we focus on how UAVs intervene with UGVs to improve the overall intelligence of the system, while reducing the burden of communication and computation. The contributions of this paper can be summarized as follows. First, a new type of air-ground coordination system is proposed, which can be written as <S,D,A|S,D,A>, where both UAVs and UGVs have the ability to perceive, make decisions, and execute tasks. It extends the unidirectional intervention that existing works consider to bidirectional interventions. Second, the drift diffusion model (DDM) and model predictive control (MPC) are introduced into the NSBC framework to accurately determine the timing of intervention, to achieve an optimal trade-off between decision speed and accuracy, and to predict whether an intervention is correct and acceptable. The latter decision-making acceptance problem is formulated into an integer programming problem and solved using current state and future predictions.

The rest of this paper is organized as follows. In Section 2, we briefly introduce preliminaries of the NSBC; In Section 3, the UAV and UGV task design and decision making model based on event-triggered way is presented; In Section 4, the intervention task decision maker based on MPC is presented; In Section 5, simulation cases are studied and discussed; Section 6 concludes the paper.

2. Preliminary

Let’s briefly review the NSBC methods [21], by defining as $\sigma \in {\mathbb{R}}^m$ mathematical expressions for the behavior to be achieved (usually called tasks). $\rho \in {\mathbb{R}}^n$ is defined as a variable vector describing the system configuration. In general, they are related through the following models:

$$\sigma =f\left(\rho \right)$$

(1)

with the corresponding differential relationship:

$$\dot{\sigma }=\frac{\partial f\left(\rho \right)}{\partial \rho }\times \nu =J\left(\rho \right)\times \nu$$

(2)

where $J\left(\rho \right)\in {\mathbb{R}}^{m\times n}$ is the configuration-dependent task Jacobian matrix and $\upsilon \in {\mathbb{R}}^n$ is the system velocity. The reference velocity ${\nu }_d$ is to act at the differential level by inverting the (locally linear) mapping and pursueminimum-norm velocity, leading to the least-squares solution:

$${\nu }_d=J^{†}{\dot{\rho }}_d=J^T{\left(J{J}^T\right)}^{-1}{\dot{\rho }}_d$$

(3)

where ${\rho }_d$ is the reference trajectory, $J^{†}$ is the pseudo-inverse of the Jacobian matrix. Since the discrete-time integration of the reference velocity will cause the numerical drift of the reconstructed position, the following closed loop inverse kinematics (CLIK) form is used to compensate for the drift:

$${\nu }_d=J^{†}({\dot{\rho }}_d+\wedge {\tilde{\rho }}_d)$$

(4)

where $\dot{{\rho }_d}$ is the derivative of the desired task function. $\mathrm{\Lambda }$ is a suitable constant positive definite matrix of gains. ${\tilde{\rho }}_d={\rho }_d-\rho$ is the task error.

Consider there are N tasks, and each task is assigned a priority (expressed by subscript i, i = 1 means the highest priority ). The NSB solution to the task combination can be formulated in an iterative way, defining the velocity vector as follows:

$${\upsilon }^i={\upsilon }_{d,i}+N_i\times {\upsilon }^{i+1},i=1,2,\cdots ,N$$

(5)

where ${\upsilon }^{(n+1)}=0$, ${\upsilon }^1={\upsilon }_d$, and the matrix $N_i=(I-J_i^{†}{J}_i)$ is the null space projector matrix of Jacobian. In sum, corresponding to a lower priority task, are projected onto the null space of the immediately higher priority task; then, eventually, conflicting velocity components are cut off, before being added to higher priority task velocity components. The geometric model of its comprehensive velocity output is shown in Figure 1.

3. Framework Design of Air-Ground Coordination Systems

This paper proposes an air-ground coordination system framework for UAVs to intervene UGVs through an event-triggered mechanism. As shown in Figure 2, The DDM is employed for simple decision-making modeling by accumulating decision variables. Bayes’ risk criterion is used to achieve the optimal trade-off between decision speed and accuracy, which is in charge of setting the decision threshold. The combination of the two achieves an accurate judgment on the timing of intervention [22]. When the decision variable does not reach the threshold, intervention is triggered, hence reducing the communication resources needed. To solve task conflicts, DDM is embedded into the NSBC framework for UAVs to determine whether and when to intervene in UGVs. When the decision variable has not reached the threshold, the UAVs and UGVs perform their own basic tasks. When the decision variable reaches the decision threshold, UGVs can no longer rely on their own intelligence, and the UAVs need to intervene to help make decisions.

3.1. Task Planning Layer Task Design

3.1.1. UGV Basic Task Function Design

UGV Move-to-Target Task Function Design

The movement of the UGV to the target point is defined as the task of moving to the target point. Once the target point is reached, the task is completed and the UGV stops moving. Define the location of the target point ${\rho }_g={\left[\begin{array}{ccc}{x}_g& y_g& z_g\end{array}\right]}^T$, location of UAV. $\rho$ as a controllable task variable ${\sigma }_m=\rho$. Define the position to reach the target point as the desired function ${\sigma }_{md}={\rho }_g$. Then, the output of the UGV motion task is:

$${\upsilon }_{mj}=J_{mj}^{†}({\dot{\sigma }}_{mj}+B{\tilde{\sigma }}_{mj})$$

(6)

where B is defined as the UGV motion task gain. $J_{mj}^{†}$ is the Jacobian pseudo-inverse matrix for the motion task. ${\tilde{\sigma }}_{mj}={\sigma }_{md}-{\sigma }_m$ is the task errors.

UGV Obstacles-Avoidance Task Function Design

The UGV needs to avoid obstacles detected by the sensor when moving along the reference trajectory to the target point. In order to ensure the safety of the UGV during its movement, obstacle avoidance tasks can provide basic guarantees. Define the location of UGV is $\rho ={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^T$. Obstacle location is ${\rho }_o={\left[\begin{array}{ccc}{x}_o& y_o& z_o\end{array}\right]}^T$. The obstacle avoidance task function is:

$${\sigma }_a=D=\sqrt{{(x-x_o)}^2+{(y-y_o)}^2+{(z-z_o)}^2}$$

(7)

Define the desired function of obstacle avoidance task as ${\sigma }_{ad}=d$, which d is the obstacle avoidance safety distance. Then, the output of the obstacle avoidance task function is:

$${\nu }_a=J_{aj}^{†}({\dot{\sigma }}_{ad}+A{\tilde{\sigma }}_{aj})$$

(8)

where A is defined as the UGV obstacles-avoidance task gain. $J_{aj}^{†}$ is the Jacobian pseudo-inverse matrix for the obstacles-avoidance task. ${\tilde{\sigma }}_{aj}={\sigma }_{ad}-{\sigma }_a$ is the obstacles-avoidance task errors.

3.1.2. UAV Basic Task Function Design

UAV Formation Task Function Design

UAVs and UGVs maintain a formation in order to effectively sense the movement of UGV and prepare for subsequent UAV landings. Define the task variable that the UAV keeps at the center of the UGVs as

$${\sigma }_c=\frac{1}{n}({\rho }_1+{\rho }_2+\cdots +{\rho }_n)$$

(9)

where ${\rho }_i,i=1,2\cdots ,n.$ represents the location of UGVs. Define the desired formation task function as ${\sigma }_{cd}$. UAV formation task output functions as follows.

$${\upsilon }_c=J_{cj}^{†}({\dot{\sigma }}_{cd}+C{\tilde{\sigma }}_{cj})$$

(10)

where C is defined as the UAV formation task gain. $J_{cj}^{†}$ is the Jacobian pseudo-inverse matrix for the formation task. ${\tilde{\sigma }}_{cj}={\sigma }_{cd}-{\sigma }_c$ is the formation task errors.

UAV Obstacles-Avoidance Task Function Design

Although the working environment of the UGV is relatively simple for the ground environment, it is necessary to design the task of avoiding obstacles.The task design process is the same as the obstacle avoidance task design of the UGV. Therefore, the output of the UAV obstacle avoidance task is

$${\nu }_{au}=J_{auj}^{†}({\dot{\sigma }}_{aud}+A{\tilde{\sigma }}_{auj})$$

(11)

where A is defined as the UAV obstacles-avoidance task gain. $J_{auj}^{†}$ is the Jacobian pseudo-inverse matrix for the UAV obstacles-avoidance task. ${\sigma }_{aud}$ is the UAV desired obstacle-avoidance task function, and ${\sigma }_{au}$ is the UAV obstacles-avoidance task function. ${\tilde{\sigma }}_{auj}={\sigma }_{aud}-{\sigma }_{au}$ is the UAV obstacles-avoidance task errors.

3.1.3. Composite Task Function Design

Composite task refers to the combination of multiple basic tasks according to task priority. Define ${\sigma }_b\in {\mathbb{R}}^{m_b}$ as the $b_{th}$ task function, $\forall 1\le b\le r$, where $m_b$ is the dimension of the task space. Moreover, we further define the task hierarchy which follows the rules [23]:

(1)

Assume that b = r is the lowest priority, and b = 1 is the top priority. Here, $m_b$ > $m_a$ implies that $m_b$ is the index of a lower priority than $m_a$; a task of priority $m_b$ may not disturb another task of priority $m_a$. The lower-priority tasks are executed in the null space of all higher priority tasks.

(2)

The mappings from the velocities to the task velocities are captured by the task Jacobian matrix $J_b\in {\mathbb{R}}^{m_b\times n}$, $\forall 1\le b\le r$.

(3)

The dimension $m_r$ of the lowest level task may be greater than $n-{\sum }_{b=1}^{r-1}{m}_b$ so that n is ensured to be greater than the total dimension of all tasks.

Following the aforementioned rules, the composite task velocity of UAV or UGV can be obtained by Equation (5).

3.2. UAV Intervention Task Design

In this paper, only the supervision behavior and intervention behavior of UAV to UGV are considered. The supervision behavior is defined by UAV monitoring the UGV task execution process without intervention, until failure is detected. The intervention behavior is defined by UAV intervening UGV. For the UAV supervision behavior, there is no task input to the UGV. Instead, for the UAV intervention behavior, task input to the UGV should be provided. Thus, the task corresponding to the UAV intervention behavior should be designed.The intervention task will be set as the highest priority task execution of the UGV, and the original task of the UGV will be projected onto the null space of the intervention task of the UAV. Define the desired UAV intervention task function as:

$${\sigma }_{int}=f\left({\rho }_{int}\right)$$

(12)

where ${\rho }_{int}$ is the is the real-time position of the UGV that the UAV would intervene. The derivative of the UAV intervention task is given by

$${\dot{\sigma }}_{int}=J_{int}{\upsilon }_{int}$$

(13)

where $J_{int}$ is the intervention Jacobian matrix. Therefore, the output of the UAV intervention task is

$${\nu }_{int}=J_{int}^{†}({\dot{\sigma }}_{int}+{\mathrm{\Lambda }}_{int}{\tilde{\sigma }}_{it})$$

(14)

where ${\mathrm{\Lambda }}_{int}$ is defined as the UAV intervention task gain. $J_{int}^{†}$ is the Jacobian pseudo-inverse matrix for the UAV intervention task. ${\sigma }_{in}$ is the UAV intervention task function. ${\tilde{\sigma }}_{it}={\sigma }_{int}-{\sigma }_{in}$ is the UAV intervention task error.

Assumption 1.

UAVs can give intervention tasks ${\nu }_{int}$.

Remark 1.

Intervention tasks can be given by fuzzy logic [24], reinforcement learning [25], neural network [26] method.

3.3. Decision-Making Layer Design

The decision-making layer consists of DDM and Bayes Risk criteria. The DDM is a cognitive process modeling method in the “two-choice forced decision-making problem”, which is suitable for simple decision-making process modeling. This model accumulates decision information under external noise. When the accumulated decision information reaches any decision threshold, the choice corresponding to the decision threshold is selected as the final decision result. In this paper, DDM is used as the event-triggered way for UAV intervene in UGV, embedded in the NSBC framework. The formula for the DDM of UAV is as follows.

$$d{\tilde{\rho }}_j={\tilde{\nu }}_{j}dt+{\sigma }_{j}d{w}_j\left(t\right)$$

(15)

The decision variables should be selected to reflect the progress of the robot to complete the task. In this paper, the task error of the UGV of the NSBC method is selected as the decision variable of the UAV. The task error is ${\tilde{\rho }}_j={\rho }_{rdj}-{\rho }_j$, where ${\rho }_{rdj}$ is the preset trajectory and ${\rho }_j$ is the trajectory planned by NSBC. ${\tilde{\upsilon }}_j={\nu }_{rdj}-{\upsilon }_j$ is the drift rate, which characterizes the amount of change of the decision variable per unit time. ${\sigma }_{j}d{w}_j\left(t\right)$ is white noise conforming to Gaussian distribution, which represents the influence of noise during the accumulation process of decision information.

The generation of UAV decision threshold requires the introduction of Bayes’ risk criterion function. This function can minimize the decision risk and realize the trade-off between decision speed and accuracy. The Bayes’ risk criterion function is the weighted sum of decision time (T) and decision deviation (E). The formula is as follows:

$$B=c_{1}T+c_{2}E$$

(16)

where $c_1$ and $c_2$ are the correlation coefficients of decision time and decision deviation, respectively. The formulae for decision time T and decision deviation E are as follows.

$$T=\frac{{\varsigma }_j}{{\tilde{\upsilon }}_j}tanh\left(\frac{{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}\right)+(\frac{2{\varsigma }_j(1-e^{\frac{-2{\tilde{\rho }}_{j,0}{\tilde{\nu }}_j^2}{{\sigma }^2}})}{{\tilde{\upsilon }}_j(e^{\frac{2{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}}-e^{\frac{-2{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}})}-{\tilde{\rho }}_{j,0})$$

(17)

$$E=\frac{1}{1+e^{\frac{2{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}}}-\frac{(1-e^{\frac{-2{\tilde{\rho }}_{j,0}{\tilde{\nu }}_j^2}{{\sigma }^2}})}{(e^{\frac{2{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}}-e^{\frac{-2{\varsigma }_j{\tilde{\upsilon }}_j}{{\sigma }^2}})}$$

(18)

where ${\tilde{\rho }}_{j,0}$ is the decision deviation at the initial moment. Since the decision deviation E decreases exponentially with the increase of the threshold ${\varsigma }_j$, and the decision time T increases with the increase of ${\varsigma }_j$, the Bayes’ risk criterion function has a minimum value. By minimizing this function and solving for $\frac{\partial B}{\partial \varsigma }=0$, the decision threshold ${\varsigma }_j$ is obtained.

$$\frac{{\sigma }^2(e^{\frac{2\varsigma {\tilde{\upsilon }}_j}{{\sigma }^2}}-e^{\frac{-2\varsigma {\tilde{\upsilon }}_j}{{\sigma }^2}})}{2{\tilde{\upsilon }}_j^2}+\frac{2{\varsigma }_j}{{\tilde{\upsilon }}_j}=\frac{c_1}{c_2}$$

(19)

At this time, when the decision variable has not reached the decision threshold, the UGV maintains its original task output. When the decision variable reaches the decision threshold, it indicates that the UGV is currently unable to complete the task with its own intelligence, and the UAV needs to give an intervention task to intervene the UGV to get out of the predicament. The original task of the UGV is projected onto the null space of the intervention task of the UAV, ensuring that once the intervention task is given, it will be executed as the highest priority task, and the original task of the UGV will be partially executed. At this time, the output formula of the UGV task is as follows:

$${\upsilon }_{ugv}=\left\{\begin{array}{c}{\upsilon }_{ugv},{\tilde{\rho }}_j<{\varsigma }_j\hfill \\ {\upsilon }_{int}+(I-J_{int}^{†}{J}_{int}){\upsilon }_{ugv},{\tilde{\rho }}_j\ge {\varsigma }_j\hfill \end{array}\right.$$

(20)

4. Optimization Layer Design Based on MPC

4.1. Optimal Control Formulation

Taking into account that the UAV may be affected by visual occlusion, inaccurate sensor measurement, and disturbance of the fuselage itself during the execution of the task, an undesirable intervention task is given. As a result, UGV performs unnecessary intervention tasks, and even worse results. Therefore, it is necessary to introduce the intervention task decision maker based on MPC to determine whether the UGV accepts the intervention task. This boils down to the 01 integer programming problem [27].

This paper proposes an intervention task decision maker based on MPC. By establishing the optimization problem and adding related constraints to solve it, the overall task performance including UGV intervention task can be optimized.

Consider a group of n UGVs, Define the state of the UGV as:

$$\rho \left(t\right)={\rho }_i\left(t\right),i=1,2,\cdots n$$

(21)

where each element represents the state of each UGV. Define a set of binary vectors:

$$\left\{\begin{array}{c}{\omega }_i^j\left(t\right)=[{\omega }_{i,1}^j{\left(t\right)}^T,{\omega }_{i,1}^j{\left(t\right)}^T]\in \{0,1\}\hfill \\ \begin{array}{c}i=1,\cdots ,n\hfill \\ j=1,2\hfill \end{array}\hfill \end{array}\right.$$

(22)

where j Represents two possible choices of whether to accept the intervention task. The kinematics of each UGV can be written as a convex combination of kinematics in different modes:

$${\dot{\rho }}_i\left(t\right)={\omega }_{i,1}^j\left(t\right)[v_{int}\left(t\right)+(I-J_{int}^{†}{J}_{\mathrm{int}})v_{ugv}\left(t\right)]+{\omega }_{i,2}^j\left(t\right)v_{ugv}\left(t\right),i=1,\cdots ,n$$

(23)

where $v_{int}\left(t\right)$ is the intervention task issued by the UAV at time t, and $v_{ugv}\left(t\right)$ is the original composite task of the UGV at time t. One of the important constraints on ${\omega }_i\left(t\right)$ is:

$$\left\{\begin{array}{c}{\omega }_{i,1}^j+{\omega }_{i,2}^j=1\hfill \\ {\omega }_{i,1}^j,{\omega }_{i,2}^j\in \{0,1\}\hfill \end{array}\right.$$

(24)

Through this constraint, it is ensured that the UGV will either accept the intervention task of the UAV and be executed as the highest priority task, and the original composite task of the UGV will be projected to the null space of the intervention task of the UAV, or it does not accept the intervention task of the UAV and maintains the original composite task output. Define a cost function

$$L\left(x\right)={\int }_0^T\sum _{i=1}^n(a_i{∥{\rho }_{rdj}^i\left(t\right)-{\rho }_i\left(t\right)∥}^2+b_i{∥u_i∥}^2)dt,i=1,\cdots n,$$

(25)

which consists of combined task errors of the NSBC from all UGVs plus slack variables, each weighted by positive parameters $a_i$ and $b_i$. As a result, an OCP can be formulated as

$$\underset{{\rho }_i,w,u}{min}L\left({\rho }_i\right)$$

(26a)

$$\begin{array}{cc}\hfill & s.t.{\rho }_i\left(0\right)={{\stackrel{⌢}{\rho }}_i}_0\hfill \end{array}$$

(26b)

$$\left\{\begin{array}{c}{\omega }_{i,1}^j+{\omega }_{i,2}^j=1\hfill \\ {\omega }_{i,1}^j,{\omega }_{i,2}^j\in \{0,1\}\hfill \end{array}\right.$$

(26c)

$$\left\{\begin{array}{c}∥{\rho }_i\left(t\right)-{\rho }_o^i\left(t\right)∥\ge d_i-u_i\left(t\right)\hfill \\ i=1,\cdots ,n\hfill \end{array}\right.$$

(26d)

$$u_i\left(t\right)\ge 0,\forall t\in [0,T]$$

(26e)

where ${\widehat{\rho }}_{i0}$ is the initial state of the i-th UGV. ${\rho }_o^i\left(t\right)$ is the Obstacle position detected by the i-th UGV at time t. $d_i$ is the corresponding safety distance. $u_i\left(t\right)$ is the slack vectors and T is the prediction horizon. slack variable $u_i$ as a soft constraint for the optimization problem. The purpose of adding $u_i$ to constraints is to effectively obtain optimized solutions.

4.2. Real-Time Model Predictive Cotrol Algorithm

During the task execution process, problem (26) is difficult to solve because it contains integer variables. In this paper, we employ a real-time MPC algorithm with “first discretize then optimize” methodology and an outer-convexfication with integer relaxation technique [28]. First, we relax the binary variable ${\omega }_i$ to real number in the range of 0 to 1, that is $\widehat{\omega }\left(t\right)\in \left[0,1\right]$. Then, through the multiple shooting method [29], problem (26) can be converted to a nonlinear programming problem:

$$\underset{{\rho }_i,\stackrel{⌢}{\omega },u}{min}{\sum }_{k=0}^{N}L(x_{s|k},u_{s|k})$$

(27a)

$$\begin{array}{cc}\hfill & s.t.x_{s|0}={\stackrel{⌢}{x}}_0\hfill \end{array}$$

(27b)

$$\left\{\begin{array}{c}{\omega }_{i,1,s|k}^j+{\omega }_{i,2,s|k}^j=1\hfill \\ {\omega }_{i,1,s|k}^j,{\omega }_{i,2,s|k}^j\in \{0,1\}\hfill \end{array}\right.$$

(27c)

$$\Vert {\rho }_{i,s|k}\left(t\right)-{\rho }_0^i\left(t\right)\Vert \ge d_i-u_{i,s|k}$$

(27d)

$$u_{i,s|k}\ge 0,$$

(27e)

$$i=1,2,\dots n$$

(27f)

$$k=0,1,\dots N$$

(27g)

where N represents that the prediction time domain T is divided into N parts, that is each time interval is $\Delta t=\frac{T}{N}$. s|k is the kth-step predicted at sampling time s. The NLP can be solved efficiently by standard NLP solvers using sequential quadratic programming or interior point methods. Finally, a sum-up-rounding (SUR) step can be employed to obtain the binary variable ${\omega }_i\left(t\right)$ from $\widehat{\omega }\left(t\right)$ [28]. The SUR step reads as

$$h_{s|k}^j={\sum }_{r=0}^k{\stackrel{⌢}{\omega }}_{s|r}^j\Delta t-{\sum }_{r=0}^{k-1}{\omega }_{s|r}^j\Delta t$$

(28)

$${\omega }_{s|k}^j=\left\{\begin{array}{cc}1& if\begin{array}{cc}{h}_{s|k}^j\ge h_{s|k}^r& \forall r\ne j\&j<r\forall r:h_{s|k}^j=h_{s|k}^r\end{array}\\ 0& otherwise\end{array}\right.$$

(29)

5. Simulation

In this part, consider an UAV and two UGVs moving in the x-y-z three dimensional space, where each robot is modeled as a first-order system. The goal of the entire air-ground coordination system is to move to the target point and avoid obstacles in an unknown environment. The UAV is equipped with cameras to sense UGVs ground environment; UGVs are equipped with sensors to sense surrounding obstacles. The values of parameters used in this simulation are shown in

Table 1. Parameter values used in the simulation.

Parameter		Value
initial position		${\left[\begin{array}{ccc}1& 0.5& 0\end{array}\right]}^T{\left[\begin{array}{ccc}-1& 0.5& 0\end{array}\right]}^T{\left[\begin{array}{ccc}0.2& 1& 2\end{array}\right]}^T$
obstacles position		$\left[\begin{array}{ccc}1.7& 4& 0\end{array}\right]\left[\begin{array}{ccc}4& 7& 0\end{array}\right]$$\left[\begin{array}{ccc}-0.5& 5& 0\end{array}\right]$
target position		${\left[\begin{array}{ccc}3.3& 13& 0\end{array}\right]}^T$${\left[\begin{array}{ccc}-1& 13& 0\end{array}\right]}^T$${\left[\begin{array}{ccc}1.1& 13& 2\end{array}\right]}^T$
UGV1 preset trajectories		${\left[\begin{array}{ccc}3.3& 1t& 0\end{array}\right]}^T$
UGV2 preset trajectories		${\left[\begin{array}{ccc}1.3& 1t& 0\end{array}\right]}^T$
safe distance		2 m
task gain A,B,C		3,2.5,1.5
DDM parameter $c_1$,$c_2$,${\sigma }_j$,${\varsigma }_j$		0.5,1,10,3
MPC sampling frequence		20 Hz
MPC prediction horizon		3 s
MPC grid point number		60

.

First, this paper proposes three verification methods for the air-ground collaborative system framework. (a) The UGV encountered a local minima point, and the UAV did not take intervention measures. (b) The UGV encounters a local minima point, and the UAV takes effective intervention to prompt the UGV to get rid of the extreme point. (c) The UGV encounters a local minima point, the UAV gives a bad intervention, and the UGV chooses not to accept the intervention task.

Next, two case studies are given to better demonstrate the advantages of the proposed UAV-UGV coordination system’s event-triggered intervention framework.

5.1. Case A

In this case, the effective intervention of UAV is verified by comparing method (a) and method (b). In the process of UGV performing tasks, there may be situations in which they cannot rely on their own intelligence to solve problems. For example, when a UGV faces two obstacles and performs an obstacle avoidance task, the sum of the speed vectors of the two obstacle avoidance tasks will have the same magnitude as the speed vector moving to the target point, but the direction will be opposite, which makes it stuck in the local minima point. Trajectory diagram of the air-ground system with no intervention measures taken by UAV and effective intervention measures taken by UAV shown in Figure 3.

In method (a), the UGV2 can complete the task by virtue of its own intelligence. Since the UAV did not effectively intervene in the UGV1, the UGV1 fell into a minimum value at (3.3 m, 5.1 m). The UAV’s decision variable for UGV1 keeps increasing until it exceeds the decision threshold, but the intervention task is not triggered. Through the effective intervention of UAV in method (b) at time 14.45 s, after the decision variable of UGV1 exceeds the threshold, it will fall back below the threshold after 1.7 s. It shows that, after the effective intervention of the UAV, the UGV1 has been able to complete the task with its own intelligence. The decision variables of the UAV are shown in Figure 4.

Figure 5 shows the distances between the UGVs and their nearby obstacles. Once the UGV is moving into the range of the obstacle or another UGV, the obstacle avoidance task is activated with a higher priority to avoid collision. Method (a) shows that the UGV1 is stuck in the extreme point and cannot move, and the distance from the obstacle is constant at 2 m. Method (b) shows that the UGV1 can get rid of the extreme point by effective intervention.

Figure 6 shows that in method (b), only when the decision variable reaches the decision threshold, will the UAV launch an intervention task. After the MPC intervention task decision maker, the UGV chooses to accept the intervention task. Method (a): Since it does not involve event triggering, it is not shown here.

5.2. Case B

In this case, by comparing method (b) and method (c), it is verified whether the UGV accepts the undesirable intervention task given by the UAV, so as to ensure its own safety. In method (b), due to factors such as disturbance or visual occlusion, the UAV gives the wrong intervention task, which causes UGV to collide with the sudden obstacle and cause danger. In method (c), since the MPC intervention task decider optimizes the intervention task and the UGV’s original composite task, when the wrong intervention task is given, the UGV chooses not to accept the intervention task. Until an effective intervention task is given, the UGV chooses to perform the intervention task. Please note that, at this time, due to the formation task of the UAV, the UAV’s trajectory is also shifted. The trajectory of method (c) and method (b) are shown in Figure 7.

In method (b), the wrong intervention task causes the UGV to crash into sudden obstacles (5.5 m, 5.8 m), which causes the UAV to continuously increase the decision variables of the UGV1 due to the failure of the UGV1. In method (c), although the UAV’s decision variables will continue to increase due to the maintenance of its original task until UAV effectively intervenes task to make the UGV get rid of the extreme point, at least the safety of the UGV is protected. As shown in Figure 8.

It can be seen from Figure 9 that the wrong intervention task of method (b) causes the distance between the UGV and the sudden obstacle to be zero. Method (c) effectively avoids this situation, even if a little time is sacrificed.

Figure 10 shows thar Method (b) suggests that UGV continue to receive the wrong intervention tasks after the event is triggered. Method (c) shows that after MPC optimization, even if the UAV sends out an intervention task, the UGV chooses not to accept it until an effective intervention task is given.

6. Conclusions and Future Work

This paper has proposed a new type of event-triggered mechanism-based air-ground coordination system for UAVs intervening in UGVs. The DDM has been embedded into NSBC to reduce the communication burden, and the intervention task decision maker based on MPC is designed to ensure that UGVs only accept safe and effective intervention tasks. At the end of this paper, two cases are studied to compare the performance of the system in situations without intervention, with direct intervention (correct or wrong), and with optimal intervention selected from the MPC. The results show that the UAV intervention task is triggered only when the decision variable reaches the decision threshold. The UGV can identify and reject wrong intervention tasks, ensuring its own safety.

In this paper, it is assumed that UAVs can give proper intervention tasks. However, no detailed description of how to design the intervention tasks is given. In future research, we will discuss how drones can select appropriate intervention tasks from the behavior database. At the same time, although DDM is used as a perceptual decision-making model for simulating humans, there remains a question that the intervention frequency may be infinite, i.e., Zeno behavior. We will consider this as the next step in research.