A Stochastic Characterization of the Capture Zone in Pursuit-Evasion Games

Abstract

Pursuit-evasion games are used to define guidance strategies for multi-agent planning problems. Although optimal strategies exist for deterministic scenarios, in the case when information about the opponent players is imperfect, it is important to evaluate the effect of uncertainties on the estimated variables. This paper proposes a method to characterize the game space of a pursuit-evasion game under a stochastic perspective. The Mahalanobis distance is used as a metric to determine the levels of confidence in the estimation of the Zero Effort Miss across the capture zone. This information can be used to gain an insight into the guidance strategy. A simulation is carried out to provide numerical results.

1. Introduction

Pursuit-evasion differential games have been applied to autonomous vehicles guidance problems in various contexts ranging from missile guidance [1], to spacecraft orbital maneuvers [2], and mobile robots [3]. Since their introduction in [4], other versions have been formulated, including stochastic [5] and multi-agent [6] games.

A common solution for simplifying the architectures and reducing the weights and costs of autonomous vehicles is to adopt a single instrument (bearing or range sensor) for target tracking or navigation [7,8]. This work focuses on a missile application, but the results can be easily extended to other scenarios and to different sets of measurements. In missile systems, passive sensors like electro-optical or optical seekers are often used to provide the target direction with respect to the vehicle. This bearings-only measurement system comes at the cost of not having information on the range to the target when an optimal guidance law is employed. To enhance the observability of the range and, therefore, to improve the performance of the engagement, it is necessary to deviate from the optimal guidance strategy. Different strategies for optimizing these maneuvers have been extensively studied in the literature. Reference [9] looks at the maneuver that maximizes the determinant of the Fisher information matrix, while [10] maximizes the eigenvalues of the normalized error covariance matrix; reference [11] tries to maintain the line of sight rate larger than a certain threshold, while [12] imposes different intercept angles between consecutive pursuers, and [13] uses a performance measure of observability based on geometric conditions.

Two stochastic metrics that have found application in the study of maneuvers effects on target estimation with different sets of sensors are the Cramér–Rao lower bound (CRLB) [14,15] and the Fisher information [16,17]. The CRLB returns an indication of the performance of a maximum likelihood estimator in terms of error covariance. The Fisher information is related to the CRLB by an inverse relationship, as will be shown later in the paper. An issue related to the use of these metrics in missile applications is that, for an unobservable system, their numerical computation can be prone to errors because the Fisher information matrix would result in being nonsingular [18].

Rather than a new sub-optimal guidance law to optimize maneuvers, this paper proposes a method for characterizing the game space of a pursuit-evasion game using another stochastic metric, the Mahalanobis distance, which can be calculated independently from the CRLB and the Fisher information and thus will suffer less from numerical issues. The proposed method allows for obtaining a map of the confidence in the estimation of the main variable of a pursuit-evasion guidance law, called the Zero Effort Miss (ZEM). This knowledge can be exploited as a cost to numerically minimize in guidance algorithms or as an information to feed reinforcement learning algorithms [19]. An advantage of this solution is that it is not computationally heavy, as it only involves the calculation of the Mahalanobis distance from the covariance matrix of the Kalman filter.

The rest of this paper is organized as follows: Section 2 reviews the pursuit-evasion games framework; Section 3 defines a method to characterize the capture zone of the game from a stochastic point of view; a numerical example in presented in Section 4; conclusions are given in Section 5.

2. Statement of the Problem

2.1. Engagement Description

Consider the pursuer P and the evader E represented in Figure 1 in a Cartesian reference frame of coordinates $XY$. P and E are assumed to be mass points, with constant speeds $v_P$ and $v_E$, and lateral accelerations $a_P$ and $a_E$, respectively. The distance between P and E is the relative range r. The line of sight (LOS) forms an angle $\lambda$ with the horizontal reference. The flight path angles of the pursuer and the evader are indicated as ${\gamma }_P$ and ${\gamma }_E$, respectively. The vertical coordinates of the two players are $y_P$ and $y_E$

The dynamics of the engagement assumes that the physical systems implementing the guidance commands u and v can be represented as first order systems with time constants ${\tau }_P$ and ${\tau }_E$, respectively. The set of nonlinear equations that describes the dynamics of the engagement is resumed in Equation (1):

$$\left\{\begin{array}{c}\dot{r}=v_{E}cos({\gamma }_E-\lambda )-v_{P}cos({\gamma }_P-\lambda )\triangleq V_r\hfill \\ \dot{\lambda }={\displaystyle \frac{v_{E}sin({\gamma }_E-\lambda )-v_{P}sin({\gamma }_P-\lambda )}{r}}\triangleq {\displaystyle \frac{V_{\lambda }}{r}}\hfill \\ {\dot{a}}_P={\displaystyle \frac{u-a_P}{{\tau }_P}}\hfill \\ {\dot{\gamma }}_P={\displaystyle \frac{a_P}{v_P}}\hfill \\ {\dot{a}}_E={\displaystyle \frac{v-a_E}{{\tau }_E}}\hfill \\ {\dot{\gamma }}_E={\displaystyle \frac{a_E}{v_E}}\hfill \end{array}\right.$$

(1)

Under the assumption of small LOS angle $\lambda$, the miss y can be approximated as

$$y=y_E-y_P=v_{E}sin{\gamma }_E-v_{P}sin{\gamma }_P$$

(2)

Assuming also that ${\gamma }_P$ and ${\gamma }_E$ are small, the nonlinear model of Equation (1) can be linearized obtaining a new system [20]:

$$\dot{X}=AX+Bu+Cv$$

(3)

whose state vector X is defined as:

$$X={\left[\begin{array}{cccccc}y& \dot{y}& a_P& {\gamma }_P& a_E& {\gamma }_E\end{array}\right]}^T$$

(4)

and the matrices are:

$$A=\left[\begin{array}{cccccc}0& 1& 0& 0& 0& 0\\ 0& 0& -1& 0& 1& 0\\ 0& 0& -1/{\tau }_P& 0& 0& 0\\ 0& 0& 1/v_P& 0& 0& 0\\ 0& 0& 0& 0& -1/{\tau }_E& 0\\ 0& 0& 0& 0& 1/v_E& 0\end{array}\right]$$

(5)

$$B={\left[\begin{array}{cccccc}0& 0& a_P^{max}/{\tau }_P& 0& 0& 0\end{array}\right]}^T$$

(6)

$$C={\left[\begin{array}{cccccc}0& 0& 0& 0& a_E^{max}/{\tau }_E& 0\end{array}\right]}^T$$

(7)

The control inputs u and v are normalized with the maximum lateral acceleration values $a_P^{max}$ and $a_E^{max}$, respectively, resulting in a system with bounded controls ($u/a_P^{max}\le 1,v/a_E^{max}\le 1$).

2.2. Pursuit-Evasion Games

A differential game can be set up to obtain optimal guidance strategies $u^{*}$ and $v^{*}$ for the linear system with bounded controls of Equation (3). This kind of differential game is called a pursuit-evasion game because the optimal strategies aim at minimizing (the pursuer) or maximizing (the evader) the relative distance at the final time $t_f$, called miss distance. One of the most important features of the pursuit-evasion games formulation is the definition of a structure for the game space with capture and avoidance regions where finite miss is guaranteed. Depending on the characteristics of the two players (time constants and maximum accelerations), a number of structures can be defined [20] with semipermeable bounds $Z_{\pm }^{*}$ that can be calculated integrating backwards the derivative ${\displaystyle \frac{dZ}{dt}}$ from its final condition $Z\left(t_f\right)$:

$$\begin{array}{c}{\displaystyle \frac{dZ}{dt}}=\mathsf{\Gamma }sign\left(Z\right)\\ \mathsf{\Gamma }=-a_P^{max}cos({\gamma }_{P0}-{\lambda }_0)\mathsf{\Xi }(t_{go}/{\tau }_P){\tau }_P+a_E^{max}cos({\gamma }_{E0}-{\lambda }_0)\mathsf{\Xi }(t_{go}/{\tau }_E){\tau }_E\\ \mathsf{\Xi }\left(x\right)=e^{-x}+x-1\end{array}$$

(8)

where ${\gamma }_{P0}-{\lambda }_0$ and ${\gamma }_{E0}-{\lambda }_0$ are the values of the relative angles around which the linearization has been performed.

Figure 2 shows a game structure with the bounds $Z_{\pm }^{*}$ plotted against the time-to-go $t_{go}$ to the interception for the case where $a_P^{max}>a_E^{max}$ and $a_P^{max}{\tau }_E>a_E^{max}{\tau }_P$. The capture zone $D0$, in this case, is a region where optimal strategies are arbitrary and zero miss is guaranteed. The avoidance zone $D1$ is a region where zero miss can not be achieved and the optimal commands are

$$\left\{\begin{array}{c}{u}^{*}=a_P^{max}sign\left(ZEM\right)\hfill \\ v^{*}=a_E^{max}sign\left(ZEM\right)\hfill \end{array}\right.$$

(9)

The ZEM in the previous equation is the final distance between the two players at the end of the engagement assuming no further maneuvers from the players. The $ZEM$ expression for the engagement of Equation (1) is given by:

$$ZEM=-\dot{r}\dot{\lambda }{t}_{go}+a_{E}cos({\gamma }_E-\lambda )\mathsf{\Xi }(t_{go}/{\tau }_E){\tau }_E^2-a_{P}cos({\gamma }_P-\lambda )\mathsf{\Xi }(t_{go}/{\tau }_P){\tau }_P^2$$

(10)

with $t_{go}$ being approximated as:

$$t_{go}=-{\displaystyle \frac{r}{\dot{r}}}$$

(11)

If the pursuer starts the engagement in the capture zone and, if it adopts the guidance strategy of Equation (9), the level of $ZEM$ will remain close to 0. A different guidance command will produce a larger $ZEM$, approaching the bounds $Z_{\pm }^{*}$.

2.3. Estimator in the Loop

The implementation of the guidance strategy of Equation (9) requires the knowledge of an estimate of the $ZEM$, which is made up of information on both the pursuer and the evader states. Pursuer’s related variables such as $v_P$, ${\gamma }_P$, and $a_P$ can be provided by the on-board navigation system, but the other variables need to be reconstructed through an estimator, in most cases a nonlinear Kalman filter. The vector $\widehat{X}$ of variables that need to be estimated is therefore:

$$\widehat{X}={\left[\begin{array}{ccccc}r& \lambda & a_E& {\gamma }_E& v_E\end{array}\right]}^T$$

(12)

In the case of a seeker measuring the LOS angle $\lambda$ (bearings-only measurements, BOM):

$$z_{BOM}=\lambda +{\nu }_{\lambda }=\left[\begin{array}{ccccc}0& 1& 0& 0& 0\end{array}\right]\widehat{X}+{\nu }_{\lambda }=H_{BOM}\widehat{X}+{\nu }_{\lambda }$$

(13)

If a relative range measurement is available too, Equation (13) is updated as:

$$z_{RB}=\left[\begin{array}{c}r+{\nu }_r\\ \lambda +{\nu }_{\lambda }\end{array}\right]=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\end{array}\right]\widehat{X}+\left[\begin{array}{c}{\nu }_r\\ {\nu }_{\lambda }\end{array}\right]=H_{RB}\widehat{X}+\left[\begin{array}{c}{\nu }_r\\ {\nu }_{\lambda }\end{array}\right]$$

(14)

The noise signals ${\nu }_r$ and ${\nu }_{\lambda }$, are defined as zero-mean Gaussian sequences with variance ${\sigma }_{{\nu }_r}$ and ${\sigma }_{{\nu }_{\lambda }}$, respectively. The model provided to the Kalman Filter includes a shaping filter to represent target maneuvers, which are assumed as maximum acceleration maneuvers whose starting time is uniformly distributed over the flight time. The maneuvers model provided to the filter is a white noise $\omega$ with spectral density $\psi$ through an integrator [21]. The complete model is given by:

$$\left\{\begin{array}{c}\dot{r}=V_r\hfill \\ \dot{\lambda }={\displaystyle \frac{V_{\lambda }}{r}}\hfill \\ {\dot{\gamma }}_T=a_T/v_T\hfill \\ {\dot{a}}_T=\omega \hfill \\ {\dot{v}}_T=0\hfill \end{array}\right.$$

(15)

The Jacobian matrix J associated with the previous model can be found in [10] and it is used along with the sample time ${\Delta }_T$ to calculate the state transition matrix $\mathsf{\Phi }$:

$$\mathsf{\Phi }=e^{J{\Delta }_T}$$

(16)

It is well known that, in the case of bearings-only measurements, $t_{go}$ becomes unobservable if the pursuer is maintained on the collision triangle, i.e., if an optimal guidance law such as that of Equation (9) is applied. To gain an insight on range observability, one should maneuver away from the collision triangle, at the cost of increasing the $ZEM$. This does not preclude capture until the D0 region is not abandoned, but since the $ZEM$ is only available as an estimation, there is the risk of getting too close to the borders of D0 or even to pass in the D1 zone.

3. Characterization of the Game Structure

It is very important for the pursuer to have a good estimate of the $ZEM$ so as to apply the best guidance command possible. In addition, if the adopted guidance strategy does not intend to maintain the $ZEM$ around 0, it is crucial to know how good its estimate is. Although $Z^{*}$ can be calculated exactly using a regressive $t_{go}$ value, its distance from the estimated $ZEM$ is random, since $\widehat{ZEM}$ is itself a random variable. It seems interesting, therefore, to characterize the D0 region in terms of a stochastic metric. To this purpose, the concept of Mahalanobis distance will be introduced in this section, along with the CRLB of the estimator considered in this study.

3.1. Mahalanobis Distance

The Mahalanobis distance $d_M\left(\theta \right)$ between a random variable $\widehat{\theta }\in {\mathbb{R}}^n$ and a point $\theta \in {\mathbb{R}}^n$ is defined as:

$$d_M\left(\theta \right)=\sqrt{(\widehat{\theta }-\theta ))P_{\widehat{\theta }\widehat{\theta }}^{-1}{(\widehat{\theta }-\theta )}^T}$$

(17)

where $P_{\widehat{\theta }\widehat{\theta }}$ is the covariance matrix associated with the random variable $\widehat{\theta }$. $d_M\left(\theta \right)$ represents a region in the neighborhood of $\widehat{\theta }$ where $\theta$ should be. In other words, $d_M\left(\theta \right)$ is a measurement of the confidence in the estimation of the real parameter $\theta$: a null $d_M\left(\theta \right)$ means that $\theta$ coincides with the mean of $\widehat{\theta }$; a larger $d_M\left(\theta \right)$ means that the estimation of $\theta$ is less correct. The interest of this work lies in determining the confidence on the estimation of the $ZEM$ at each value of $t_{go}$. To this end, a Mahalanobis distance for $ZEM$ is defined as:

$$d_M\left(ZEM\left(t_{go}\right)\right)=\sqrt{(\widehat{ZEM}\left(t_{go}\right)-ZEM\left(t_{go}\right))P_{ZEM}^{-1}{(\widehat{ZEM}\left(t_{go}\right)-ZEM\left(t_{go}\right))}^T}$$

(18)

The covariance $P_{ZEM}$ can be obtained at each time instant using the value of the error covariance of the Kalman filter. This can be easily done on the go, i.e., in real time with the estimator, returning a value of $d_M$ that depends on the features of the filtering algorithm (e.g., approximation of the nonlinear dynamics, tuning parameters, etc.). As a mean of comparison, another covariance can be used in the calculation, which is that obtained through the CRLB associated with the estimator. This can be interpreted as an ideal performance test, as it would return the minimum value for $P_{ZEM}$, independent from the filtering algorithm.

3.2. Cramér–Rao Bound

The CRLB is defined as the minimum estimation covariance bound of an unbiased estimator. In practice, it tells how good an estimator can theoretically be, given a noisy measurement. An estimator is called efficient if its variance is equal to the CRLB, meaning that its mean squared estimation error is the lowest possible among all unbiased estimators. Such an estimator is sometimes called not practical, as it would yield the best theorical performance. According to the Cramér–Rao theorem, the minimum variance of an unbiased estimator $\widehat{\theta }$ of the parameter $\theta$ is always larger than the inverse of the associated Fisher information matrix F [22]:

$$va{r}_{\theta }=E\left[(\widehat{\theta }-\theta ){(\widehat{\theta }-\theta )}^T\right]\ge F^{-1}$$

(19)

$$F=E\left[\left({\nabla }_{X}lnf\left(z\right|\theta )\right){\left({\nabla }_{X}lnf\left(z\right|\theta )\right)}^T\right]$$

(20)

where f is the likelihood function of the n measurements sequence z given $\theta$.

$$f\left(z\right|\theta )=\frac{1}{{\left(2\pi \right)}^{n/2}{\sigma }_{\nu }^2}exp\left\{-\frac{{(z-\theta )}^2}{2{\sigma }_{\nu }^2}\right\}$$

(21)

The CRLB of a function $g\left(\theta \right)$ of the parameter $\theta$ is given by:

$$va{r}_{g\left(\theta \right)}\ge \frac{\partial g}{\partial \theta }{F}^{-1}{\frac{\partial g}{\partial \theta }}^T$$

(22)

In this work, the function g is the $ZEM$ of Equation (10), while $\theta$ is the vector $\widehat{X}$ of Equation (12). When the estimation is carried out using a Kalman filter, as in this work, the Fisher information matrix of Equation (20) at the k-th step can be written in a recursive form [23]:

$$F_k={\left({\mathsf{\Phi }}^{-1}\right)}^T{F}_{k-1}{\mathsf{\Phi }}^{-1}+H^T{\sigma }_{\nu }^{-1}H$$

(23)

The initial condition on F is defined considering a filter with infinite initial error covariance matrix, therefore:

$$F_0=0$$

(24)

4. Numerical Example

Two numerical simulations, each one consisting of 200 Monte Carlo runs, were carried out to calculate the values of $d_M\left(ZEM\right)$ across the D0 region. The first simulation uses both bearings and range measurements as in Equation (14), while the second employs the bearings-only measurements of Equation (13). Different paths are travelled in each run of the simulation in order to cover the entire D0 region, as shown in Figure 3, where each colour represents a different run. The trajectories of both players for a single run are shown in Figure 4. The Mahalanobis distance is calculated in correspondence of each dot of Figure 3 in two ways: first using the values of the CRLB from Equation (22) as the covariance $P_{ZEM}$ in Equation (18), and then using the error covariance matrix calculated by the filter. The numerical initial values ($R_0$, ${\lambda }_0$, ${\gamma }_{E_0}$, ${\gamma }_{P_0}$, $a_{P_0}$, $a_{E_0}$) and parameters used in the simulations are reported in

Table 1. Simulation initial values and parameters.

Parameter	Value	Parameter	Value
$v_P$	3 km/s	$v_E$	1.2 km/s
$a_P^{max}$	30 g	$a_E^{max}$	10 g
$R_0$	10 km	${\lambda }_0$	$5^{\circ }$
${\gamma }_{E_0}$	$\sim \mathcal{U}$ (${30}^{\circ }$, ${10}^{\circ }$)	g	9.81 m/s${}^2$
$a_{P_0}$	0 m/s${}^2$	$a_{E_0}$	0 m/s${}^2$
${\tau }_P$	0.1 s	${\tau }_E$	0.2 s
$\psi$	${a_E^{max}}^2/t_f$	${\Delta }_T$	0.02 s
${\sigma }_{{\nu }_{\lambda }}$	0.001 rad	${\sigma }_{{\nu }_r}$	50 m

.

Figure 5 and Figure 6 show the result of the simulation for the case of bearings and range measurements. The levels of the Mahalanobis distance are associated with different colors in the maps, as indicated in the sidebar. The red regions are those where the estimation of $ZEM$ is worse, and the blue regions where it is best. The maps are generated by merging the results of all the runs at each time instant. The value of the Mahalanobis distance in Figure 5 is calculated using the CRLB, while that in Figure 6 is obtained from the filter. At the beginning of the engagement ($t_{go}>4$ s), the value of $d_M\left(ZEM\right)$ in Figure 5 is very low and increases as the engagement moves on. This is a consequence of the initialization of the Fisher information in this method (Equation (24)). Since the CRLB is the inverse of the Fisher information, the initial $P_{ZEM}$ will be very large and, therefore, $d_M\left(ZEM\right)$ will be very small at the beginning. Since in the case of Figure 6 the initial covariance is finite, the value of $d_M\left(ZEM\right)$ at the beginning is larger than in Figure 5, which is more realistic, as the uncertainty over the estimation of a variable is finite in practice. Another difference is that the levels of Mahalanobis distance obtained with the CRLB are lower than those calculated through the filter. However, this was expected as the CRLB is an ideal bound for the estimator and the performance of a practical filter is always worse. A feature in common for the two cases is that $d_M\left(ZEM\right)$ is smaller in the proximity of the upper bound $Z_{+}^{*}$. This can be explained with the observability improvement obtained when maneuvering away from the collision triangle, even though the range measurements here already provide a certain level of observability. The high levels of $d_M\left(ZEM\right)$ towards the end of the engagement suggest that then it is risky to maneuver away from the collision triangle because a last-minute maneuver from the evader might suddenly increase the $ZEM$ and cause the passage to the avoidance zone.

The case of bearings-only measurements is presented in Figure 7 and Figure 8. The results obtained with the CRLB are prone to numerical issues due to the fact that the system in this case is unobservable, and therefore a good portion of the data at the beginning of the engagement is missing in Figure 7. The matrix computed through Equation (23) is not invertible and therefore the CRLB cannot be initially calculated. The values of $d_M\left(ZEM\right)$ obtained from the filter (Figure 8) in the first instants of the engagement ($t_{go}$ > 4 s) are similar to those of Figure 6: this is no surprise as the filter is initialized in the same way in both cases. As the engagement moves on, the tendency of having smaller values of $d_M\left(ZEM\right)$ (and hence a better estimation of the $ZEM$) in the proximity of the bound is even more evident than in the previous case: the 0 level of the $ZEM$ is characterized by a red strip, while yellow and even blue layers can be found next to the bound. Since there are no range measurements here, the only way to enhance the observability is to maneuver away from the collision triangle, evidently. As in the case of bearings and range measurements, the values of $d_M\left(ZEM\right)$ are larger when calculated through the filter (Figure 8) than when computed through the CRLB (Figure 7).

5. Conclusions

Pursuit-evasion games offer a compact solution to the problem of target interception or rendez-vous for autonomous vehicles, but need to rely on a good estimation of the variables needed in the guidance formulation. This is not always possible, as the number of on-board sensors is often limited by weights and cost constraints.

This paper has presented a method for characterizing the capture region of a pursuit-evasion game in terms of the confidence on the estimation of the $ZEM$. The method consists of calculating the Mahalanobis distance associated with the $ZEM$ estimated by the on-board nonlinear filter. A comparison with the Mahalanobis distance obtained through the CRLB showed that the proposed method provides consistent results, which are less prone to numerical issues than the other.

These results can be used to design modern guidance laws that consider observability issues in their formulation, in addition to the classic considerations on miss distance and control effort minimization. This is especially valuable in scenarios where not all the necessary sensors are available, but there is a demand for high performance in terms of accuracy.