Defense against Adversarial Swarms with Parameter Uncertainty

Abstract

This paper addresses the problem of optimal defense of a high-value unit (HVU) against a large-scale swarm attack. We discuss multiple models for intra-swarm cooperation strategies and provide a framework for combining these cooperative models with HVU tracking and adversarial interaction forces. We show that the problem of defending against a swarm attack can be cast in the framework of uncertain parameter optimal control. We discuss numerical solution methods, then derive a consistency result for the dual problem of this framework, providing a tool for verifying computational results. We also show that the dual conditions can be computed numerically, providing further computational utility. Finally, we apply these numerical results to derive optimal defender strategies against a 100-agent swarm attack.

1. Introduction

Swarms are characterized by large numbers of agents which act individually, yet produce collective, herd-like behaviors. Implementing cooperating swarm strategies for a large-scale swarm is a technical challenge which can be considered to be from the “insider’s perspective”. It assumes inside control over the swarm’s operating algorithms. However, as large-scale ‘swarm’ systems of autonomous systems become achievable—such as those proposed by autonomous driving, UAV package delivery, and military applications—interactions with swarms outside our direct control become another challenge. This generates its own “outsider’s perspective” issues.

In this paper, we look at the specific challenge of protecting an asset against an adversarial swarm. Autonomous defensive agents are tasked with protected a high-value unit (HVU) from an incoming swarm attack. The defenders do not fully know the cooperating strategy employed by the adversarial swarm. Nevertheless, the task of the defenders is to maximize the probability of survival of the HVU against an attack by such a swarm. This challenge raises many issues—for instance, how to search for the swarm [1], how to observe and infer swarm operating algorithms [2], and how to best defend against the swarm given algorithm unknowns, and only limited, indirect control through external means. In this paper, we restrict ourselves to the last issue. However, these problems share multiple technical challenges. The preliminary approach we apply in this paper demonstrates some basic methods which we hope will stimulate the development of more sophisticated tools.

For objectives achieved via external control of the swarm, several features of swarm behavior must be characterized: capturing the dynamic nature of the swarm, tracking the collective risk profile created by a swarm, and engaging with a swarm via dynamic inputs, such as autonomous defenders. The many modeling layers create a challenge for generating an effective response to the swarm, as model uncertainty and model error are almost certain. In this paper, we look at several dynamic systems where the network structure is determined by parameters. These parameters set neighborhood relations and interaction rules. Additional parameters establish defender input and swarm risk.

We consider the generation of optimal defense strategies given uncertainty in parameter values. We demonstrate that small deviances in parameter values can have catastrophic effects on defense trajectories optimized without taking error into account. We then demonstrate the contrasting robustness of applying an uncertain parameter optimal control framework instead of optimizing with nominal values. The robustness against these parameter values suggests that refined parameter knowledge may not be necessary given appropriate computational tools. These computational tools—and the modeling of the high-dimensional swarm itself—are expensive. To assist with this issue, we provide dual conditions for this problem in the form of a Pontryagin minimum principle and prove the consistency of these conditions for the numerical algorithm. These dual conditions can, thus, be computed from the numerical solution of the computational method and provide a tool for solution verification and parameter sensitivity analysis.

Although in this paper, optimal strategies against swarms motivate the framework of uncertain parameter optimal control, and the subsequent development of the dual conditions, both the framework and the dual conditions have many applications beyond swarm defense. Optimal control with parameter uncertainty is relevant to robotics—where parts, such as wheels, may have small size and calibration uncertainties; aerospace—where both components and exogeneous factors, such as wind, may be modeled using parameter uncertainty; and search and rescue—where the location of a target object can be considered a parameter uncertainty [3,4]. It is also an instance of mean-field optimal control (which includes this framework, but also more general probability distributions), which is finding application in the training of neural networks [5]. The dual conditions provided in this paper provide both a tool for verification of numerical solutions, as well as another potential route for generating numerical solutions.

The structure of this paper is as follows. Section 2 provides examples of dynamic swarming models and extensions for defensive interactions. Section 3 discusses optimization challenges and describes a general uncertain parameter optimal control framework that this problem could be addressed with. Section 4 provides a proof of the consistency of the dual problem for this control framework, which expands on the results initially presented in the conference paper [6]. Section 5 gives an example of numerical implementation that demonstrates optimal defense against a large-scale swarm of 100 agents. Section 6 discusses the results and future work.

2. Modeling Adverserial Swarms

2.1. Cooperative Swarm Models

The literature on the design of swarm strategies which produce coherent, stable collective behavior has become vast. A quick review of the literature points to two main trends/categories in swarm behavior design. The first relies on dynamic modeling of the agents and potential functions to control their behavior (see [7,8] and references therein). The second trend relates to the use of rules to describe agents’ motion and local rule-based algorithms to control them [9,10].

We present two examples of dynamic swarming strategies from the literature. These examples are illustrative of the forces considered in many swarming models:

• collision avoidance between swarm members;
• alignment forces between neighboring swarm members;
• stabilizing forces.

These intra-swarm goals are aggregated to provide a swarm control law, which we will refer to as $F_S$, to each swarm agent. Both example models in this paper share the same double integrator form with respect to this control law. For n swarm agents, the dynamics are defined by

$$\begin{array}{cc}\hfill & {\ddot{x}}_i=u_i.i=1,\dots ,n,\hfill \end{array}$$

(1)

$$u_i=F_S(x_i,{\dot{x}}_i,\forall j\ne i:x_j,{\dot{x}}_j|\theta ).$$

(2)

2.1.1. Example Model 1: Virtual Body Artificial Potential

In this model [11,12], swarm agents track to a virtual body (or bodies) guiding their course, while also reacting to intra-swarm forces of collision avoidance and group cohesion. The input $u_i$ is the sum of intra-swarm forces, virtual body tracking, and a velocity dampening term. In addition, in this adversarial scenario, swarm agents are influenced to avoid intruding defense agents. The intra-swarm force between two swarm agents has magnitude $f_I$ and is a gradient of an artificial potential $V_I$. Let

$$x_{ij}=x_i-x_j.$$

(3)

The artificial potential $V_I$ depends on the distance $\left|\right|x_{ij}\left|\right|$ between swarm agents i and j. The artificial potential $V_I$ is defined as:

$$V_I=\left\{\begin{array}{c}\alpha \left(ln\left(\left|\right|x_{ij}\left|\right|\right)+\frac{d_0}{\left|\right|x_{ij}\left|\right|}\right),0<\left|\right|x_{ij}\left|\right|<d_1\hfill \\ \alpha \left(ln\left(d_1\right)+\frac{d_0}{d_1}\right),\left|\right|x_{ij}\left|\right|\ge d_1\hfill \end{array}\right.$$

(4)

where $\alpha$ is a scalar control gain, and $d_0$ and $d_1$ are scalar constants for distance ranges. Then the magnitude of interaction force is given by

$$f_I=\left\{\begin{array}{c}{\nabla }_{\left|\right|x_{ij}\left|\right|}{V}_I,0<\left|\right|x_{ij}\left|\right|<d_1\hfill \\ 0,\left|\right|x_{ij}\left|\right|\ge d_1\hfill \end{array}\right.$$

(5)

The swarm body is guided by ‘virtual leaders’, non-corporeal reference trajectories which lead the swarm. We assign a potential $V_h$ on a given swarm agent i associated with the k-th virtual leader, defined with the distance $\left|\right|h_{ik}\left|\right|$ between the swarm agent i and leader k. Mirroring the parameters $\alpha$, $d_0$, and $d_1$ defining $V_I$, we assign $V_h$ the parameters ${\alpha }_h$, $h_0$, and $h_1$. An additional dissipative force $f_{v_i}$ is included for stability. The control law $u_i$ for the vehicle i associated with m defenders is given by

$$\begin{array}{cc}\hfill u_i& =-\sum _{j\ne i}^n{\nabla }_{x_i}{V}_I\left(x_{ij}\right)-\sum _{k=1}^m{\nabla }_{x_i}{V}_h\left(h_{ik}\right)+f_{v_i}\hfill \\ \hfill & =-\sum _{j\ne i}^n\frac{f_I\left(x_{ij}\right)}{\left|\right|x_{ij}\left|\right|}{x}_{ij}-\sum _{k=1}^m\frac{f_h\left(h_{ik}\right)}{\left|\right|h_{ik}\left|\right|}{h}_{ik}+f_{v_i}.\hfill \end{array}$$

(6)

2.1.2. Example Model 2: Reynolds Boid Model

In this model [8,13], for radius r, $j=1,\cdots ,N$, define the neighbors of agent i at position $x_i\in {\mathbb{R}}^n$ by the set

$${\mathcal{N}}_i=\left\{j\right|j\ne i\wedge \parallel x_i-x_j\parallel <r\}$$

(7)

Swarm control is designated by three forces.

Alignment of velocity vectors:

$$f_{al}=-w_{al}\left({\dot{x}}_i-\frac{1}{|{\mathcal{N}}_i|}\sum _{j\in {\mathcal{N}}_i}{\dot{x}}_j\right)$$

(8)

Cohesion of swarm:

$$f_{coh}=-w_{coh}\left(x_i-\frac{1}{|{\mathcal{N}}_i|}\sum _{j\in {\mathcal{N}}_i}{x}_j\right)$$

(9)

Separation between agents:

$$f_{sep}=-w_{sep}\frac{1}{|{\mathcal{N}}_i|}\left(\sum _{j\in {\mathcal{N}}_i}\frac{x_j-x_i}{\parallel x_i-x_j\parallel }\right)$$

(10)

for positive constant parameters $w_{al}$, $w_{coh}$, $w_{sep}$.

$$u_i=f_{al}+f_{coh}+f_{sep}$$

(11)

2.2. Adversarial Swarm Models

The previous subsection provides several examples of inner swarm cooperative forces, $F_S$. In order to enable adversarial behavior and defense, these inner swarm cooperative forces need to be supplemented by additional forces of exogenous input into the collective. As written, the above cooperative swarming models neither respond to outside agents nor ‘attack’ (swarm towards) a specific target. We, thus, supplement the control laws above with two additional forces. The first, we refer to as $F_{HVU}$; the goal of the swarm, in this paper, is limited to tracking an HVU. An example of $F_{HVU}$ is provided in the example of Section 5, in Equation (28).

We also supplement by an adversarial force, which we refer to as $F_D$. The review [7] discusses several approaches to adversarial control. Examples include containment strategies modeled after dolphins [14], sheep-dogs [15,16], and birds of prey [17]. In [18], the authors studied the interaction between two swarms, one of which could be considered adversarial. In these examples of adversarial swarm control, the mechanism of interaction and defense is provided through the swarm’s own pursuit and evasion responses. This indirectly uses the swarm’s own response strategy against it—an approach which can be termed ‘herding’.

In addition to herding reactions, one can consider more direct additional forces of disruption, to model neutralizing swarm agents and/or physically remove them from the swarm. One form this can take, for example, is the removal of agents from the communications network, as considered in [19]. Another approach is taken in [20], which uses survival probabilities based on damage attrition. Defenders and the attacking swarm engage in mutual damage attrition while the swarm also damages the HVU when in proximity to it. Probable damage between agents is tracked as damage rates over time, where the rate of damage is based on features such as distance between agents and angle of attack. The damage rate at time t provides the probability of a successful ‘hit’ in time period $[t,t+\Delta t]$. The probability of agent survival can be modeled based on the aggregate number of hits it takes to incapacitate the agent. The authors of [20] provide derivations for multiple possibilities, such as single-shot destruction and N-shot destruction. These probabilities take the form of ODE equations. Tracking survival probabilities thus adds an additional state to the dynamics of each agent—a survival probability state.

We, thus, summarize a control scheme with HVU target-tracking and herding driven by the reactive forces of collision avoidance with the defenders as the following, for HVU states $y_0$ and defender states $y_k$, $k=1,\cdots ,K$:

$$\begin{array}{ccc}\hfill u_i=& F_S(x_i,{\dot{x}}_i,\forall j\ne i:x_j,{\dot{x}}_j|\theta )\hfill & \hfill \leftarrow \mathit{intra-swarm}\\ \hfill +& F_{HVU}(x_i,{\dot{x}}_i,y_0,{\dot{y}}_0|\theta )\hfill & \hfill \leftarrow \mathit{target\; tracking}\\ \hfill +& F_D(x_i,{\dot{x}}_i,\forall k:y_k,{\dot{y}}_k|\theta )\hfill & \hfill \leftarrow \mathit{herding\; and/or\; damage}\end{array}$$

(12)

Example Attrition Model: Single-Shot Destruction

From [20]: let $P_0\left(t\right)$ be the probability the HVU has survived up to time t, $P_k\left(t\right)$, $k=1,\cdots ,K$, the probability defender k has survived, and $Q_j\left(t\right)$, $j=1,\cdots ,N$ the probability swarm attacker j has survived. Let $d_y^{j,k}(x_j\left(t\right),y_k\left(t\right))$ be the damage the defender $y_k$ inflicts on swarm attacker $x_j$ and let $d_x^{k,j}(y_k\left(t\right),x_j)$ be the damage the swarm attacker $x_j$ inflicts on the defender $y_k$, with the HVU represented by $k=0$.

Then the survival probabilities for attackers and defenders from single-shot destruction are given by the coupled ODEs:

$$\left\{\begin{array}{cc}{\dot{Q}}_j\left(t\right)=-Q_j\left(t\right)\sum _{k=1}^K{P}_k\left(t\right)d_y^{j,k}(x_j\left(t\right),y_k\left(t\right)),\hfill & Q_j\left(0\right)=1\hfill \\ {\dot{P}}_k\left(t\right)=-P_k\left(t\right)\sum _{j=1}^N{Q}_j\left(t\right)d_x^{k,j}(y_k\left(t\right),x_j\left(t\right)),\hfill & P_k\left(0\right)=1\hfill \end{array}\right.$$

for $j=1,\cdots ,N$, $k=0\cdots ,K$.

3. Problem Formulation

The above models depend on a large number of parameters. The dynamic swarming model coupled with attrition functions results in over a dozen key parameters, and many more would result from a non-homogeneous swarm. A concern would be that this adds too much model specificity, making optimal defense strategies lack robustness due to sensitivity to the specific set of model parameters. This concern turns out to be justified. When defense strategies are optimized for fixed, nominal parameter values, they display catastrophic failure for small perturbations of certain parameters, as can be seen in Figure 1. In fact, the plots included in Figure 1 clearly demonstrate that the sensitivity of the cost with respect to the uncertain parameters is highly non-linear. Thus, generating robust defense strategies requires a more sophisticated formalism introduced in the next Section 3.1.

3.1. Uncertain Parameter Optimal Control

The class of problems addressed by the computational algorithm is defined as follows:

Problem P. 

Determine the function pair $(x,u)$ with $x\in W_{1,\infty }([0,T]\times \mathrm{\Theta };{\mathbb{R}}^{n_x})$, $u\in L_{\infty }([0,T];{\mathbb{R}}^{n_u})$ that minimizes the cost

$$\begin{array}{c}\hfill J[x,u]={\int }_{\mathrm{\Theta }}\left[F\left(x(T,\theta ),\theta \right)+{\int }_0^{T}r(x(t,\theta ),u\left(t\right),t,\theta )dt\right]d\theta \end{array}$$

(13)

subject to the dynamics

$$\begin{array}{cc}\hfill \frac{\partial x}{\partial t}(t,\theta )& =f\left(x\right(t,\theta ),u(t),\theta ),\hfill \end{array}$$

(14)

initial condition $x(0,\theta )=x_0\left(\theta \right)$, and the control constraint $g\left(u\right(t\left)\right)\le 0$ for all $t\in [0,T]$. The set $L_{\infty }([0,T];{\mathbb{R}}^{n_u})$ is the set of all essentially bounded functions, $W_{1,\infty }([0,T]\times \mathrm{\Theta };{\mathbb{R}}^{n_x})$ the Sobolev space of all essentially bounded functions with essentially bounded distributional derivatives, and $F:{\mathbb{R}}^{n_x}\times {\mathbb{R}}^{n_{\theta }}\mapsto \mathbb{R}$, $r:{\mathbb{R}}^{n_x}\times {\mathbb{R}}^{n_u}\times \mathbb{R}\times {\mathbb{R}}^{n_{\theta }}\mapsto \mathbb{R}$, $g:{\mathbb{R}}^{n_u}\mapsto {\mathbb{R}}^{n_g}$. Additional conditions imposed on the state and control space and component functions are specified in Appendix A.

In Problem $\mathbf{P}$, the set $\mathrm{\Theta }$ is the domain of a parameter $\theta \in {\mathbb{R}}^{n_{\theta }}$. The format of the cost functional is that of the integral over $\mathrm{\Theta }$ of a Mayer–Bolza type cost with parameter $\theta$. This parameter can represent a range of values for a feature of the system, such as in ensemble control [21], or a stochastic parameter with a known probability density function.

For computation of numerical solutions, we introduce an approximation of Problem $\mathbf{P}$, referred to as Problem ${\mathbf{P}}^{\mathbf{M}}$. Problem ${\mathbf{P}}^{\mathbf{M}}$ is created by approximating the parameter space, $\mathrm{\Theta }$, with a numerical integration scheme. This numerical integration scheme is defined in terms of a finite set of M nodes ${\left\{{\theta }_i^M\right\}}_{i=1}^M$ and an associated set of M weights ${\left\{{\alpha }_i^M\right\}}_{i=1}^M\subset \mathbb{R}$ such that

$${\int }_{\mathrm{\Theta }}h\left(\theta \right)d\theta =\underset{M\to \infty }{lim}\sum _{i=1}^{M}h\left({\theta }_i^M\right){\alpha }_i^M.$$

(15)

given certain function smoothness assumptions. See Appendix A Assumption A1 for formal assumptions. Throughout the paper, M is used to denote the number of nodes used in this approximation of parameter space.

For a given set of nodes ${\left\{{\theta }_i^M\right\}}_{i=1}^M$, and control $u\left(t\right)$, let ${\overline{x}}_i^M\left(t\right)$, $i=1,\cdots ,M$, be defined as the solution to the ODE created by the state dynamics of Problem $\mathbf{P}$ evaluated at ${\theta }_i^M$:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\frac{d{\overline{x}}_i^M}{dt}\left(t\right)=f({\overline{x}}_i^M\left(t\right),u\left(t\right),{\theta }_i^M)\hfill \\ {\overline{x}}_i^M\left(0\right)=x_0\left({\theta }_i^M\right),\hfill \end{array}\right.i=1,\cdots ,M.\end{array}$$

(16)

Let ${\overline{X}}^M\left(t\right)=[{\overline{x}}_1^M\left(t\right),\dots ,{\overline{x}}_M^M\left(t\right)]$. The system of ODEs defining ${\overline{X}}^M$ has dimension $n_x\times M$, where $n_x$ is the dimension of the original state space and M is the number of nodes. The numerical integration scheme for parameter space creates an approximate objective functional, defined by:

$${\overline{J}}^M[{\overline{X}}^M,u]=\sum _{i=1}^M\left[F\left({\overline{x}}_i^M\left(T\right),{\theta }_i^M\right)+{\int }_0^{T}r({\overline{x}}_i^M\left(t\right),u\left(t\right),t,{\theta }_i^M)dt\right]{\alpha }_i^M.$$

(17)

In [4], the consistency of ${\mathbf{P}}^{\mathbf{M}}$ is proved. This is the property that, if optimal solutions to Problem ${\mathbf{P}}^{\mathbf{M}}$ converge as the number of nodes $M\to \infty$, they converge to feasible, optimal solutions of Problem $\mathbf{P}$. See [4] for detailed proof and assumptions.

3.2. Computational Efficiency

The computation time of the numerical solution to the discretized problem defined in Equations (16) and (17) will depend on the value of M. Ideally, it should be sufficiently small so as to allow for a fast solution. On the other hand, a value of M that is too small will result in a solution that is not particularly useful, i.e., too far from the optimal. Naturally, the question arises: how far is a particular solution from the optimal? One tool for assessing this lies in computing the Hamiltonian and is addressed in Section 4.

4. Consistency of Dual Variables

The dual variables provide a method to determine the solution of an optimal control problem or a tool to validate a numerically computed solution. For numerical schemes based on direct discretization of the control problem, analyzing the properties of the dual variables and their resultant Hamiltonian may also lead to insight into the validity of an approximation scheme [22,23]. This could be especially helpful in high-dimensional problems, such as swarming, where parsimonious discretization is crucial to computational tractability.

Previous work shows the consistency of the primal variables in approximate Problem ${\mathbf{P}}^{\mathbf{M}}$ to the original parameter uncertainty framework of Problem $\mathbf{P}$. Here, we build on that and prove the consistency of the dual problem of Problem $\mathbf{P}$ as well. This theoretical contribution is diagrammed in Figure 2. The consistency of the dual problem in parameter space enables approximate computation of the Hamiltonian from numerical solutions.

In [24], necessary conditions for Problem $\mathbf{P}$ were established. These conditions are as follows:

Problem${\mathbf{P}}^{\lambda }$ 

[([24], pp. 80–82)]. If $(x^{*},u^{*})$ is an optimal solution to Problem $\mathbf{P}$, then there exists an absolutely continuous costate vector ${\lambda }^{*}(t,\theta )$, such that for $\theta \in \mathrm{\Theta }$:

$$\frac{\partial {\lambda }^{*}}{\partial t}(t,\theta )=-\frac{\partial H(x^{*},{\lambda }^{*},u^{*},t,\theta )}{\partial x},$$

$${\lambda }^{*}(T,\theta )=\frac{\partial F(x^{*}(T,\theta ),\theta )}{\partial x}$$

(18)

where H is defined as:

$$H(x,\lambda ,u,t,\theta )=$$

$$\lambda f\left(x\right(t,\theta ),u(t),\theta )+r\left(x\right(t,\theta ),u(t),t,\theta ).$$

(19)

Furthermore, the optimal control $u^{*}$ satisfies

$$u^{*}\left(t\right)=\underset{u\in U}{argmin}\mathbf{H}(x^{*},{\lambda }^{*},u,t),$$

where $\mathbf{H}$ is given by

$$\mathbf{H}(x,\lambda ,u,t)={\int }_{\mathrm{\Theta }}H(x,\lambda ,u,t,\theta )d\theta .$$

(20)

Because Problem ${\mathbf{P}}^{\mathbf{M}}$ is a standard non-linear optimal control problem, it admits a dual problem as well. Problem ${\mathbf{P}}^{\mathbf{M}\lambda }$, provided by the Pontryagin minimum principle (a survey of minimum principle conditions is given by [25]). Applied to ${\mathbf{P}}^{\mathbf{M}}$ this generates:

Problem ${\mathbf{P}}^{\mathbf{M}\lambda }$. 

For feasible solution $({\overline{X}}^M,u)$ to Problem ${\mathbf{P}}^{\mathbf{M}}$, find $\overline{\Lambda }\left(t\right)=[{\overline{\lambda }}_1^M\left(t\right),\cdots {\overline{\lambda }}_M^M\left(t\right)]$, ${\overline{\lambda }}_i^M:[0,T]\to {\mathbb{R}}^{N_x}$, that satisfies the following conditions:

$$\frac{d{\overline{\lambda }}_i^M}{dt}\left(t\right)=-\frac{\partial {\overline{H}}^M({\overline{x}}_i^M,{\overline{\lambda }}_i^M,u,t)}{\partial x_i^M},$$

$${\overline{\lambda }}_i^M\left(T\right)={\alpha }_i^M\frac{\partial F({\overline{x}}_i^M,{\theta }_i^M)}{\partial {\overline{x}}_i^M},$$

(21)

where ${\overline{H}}^M$ is defined as:

$${\overline{H}}^M({\overline{X}}^M,{\overline{\Lambda }}_M,u,t)=\sum _{i=1}^M\left[{\overline{\lambda }}_i^{M}f({\overline{x}}_i^M\left(t\right),u\left(t\right),{\theta }_i^M)+{\alpha }_i^{M}r({\overline{x}}_i^M\left(t\right),u\left(t\right),t,{\theta }_i^M)\right].$$

(22)

An alternate direction from which to approach solving Problem $\mathbf{P}$ overall is to approximate the necessary conditions of Problem $\mathbf{P}$, i.e., Problem ${\mathbf{P}}^{\lambda }$, directly rather than to approximate Problem $\mathbf{P}$. This creates the system of equations:

$$\frac{d\lambda }{dt}(t,{\theta }_i^M)=-\frac{\partial H(x,u,t,{\theta }_i^M)}{\partial x}$$

$$\lambda (T,{\theta }_i^M)=\frac{\partial F(x(T,{\theta }_i^M),{\theta }_i^M)}{\partial x}$$

(23)

for $i=1,\cdots ,M$, where H is defined as:

$$H(x,\lambda ,u,t,\theta )=\lambda f\left(x\right(t,\theta ),u(t),\theta )+r\left(x\right(t,\theta ),u(t),t,\theta ).$$

This system of equations can be re-written in terms of the quadrature approximation of the stationary Hamiltonian defined in Equation (20). Define

$${\tilde{H}}^M(x,\lambda ,u,t):=\sum _{i=1}^M{\alpha }_i^{M}H(x(t,{\theta }_i^M),\lambda (t,{\theta }_i^M),u\left(t\right),t,{\theta }_i^M).$$

Let

$$\tilde{\Lambda }\left(t\right)=[{\tilde{\lambda }}_1^M\left(t\right),\cdots {\tilde{\lambda }}_M^M\left(t\right)]=[\lambda (t,{\theta }_1^M),\cdots ,\lambda (t,{\theta }_M^M)]$$

and let

$${\tilde{X}}_M=[{\tilde{x}}_1^M\left(t\right),\cdots ,{\tilde{x}}_M^M\left(t\right)]$$

denote the semi-discretized states from Equation (16). Equation (23) can then be written as:

$$\frac{d{\tilde{\lambda }}_i^M}{dt}\left(t\right)=-\frac{1}{{\alpha }_i^M}·\frac{\partial {\tilde{H}}^M({\tilde{X}}_M,\tilde{\Lambda },u,t)}{\partial {\tilde{x}}_i^M}$$

$${\tilde{\lambda }}_i^M\left(T\right)=\frac{\partial F({\tilde{x}}_i^M\left(T\right),{\theta }_i^M)}{\partial {\tilde{x}}_i^M}$$

(24)

for $i=1,\cdots ,M$. Thus, we reach the following discretized dual problem:

Problem ${\mathbf{P}}^{\lambda \mathbf{M}}$. 

For feasible controls u and solutions ${\tilde{X}}_M$ to Equation (16), find $\tilde{\Lambda }\left(t\right)=[{\tilde{\lambda }}_1^M\left(t\right),\cdots {\tilde{\lambda }}_M^M\left(t\right)]$, ${\tilde{\lambda }}_i^M:[0,T]\to {\mathbb{R}}^{n_x}$, that satisfies the following conditions:

$$\frac{d{\tilde{\lambda }}_i^M}{dt}\left(t\right)=-\frac{1}{{\alpha }_i^M}·\frac{\partial {\tilde{H}}^M({\tilde{X}}_M,\tilde{\Lambda },u,t)}{\partial {\tilde{x}}_i^M},$$

$${\tilde{\lambda }}_i^M\left(T\right)=\frac{\partial F({\tilde{x}}_i^M,{\theta }_i^M)}{\partial {\tilde{x}}_i^M},$$

(25)

where ${\tilde{H}}^M$ is defined as:

$${\tilde{H}}^M({\tilde{X}}_M,{\tilde{\Lambda }}_M,u,t)=\sum _{i=1}^M\left[{\alpha }_i^M{\tilde{\lambda }}_i^{M}f({\tilde{x}}_i^M\left(t\right),u\left(t\right),{\theta }_i^M)+{\alpha }_i^{M}r({\tilde{x}}_i^M\left(t\right),u\left(t\right),t,{\theta }_i^M)\right].$$

(26)

Lemma 1.

The mapping:

$$({\overline{x}}_i^M,\overline{u})\mapsto ({\tilde{x}}_i^M,\tilde{u}),\frac{{\overline{\lambda }}_i^M}{{\alpha }_i^M}\mapsto {\tilde{\lambda }}_i^M,$$

for $i=1,\cdots ,M$ is a bijective mapping from solutions of Problem ${\mathbf{P}}^{\mathbf{M}\lambda }$ to Problem ${\mathbf{P}}^{\lambda \mathbf{M}}$.

Proof. 

In the case of this particular problem, unlike standard control, the collocation of the relevant dynamics involves no approximation of differentiation (since the discretization is in the parameter domain rather than the time domain), and, thus, the mapping of covectors between Problem ${\mathbf{P}}^{\mathbf{M}\lambda }$ and Problem ${\mathbf{P}}^{{\tilde{H}}^M({\tilde{X}}_M,{\tilde{\Lambda }}_M,u,t)=\lambda M}$ is straightforward and simply constructively provided by the lemma itself. The two mappings of the lemma, $({\overline{x}}_i^M,\overline{u})\mapsto ({\tilde{x}}_i^M,\tilde{u})$ (identity mapping) and $\frac{{\overline{\lambda }}_i^M}{{\alpha }_i^M}\mapsto {\tilde{\lambda }}_i^M$ (scaling by $\frac{1}{{\alpha }_i^M}$) are both bijections. □

Theorem 1.

Let ${\{{\tilde{X}}_M,{\tilde{\Lambda }}_M,u_M\}}_{M\in V}$ be a sequence of solutions for Problem ${\mathbf{P}}^{\lambda \mathbf{M}}$ with an accumulation point $\{{\tilde{X}}^{\infty },{\tilde{\Lambda }}^{\infty },u^{\infty }\}$. Let $(x^{\infty },{\lambda }^{\infty },u^{\infty })$ be the solutions to Problem ${\mathbf{P}}^{\lambda }$ for the control $u^{\infty }$. Then

$$\underset{M\in V}{lim}{\tilde{H}}^M({\tilde{X}}_M,{\tilde{\Lambda }}_M,u_M,t)=\mathbf{H}(x^{\infty },{\lambda }^{\infty },u^{\infty },t)$$

where ${\tilde{H}}^M$ is the Hamiltonian of Problem ${\mathbf{P}}^{\lambda \mathbf{M}}$ as defined by Equation (26) and $\mathbf{H}$ is the Hamiltonian of Problem $\mathbf{P}$ as defined by Equation (20). The proof of this theorem can be found in the Appendix B.

The convergence of the Hamiltonians of the approximate, standard control problems to the Hamiltonian of the general problem, $\mathbf{H}(x^{\infty },{\lambda }^{\infty },u^{\infty },t)$, means that many of the useful features of the Hamiltonians of standard optimal control problems are preserved. For instance, it is straightforward to show that the satisfaction of Pontryagin’s minimum principle by the approximate Hamiltonians implies minimization of $\mathbf{H}(x^{\infty },{\lambda }^{\infty },u^{\infty },t)$ as well. That is, that

$$\mathbf{H}(x^{\infty },{\lambda }^{\infty },u^{\infty },t)\le \mathbf{H}(x^{\infty },{\lambda }^{\infty },u,t)$$

for all feasible u. Furthermore, when applicable, the stationarity properties of the standard control Hamiltonian, such as a constant-valued Hamiltonian in time-invariant problems, or stationarity with respect to $u\left(t\right)$ in problems with open control regions, are also preserved.

5. Numerical Example

In a slight refashioning of the notation in the Section 2.2, Equation (12), let the parameter vector $\theta$ be defined by all the unknown parameters defining the interaction functions. Assuming prior distribution $\varphi \left(\theta \right)$ over these unknowns and parameter bounds $\mathrm{\Theta }$, we construct the following optimal control problem for robustness against the unknown parameters.

Problem SD (Swarm Defense). 

For K defenders and N attackers, determine the defender controls $u_k\left(t\right)$ that minimize:

$$J={\int }_{\theta }\left[1-P_0(t_f,\theta )\right]\varphi \left(\theta \right)d\theta$$

(27)

subject to:

$$\left\{\begin{array}{cc}{\dot{y}}_k\left(t\right)=f(y_k\left(t\right),u_k\left(t\right)),\hfill & y_k\left(0\right)=y_{k0}\hfill \\ {\ddot{x}}_j(t,\theta )=\hfill & \\ F_S(t,\theta )+F_{HVU}(t,\theta )+F_D(t,\theta ),\hfill & x_j(0,\theta )=x_{j0}\left(\theta \right)\hfill \\ {\dot{Q}}_j(t,\theta )=\hfill & \\ -Q_j(t,\theta )\sum _{k=1}^K{P}_k(t,\theta )d_y^{j,k}(x_j(t,\theta ),y_k\left(t\right)),\hfill & Q_j(0,\theta )=1\hfill \\ {\dot{P}}_k\left(t\right)=\hfill & \\ -P_k(t,\theta )\sum _{j=1}^N{Q}_j(t,\theta )d_x^{k,j}(y_k\left(t\right),x_j(t,\theta )),\hfill & P_k(0,\theta )=1\hfill \end{array}\right.$$

for swarm attackers $j=1,\cdots ,N$ and controlled defenders $k=1\cdots ,K$.

We implement Problem SD for both swarm models in Section 2.1, for a swarm of $N=100$ attackers and $K=10$ defenders.

5.1. Example Model 1: Virtual Body Artificial Potential

The cooperative swarm forces $F_S$ are defined with the Virtual Body Artificial Potential of Section 2.1 with parameters $\alpha$, $d_0$ and $d_1$. In lieu of a potential for the virtual leaders, we assign the HVU tracking function:

$$f_{HVU}=-\frac{K_1(x_i-y_0)}{\parallel x_i-y_0\parallel }$$

(28)

where $y_0\in {\mathbb{R}}^3$ is the position of the HVU. The dissipative force $f_{v_i}=-K_2{\dot{x}}_i$ is employed to guarantee stability of the swarm system. $K_1$ and $K_2$ are positive constants. The swarm’s collision avoidance response to the defenders is defined by Equation (4) with parameters ${\alpha }_h$, $h_0$ and $h_1$. Since there is only a repulsive force between swarm members and defenders, not an attractive force, we set $h_1=h_0$. For attrition, we use the the damage function defined in Equation (21) of [20]:

$$F_D=\lambda \Phi \left(\frac{F-a{r}^2}{\sigma }\right),r={\parallel x_i-y_j\parallel }_2$$

(29)

where $\Phi$ is the cumulative normal distribution and ${\parallel ·\parallel }_2$ is the vector 2-norm. This function smoothly penalizes proximity, with the impact decreasing with distance. The parameters $\lambda$, F, a, and $\sigma$ shape the steepness of this function and the decline of damage over distance. For the damage rate of defenders inflicted on attackers, we calibrate by the parameters ${\lambda }_D$, ${\sigma }_D$. For the damage rate of attackers inflicted on defenders, we calibrate by the parameters ${\lambda }_A$, ${\sigma }_A$. In both cases, the parameters F and a in [20] are set to $F=0$, $a=1$.

Table 1. Model 1 Fixed Parameter Values.

Parameter	Value	Meaning
$t_f$	45	final time
$K_1$	5	tracking coefficient
K	10	number of defenders
$h_1$	$h_0$	interaction parameter
${\lambda }_D$	2	defender weapon intensity
${\sigma }_D$	2	defender weapon range
N	100	number of attackers
$K_2$	5	dissipative force

provides the parameter values that remain fixed in each simulation, and and

Table 2. Model 1 Varied Parameter Values.

Parameter	Nominal	Range	Meaning
$\alpha$	$0.5$	[0.1, 0.9]	control gain
$d_0$	1	[0.5, 1.5]	lower range limit
$d_1$	6	[4, 8]	upper range limit
${\lambda }_A$	$0.05$	[0.01, 0.09]	weapon intensity
${\sigma }_A$	2	[1.5. 2.5]	weapon range
${\alpha }_h$	6	[5, 7]	herding intensity
$h_0$	3	[2, 4]	herding range

provides the parameters we consider as uncertain.

We first use the nominal parameter values provided in Table 1 and Table 2 to find a nominal solution defender trajectories that result in the minimum probability of HVU destruction. With the results of these simulations as a reference point, we consider as uncertain each of the parameters that define attacker swarm model and weapon capabilities. In this simulation, these parameters are considered individually. The number of discretization nodes for parameter space was chosen by examination of the Hamiltonian. To illustrate this method and the results obtained in Section 4 we compute Hamiltonians for the Problem SD and Model 1 with $\theta =d_0,d_0\in [0.5,1.5]$ and $M=[5,8,11]$. As M increases the sequence of Hamiltonians should converge to the optimal Hamiltonian for the Problem SD. For Problem SD that should result in a constant, zero-valued Hamiltonian. Figure 3 shows the respective Hamiltonians for $M=[5,8,11]$. The value $M=11$ was chosen for simulations, based on the approximately zero-valued Hamiltonian it generates.

We compare the performance of the solution generated using uncertain parameter optimal control Problem SD versus a solution obtained with the nominal values. Figure 4 shows the nominal solution trajectories. The comparitive results of the nominal solutions vs the uncertain parameter control solutions are shown in Figure 5, where the performance of each is shown for different parameters values.

As seen in Figure 5 the trajectories generated by optimization using the nominal values perform poorly over a range of $\alpha$, $d_0$, ${\sigma }_A$, ${\alpha }_k$ and $h_0$. In the case of $h_0$, for example, this is because the attackers are less repelled by the defenders when $h_0$ is decreased, and they are more able to destroy the HVU from a longer distance as ${\sigma }_A$ is increased. The parameter uncertainty solution, however, demonstrates that using the uncertain parameter optimal control framework a solution can be provided which is robust over a range of parameter values. We contrast these results with the case of uncertain parameters $d_1$ and ${\lambda }_A$, also shown in Figure 5. It can be seen that robustness improvements are modest to non-existent for these parameters. This suggests an insensitivity of the problem $d_1$ and ${\lambda }_A$ parameters. This kind of analysis can be used to guide inference and observability priorities.

5.2. Example Model 2: Reynolds Boid Model

To demonstrate flexibility of the proposed framework to include diverse swarm models we have applied the same analysis as was done in Section 5.1 to the Reynolds Boid Model introduced in Section 2.1. We apply the same HVU tracking function as Equation (28). The herding force $F_D$ of the defenders repelling attackers is applied as a separation force in the form of Equation (10). The fixed parameter values are the same as those in Table 1; the uncertain parameters and ranges are given in

Table 3. Model 2 Varied Parameter Values.

Parameter	Nominal	Range	Meaning
${\lambda }_A$	$0.05$	[0.01, 0.09]	weapon intensity
${\sigma }_A$	2	[1.5, 2.5]	weapon range
$r_{al}$	2	[1.5, 2.5]	alignment range
$w_{al}$	$0.75$	[0.25, 1.25]	alignment intensity
$r_{coh}$	2	[1.5, 2.5]	cohesion range
$w_{coh}$	$0.75$	[0.25, 1.25]	cohesion intensity
$r_{sep}$	1	[0.5, 1.5]	separation range
$w_{sep}$	$0.5$	[0.1, 0.9]	separation intensity
$r_I$	2	[1.5, 2.5]	herding range
$w_I$	$4.5$	[3.5, 5.5]	herding intensity

. The results are shown in Figure 6. Again, we see that the tools developed in this paper can be used to gain an insight into the robustness properties of the nominal versus uncertain parameter solutions. For example, we can see that the uncertain parameter solutions perform much better than the nominal ones for the cases where $\lambda$, $\sigma$ and $w_I$ are uncertain.

6. Conclusions

In this paper, we have built on our previous work on developing an efficient numerical framework for solving uncertain parameter optimal control problems. Unlike uncertainties introduced into systems due to stochastic “noise”, parameter uncertainties do not average or cancel out in regard to their effects. Instead, each possible parameter value creates a specific profile of possibility and risk. The uncertain optimal control framework which has been developed for these problems exploits this inherent structure by producing answers which have been optimized over all parameter profiles. This approach takes into account the possible performance ranges due to uncertainty, while also utilizing what information is known about the uncertain features, such as parameter domains and prior probability distributions over the parameters. Thus, we are able to contain risk, while providing plans which have been optimized for performance under all known conditions. The results reported in this paper include analysis of the consistency of the adjoint variables of the numerical solution. In addition, the paper includes a numerical analysis of a large scale adversarial swarm engagement that clearly demonstrates the benefits of using the proposed framework.

There are many directions for future work for the topics of this paper. The numerical simulations in this paper consider the parameters individually, as one-dimensional parameter spaces. However, Problem P allows for multi-dimensional parameter spaces. A more dedicated implementation, taking advantage of the parallelizable form of Equation (16), for example, could certainly manage several simultaneous parameters. Exponential growth as parameter space dimension increases is an issue for both the quadrature format of Equation (15) and handling of the state space size for Equation (16). This can be somewhat mitigated by using sparse grid methods for high-dimensional integration to define the nodes in Equation (15). For large enough sizes, Monte Carlo sampling, rather than quadrature might be more appropriate for designating parameter nodes.

A further direction for future work would be to incorporate these methods into the design of more responsive closed-loop control solutions. The optimization methods in this paper provide open-loop controls. While useful, closed-loop controls would be more ideal for dynamic situations with uncertainty. There are many ways, however, that open-loop solutions can provide stepping stones to developing closed-loop solutions. For instance, Ref. [26] utilizes closed-loop solutions to train a neural network to learn an optimal closed-loop control strategy. Open-loop solutions can also be used to provide initial guesses to discretized closed-loop optimizations, seeding the optimization algorithm.

Another direction for future work is in the greater application of the duality results of Section 4. The numerical results in this paper simply utilize the Hamiltonian consistency. The proof of Theorem 1, however, additionally demonstrates the consistency of the adjoint variables for the problem. As the results demonstrate, parameter sensitivity for these swarm models is highly non-linear. The numerical solutions of Section 5 are able to demonstrate this sensitivity by applying the solution to varied parameter values. However, this is actually a fairly expensive method for a large swarm, as it involves re-evaluation of the swarm ODE for each parameter value. More importantly, it would not be scalable to high-dimensional parameter spaces, as the exponential growth of that approach to sensitivity analysis would be unavoidable. The development of an analytical adjoint sensitivity method for this problem could be of great utility for paring down numerical simulations to only focus on the parameters most relevant to success.