Necessary Optimality Conditions for a Class of Control Problems with State Constraint

Abstract

An elementary approach to a class of optimal control problems with pathwise state constraint is proposed. Based on spike variations of control, it yields simple proofs and constructive necessary conditions, including some new characterizations of optimal control. Two examples are discussed.

1. Introduction

Necessary optimality conditions for control problems with pathwise state constraints have been widely studied since the beginnings of optimal control theory [1], and this domain of research is still vivid nowadays [2]. Most of the existing approaches may be divided into two streams [3]. The first one, characterized by the use of classical methods of analysis with often heuristic proofs, yields results of limited generality (see the review [4], and [3]). The other is based on the abstract theory of infinite-dimensional optimization and its results encompass a wide class of problems, with rigorous but demanding proofs (see [5,6,7,8,9]). However, these results are difficult for practical verification, because of too general characterizations of the adjoint variables and multiplier functions. Generally, the existing approaches are hardly constructive, meaning they do not give sufficient indications how to improve a nonoptimal control.

We propose an elementary approach to necessary optimality conditions for problems in Mayer form with free final state and a scalar state constraint. The controls are scalar functions and nontangentiality is assumed at all entry and exit points. As is well known, the proof of the minimum principle with free final state and without pathwise state constraints can be made elementary and simple by considering the cost increment caused by a single spike variation of control. Our first purpose is to show that a similar proof technique may be effective when a pathwise state constraint is present, with the difference that additionally a coordinated pair of spikes is used. A second purpose is to extend the known results for state constraints of index one, mainly to nonregular problems in which the optimal control and the corresponding state trajectory may at the same time take values on the boundaries of their respective admissible sets. In particular, we allow for discrete sets of admissible control values. From a conceptual point of view, this work also offers a clear geometrical interpretation of the results. On the practical side, an advantage of our approach is that the obtained conditions are readily verifiable and constructive: if they are not fulfilled, a gradient optimization procedure can be indicated and initialized which guarantees an improvement of the control, up to numerical precision (as in the method of Monotone Structural Evolution [10]). Of course, the other approaches clearly prevail in a wider perspective, when problems of greater complexity are also taken into account. They then produce optimality conditions, which can be effectively used in optimal control computations (see [11,12,13,14]).

Consider a control system described by a state equation

$$\dot{x}(t)=f(x(t),u(t)), t\in [0,T], x(0)=x_0, x(t)\in R^n,$$

(1)

with a given initial condition $x_0$ and a given time horizon T. The controls $u:[0,T]\to R$ are piecewise continuous functions of time, taking values in a given set U, that is, they belong to $PC(0,T;U)$. 1 The function $f:R^n\times R\to R^n$ is of class $C^1$ in its both arguments. We make a general assumption that all solutions of (1) appearing in the sequel are well defined in the whole time interval $[0,T]$. The state is subject to a scalar pathwise constraint,

$$g(x(t))\le 0, t\in [0,T].$$

(2)

The function $g:R^n\to R$ is of class $C^2$, and $\partial g(\xi )\ne 0$ if $g(\xi )=0$. We assume $g(x_0)\le 0$. A performance index (or cost),

$$Q(u)=q(x(T)),$$

is minimized on the trajectories of (1). The function $q:R^n\to R$ is of class $C^1$.

For a control $u\in PC(0,T;U)$, let x be the corresponding solution of the initial value problem (1). The control u is admissible if the trajectory x satisfies the state constraint (2). The control u is optimal if it is admissible and minimizes the cost Q in the set of all admissible controls. A boundary interval of u is defined as any nonempty and right-open interval of time in which $g(x(t))\equiv 0$. Any nonempty and right-open interval of time, such that $g(x(t))<0$ for every t in that interval, is nonboundary. If $[t_1,t_2[$ is an inclusion-maximal boundary interval of u and $t_1>0$, then $t_1$ is called an entry point of u. If $t_2<T$, then $t_2$ is an exit point. Denote

$$\dot{g}(\xi ,v)=\partial g{(\xi )}^{T}f(\xi ,v) \mathrm{for} \xi \in R^n, v\in R.$$

(3)

The derivative of the function $t\mapsto g(x(t))$ along the trajectories of (1) is equal to $\dot{g}(x(t),u(t))$.

For admissible controls we introduce the concept of verifiability, aiming to distinguish the controls to which the spike technique of (non)optimality verification, developed below, can be effectively applied. Let u be an admissible control with the corresponding state trajectory x. We call this control verifiable if

(i)

it has a finite number of inclusion-maximal boundary intervals,

(ii)

an implication holds that if $g(x(t))=0$ for a certain t, then t belongs to the closure of some boundary interval of u, 2

(iii)

the conditions of nontangentiality

$$\dot{g}(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))>0,\hspace{1em}\dot{g}(x(t_{\mathrm{ex}}),u(t_{\mathrm{ex}}))<0$$

(4)

hold at all entry points $t_{\mathrm{en}}$ and all exit points $t_{\mathrm{ex}}$ of u,

(iv)

there is an open set $X_u\subset R^n$ containing all points $x(t)$ such that $g(x(t))=0$, and there is a $C^1$ function $w:X_u\to R$ such that

$$\left(w(\xi )\in U \mathrm{and} \dot{g}(\xi ,w(\xi ))=0\right)\hspace{1em}\mathrm{if}\hspace{1em}\left(\xi \in X_u \mathrm{and} g(\xi )\le 0\right).$$

Note that if (4) is true, then the functions u and $t\mapsto \dot{g}(x(t),u(t))$ are discontinuous at $t_{\mathrm{en}}$ and $t_{\mathrm{ex}}$. Claim (iv) may be regarded as a weakened form of the assumption that the state constraint (2) is of index (or order) one (cf. [3,4,8]). From the implicit function theorem it follows that if $\xi \in X_u$, $g(\xi )\le 0$ and ${\partial }_2\dot{g}(\xi ,w(\xi ))\ne 0$, then

$$\partial w(\xi )=-\frac{{\partial }_1\dot{g}(\xi ,w(\xi ))}{{\partial }_2\dot{g}(\xi ,w(\xi ))}=-\frac{{\partial }_{1}f(\xi ,w(\xi ))\partial g(\xi )+{\partial }^{2}g(\xi )f(\xi ,w(\xi ))}{\partial g{(\xi )}^T{\partial }_{2}f(\xi ,w(\xi ))}.$$

For any verifiable control u define a function $F:[0,T]\times R^n\to R^n$

$$F(t,\xi )=\{\begin{array}{c}f(\xi ,w(\xi )),\hspace{1em}t\in {\Theta }_u,\hspace{1em}\xi \in X_u\hfill \\ f(\xi ,u(t)),\hspace{1em}\mathrm{elsewhere},\hfill \end{array}$$

where ${\Theta }_u$ denotes the union of all boundary intervals of u. Obviously, the corresponding state trajectory satisfies $\dot{x}(t)=F(t,x(t))$ for almost all $t\in [0,T]$.

2. The One-Spike Control Variation and Trajectory Variation

Let u be a verifiable control, and x, the corresponding state trajectory. Denote

$$U_t=\{\begin{array}{c}\{v\in U:\dot{g}(x(t),v)<0\},\hspace{1em}g(x(t))=0\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}U,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}g(x(t))<0.\end{array}$$

For any $\tau \in [0,T[$, any $v\in U_{\tau }$, and any sufficiently small $\epsilon >0$, we shall define a control $u^{\epsilon }\in PC(0,T;U)$. We also define $x^{\epsilon }$ as the solution of the initial value problem

$${\dot{x}}^{\epsilon }(t)=f(x^{\epsilon }(t),u^{\epsilon }(t)), t\in [0,T], x^{\epsilon }(0)=x_0.$$

We put $u^{\epsilon }(t)=u(t)$ if $t<\tau$, and $u^{\epsilon }(t)=v$ if $\tau \le t<\tau +\epsilon$. To define $u^{\epsilon }(t)$ for $t\ge \tau +\epsilon$, suppose first that $\tau \notin {\Theta }_u$. Then

(i)

$u^{\epsilon }(t)=w(x^{\epsilon }(t))$ if $g(x^{\epsilon }(t_1))=0$ for some $t_1\le t$, and u has no exit points in $[t_1,t]$,

(ii)

$u^{\epsilon }(t)=u(t_{\mathrm{en}}-)$ if $g(x(t))=0$ and $g(x^{\epsilon }(t))<0$, where $t_{\mathrm{en}}$ is the greatest entry point of u less than or equal to t,

(iii)

$u^{\epsilon }(t)=u(t)$ otherwise.

Let now $\tau$ belong to $\theta$, an inclusion-maximal boundary interval of u, and $t\ge \tau +\epsilon$. Then $u^{\epsilon }(t)=w(x^{\epsilon }(t))$ for $t\in \theta$, and (i), (ii), (iii) are valid for $t\notin \theta$.

The spike variation of control is the difference $u^{\epsilon }-u$. Note that the control $u^{\epsilon }$ is admissible for every sufficiently small positive $\epsilon$.

Lemma 1.

The trajectory increment$\Delta x=x^{\epsilon }-x$satisfies

$$\Delta x(t)=\epsilon \delta x(t)+o(\epsilon )$$

(5)

for every$t\in [\tau ,T]$, where the trajectory variation$\delta x:[\tau ,T]\to R^n$is absolutely continuous except, possibly, at the entry points of u, and independent of$\epsilon$. For almost every$t\in [\tau ,T]$

$$\delta \dot{x}(t)={\partial }_{2}F{(t,x(t))}^T\delta x(t),$$

(6)

moreover,

$$\delta x(\tau )=f(x(\tau ),v)-f(x(\tau ),u(\tau ))$$

(7)

and at every entry point$t_{\mathrm{en}}>\tau$of u

$$\delta x(t_{\mathrm{en}}+)=Z{(t_{\mathrm{en}})}^T\delta x(t_{\mathrm{en}}-).$$

(8)

Here

$$Z(t_{\mathrm{en}})=I-\frac{\partial g(x(t_{\mathrm{en}}))\Delta f{(t_{\mathrm{en}})}^T}{\partial g{(x(t_{\mathrm{en}}))}^T\Delta f(t_{\mathrm{en}})}$$

(9)

$$\Delta f(t_{\mathrm{en}})=f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))-f(x(t_{\mathrm{en}}),w(x(t_{\mathrm{en}}))).$$

Proof. 

If the control u has no entry points in $]\tau ,T[$, the lemma is obviously true by virtue of the classical theorems on ordinary differential equations. Suppose that u has exactly one entry point $t_{\mathrm{en}}$ in $]\tau +\epsilon ,T[$. From the mentioned theorems it directly follows that (5) holds in the time interval $[\tau ,t_{\mathrm{en}}[$, with an absolutely continuous function $\delta x$, satisfying (7) and (6) in that interval. We shall prove that the relationships (5) and (6) may be extended to the whole interval $[\tau ,T]$, with the function $\delta x$ absolutely continuous in $]t_{\mathrm{en}},T]$. To this end, let us first notice that for every sufficiently small $\epsilon >0$ the control $u^{\epsilon }$ has an entry point $t_{\mathrm{en}}^{\epsilon }=t_{\mathrm{en}}+\epsilon \delta t_{\mathrm{en}}+o(\epsilon )$, where $\delta t_{\mathrm{en}}$ is a real number independent of $\epsilon$. This follows from the verifiability of u (see (4)) and from the construction of $u^{\epsilon }$. Let $\Delta t_{\mathrm{en}}=t_{\mathrm{en}}^{\epsilon }-t_{\mathrm{en}}$. To fix attention, assume $\Delta t_{\mathrm{en}}>0$. We then have

$$\begin{gathered} x(t_{\mathrm{en}}^{\epsilon })=x(t_{\mathrm{en}})+f(x(t_{\mathrm{en}}),w(x(t_{\mathrm{en}})))\Delta t_{\mathrm{en}}+o(\Delta t_{\mathrm{en}}) \\ {x}^{\epsilon }(t_{\mathrm{en}}^{\epsilon })=x^{\epsilon }(t_{\mathrm{en}})+f(x^{\epsilon }(t_{\mathrm{en}}),u^{\epsilon }(t_{\mathrm{en}}-))\Delta t_{\mathrm{en}}+o(\Delta t_{\mathrm{en}}) \\ =x^{\epsilon }(t_{\mathrm{en}})+f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))\Delta t_{\mathrm{en}}+o(\Delta t_{\mathrm{en}}), \end{gathered}$$

as $f(x^{\epsilon }(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))=f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))$$+{\partial }_{1}f{(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))}^T\Delta x(t_{\mathrm{en}})$$+o(\Delta x(t_{\mathrm{en}}))$. Hence

$$\Delta x(t_{\mathrm{en}}^{\epsilon })=\Delta x(t_{\mathrm{en}})+\Delta f(t_{\mathrm{en}})\Delta t_{\mathrm{en}}+o(\Delta t_{\mathrm{en}}).$$

(10)

By the definition of entry points, $g(x(t_{\mathrm{en}}))=g(x^{\epsilon }(t_{\mathrm{en}}^{\epsilon }))=0$. Thus

$$\begin{gathered} g(x^{\epsilon }(t_{\mathrm{en}}^{\epsilon }))=g(x(t_{\mathrm{en}})+\Delta x(t_{\mathrm{en}})+f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))\Delta t_{\mathrm{en}}+o(\Delta t_{\mathrm{en}})) \\ =\partial g{(x(t_{\mathrm{en}}))}^T(\Delta x(t_{\mathrm{en}})+f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))\Delta t_{\mathrm{en}})+o(\Delta t_{\mathrm{en}})=0, \end{gathered}$$

and so

$$\Delta t_{\mathrm{en}}=-\frac{\partial g{(x(t_{\mathrm{en}}))}^T\Delta x(t_{\mathrm{en}})}{\partial g{(x(t_{\mathrm{en}}))}^{T}f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))}+o(\Delta t_{\mathrm{en}}).$$

Substituting this into (10), we obtain

$$\Delta x(t_{\mathrm{en}}^{\epsilon })=\left(I-\frac{\Delta f(t_{\mathrm{en}})\partial g{(x(t_{\mathrm{en}}))}^T}{\partial g{(x(t_{\mathrm{en}}))}^{T}f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))}\right)\Delta x(t_{\mathrm{en}})+o(\Delta t_{\mathrm{en}}).$$

As $\partial g{(x(t_{\mathrm{en}}))}^{T}f(x(t_{\mathrm{en}}),u(t_{\mathrm{en}}-))=\partial g{(x(t_{\mathrm{en}}))}^T\Delta f(t_{\mathrm{en}})$ and $\Delta x(t_{\mathrm{en}})=\epsilon \delta x(t_{\mathrm{en}}-)+o(\epsilon )$, we get

$$\Delta x(t_{\mathrm{en}}^{\epsilon })=\epsilon Z{(t_{\mathrm{en}})}^T\delta x(t_{\mathrm{en}}-)+o(\epsilon ).$$

Defining $\delta x(t_{\mathrm{en}}+)$ by (8), we arrive at the extension of (5) and (6) to $[\tau ,T]$ because of the same classical theorems on differential equations. For $\Delta t_{\mathrm{en}}<0$, an analogous argument leads to the same result. The proof can be easily generalized to an arbitrary finite number of entry points. □

3. The Adjoint Function and The One-Spike Necessary Optimality Condition

As in Section 2, let u be a verifiable control and x, the corresponding solution of (1). With every such control we associate an adjoint function $\psi :[0,T]\to R^n$, defined as a solution of the adjoint equation

$$\dot{\psi }(t)=-{\partial }_{2}F(t,x(t))\psi (t),$$

(11)

absolutely continuous except at the entry points of u and satisfying the final condition

$$\psi (T)=\partial q(x(T)).$$

(12)

At every entry point $t_{\mathrm{en}}$, $\psi (t_{\mathrm{en}})=\psi (t_{\mathrm{en}}+)$ and

$$\psi (t_{\mathrm{en}}-)=Z(t_{\mathrm{en}})\psi (t_{\mathrm{en}}+).$$

(13)

Let $\tau$ be an arbitrary point from $[0,T[$, and $\delta x$, the trajectory variation determined in Lemma 1. It is easy to notice that the function $t\mapsto \psi {(t)}^T\delta x(t)$ is constant in the whole time interval $[\tau ,T]$. Indeed, its derivative $\dot{\psi }{(t)}^T\delta x(t)+\psi {(t)}^T\delta \dot{x}(t)$ equals zero at every t where $\psi$ and $\delta x$ are differentiable, and at the entry points $t_{\mathrm{en}}>\tau$ we have by virtue of (13) and (8): $\psi {(t_{\mathrm{en}}-)}^T\delta x(t_{\mathrm{en}}-)=$$\psi {(t_{\mathrm{en}}+)}^{T}Z{(t_{\mathrm{en}})}^T\delta x(t_{\mathrm{en}}-)=$$\psi {(t_{\mathrm{en}}+)}^T\delta x(t_{\mathrm{en}}+)$. Thus,

$$\psi {(\tau )}^T(f(x(\tau ),v)-f(x(\tau ),u(\tau )))=\partial q{(x(T))}^T\delta x(T).$$

Define the pre-Hamiltonian $H:R^n\times R^n\times R\to R$, $H(\psi ,x,u)={\psi }^{T}f(x,u)$, and its increment

$$\Delta H(\tau ,v)=H(\psi (\tau ),x(\tau ),v)-H(\psi (\tau ),x(\tau ),u(\tau ))$$

(14)

for any $v\in U$ and $\tau \in [0,T]$ (note that $\Delta H$ is only defined for a uniquely predetermined control u). We can now express the value of cost on the control $u^{\epsilon }$, defined in Section 2

$$Q(u^{\epsilon })=q(x^{\epsilon }(T))=Q(u)+\epsilon \partial q{(x(T))}^T\delta x(T)+o(\epsilon )=Q(u)+\epsilon \Delta H(\tau ,v)+o(\epsilon ).$$

A sufficient condition for the existence of spike variations which improve the cost is a straightforward consequence.

Lemma 2.

Assume that$\tau \in [0,T[$, $v\in U_{\tau }$, and$\Delta H(\tau ,v)<0$. Then the control$u^{\epsilon }$is admissible and$Q(u^{\epsilon })<Q(u)$for every sufficiently small$\epsilon >0$.

A theorem on optimal control of the minimum principle type follows from Lemma 2.

Theorem 1.

Assume that the control u is optimal. Then:

(i)

$\Delta H(t,v)\ge 0$for every$t\in [0,T]$and every$v\in U_t$,

(ii)

the function$[0,T]\ni t\mapsto \chi (t)=H(\psi (t),x(t),u(t))$is constant.

Proof. 

Conclusion (i) is a direct consequence of Lemma 2. Conclusion (ii) for the nonboundary intervals is proved exactly as in the classical proofs of the minimum principle without pathwise state constraints. In the interior of every boundary interval, the function $\chi$ is of class $C^1$ with the derivative identically zero. The continuity of $\chi$ at entry points readily follows from (13) and (9). Let now $t_{\mathrm{ex}}$ be an exit point of u. By (i) and (4), there is a $\delta >0$ such that $\Delta H(t,u(t_{\mathrm{ex}}))\ge 0$ for all $t\in [t_{\mathrm{ex}}-\delta ,t_{\mathrm{ex}}[$, and $\Delta H(t,w(x(t_{\mathrm{ex}})))\ge 0$ for all $t\in ]t_{\mathrm{ex}},t_{\mathrm{ex}}+\delta ]$. The continuity of $\chi$ at $t_{\mathrm{ex}}$ is shown by limit passages: $t\to t_{\mathrm{ex}}-$ for the first of these inequalities, and $t\to t_{\mathrm{ex}}+$ for the second. □

Corollary 1.

Assume that the control u is optimal and$g(x(t))=0$for some$t\in [0,T]$. Then$\Delta H(t,v)\ge 0$or$\dot{g}(x(t),v)\ge 0$for every$v\in U$.

4. The Two-Spike Necessary Optimality Condition

For a verifiable control u and the corresponding state trajectory x, we shall define a two-spike control variation. Let $\theta$ be an inclusion-maximal boundary interval of u. For any quintuple ${\tau }_1,{\tau }_2,v_1,v_2,\eta$ such that ${\tau }_1,{\tau }_2\in \theta$, ${\tau }_1<{\tau }_2$, $v_1,v_2\in U$, $\eta >0$, and for any sufficiently small $\epsilon >0$ we define a control $u^{\epsilon }\in PC(0,T;U)$ (not to be confused with the control $u^{\epsilon }$ defined in Section 2) and the corresponding state trajectory $x^{\epsilon }$. We put $u^{\epsilon }(t)=u(t)$ if $t<{\tau }_1$, $u^{\epsilon }(t)=v_1$ if ${\tau }_1\le t<{\tau }_1+\epsilon$, $u^{\epsilon }(t)=v_2$ if ${\tau }_2\le t<{\tau }_2+\eta \epsilon$, and $u^{\epsilon }(t)=$$w(x^{\epsilon }(t))$ for any other t in $\theta$. Points (i), (ii) and (iii) of the definition in Section 2 apply to all the remaining values of t. The two-spike control variation is the difference $u^{\epsilon }-u$.

The control u may sometimes be improved even if it fulfills the necessary optimality condition (i) of Theorem 1. We shall now give conditions, sufficient for the existence of a two-spike control variation in $\theta$ which is admissible and guarantees a cost improvement.

Lemma 3.

Assume that${\tau }_1,{\tau }_2\in \theta$, ${\tau }_1<{\tau }_2$, $v_1,v_2\in U$, $\eta >0$, and

$$\dot{g}(x({\tau }_1),v_1)<0$$

(15)

$$\dot{g}(x({\tau }_1),v_1)+\eta \dot{g}(x({\tau }_2),v_2)<0$$

(16)

$$\Delta H({\tau }_1,v_1)+\eta \Delta H({\tau }_2,v_2)<0.$$

(17)

Then for every sufficiently small$\epsilon >0$the control$u^{\epsilon }$is admissible, and$Q(u^{\epsilon })<Q(u)$.

Proof. 

It follows from the definition of $u^{\epsilon }$ and the inequalities (15), (16) that for every sufficiently small $\epsilon >0$ the function $t\mapsto g(x^{\epsilon }(t))$ is negative in the time interval $]{\tau }_1,\sup \theta ]$ and constant in the intervals $[{\tau }_1+\epsilon ,{\tau }_2]$ and $[{\tau }_2+\eta \epsilon ,\sup \theta ]$,

$$g(x^{\epsilon }(t))=\epsilon \dot{g}(x({\tau }_1),v_1)+o(\epsilon ), t\in [{\tau }_1+\epsilon ,{\tau }_2],$$

$$g(x^{\epsilon }(t))=\epsilon (\dot{g}(x({\tau }_1),v_1)+\eta \dot{g}(x({\tau }_2),v_2))+o(\epsilon ),t\in [{\tau }_2+\eta \epsilon ,\hspace{0.17em}\sup \theta ].$$

From this we infer that the control $u^{\epsilon }$ is admissible for all sufficiently small $\epsilon >0$. Reasoning similarly as in Section 3 and using the adjoint function defined therein, we estimate the value of the performance index on the control $u^{\epsilon }$.

$$Q\left(u^{\epsilon }\right)=Q\left(u\right)+\epsilon (\Delta H({\tau }_1,v_1)+\eta \Delta H({\tau }_2,v_2)+o(\epsilon ).$$

Hence, (17) is a sufficient condition for the two-spike control variation to reduce the cost for every sufficiently small $\epsilon >0$. □

Lemma 4.

Assume that${\tau }_1,{\tau }_2\in \theta$, ${\tau }_1<{\tau }_2$, $v_1,v_2\in U$, and

$$\dot{g}(x({\tau }_1),v_1)<0, \Delta H({\tau }_1,v_1)\ge 0$$

$$\dot{g}(x({\tau }_2),v_2)\ge 0, \Delta H({\tau }_2,v_2)<0.$$

(18)

Assume also that if$\dot{g}(x({\tau }_2),v_2)>0$, then

$$\frac{\Delta H({\tau }_1,v_1)}{\dot{g}(x({\tau }_1),v_1)}>\frac{\Delta H({\tau }_2,v_2)}{\dot{g}(x({\tau }_2),v_2)}.$$

(19)

Under these assumptions there is an$\eta >0$, such that for every sufficiently small$\epsilon >0$the control$u^{\epsilon }$is admissible, and$Q(u^{\epsilon })<Q(u)$.

Proof. 

We shall show that the assumptions of Lemma 3 follow from the assumptions of Lemma 4. Denote

$${\eta }_1=-\frac{\Delta H({\tau }_1,v_1)}{\Delta H({\tau }_2,v_2)} \mathrm{and} {\eta }_2=-\frac{\dot{g}(x({\tau }_1),v_1)}{\dot{g}(x({\tau }_2),v_2)} \mathrm{for} \dot{g}(x({\tau }_2),v_2)>0.$$

Of course ${\eta }_1\ge 0$ and ${\eta }_2>0$. The inequality (15) is obvious. In view of (18), (17) is true for every $\eta >{\eta }_1$. If $\dot{g}(x({\tau }_2),v_2)=0$, then (16) holds for every $\eta >0$, and the assumptions of Lemma 3 are satisfied. If $\dot{g}(x({\tau }_2),v_2)>0$, then the inequality (16) holds for $\eta <{\eta }_2$. We thus have a two-sided bound on $\eta$, ${\eta }_1<\eta <{\eta }_2$. The interval of admissible values of $\eta$ is nonempty if ${\eta }_1<{\eta }_2$, and this inequality follows from (19). □

By contradicting the sufficient nonoptimality conditions of Lemma 4 we obtain new necessary conditions of optimality.

Theorem 2 (main result).

Assume that the control u is optimal and verifiable, and has a boundary interval$\theta$. Let also$t_1,t_2\in \theta$, $t_1\le t_2$, and$v_1,v_2\in U$. Under these assumptions:

(i)

if$\dot{g}(x(t_1),v_1)<0$and$\Delta H(t_1,v_1)=0$, then$\Delta H(t_2,v_2)\ge 0$,

(ii)

if$\dot{g}(x(t_2),v_2)=0$and$\Delta H(t_2,v_2)<0$, then$\dot{g}(x(t_1),v_1)\ge 0$,

(iii)

if$\dot{g}(x(t_1),v_1)<0$and$\Delta H(t_2,v_2)<0$, then$\dot{g}(x(t_2),v_2)>0$and

$$\frac{\Delta H(t_1,v_1)}{\dot{g}(x(t_1),v_1)}\le \frac{\Delta H(t_2,v_2)}{\dot{g}(x(t_2),v_2)}.$$

(20)

Proof. 

Let first $t_1<t_2$. Suppose, contrary to (i), that $\dot{g}(x(t_1),v_1)<0$, $\Delta H(t_1,v_1)=0$ and $\Delta H(t_2,v_2)<0$. From Corollary 1, $\dot{g}(x(t_2),v_2)\ge 0$. By Lemma 4, this contradicts the assumption that u is optimal. The implication (ii) is similarly proved. Let $\dot{g}(x(t_2),v_2)=0$, $\Delta H(t_2,v_2)<0$ and $\dot{g}(x(t_1),v_1)<0$. By Corollary 1, $\Delta H(t_1,v_1)\ge 0$ and the assumptions of Lemma 4 are fulfilled (with ${\tau }_1:=t_1$ and ${\tau }_2:=t_2$). To prove (iii), assume that $\dot{g}(x(t_1),v_1)<0$ and $\Delta H(t_2,v_2)<0$. It follows from Corollary 1 that $\Delta H(t_1,v_1)\ge 0$ and $\dot{g}(x(t_2),v_2)\ge 0$. If $\dot{g}(x(t_2),v_2)=0$, the assumptions of Lemma 4 hold. If $\dot{g}(x(t_2),v_2)>0$, the inequality opposite to (19), that is (20), is true.

Let now $t_1=t_2={\tau }_1$. The proof goes similarly, however, we additionally have to use a simple observation (rc): the functions $t\mapsto \dot{g}(x(t),v)$ and $t\mapsto \Delta H(t,v)$ are right-continuous in $[0,T[$ for every $v\in U$. Let $\dot{g}(x({\tau }_1),v_1)<0$, $\Delta H({\tau }_1,v_1)=0$ and $\Delta H({\tau }_1,v_2)<0$. By virtue of (rc) and Corollary 1, there is a ${\tau }_2>{\tau }_1$ such that $\Delta H({\tau }_2,v_2)<0$ and $\dot{g}(x({\tau }_2),v_2)\ge 0$. Lemma 4 then gives a contradiction. To prove (ii), assume $\dot{g}(x({\tau }_1),v_2)=0$, $\Delta H({\tau }_1,v_2)<0$ and $\dot{g}(x({\tau }_1),v_1)<0$. By (rc) and Corollary 1, $\Delta H({\tau }_1,v_1)\ge 0$ and there is a $\widehat{\tau }\in \theta$, $\widehat{\tau }>{\tau }_1$, such that $\Delta H({\tau }_2,v_2)<0$ and $\dot{g}(x({\tau }_2),v_2)\ge 0$ for every ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$. If $\dot{g}(x({\tau }_2),v_2)=0$ for some ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$, Lemma 4 again yields a contradiction. If $\dot{g}(x({\tau }_2),v_2)>0$ for every ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$, then (19) holds for every ${\tau }_2>{\tau }_1$ sufficiently close to ${\tau }_1$, since $\dot{g}(x({\tau }_2),v_2)\to 0$ as ${\tau }_2\to {\tau }_1+$, and so Lemma 4 gives a contradiction. We shall now prove (iii). Let $\dot{g}(x({\tau }_1),v_1)<0$ and $\Delta H({\tau }_1,v_2)<0$. It follows from (rc) and Corollary 1 that $\Delta H({\tau }_1,v_1)\ge 0$ and there is a $\widehat{\tau }\in \theta$, $\widehat{\tau }>{\tau }_1$, such that $\Delta H({\tau }_2,v_2)<0$ and $\dot{g}(x({\tau }_2),v_2)\ge 0$ for every ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$. If $\dot{g}(x({\tau }_2),v_2)=0$ for some ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$, a contradiction follows from Lemma 4. Similarly, Lemma 4 gives a contradiction if for some ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$ the relationships $\dot{g}(x({\tau }_2),v_2)>0$ and (19) are fulfilled. In consequence, the inequalities $\dot{g}(x({\tau }_2),v_2)>0$ and (20) with $t_1:={\tau }_1$ and $t_2:={\tau }_2$ hold true for every ${\tau }_2\in ]{\tau }_1,\widehat{\tau }]$. By (rc), $\Delta H({\tau }_2,v_2)\to$$\Delta H({\tau }_1,v_2)<0$ and $\dot{g}(x({\tau }_2),v_2)\to$$\dot{g}(x({\tau }_1),v_2)\ge 0$ as ${\tau }_2\to {\tau }_1+$.

If $\dot{g}(x({\tau }_1),v_2)=0$, then (19) is true for all ${\tau }_2>{\tau }_1$, sufficiently close to ${\tau }_1$. We have thus come to a contradiction. Hence $\dot{g}(x({\tau }_1),v_2)>0$, and the inequality (20) holds by virtue of (rc). □

5. A Geometrical Interpretation and a Minimum Condition

Let u be a verifiable control with a boundary interval $\theta$. The corresponding state and adjoint trajectories are denoted by x and $\psi$, respectively. Define a family of sets

$$C_t=\{y\in R^2:y_1=\dot{g}(x(t),v),\hspace{0.17em}{y}_2=\Delta H(t,v),\hspace{0.17em}v\in U\}, t\in \theta .$$

It readily follows from this definition that $0\in C_t$ for every $t\in \theta$. In the sequel we implicitly assume that $\left|C_t\right|>1$.

We shall now characterize the properties of the sets $C_t$ which result from control optimality. For an arbitrary nonzero vector $y\in R^2$, define $\arg y$ as the angle between $\mathrm{col}(1,0)$ and y, measured anticlockwise and taking values in the interval $]-\pi ,\pi ]$. Let also

$${\varphi }_{\min }(t)=\inf \{\arg y:y\in C_t\backslash \{0\left\}\right\}, {\varphi }_{\max }(t)=\sup \{\arg y:y\in C_t\backslash \{0\left\}\right\}$$

for every $t\in \theta$. Corollary 1 says that if u is optimal, then $C_t$ has no points in quadrant III of the coordinate system $y_1{y}_2$. The following theorem is a straightforward consequence of that corollary and of Theorem 2.

Theorem 3.

Assume that the control u is optimal. Then

(i)

$-\frac{1}{2}\pi \le {\varphi }_{\min }(t)\le {\varphi }_{\max }(t)\le \pi$for every$t\in \theta$,

(ii)

${\varphi }_{\max }(t_1)-{\varphi }_{\min }(t_2)\le \pi$for every pair$t_1,t_2\in \theta$such that$t_1\le t_2$.

From this it easily follows that if the control u is optimal and ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$ for every $t\in \theta$, then $\frac{1}{2}\pi \le {\varphi }_{\max }(t)\le \pi$ for every $t\in \theta$, and the function ${\varphi }_{\max }$ is nondecreasing in $\theta$.

It proves useful to describe the consequences of control optimality in terms of the straight lines supporting the sets $C_t$ at zero. This allows easier verification of the necessary conditions of optimality, and also expressing a partial optimality criterion as a minimum condition imposed on the extended pre-Hamiltonian. We say that a straight line is a supporting line of $C_t$ at the origin (SLO) if it is given by ${\rho }^{T}y=0$ with $\rho ,y\in R^2$, $\rho \ne 0$, and ${\rho }^{T}y\ge 0$ for every $y\in C_t$. Generally, the set $C_t$ may have many SLOs, whether the control is optimal or not. If u is optimal, then every set $C_t$, $t\in \theta$, has an SLO with $\rho \ge 0$. The set $C_t$ has a unique SLO if and only if ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$.

The equality ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$ occurs in two practically important situations (mutually nonexclusive). One of them, in which the right-hand side of the system equation (1) is affine in control, will be discussed in Section 7. Here we consider the other situation, in which $C_t$ has a tangent at the origin. A sufficient condition for that reads

$$u(t)\in \mathrm{int}U \mathrm{and} |{\partial }_2\dot{g}(x(t),u(t))|+|{\partial }_2\Delta H(t,u(t))|>0.$$

(21)

Under this condition, the tangent has the equation ${\rho }^{T}y=0$ with

$${\rho }_1={\partial }_2\Delta H(t,u(t)),\hspace{1em}{\rho }_2=-{\partial }_2\dot{g}(x(t),u(t)).$$

Of course, if $C_t$ has both an SLO and a tangent at the origin, they coincide.

Suppose u is optimal and ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$ for every $t\in \theta$. Then every set $C_t$, $t\in \theta$, has a unique SLO. The SLO is vertical if ${\varphi }_{\max }(t)=\frac{1}{2}\pi$; if ${\varphi }_{\max }(t)>\frac{1}{2}\pi$, the SLO equation may be written as $y_2=p(t)y_1$ with a nonpositive directional coefficient $p(t)=\tan {\varphi }_{\max }(t)$. If, additionally, the condition (21) is fulfilled with ${\partial }_2\dot{g}(x(t),u(t))\ne 0$, then

$$p(t)=\underset{\begin{array}{l}v\to u(t)\\ v\ne u(t)\end{array}}{\lim }\frac{\Delta H(t,v)}{\dot{g}(x(t),v)}=\frac{{\partial }_2\Delta H(t,u(t))}{{\partial }_2\dot{g}(x(t),u(t))}=\frac{{\partial }_{3}H(\psi (t),x(t),u(t))}{{\partial }_2\dot{g}(x(t),u(t))}.$$

(22)

The function p thus defined is nondecreasing in all that part of $\theta$ where it is determined.

Define the extended pre-Hamiltonian $\widehat{H}(\psi ,p,x,v)={\psi }^{T}f(x,v)-p\dot{g}(x,v)$. If the control u is optimal and the function p is determined as above in all the interval $\theta$, then the following minimum condition is straightforward by the properties of SLO

$$\widehat{H}(\psi (t),p(t),x(t),u(t))\le \widehat{H}(\psi (t),p(t),x(t),v) \forall v\in U \forall t\in \theta .$$

(23)

This necessary optimality condition is similar to the minimum condition of indirect adjoining. We postpone a discussion of relations with the classical results to Section 9.

6. Example 1

In this example we apply the above necessary conditions to verify optimality of two controls, the first of which is optimal, and the second is not. We show that the nonoptimality is easily detected. The control system is described by state equations

$${\dot{x}}_1=2-\frac{1}{2}{x}_1+(x_1-2)x_2+au(b-u), {\dot{x}}_2=u,$$

with the initial conditions $x_1(0)=x_2(0)=1$. The set of admissible control values consists of three elements, $U=\{-1,\hspace{0.17em}0,\hspace{0.17em}1\}$. The state is subject to a pathwise constraint $g(x)=x_2-1\le 0$. The cost to be minimized is given by $Q(u)=\frac{1}{2}{(x_1(T)-6)}^2+\frac{1}{2}{x}_2{(T)}^2$. We take $a=0.125$, $b=1.025$, $T=4$.

Let u be a verifiable control, and x and $\psi$, respectively, the corresponding state trajectory and adjoint function. Let us write the pre-Hamiltonian

$$H(\psi ,x,u)={\psi }_1(2-\frac{1}{2}{x}_1+(x_1-2)x_2+au(b-u))+{\psi }_{2}u$$

and the adjoint equations in nonboundary intervals of time

$${\dot{\psi }}_1=(\frac{1}{2}-x_2){\psi }_1, {\dot{\psi }}_2=(2-x_1){\psi }_1.$$

The adjoints satisfy the final conditions ${\psi }_1(T)=x_1(T)-6$, ${\psi }_2(T)=x_2(T)$. As $\dot{g}(x,u)=u$, we have $w(x)=0$. In consequence, the state and adjoint equations in the boundary intervals take the form

$${\dot{x}}_1=\frac{1}{2}{x}_1, {\dot{x}}_2=0$$

$${\dot{\psi }}_1=-\frac{1}{2}{\psi }_1, {\dot{\psi }}_2=(2-x_1){\psi }_1.$$

To determine the behavior of the adjoint function at entry points, we calculate the matrix (9)

$$Z(t_{\mathrm{en}})=\left[\begin{array}{cc}1& 0\\ a(u(t_{\mathrm{en}}-)-b)& 0\end{array}\right].$$

Hence by (13), ${\psi }_1(t_{\mathrm{en}}-)={\psi }_1(t_{\mathrm{en}}+)$ and ${\psi }_2(t_{\mathrm{en}}-)=a(u(t_{\mathrm{en}}-)-b){\psi }_1(t_{\mathrm{en}}+)$. In accordance with (14), $\Delta H(t,v)=(ab{\psi }_1(t)+{\psi }_2(t))v-a{\psi }_1(t)v^2$ in every boundary interval.

It is evident that the nontangentiality conditions (4) are fulfilled at all entry and exit points. The optimality of u should be verified with Theorem 1(i) in the whole interval $[0,T]$, and additionally with Theorem 2 or 3 in the boundary intervals. Every set $C_t$ introduced in Section 5 consists of three points, $C_t=$$\{y(t),0,z(t)\}$, where $y_1(t)=-1$, $y_2(t)=\Delta H(t,-1)$, $z_1(t)=+1$, and $z_2(t)=\Delta H(t,+1)$. If the control u is optimal, then it follows from part (i) of Theorem 3 that $y_2(t)\ge 0\hspace{1em}\forall t\in \theta$, and from part (ii), that $-y_2(t_1)\le$$z_2(t_2)$ for every pair $t_1,t_2\in \theta$ such that $t_1\le t_2$. By Lemmas 2 and 4, u is nonoptimal if $y_2(t)<0$ for some $t\in \theta$ or $-y_2(t_1)>z_2(t_2)$ for some $t_1,t_2\in \theta$, $t_1\le t_2$.

Example 1a.

A numerically computed approximation of optimal control and optimal state trajectory is presented in Figure 1. The control has discontinuities at $s_1=0.49483839$, $s_2=0.99973751$, $s_3=1.4945759$ and $s_4=3.1366809$. Figure 2 shows the corresponding adjoint trajectory. Let us verify the necessary conditions of optimality. Figure 3 shows that the condition of Theorem 1(i) is fulfilled. In the boundary interval $\theta =[s_3,s_4[$, we additionally have to verify the conditions of Theorem 2 or 3. It can be seen in Figure 3 that $\Delta H(t,-1)>0$ for all $t\in \theta$. Thus, implications (i) and (ii) of Theorem 2 are vacuously true for all $v_1$,$v_2$,$t_1$,$t_2$ satisfying the assumptions of the theorem, and so is (iii) except the case where $v_1=-1$, $v_2=1$ and $\Delta H(t_2,1)<0$. In that case, $\dot{g}(x(t_2),v_2)=1$ and (20) reads $-\Delta H(t_1,-1)\le \Delta H(t_2,1)$. As can be checked by inspection, (20) holds for all $t_1,t_2\in \theta$, $t_1\le t_2$, and so (iii) is true. Alternatively and equivalently, we can use Theorem 3. We see in Figure 4 that $-\frac{1}{2}\pi <{\varphi }_{\min }(t)<{\varphi }_{\max }(t)$ and ${\varphi }_{\max }(t)-\pi <0$ for every $t\in \theta$, then conclusion (i) of Theorem 3 is true. It can be also seen that ${\varphi }_{\max }(t_1)-\pi <{\varphi }_{\min }(t_2)$ for every $t_1\in \theta$ and $t_2\in [t_1,s_4[$, and so conclusion (ii) holds too. Thus, the necessary optimality conditions of Theorems 1, 2 and 3 are satisfied.

Example 1b.

Consider a nonoptimal control

$$u(t)=\{\begin{array}{c}0,\hspace{1em}0\le t<s_1\\ -1,\hspace{1em}{s}_1\le t\le T,\end{array}$$

where $s_1=3.2429037$. It is plotted in Figure 5 together with the corresponding state trajectory. The adjoint function is depicted in Figure 6. As follows from Figure 7, the necessary optimality condition of Theorem 1(i) is satisfied and in consequence, there are no one-spike variations described in Section 2 which guarantee an improvement of the cost. Let us now check the conditions of Theorem 3 in the boundary interval $\theta =[0,s_1[$. To this end we define ${\overrightarrow{\varphi }}_{\min }(t)=\inf \{{\varphi }_{\min }(s):t<s\le s_1\}$ and ${\overleftarrow{\varphi }}_{\max }(t)=\sup \{{\varphi }_{\max }(s):0\le s<t\}$ for $t\in \theta$. The inequalities ${\varphi }_{\min }(t)\le {\varphi }_{\max }(t)\le \pi$ in conclusion (i) of Theorem 3 directly follow from the definitions. Figure 8 shows that ${\overrightarrow{\varphi }}_{\min }(t)>-\frac{1}{2}\pi$, and the more so ${\varphi }_{\min }(t)>-\frac{1}{2}\pi$ for every $t\in \theta$. Let us rewrite conclusion (ii) of the theorem in an equivalent form, ${\overleftarrow{\varphi }}_{\max }(t)-\pi \le {\overrightarrow{\varphi }}_{\min }(t)$$\forall t\in \theta$. Figure 8 shows that this inequality holds only in $[t^{\ast },s_1[$, with $t^{\ast }>0$. This proves that the control u is not optimal and there are two-spike variations in the boundary interval (defined in Section 4) which yield a cost reduction for any sufficiently small positive value of the parameter $\epsilon$. A closer analysis of the conditions of Lemma 4 shows that the difference ${\tau }_2-{\tau }_1$, that is, the distance between the spikes in such a variation cannot be arbitrarily small.

7. The Control Affine Case

Consider the system (1) with the function f affine in control, $f(x,u)=a(x)+b(x)u$. Many of the results obtained so far may then be significantly simplified, or even strengthened. In this section, u stands for a certain verifiable control, x for the corresponding state trajectory, and $\psi$ for the corresponding adjoint. We also denote

$${\alpha }_1(t)=\partial g{(x(t))}^{T}b(x(t)), {\alpha }_2(t)=\psi {(t)}^{T}b(x(t)).$$

In consequence we have $\Delta H(t,v)={\alpha }_2(t)(v-u(t))$, where $t\in [0,T]$ and $v\in U$. The formula (3) in every boundary interval $\theta$ of u may be written as $\dot{g}(x(t),v)=$${\alpha }_1(t)(v-u(t))$, for $t\in \theta$ and $v\in U$. The equality (9) is simplified to

$$Z(t_{\mathrm{en}})=I-\frac{\partial g(x(t_{\mathrm{en}}))b{(x(t_{\mathrm{en}}))}^T}{{\alpha }_1(t_{\mathrm{en}})}.$$

(24)

It follows from (13) that the left-hand limit of the switching function ${\alpha }_2$ equals zero at every entry point $t_{\mathrm{en}}$, ${\alpha }_2(t_{\mathrm{en}}-)=0$. If the control u is optimal, then ${\alpha }_2$ vanishes at every exit point $t_{\mathrm{ex}}$, ${\alpha }_2(t_{\mathrm{ex}})=0$. Indeed, ${\alpha }_2(t_{\mathrm{ex}})u(t_{\mathrm{ex}}-)={\alpha }_2(t_{\mathrm{ex}})u(t_{\mathrm{ex}})$ by Theorem 1(ii), and $u(t_{\mathrm{ex}}-)\ne u(t_{\mathrm{ex}})$ by (4).

We shall now formulate the results of Section 3 for the control affine case, beginning with Lemma 2.

Lemma 5.

Assume that$\tau \in [0,T[$, $v\in U_{\tau }$, and${\alpha }_2(\tau )(v-u(\tau ))<0$. Then for every sufficient-ly small$\epsilon >0$, the control$u^{\epsilon }$defined inSection 2is admissible and$Q(u^{\epsilon })<Q(u)$.

Theorem 1(i) takes the following form.

Theorem 4.

Assume that the control u is optimal. Then${\alpha }_2(t)(v-u(t))\ge 0$for every$t\in [0,T]$and every$v\in U_t$.

Corollary 2.

Assume that u is optimal and$t\in [0,T]$. Then the following implications hold:

(i)

if$\left(g(x(t))<0 and {\alpha }_2(t)>0\right)$or$\left(t\in {\Theta }_u,\hspace{0.17em}{\alpha }_1(t)>0 and {\alpha }_2(t)>0\right)$, then$\min U$exists and$u(t)=\min U$,

(ii)

if$\left(g(x(t))<0 and {\alpha }_2(t)<0\right)$or$\left(t\in {\Theta }_u,\hspace{0.17em}{\alpha }_1(t)<0 and {\alpha }_2(t)<0\right)$, then$\max U$exists and$u(t)=\max U$.

From here till the end of this section, u is a verifiable control with a boundary interval $\theta$. Let us pass to the results of Section 4. For all $t\in [0,T]$ such that ${\alpha }_1(t)\ne 0$, define

$$p(t)=\frac{{\alpha }_2(t)}{{\alpha }_1(t)}.$$

(25)

Note that this is an extension of the function p given by (22). The following lemma is an immediate consequence of Lemma 4.

Lemma 6.

Assume that${\tau }_1,{\tau }_2\in \theta$, ${\tau }_1<{\tau }_2$, $v_1,v_2\in U$, and

$${\alpha }_1({\tau }_1)(v_1-u({\tau }_1))<0, {\alpha }_2({\tau }_1)(v_1-u({\tau }_1))\ge 0$$

$${\alpha }_1({\tau }_2)(v_2-u({\tau }_2))\ge 0, {\alpha }_2({\tau }_2)(v_2-u({\tau }_2))<0.$$

Let also$p({\tau }_1)>p({\tau }_2)$if${\alpha }_1({\tau }_2)\ne 0$. Then there is an$\eta >0$, such that for every sufficiently small$\epsilon >0$the control$u^{\epsilon }$defined inSection 4is admissible and$Q(u^{\epsilon })<Q(u)$.

The question arises how to choose $\eta$ in the construction of $u^{\epsilon }$ under the assumptions of Lemma 6. It follows from the proof of Lemma 4 that if ${\alpha }_1({\tau }_2)=0$, then $\eta$ may be any positive number, whereas if ${\alpha }_1({\tau }_2)\ne 0$, then ${\eta }_1<\eta <{\eta }_2$ with

$${\eta }_1=-\frac{{\alpha }_2({\tau }_1)(v_1-u({\tau }_1))}{{\alpha }_2({\tau }_2)(v_2-u({\tau }_2))}, {\eta }_2=-\frac{{\alpha }_1({\tau }_1)(v_1-u({\tau }_1))}{{\alpha }_1({\tau }_2)(v_2-u({\tau }_2))}$$

(26)

If the control u has nonextremal values in the boundary interval $\theta$, a simple consequence follows from Lemma 6.

Corollary 3.

Assume that$v<u(t)<\widehat{v}$and${\alpha }_1(t){\alpha }_2(t)<0$for some$v,\widehat{v}\in U$and every$t\in \theta$. Let also the function p be strictly decreasing in$\theta$. Then there exist${\tau }_1,{\tau }_2\in \theta$,$v_1,v_2\in U$and$\eta >0$, such that the control$u^{\epsilon }$defined inSection 4is admissible and$Q(u^{\epsilon })<Q(u)$for every sufficiently small$\epsilon >0$.

The following theorem is a straightforward consequence of Theorem 2.

Theorem 5.

Assume that the control u is optimal,$t_1,t_2\in \theta$,$t_1\le t_2$, and$v_1,v_2\in U$. Under these assumptions:

(i)

if${\alpha }_1(t_1)(v_1-u(t_1))<0$and${\alpha }_2(t_1)=0$, then${\alpha }_2(t_2)(v_2-u(t_2))\ge 0$,

(ii)

if${\alpha }_1(t_2)=0$and${\alpha }_2(t_2)(v_2-u(t_2))<0$, then${\alpha }_1(t_1)(v_1-u(t_1))\ge 0$,

(iii)

if${\alpha }_1(t_1)(v_1-u(t_1))<0$and${\alpha }_2(t_2)(v_2-u(t_2))<0$, then${\alpha }_1(t_2)\ne 0$and$p(t_1)\le p(t_2)<0$.

Corollary 4.

Assume that u is optimal,$v_1,v_2\in U$,${\alpha }_1(t)(v_1-u(t))<0$and${\alpha }_2(t)(v_2-u(t))<0$for every$t\in \theta$. Then the function p is negative and nondecreasing in$\theta$.

The analysis of Section 5 applied to the control affine case leads to the following conclusions. Every set $C_t$ is included in a certain straight line in $R^2$, passing through the origin. If ${\alpha }_1{(t)}^2+{\alpha }_2{(t)}^2>0$, this line has parametric equations $y_1(s)=$${\alpha }_1(t)s$, $y_2(s)=$${\alpha }_2(t)s$, $s\in R$. If $\left|C_t\right|=2$, then ${\varphi }_{\max }(t)={\varphi }_{\min }(t)$, and if $\left|C_t\right|>2$, then either ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$ or ${\varphi }_{\max }(t)=$${\varphi }_{\min }(t)$. Theorem 3 remains unchanged.

Suppose that the control u is optimal. We then have by Theorem 3 that for every $t\in \theta$ the set $C_t$ has an SLO given by ${\rho }^{T}y=0$ with $\rho \ge 0$. If ${\alpha }_1{(t)}^2+{\alpha }_2{(t)}^2>0$, we can put ${\rho }_1=|{\alpha }_2(t)|$ and ${\rho }_2=|{\alpha }_1(t)|$. Let us now assume that for every $t\in \theta$ there are $v,\widehat{v}\in U$ such that $v<u(t)<\widehat{v}$, and in consequence ${\varphi }_{\max }(t)={\varphi }_{\min }(t)+\pi$. Let also $\frac{1}{2}\pi <{\varphi }_{\max }(t)<\pi$. It then follows from the reasoning in Section 5 that the equality $y_2=p(t)y_1$ holds for every $t\in \theta$ and every $y\in C_t$, with the function p (25) negative and nondecreasing in the interval $\theta$.

Finally, note that the minimum condition on the extended pre-Hamiltonian (23) is trivially satisfied with equality (independently of whether the control u is optimal or not).

8. Example 2: The Pendulum on a Cart

We shall now consider a problem with the right-hand side of the state Equation (1) affine in control. The system is described by state equations

$${\dot{x}}_1=x_3$$

$${\dot{x}}_2=x_4$$

$${\dot{x}}_3=f_3(x,u)=\frac{u-x_4^2\sin x_2+\sin x_2\cos x_2}{1+{\sin }^2{x}_2}$$

$${\dot{x}}_4=f_4(x,u)=\frac{(u-x_4^2\sin x_2)\cos x_2+2\sin x_2}{1+{\sin }^2{x}_2}.$$

(27)

The initial state $x(0)=x_0$ and the time horizon T are fixed. The performance index

$$Q(u)=\frac{1}{2}x{(T)}^{T}x(T)$$

is minimized subject to control bounds and a pathwise state constraint

$$u_{\min }\le u(t)\le u_{\max }, g(x(t))=x_3(t)-x_{3\max }\le 0, t\in [0,T].$$

We write the pre-Hamiltonian

$$H={\psi }_1{x}_3+{\psi }_2{x}_4+{\psi }_3{f}_3+{\psi }_4{f}_4$$

and the adjoint equations in the nonboundary time intervals

$$\begin{array}{c}{\dot{\psi }}_1=0\hfill \\ {\dot{\psi }}_2=-{\psi }_3\frac{\partial f_3}{\partial x_2}-{\psi }_4\frac{\partial f_4}{\partial x_2}\hfill \\ {\dot{\psi }}_3=-{\psi }_1\hfill \\ {\dot{\psi }}_4=-{\psi }_2+\frac{2x_4\sin x_2({\psi }_3+{\psi }_4\cos x_2)}{1+{\sin }^2{x}_2},\hfill \end{array}$$

where

$$\frac{\partial f_3}{\partial x_2}=\frac{\cos 2x_2-x_4^2\cos x_2-f_3\sin 2x_2}{1+{\sin }^2{x}_2}$$

$$\frac{\partial f_4}{\partial x_2}=\frac{2\cos x_2-u\sin x_2-x_4^2\cos 2x_2-f_4\sin 2x_2}{1+{\sin }^2{x}_2}.$$

The adjoint function satisfies the final condition $\psi (T)=x(T)$. The state equations in the bound-ary intervals are obtained by the substitution $u=w(x)=(x_4^2-\cos x_2)\sin x_2$ in (27), which gives

$${\dot{x}}_1=x_3, {\dot{x}}_2=x_4, {\dot{x}}_3=0, {\dot{x}}_4=\sin x_2.$$

Hence the pre-Hamiltonian and the adjoint equations in the boundary intervals read

$$H={\psi }_1{x}_3+{\psi }_2{x}_4+{\psi }_4\sin x_2$$

$${\dot{\psi }}_1=0, {\dot{\psi }}_2=-{\psi }_4\cos x_2, {\dot{\psi }}_3=-{\psi }_1, {\dot{\psi }}_4=-{\psi }_2.$$

At every entry point $t_{\mathrm{en}}$ the jump condition (13) is valid with the matrix (24), whence ${\psi }_i(t_{\mathrm{en}}-)={\psi }_i(t_{\mathrm{en}}+)$, $i=1,\hspace{0.17em}2,\hspace{0.17em}4$, ${\psi }_3(t_{\mathrm{en}}-)=-\cos x_2(t_{\mathrm{en}}){\psi }_4(t_{\mathrm{en}}+)$. We further compute

$${\alpha }_1(t)={(1+{\sin }^2{x}_2(t))}^{-1},$$

$${\alpha }_2(t)=({\psi }_3(t)+{\psi }_4(t)\cos x_2(t)){(1+{\sin }^2{x}_2(t))}^{-1},$$

and from (25), $p(t)={\psi }_3(t)+{\psi }_4(t)\cos x_2(t)$ for every $t\in [0,T]$. Note that ${\alpha }_1(t)$ is always positive, and $p(t)$ and ${\alpha }_2(t)$ have the same sign.

Assume that the control u is optimal. It follows from Corollary 2 that for every t in any nonboundary interval of u

$$u(t)=\{\begin{array}{c}{u}_{\min },\hspace{1em}{\alpha }_2(t)>0\hfill \\ u_{\max },\hspace{1em}{\alpha }_2(t)<0.\hfill \end{array}$$

Let now $\theta$ be a boundary interval of u. We infer from Corollary 2 that if $t\in \theta$, then $u(t)=u_{\min }$ or ${\alpha }_2(t)\le 0$. If these relations are not satisfied at some $t\in \theta$, then the control u can be improved in accordance with Lemma 5, by means of a spike control variation described in Section 2. We deduce from Theorem 5 that if $t_1,t_2\in \theta$, $t_1\le t_2$, and $u(t_1)>u_{\min }$, then

(i)

$u(t_2)=u_{\max }$ if ${\alpha }_2(t_1)=0$ and ${\alpha }_2(t_2)<0$,

(ii)

$u(t_2)=u_{\min }$ if ${\alpha }_2(t_1)=0$ and ${\alpha }_2(t_2)>0$,

(iii)

$p(t_1)\le p(t_2)<0$ if ${\alpha }_2(t_2)\ne 0$ and $u_{\min }<u(t_2)<u_{\max }$.

If some of the necessary conditions of Theorem 5 or Corollary 4 are not fulfilled, then—as follows from Lemma 6—the control u can be improved with the use of a two-spike control variation (Section 4).

We shall now numerically analyze two cases, taking $x_0=\mathrm{col}(-0.4,\hspace{0.17em}3.5,\hspace{0.17em}1,-1.1)$, $u_{\min }=-4$, $u_{\max }=4$, $T=1.5$, and $x_{3\max }=1$.

Example 2a.

The optimal control in the considered problem is of the form

$$u(t)=\{\begin{array}{c}{u}_{\min },\hspace{1em}\hspace{1em}\hspace{1em}0\le t<s_1\hfill \\ u_{\max },\hspace{1em}\hspace{1em}\hspace{1em}{s}_1\le t<s_2\hfill \\ w(x(t)),\hspace{1em}\hspace{1em}{s}_2\le t<s_3\hfill \\ u_{\min },\hspace{1em}\hspace{1em}\hspace{1em}{s}_3\le t\le T,\hfill \end{array}$$

where $s_1=0.20359164$, $s_2=0.36709680$, $s_3=1.1925492$, with $Q(u)=2.6850568$. Figure 9 shows the control u, the switching function ${\alpha }_2$, and the function p (25). It is easy to see that the necessary optimality conditions of Theorems 4 and 5 are fulfilled in the whole interval $[0,T]$. The optimal state trajectory is depicted in Figure 10. Notice the cusps of $x_3$ at $t_{\mathrm{en}}=s_2$ and $t_{\mathrm{ex}}=s_3$, indicating that u satisfies the nontangentiality conditions and is verifiable.

Example 2b.

Consider a verifiable, but nonoptimal control with a boundary interval

$$u(t)=\{\begin{array}{c}w(x(t)),\hspace{1em}0\le t<s_1\hfill \\ u_{\min },\hspace{1em}\hspace{1em}{s}_1\le t\le T\end{array}, s_1=1.1825443$$

The corresponding value of cost is $Q(u)=2.7408438$. Figure 11 presents the control u, the switching function ${\alpha }_2$, and the function p (25). It can be seen that the necessary conditions of Theorem 4 and Corollary 2 are satisfied, which means that there are no one-spike control variations described in Section 2, guaranteeing an improvement of the cost. Figure 12 shows the state trajectory. The plot of $x_3$ has a cusp at the exit point $t_{\mathrm{ex}}=s_1$, and so the control u is verifiable. We can also see in Figure 11 that in the time interval [0, 0.2948] the function p is decreasing, hence it is possible to construct a two-spike control variation in that interval (according to Section 4) which reduces the cost. In order to verify this numerically, consider also Figure 13 which presents a contour plot of the difference $p({\tau }_2)-p({\tau }_1)$. Let for instance ${\tau }_1=0$, ${\tau }_2=0.2948$, $v_1=-4$, $v_2=4$ (red cross on the left y-axis). By (26), the parameter $\eta$ may have an arbitrary value from the interval $]0.5410,\hspace{0.17em}0.7134[$; we choose $\eta =0.713$. Figure 14 demonstrates the dependence of the cost increment on the width of the first spike $\epsilon$. The greatest improvement takes place at $\epsilon \cong 0.073$. For $\epsilon =0.073$, Figure 15 shows the state trajectory, and Figure 16, an enlargement of the plot of $x_3^{\epsilon }$.

9. Connections with Some Classical Results

There are essential connections between some of our results presented in Section 5 and Section 7, and certain classical results obtained by the so called indirect adjoining method, dating back to the works of R.V. Gamkrelidze, A.E. Bryson, H. Maurer, D.H. Jacobson, and many others (see [1,3,4,5]). As we have no space to discuss all similarities and analogies that can be found in the vast literature, we shall concentrate on one representative theorem due to H. Maurer [3]. We shall use a reduced version of that theorem, specialized to the case of state constraint of order one, verifiable control, fixed initial state and free final state.

Consider the optimal control problem formulated in Section 1, with the additional assumption that U is a closed interval with nonempty interior. Define

$$H^1(x,u,{\lambda }^1,{\eta }^1)={({\lambda }^1)}^{T}f(x,u)+{\eta }^1\dot{g}(x,u), {\lambda }^1\in R^n, {\eta }^1\in R.$$

Theorem 6 ([3], Theorem 5.1).

Let u be a verifiable optimal control and x, the corre-sponding state trajectory. Suppose that f and g are of class$C^2$, and let${\partial }_2\dot{g}(x(t),u(t))\ne 0$and$u(t)\in \mathrm{int} U$for every t in any boundary interval. Additionally, assume that there are finitely many entry points. Then there exist a number${\lambda }_0\ge 0$and functions${\lambda }^1:[0,T]\to R^n$,${\eta }^1:[0,T]\to R$such that

$${\dot{\lambda }}^1=-{\partial }_1{H}^1(x,u,{\lambda }^1,{\eta }^1)=-{\partial }_{1}f(x,u){\lambda }^1-{\eta }^1{\partial }_1\dot{g}(x,u)$$

(28)

$${\lambda }^1(T)={\lambda }_0\partial q(x(T)).$$

(29)

The following jump condition holds

$${\lambda }^1(t_{\mathrm{en}}+)={\lambda }^1(t_{\mathrm{en}}-)-{\beta }^1(t_{\mathrm{en}})\partial g(x(t_{\mathrm{en}})), {\beta }^1(t_{\mathrm{en}})\ge 0,$$

(30)

at every entry point$t_{\mathrm{en}}$, and${\lambda }^1$is continuous at every exit point.The function${\eta }^1$satisfies${\eta }^1(t)g(x(t))=0$on$[0,T]$and is a$C^1$function in the interior$]t_1,t_2[$of every boundary interval, given by

$${\eta }^1(t)=-{\lambda }^1{(t)}^T\frac{{\partial }_{2}f(x(t),u(t))}{{\partial }_2\dot{g}(x(t),u(t))}.$$

(31)

Moreover,${\eta }^1(t)\ge 0$and${\dot{\eta }}^1(t)\le 0$for$t_1<t<t_2$. It also holds that for a.e.$t\in [0,T]$

$$\underset{v\in U}{\min }{H}^1(x(t),v,{\lambda }^1(t),{\eta }^1(t))=H^1(x(t),u(t),{\lambda }^1(t),{\eta }^1(t))=const.$$

(32)

Let us first notice that if ${\lambda }^1$ in (31) is identical with the adjoint $\psi$, then the multiplier ${\eta }^1$ is equal to the function $-p$ given by (22) in Section 5. The function $-p$, similarly to ${\eta }^1$, is nonnegative and nonincreasing in every boundary interval. The adjoint Equation (28) is then identical with (11) almost everywhere. Indeed, in every boundary interval the Equation (11) takes the form

$$\dot{\psi }=-{\partial }_{1}f(x,w(x))\psi -\partial w(x){\partial }_{2}f{(x,w(x))}^T\psi .$$

By virtue of Section 1 and (22),

$$\partial w(x)=-\frac{{\partial }_1\dot{g}(x,u)}{{\partial }_2\dot{g}(x,u)}, p=\frac{{\psi }^T{\partial }_{2}f(x,u)}{{\partial }_2\dot{g}(x,u)}.$$

Hence if ${\eta }^1=-p$, then ${\eta }^1{\partial }_1\dot{g}(x,u)=\partial w(x){\partial }_{2}f{(x,w(x))}^T\psi$. The final condition (29) coin-cides with (12) if ${\lambda }_0=1$. This last equality is ensured in [3] by special regularity conditions. Further, it is evident that the jump condition (13) can be written in the form

$$\psi (t_{\mathrm{en}}-)=\psi (t_{\mathrm{en}}+)-\frac{\Delta H(t_{\mathrm{en}},u(t_{\mathrm{en}}-))}{\dot{g}(t_{\mathrm{en}},u(t_{\mathrm{en}}-))}\partial g(x(t_{\mathrm{en}})).$$

Thus, in this case (30) is identical with (13) if

$${\beta }^1(t_{\mathrm{en}})=-\frac{\Delta H(t_{\mathrm{en}},u(t_{\mathrm{en}}-))}{\dot{g}(t_{\mathrm{en}},u(t_{\mathrm{en}}-))}\ge 0.$$

In the control affine case discussed in Section 7 the condition (13) is readily transformed to

$$\psi (t_{\mathrm{en}}-)=\psi (t_{\mathrm{en}}+)-p(t_{\mathrm{en}}+)\partial g(x(t_{\mathrm{en}})),$$

with p determined by (25). Hence, we have ${\beta }^1(t_{\mathrm{en}})=-p(t_{\mathrm{en}}+)\ge 0$ by Corollary 4. We skip the proof of the inequality sign in the general case, which is more complicated. The identity of the adjoints ${\lambda }^1$ and $\psi$ entails the equivalence between the minimum conditions (32) and (23). In conclusion, the results obtained in this work for the case where in the boundary intervals $u(t)\in \mathrm{int}U$, ${\varphi }_{\max }(t)=$${\varphi }_{\min }(t)+\pi$ and $p(t)$ is given by (22) or (25), are in agreement with Theorem 5.1 in [3].

Finally, note that this work’s approach does not require that the optimal control in the boundary intervals takes values in the interior of U, whereas that assumption is essential in [3]. Also, in contrast to [3], we give an explicit representation of the jump of the adjoint function (Section 3, see also [11,12]).