Approximately Optimal Control of Nonlinear Dynamic Stochastic Problems with Learning: The OPTCON Algorithm

Abstract

OPTCON is an algorithm for the optimal control of nonlinear stochastic systems which is particularly applicable to economic models. It delivers approximate numerical solutions to optimum control (dynamic optimization) problems with a quadratic objective function for nonlinear economic models with additive and multiplicative (parameter) uncertainties. The algorithm was first programmed in C# and then in MATLAB. It allows for deterministic and stochastic control, the latter with open loop (OPTCON1), passive learning (open-loop feedback, OPTCON2), and active learning (closed-loop, dual, or adaptive control, OPTCON3) information patterns. The mathematical aspects of the algorithm with open-loop feedback and closed-loop information patterns are presented in more detail in this paper.

1. Introduction

Optimal control of stochastic processes is a topic which occurs in many contexts of applied mathematics such as engineering, biology, chemistry, economics, and management science. These techniques are used to steer the behavior of a plant, natural, human or social system over a certain (finite or infinite) time period where the system is driven by differential or difference equations whose time path can be influenced by an external controller. The controller (decision maker, policy maker) aims at achieving the best trajectory of his/her controls according to some criterion (performance measure, objective function) over that time horizon.

Unfortunately, only a few special cases can be solved explicitly, in particular the linear-quadratic-Gaussian problem, where the dynamic system is linear, the objective function is quadratic, and the stochastics are confined to additive normally distributed noise affecting the system dynamics (e.g., [1]). In this case, it is possible to separate the estimation of the state and the optimization of its behavior due to the property of certainty equivalence (the separation theorem). For most practical applications, however, especially those where the parameters of the system are not known precisely, an exact solution to the optimal stochastic control problem is not available and some kind of approximation is required. One of the reasons is the curse of dimensionality, which prevents the solving of dynamic functional equations for multivariable systems [2,3]. This is particularly complicated by the so-called dual effect of controls preventing the separation of estimation and control in adaptive processes; it was discovered by [4] that “the control must have a reciprocal, ‘dual’ character: to a certain extent they must be of a probing nature, but also controlling to a known degree” ([4], p. 31). The dynamic functional equations for general problems of this kind are known (see, e.g., [5]) but cannot be solved except in very special cases; hence approximations are required (see [6,7,8]).

In this paper, we present the OPTCON algorithm for calculating numerically (approximately) optimal control solutions to nonlinear dynamic stochastic optimization problems without and with learning. The algorithm was developed in three stages (OPTCON1, OPTCON2, and OPTCON3) over a long period. OPTCON1 determines optimal controls for deterministic problems and stochastic problems based on the open-loop information pattern [9]. OPTCON2 solves problems with passive learning about the unknown parameters of the system model [10], so-called open-loop feedback policies (see [11] for the terminology). Finally, OPTCON3 also includes active learning (dual, adaptive control) according to an approximation procedure initiated by [12,13,14], and adapted to linear economic systems by [15,16], cf. [17]. The present paper provides mathematical details for OPTCON3, which is the most sophisticated version of the OPTCON algorithm; in [18] we concentrate on computations aspects and applications of OPTCON3.

The OPTCON algorithm is applicable for stochastic control problems with the following properties: The model of the process is multivariable, formulated in discrete time, and described by a system of nonlinear difference equations with known functional form but additive noise and (possibly) unknown parameters. The state is stochastically observable and controllable. No inequality constraints on states or controls are given. Open-loop, open-loop feedback, and closed-loop information structures can be considered. The objective function is quadratic and perfectly known; it is formulated in tracking form but can easily be transformed to the general quadratic form. Quadratic functions can be interpreted as second-order Taylor approximations to more general functional forms. There is only one decision maker; for decentralized problems, see the OPTGAME algorithm [19].

The paper has the following structure. In Section 2 the relevant basic background is provided, namely the class of problems to be solved by the algorithm as well as some basic information about the linear-quadratic framework and nonlinear system solving algorithms. Section 3 presents the OPTCON algorithm stepwise, starting with the basic version (OPTCON1) and then introducing the more advanced versions OPTCON2 and OPTCON3 for handling stochastic components in the optimal control process. In Section 4 some details on computational time and accuracy of the OPTCON algorithm are presented. Section 5 concludes.

2. Theoretical Background

2.1. Optimal Control Problem

We consider optimal control problems with a quadratic objective function and a nonlinear multivariate discrete-time dynamic system under additive and parameter uncertainties. The basis for an optimal control problem is a deterministic dynamic system to be controlled (plant, firm, economy, etc.) in discrete time in the form:

$$x_t=f(x_{t-1},x_t,u_t,z_t),t=1,\dots ,T,$$

(1)

where:

-

$x_t\in R^n$ is a vector of state variables that describes the state of the system at any point in time t,

-

$u_t\in R^m$ is a vector of control variables; we assume that the decision maker determines the values of the control variables exactly (without randomization) according to the approximately optimal solution of the problem,

-

$z_t\in R^l$ denotes a vector of non-controlled deterministic exogenous variables, whose values are known to the decision maker at time t,

-

T denotes the terminal time period of the finite planning horizon.

The function f is assumed to be twice continuously differentiable with respect to all of its arguments.

We assume that there is some true law of motion given by Equation (1) in the background which is at least partially unknown to the policy maker while the function f is known to him/her. The policy maker faces two sources of uncertainty, additive and parameter uncertainties where:

-

$\theta \in R^p$ denotes a vector of stochastic variables of system parameters (parameter uncertainty),

-

$\overline{\theta }\in R^p$ is a vector of true parameters whose values are assumed to be constant but unknown to the policy maker,

-

${\epsilon }_t\in R^n$ is a vector of additive stochastic disturbances (system error).

${\theta }_t$ and ${\epsilon }_t$ are assumed to be independent Gaussian random vectors with expectations $\widehat{\theta }$ and $O_n$ respectively and covariance matrices ${\Sigma }^{\theta \theta }$ and ${\Sigma }^{\epsilon \epsilon }$ respectively.

Including uncertainty, the system (1) can be written as:

$$x_t=f(x_{t-1},x_t,u_t,{\theta }_t,z_t)+{\epsilon }_t,t=1,\dots ,T.$$

(2)

The optimal control problem to be solved approximately assumes that there is a modeler who wishes to control the system (2). It means that the controller wishes to bring the state variables, using the control variables, as close as possible to some pre-defined desired time path, according to a given optimality criterion. The OPTCON algorithm allows for the optimal control of the system (2) using a quadratic objective function. To this end the modeler needs to define the following variables:

-

${\tilde{x}}_t\in R^n$ are given target (‘ideal’) values (for $t=1,\dots ,T$) of the state variables,

-

${\tilde{u}}_t\in R^m$ are given target (‘ideal’) values (for $t=1,\dots ,T$) of the control variables,

-

$W_t$ is an $\left(\right(n+m)\times (n+m\left)\right)$ symmetric matrix defining the relative weights of the state and control variables in the objective function. In a frequent special case, $W_t$ includes a discount factor $\alpha$ with $W_t={\alpha }^{t-1}W$.

The resulting intertemporal objective function is given in quadratic tracking form:

$$J=E\left[\sum _{t=1}^T{L}_t(x_t,u_t)\right],$$

(3)

with

$$L_t(x_t,u_t)=\frac{1}{2}{\left(\begin{array}{c}{x}_t-{\tilde{x}}_t\\ u_t-{\tilde{u}}_t\end{array}\right)}^{\prime }{W}_t\left(\begin{array}{c}{x}_t-{\tilde{x}}_t\\ u_t-{\tilde{u}}_t\end{array}\right).$$

(4)

The OPTCON algorithm allows us to find approximate optimal control solutions which minimize the functions (3) and (4) and satisfy the system dynamics (2).

2.2. Linear-Quadratic Optimal Control (LQ) Framework

The novel feature of the OPTCON algorithm is the combination of a nonlinear dynamic system and the above-mentioned stochastics when solving optimal control problems. Before we can discuss this technique in detail, a few words should be said about the underlying linear-quadratic optimal control framework. The dynamic system is given in linear form as

$$x_t=A{x}_{t-1}+B{u}_t+c_t.$$

(5)

The corresponding optimal control objective function can be written in the same form as in (3) and (4). Using well-known LQ optimization techniques, the rules for defining control variables can be found iteratively backwards in time using dynamic Riccati equations. We start from the familiar LQ framework (e.g., [20]), which was adapted to economic models by [15,21,22]. The same standard technique is applied in the OPTCON algorithm whenever an optimal control for a linearized system should be obtained.

Furthermore, for computational reasons it is useful to transfer the objective function from the quadratic tracking form as used in Equation (4) into the general quadratic form:

$$L_t(x_t,u_t)=\frac{1}{2}{\left(\begin{array}{c}{x}_t\\ u_t\end{array}\right)}^{\prime }{W}_t\left(\begin{array}{c}{x}_t\\ u_t\end{array}\right)+{\left(\begin{array}{c}{x}_t\\ u_t\end{array}\right)}^{\prime }\left(\begin{array}{c}{w}_t^x\\ w_t^u\end{array}\right)+w_t^c,$$

(6)

where

$$\left(\begin{array}{c}{w}_t^x\\ w_t^u\end{array}\right)=-W_t\left(\begin{array}{c}{\tilde{x}}_t\\ {\tilde{u}}_t\end{array}\right)and{w}_t^c=\frac{1}{2}{\left(\begin{array}{c}{\tilde{x}}_t\\ {\tilde{u}}_t\end{array}\right)}^{\prime }{W}_t\left(\begin{array}{c}{\tilde{x}}_t\\ {\tilde{u}}_t\end{array}\right).$$

The equivalence between the quadratic tracking form and the general quadratic form is shown, for instance, in [23].

2.3. Solving Nonlinear Dynamic Systems

The OPTCON algorithm allows us to find approximately optimal control solutions for nonlinear stochastic systems. In the solution process, the nonlinear dynamic system (2) should be solved in a large number of iterations. For this reason the choice of an appropriate solver is important to guarantee obtaining reliable solutions taking the computational costs into account. In the OPTCON algorithm, different solution algorithms are included attaching different importance to the trade-off between reliability and computational speed. At the moment the following methods can be used: Levenberg–Marquardt [24], trust region [25], Newton–Raphson [26], or Gauss–Seidel [27].

3. The OPTCON Algorithm

3.1. The OPTCON1 Algorithm

The first version of the OPTCON algorithm, the OPTCON1 algorithm, was developed by [9]. It allows us to calculate an open-loop (OL) solution to a nonlinear stochastic dynamic optimal control problem with a quadratic objective function under additive and parameter uncertainties. Open-loop controls either do not take account of the effect of uncertainties in the system or assume the stochastics (expectation and covariance matrices of additive and multiplicative disturbances) to be given for all time periods at the beginning of the planning horizon. The basic idea behind this algorithm is that it extends linear-quadratic stochastic optimal control techniques (see, e.g., [1]) to nonlinear problems using an iterative method of linear approximations.

The OPTCON algorithm solves the nonlinear optimal control problem iteratively by a sequence of linear approximations where the next approximation is derived using the optimal control solution of the previous one. The (optimal) solutions of the intermediate linear approximations are iterated from one time path to another until the algorithm converges. The criterion for convergence is that the difference in the values of the state and control variables of current and previous iterations is smaller than a pre-specified number. or the maximal number of iterations is reached. In each iteration the same procedure is conducted, namely linearizing and optimizing the linearized system. The system is linearized around the previous iteration’s result and the problem is solved for the resulting time-varying linearized system. The solution of the linearized model is obtained via Bellman’s principle of optimality and is used as the tentative path for the next iteration, starting off the procedure all over again. If the OPTCON algorithm converges, the solution of the last iteration is taken as the approximately optimal solution to the nonlinear problem and the value of the objective function is calculated. Figure 1 shows the structure of the OPTCON1 algorithm.

3.2. The OPTCON2 Algorithm

The second version of the algorithm (called OPTCON2, see [10,23] for more details) extends the first version with a passive learning strategy (open-loop feedback (OLF)). OLF uses the idea of re-estimating the model at the end of each time period $t=1,\dots ,T$. For that purpose, the model builder (policy maker) observes the current situation and uses the new information, namely the current values of the state variables, to improve his/her knowledge of the system.

Recall that the stochastic system includes two kinds of uncertainties, namely additive (random system errors) and multiplicative (‘structural’ errors in parameters). We assume that there is some true law of motion in the background which is unknown to the policy maker. The passive learning strategy deals with the ‘true’ parameters $\overline{\theta }$ which generate the model. However, the policy maker does not know these true parameters $\overline{\theta }$ and works with the ‘wrong’ estimated parameters $\widehat{\theta }$. The idea of the OLF strategy is to observe the outcome of the model in each time period and use this information to bring the estimated parameters $\widehat{\theta }$ closer to the true values $\overline{\theta }$.

The OLF strategy of the OPTCON2 algorithm consists of the following (rough) steps running in a forward loop.

In each time period S ($S=1,\cdots ,T$) do the following:

• Find an (approximately) optimal open-loop solution for the remaining subproblem (for the time periods from S to T) using the OPTCON1 algorithm.
• Fix the predicted solution for time period S, namely $x_S^{\ast }$ and $u_S^{\ast }$.
• Observe the realized state variables $x_S^{a\ast }$ which result from the chosen control variables ($u_S^{\ast }$), the true law of motion with parameter vector $\overline{\theta }$, and the realization of the random disturbances ${\epsilon }_S$. The main difference between $x_S^{a\ast }$ and $x_S^{\ast }$ is that the former is driven by the true dynamic process with parameter vector $\overline{\theta }$ and the latter by the estimated dynamic system with $\widehat{\theta }$.
• Update the parameter estimate $\widehat{\theta }$ (via the Kalman filter and using the difference between $x_S^{a\ast }$ and $x_S^{\ast }$) and use it in the next iteration step $S+1$.

The last step is carried out using the idea of the Kalman filter (see, e.g., [28] or [29]). The updating procedure according to the Kalman filter consists of two distinct phases, prediction and correction. First (the prediction phase), the predicted values of the state variables ${\widehat{x}}_{S/S-1}$, the vector of parameters ${\widehat{\theta }}_{S/S-1}$, and the covariances of the parameters ${\Sigma }_{S/S-1}^{\theta \theta }$ are calculated using the estimates from the previous time period. Next (the correction phase), these ‘a priori’ values of the state variables, the vector of the parameters, and the values of the covariances are corrected using the current observations of the state variables.

The phase of calculating the predicted values of ${\widehat{x}}_{S/S-1}$ and ${\widehat{\theta }}_{S/S-1}$ is integrated in the previous steps of the OPTCON2 algorithm. At the end of time period S the predicted values of ${\widehat{x}}_{S/S-1}=x_S^{\ast }$ are calculated and ${\widehat{\theta }}_{S/S-1}={\widehat{\theta }}_{S-1}$ is known.

The correction phase is reduced to the calculation of the parameter estimates ${\widehat{\theta }}_{S/S}$ and the parameter covariances ${\Sigma }_{S/S}^{\theta \theta }$ because the ‘corrected’ values of the state variables $x_S^{a\ast }=f(x_{S-1}^{a\ast },x_S^{a\ast },u_S^{\ast },\overline{\theta },z_S)+{\epsilon }_S$ are calculated in the previous step of the algorithm or simply observed. As explained before, the updating procedure is based on the difference between $x_S^{\ast }=f(x_{S-1}^{a\ast },x_S^{\ast },u_S^{\ast },\widehat{\theta },z_S)$ and $x_S^{a\ast }$ at the end of each time period.

The OPTCON2 algorithm allows us to find an approximate solution of the optimal control problem using a passive learning strategy. The policy maker uses real observations in each time period to update his/her knowledge about the stochastic dynamic system. In many cases it allows him/her to obtain more reliable results (see [30] for a performance study on optimal control with a passive learning strategy).

3.3. The OPTCON3 Algorithm

In this section we present the detailed description of the OPTCON3 algorithm. As mentioned above, the latest version of the OPTCON algorithm includes an active learning strategy. In the literature this strategy is also called closed-loop, adaptive, or dual control. The passive learning method in the OPTCON2 algorithm uses current observations to update the information about the dynamic system. The active learning strategy in OPTCON3 actively uses system perturbations to improve the optimal control performance by reducing the uncertainty in the future.

The procedure for finding the closed-loop solution in the OPTCON3 algorithm is based on the linear active learning control method in [15]. Similar to the iterative structure in the OPTCON2 algorithm, the optimization runs in a forward loop ($S=1,\cdots ,T$). However, in each time period, more information about future measurements is used. To this end the objective function J (as defined in Equations (3) and (4)) is extended for stochastic components. We define in each time period $J_d$ as the approximate total cost-to-go with $T-S$ periods remaining. The approximate cost-to-go is broken down into three terms: $J_d=J_D+J_C+J_P$. $J_D$ is called deterministic term and incorporates only non-stochastic components. $J_C$ is the cautionary factor that includes the stochastic elements in the current period. $J_P$ is called probing term and captures the effect of dual learning on the uncertainty in the future periods. $J_C$ and $J_P$ constitute a separate quadratic minimization problem constrained by the nonlinear system. The original system needs to be expanded to the perturbation form $\delta x_t$. The optimization problem has to be solved for the perturbed system, where $\Delta J_t^{\ast }$ is the perturbed objective function. Using Taylor expansion of nonlinear system and Bellman optimization technique, we obtain the solution $\Delta J_t^{\ast }$ as a quadratic function of $\delta x_{t-1}$. The original $J_t^{\ast }$ can be derived from $\Delta J_t^{\ast }$. To take uncertain parameters into account, all the terms and formulas need to be adjusted to the augmented system $\left[\begin{array}{c}{x}_t\\ ........\\ \theta \end{array}\right]$.

Next, a schematic structure of the OPTCON3 algorithm is presented. This structure is visualized in Figure 2.

The optimization is carried out in a forward loop from 1 to T conducting following procedure in each time period S ($S=1,\dots ,T$). The subproblem from S to T is solved via the open-loop (OL) strategy (see Figure 1 in Section 3.1). The OL solution $(x_S^{\ast },u_S^{\ast })$ for the time period S is fixed. After that the core part of the dual control strategy starts, where the system is actively perturbed to learn about it in order to minimize the uncertainty and the objective function values in the future. In the OPTCON3 algorithm a grid search method is used (instead of grid search many other methods can be applied, such as gradient optimization or a more advanced heuristic approach). In the so-called “$\pi$-loop” we create a grid of possible solutions around the existing path $(x_S^{\ast },u_S^{\ast })$. In each iteration ($\pi =1,\dots ,\Pi$), i.e., in each grid point, the optimal control solution is obtained and evaluated using the objective function. Inside the $\pi$-loop (for each $\pi$), the following steps are carried out.

An optimal open-loop solution for the subproblem from $S+1$ to T is determined. Then the OL solution $(x_{S+1}^{\ast \pi },u_{S+1}^{\ast \pi })$ for the time period $S+1$ is fixed. Next, after some auxiliary calculations (including, among other, Riccati matrices), the deterministic $J_D$, cautionary $J_C$, and probing $J_P$ terms of the cost-to-go are determined in a forward loop from $S+1$ to T. Once the iteration of the $\pi$-loop has been completed, the total approximate objective function $J_d=J_D+J_C+J_P$ can be obtained. When the search is completed, i.e., the approximately optimal path with $min{J}_d$ has been found, the obtained optimal control in time period $S+1$ is carried out by the policy maker. As a result of these controls and the true dynamic systems the actual values of the state variables can be obtained. This new information is used by the policy maker to update and to adjust the parameter estimate $\widehat{\theta }$, whereby the Kalman filter is used. Using the updated stochastic parameters, the same active learning procedure is applied for the remaining time periods from $S+2$ to T.

The OPTCON3 algorithm basically uses the technique presented in [13,15] but is augmented by approximating, in each step, the nonlinear system in a series of linear systems (iterative procedure as described in Section 3.1).

The following steps (I–IV) of the OPTCON3 algorithm describe how to obtain an approximately optimal dual control solution to a stochastic problem.

The algorithm needs the following input values (A legitimate question would be how to obtain the required (tentative) inputs. It obviously depends on the research question. In the case of some numerical experiments, one can resort to Monte Carlo simulations. In the case of a real-world application, one can estimate system parameters by an appropriate method or simply guess some values):

f	system function
$x_0={\stackrel{\circ }{x}}_0$	initial values of state variables
${\left({\stackrel{\circ }{u}}_t\right)}_{t=1}^T$	tentative path of control variables
${\widehat{\theta }}_0=\widehat{\theta }$	expected values of system parameters
${\Sigma }_0^{\theta \theta }={\Sigma }^{\theta \theta }$	covariance matrix of system parameters
${\Sigma }^{\epsilon \epsilon }$	covariance matrix of system noise
${\left({\epsilon }_t\right)}_{t=1}^T$	system noises
${\left(z_t\right)}_{t=1}^T$	path of exogenous variables
${\left({\tilde{x}}_t\right)}_{t=1}^T$	target path for state variables
${\left({\tilde{u}}_t\right)}_{t=1}^T$	target path for control variables
$W^{xx}$, $W^{ux}$, $W^{uu}$	weighting matrices of objective function
$w_t^u$, $w_t^x$	weighting vectors of objective function
$\alpha$	discount factor of objective function.

As a result of the algorithm, an approximately optimal solution ${\left(x_t^{a\ast }\right)}_{t=1}^T$, ${\left(u_t^{\ast }\right)}_{t=1}^T$ and the corresponding value of the objective function $J^{\ast }$ are obtained.

Step I: Do the following search steps [1], [2] and [3] for each $S=1,\dots ,T$:

Step I-[1]: Find the OL solution for the subproblem $(S,\dots ,T)$: apply the procedure already used in OPTCON1. Fix $(x_S^{\ast },u_S^{\ast })$.

Step I-[2]: Run a grid search of size $\Pi$ around $(x_S^{\ast },u_S^{\ast })$, i.e., perform steps (A)–(G) for each $\pi =1,\dots ,\Pi$:

Step I-2A: Find the OL solution ${(x_t^{\ast \pi },u_t^{\ast \pi })}_{S+1}^T$ for the subproblem ($S+1$, …, T).

• The nonlinearity loop is run until the stop criterion is fulfilled (the stop criterion is fulfilled when the difference between the values of the current and the previous iteration is smaller than a pre-specified number or the maximum number of iterations is achieved). As a result the approximately optimal solution ${(x_t^{\ast \pi },u_t^{\ast \pi })}_{S+1}^T$ has been found. Then go to the next step I-2B.

Notes:

-

After several runs of the nonlinearity loop, only the solution ($x_{S+1}^{\ast \pi },u_{S+1}^{\ast \pi })$ for the time period $S+1$ will be taken as the optimal (nominal) solution. The calculations of the pairs $(x_{t^{\prime }}^{\ast \pi },u_{t^{\prime }}^{\ast \pi })$ for other periods ($t^{\prime }$ > $S+1$) have to be done again, taking into account the re-estimated parameters for all periods.

-

After linearization the parameter matrices for the linearized system are obtained: $A_t={(I-F_{x_t}^x)}^{-1}{F}_{x_{t-1}}^x$ and $B_t={(I-F_{x_t}^x)}^{-1}{F}_{u_t}^x$, where $F_{x_t}^x$, $F_{x_{t-1}}^x$, and $F_{u_t}^x$ are the derivatives of the system function f with respect to $x_t$, $x_{t-1}$, and $u_t$ respectively.

Step I-2B: Do a backward recursion to obtain auxiliary matrices for calculating components of the objective function.

Initialize the following auxiliary matrices for backward recursion:

$$\begin{array}{c}{H}_{T+1}^{xx}=O_{n\times n},h_{T+1}^x=O_n,\\ H_{T+1}^{x\theta }=O_{n\times p},H_{T+1}^{\theta \theta }=O_{p\times p}.\end{array}$$

Calculate the Riccati matrices $K_t^{xx}$, $K_t^{\theta x}$, and $K_t^{\theta \theta }$ and the auxiliary matrices ${\Lambda }_t^{xx}$, ${\Lambda }_t^{xu}$, ${\Lambda }_t^{uu}$, ${\lambda }_t^x$, ${\lambda }_t^u$, $H_t^{xx}$, $H_t^{\theta x}$, $H_t^{\theta \theta }$, and $h_t^x$ backwards in time from T to $S+1$.

$$\begin{array}{c}{K}_t^{xx}=W_t^{xx}+H_{t+1}^{xx},K_t^{\theta x}=H_{t+1}^{\theta x},K_t^{\theta \theta }=H_{t+1}^{\theta \theta },\hfill \\ k_t^x=h_{t+1}^x+x^{\prime }{W}_t^{xx}+W_t^{xu}u+w_t^x.\hfill \end{array}$$

(7)

$$\begin{array}{c}{\Lambda }_t^{xx}={\left(A_t\right)}^{\prime }{K}_t^{xx}{A}_t,\hfill \\ {\Lambda }_t^{ux}={\left(B_t\right)}^{\prime }{K}_t^{xx}{A}_t+W_t^{ux}{A}_t,\hfill \\ {\Lambda }_t^{xu}={\left({\Lambda }_t^{ux}\right)}^{\prime },\hfill \\ {\Lambda }_t^{uu}={\left(B_t\right)}^{\prime }{K}_t^{xx}{B}_t+2{\left(B_t\right)}^{\prime }{W}_t^{xu}+W_t^{uu}.\hfill \end{array}$$

(8)

$$\begin{array}{c}{\lambda }_t^x=k_t^x{A}_t,\hfill \\ {\lambda }_t^u=k_t^x{B}_t+x^{\prime }{W}_t^{xu}+u^{\prime }{W}_t^{uu}+{\left(w_t^u\right)}^{\prime }.\hfill \\ \hfill \end{array}$$

(9)

$$\begin{array}{c}{H}_t^{xx}={\Lambda }_t^{xx}-{\Lambda }_t^{xu}{\left({\Lambda }_t^{uu}\right)}^{-1}{\Lambda }_t^{ux},\hfill \\ \\ H_t^{\theta x}=[D^{\prime }{K}_t^{xx}+K_t^{\theta x}]A_t-[[D^{\prime }{K}_t^{xx}+K_t^{\theta x}]B_t+D{W}^{xu}]{\left({\Lambda }_t^{uu}\right)}^{-1}{\Lambda }_t^{ux},\hfill \\ \\ H_t^{\theta \theta }=D^{\prime }(K_t^{xx}D+K_t^{x\theta })+K_t^{\theta x}D+K_t^{\theta \theta }-[[D^{\prime }{K}_t^{xx}+K_t^{\theta x}]B_t+D{W}^{xu}]\hfill \\ \times {\left({\Lambda }_t^{uu}\right)}^{-1}[B_t[K_t^{xx}D+K_t^{x\theta }]+W^{ux}D],\hfill \\ \\ h_t^x={\lambda }_t^x-{\Lambda }_t^{xu}{\left({\Lambda }_t^{uu}\right)}^{-1}{\lambda }_t^u,\hfill \end{array}$$

(10)

where $D={(I-F_{x_t}^x)}^{-1}{F}_{\theta }^x$.

$F_{\theta }^x$ is the derivative of the system function with respect to $\theta$.

Step I-2C: Calculate the deterministic component of the approximate objective function $J_{D,S}$ and the cautionary component $J_{C,S}$:

$$\begin{array}{c}{J}_{D,S}=\frac{1}{2}{[x_S^{\ast \pi }-{\tilde{x}}_S]}^{\prime }{W}^{xx}[x_S^{\ast \pi }-{\tilde{x}}_S]+{[x_S^{\ast \pi }-{\tilde{x}}_S]}^{\prime }{W}^{xu}[u_S^{\ast \pi }-{\tilde{u}}_S]\hfill \\ \\ +\frac{1}{2}{[u_S^{\ast \pi }-{\tilde{u}}_S]}^{\prime }{W}^{uu}[u_S^{\ast \pi }-{\tilde{u}}_S]\hfill \\ \mathrm{and}\hfill \\ J_{C,S}=\frac{1}{2}tr\left(H_{S+1}^{xx}{\Sigma }_{S+1/S}^{xx}\right)+tr\left(H_{S+1}^{\theta x}{\Sigma }_{S+1/S}^{x\theta }\right)+\frac{1}{2}tr\left(H_{S+1}^{\theta \theta }{\Sigma }_{S+1/S}^{\theta \theta }\right).\hfill \end{array}$$

(11)

Step I-2D: Repeat steps [a]–[c] for each $j=S+1,\dots ,T$:

[a]:

Calculate the components $J_{D,j}$ and $J_{C,j}$:

$$\begin{array}{c}{J}_{D,j}=\frac{1}{2}{[x_j^{\ast \pi }-{\tilde{x}}_j]}^{\prime }{W}^{xx}[x_j^{\ast \pi }-{\tilde{x}}_j]+{[x_j^{\ast \pi }-{\tilde{x}}_j]}^{\prime }{W}^{xu}[u_j^{\ast \pi }-{\tilde{u}}_j]\hfill \\ +\frac{1}{2}{[u_j^{\ast \pi }-{\tilde{u}}_j]}^{\prime }{W}^{uu}[u_j^{\ast \pi }-{\tilde{u}}_j]),\hfill \\ J_{C,j}=\frac{1}{2}tr\left(K_j^{xx}{\Sigma }^{\xi \xi }\right).\hfill \end{array}$$

(12)

[b]:

Calculate the matrix ${\Sigma }_{j/j}^{\theta \theta }$:

$$\begin{array}{c}{\Sigma }_{j/j}^{\theta \theta }={\Sigma }_{j/j-1}^{\theta \theta }-{\Sigma }_{j/j-1}^{\theta x}{\left({\Sigma }_{j/j-1}^{xx}\right)}^{-1}{\Sigma }_{j/j-1}^{x\theta }\hfill \end{array}$$

(13)

with

$$\begin{array}{c}{\Sigma }_{j/j-1}^{xx}=F_{\theta }^x{\Sigma }_{j-1/j-1}^{\theta \theta }{\left(F_{\theta }^x\right)}^{\prime }+{\Sigma }_j^{\epsilon \epsilon }\hfill \\ \mathrm{and}\hfill \\ {\Sigma }_{j/j-1}^{x\theta }={\left({\Sigma }_{j/j-1}^{\theta x}\right)}^{\prime }=F_{\theta }^x{\Sigma }_{j-1/j-1}^{\theta \theta },{\Sigma }_{j/j-1}^{\theta \theta }={\Sigma }_{j-1/j-1}^{\theta \theta }.\hfill \end{array}$$

(14)

[c]:

Compute the probing component $J_{P_j}$:

$$\begin{array}{c}{J}_{P_j}=\frac{1}{2}tr\left[{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux}{\Sigma }_{j/j}^{xx}\right]\hfill \\ +tr\left[{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }(K^{xx}D+K^{x\theta })+W^{ux}D){\Sigma }_{j/j}^{\theta x}\right]\hfill \\ +\frac{1}{2}tr[((K^{\theta x}+D^{\prime }{K}^{xx})B+D^{\prime }{W}^{xu}){\left({\Lambda }^{uu}\right)}^{-1}\hfill \\ \times (B^{\prime }(K^{xx}D+K^{x\theta })+W^{ux}D){\Sigma }_{j/j}^{\theta \theta }].\hfill \end{array}$$

(15)

Step I-2E: Calculate the sum of the deterministic, cautionary, and probing terms over the periods $S,\dots ,T$:

$$J_d=(J_{D,S}+\sum _{j=S+1}^T{J}_{D,j})+(J_{C,S}+\sum _{j=S+1}^T{J}_{C,j})+\sum _{j=S+1}^T{J}_{P_j}.$$

Step I-2F: Take a new control ${\stackrel{\circ }{u}}_S={\stackrel{\circ }{u}}_S^{\pi +1}$ (a new point in the grid search) and go to step I-2A.

Step I-[3]: Choose an optimal $u_S^{\ast }$ with $minJ=J_d^{\ast }\left(u_S^{\ast }\right)$. End of grid search. Fix the corresponding $x_S^{\ast }$.

Step II: Calculate the following (a) and (b) for only one time period S:

(a)

$$\begin{array}{c}{\Sigma }_{S/S-1}^{xx}=F_{\theta }^x{\Sigma }_{S-1/S-1}^{\theta \theta }{\left(F_{\theta }^x\right)}^{\prime }+{\Sigma }_S^{\epsilon \epsilon }\hfill \\ \mathrm{and}\hfill \\ {\Sigma }_{S/S-1}^{x\theta }={\left({\Sigma }_{S/S-1}^{\theta x}\right)}^{\prime }=F_{\theta }^x{\Sigma }_{S-1/S-1}^{\theta \theta },{\Sigma }_{S/S-1}^{\theta \theta }={\Sigma }_{S-S/S-1}^{\theta \theta }.\hfill \end{array}$$

(16)

(b)

$$x_S^{a\ast }=f(x_{S-1}^{a\ast },x_S^{a\ast },u_S^{\ast },\overline{\theta },z)+{\epsilon }_S.$$

Step III: Update the parameter estimates $\widehat{\theta }$ and ${\Sigma }_{S/S}^{\theta \theta }$:

$$\begin{array}{c}{\widehat{\theta }}_{S/S}={\widehat{\theta }}_{S/S-1}+{\Sigma }_{S/S-1}^{\theta x}{\left({\Sigma }_{S/S-1}^{xx}\right)}^{-1}[x_S^{a\ast }-x_S^{\ast }]\mathrm{and}{\widehat{x}}_{S/S}=x_S^{a\ast }.\hfill \end{array}$$

(17)

$$\begin{array}{c}{\Sigma }_{S/S}^{\theta \theta }={\Sigma }_{S/S-1}^{\theta \theta }-{\Sigma }_{S/S-1}^{\theta x}{\left({\Sigma }_{S/S-1}^{xx}\right)}^{-1}{\Sigma }_{S/S-1}^{x\theta }.\hfill \end{array}$$

(18)

Step IV: Set $\widehat{\theta }={\widehat{\theta }}_{S/S}$ and ${\Sigma }^{\theta \theta }={\Sigma }_{S/S}^{\theta \theta }$, go to Step I and run the procedure for the time period $S+1$.

The OPTCON3 algorithm is finished when $S=T$ and the approximately optimal dual control and state variables have been found for all periods.

3.4. Computational Details for the AL Procedure

This subsection includes the technical computations for the AL procedure of the OPTCON3 algorithm.

The basic tasks are applying Bellman’s principle of optimality, linearization of the nonlinear system in perturbation form, minimizing the objective function, and inserting the obtained feedback rule for controls back into the objective function. In this way the (temporary) components of the optimal objective function can be derived. After that, due to the stochastic nature of the parameters ($\theta$), all the components have to be adjusted to the extended system $\left[\begin{array}{c}{x}_t\\ ........\\ \theta \end{array}\right]$.

Start with Bellman’s principle of optimality:

$$J_{T-t}^{\ast }=\underset{u_t}{min}E\{L_t(x_t,u_t)+J_{T-(t+1)}^{\ast }\left(x_t\right)\}.$$

(19)

The last term of (19) is

$$J_{T-(t+1)}^{\ast }\left(x_t\right)=\underset{u_{t+1}}{min}E\left\{\dots \underset{u_{T-1}}{min}E\{\sum _{j=t+1}^T{L}_j(x_j,u_j)/P^{T-1}\}\dots /P^{t+1}\}\right\},$$

where $P^t=({\widehat{x}}_{t/t},{\Sigma }_{t/t})$.

For ${\sum }_{j=t+1}^T{L}_j(x_j,u_j)$ use the (Taylor) series expansion:

$$\begin{array}{c}\sum _{j=t+1}^T{L}_j(x_j,u_j)=\sum _{j=t+1}^T{L}_j(x_{oj},u_{oj})+\sum _{j=t+1}^T(L_{jx}\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{L}_{jxx}\delta x_j\\ +\delta x_j^{{}^{\prime }}{L}_{jxu}\delta u_j+L_{ju}\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{juu}\delta u_j),\end{array}$$

where $L_{jx}$, $L_{ju}$ are the gradients and $L_{jxx}$, $L_{jxu}$, and $L_{juu}$ are the Hessian matrices; $\delta x_j=x_j-x_{oj}$, $\delta u_j=u_j-u_{oj}$, and $(x_{oj},u_{oj})$ is a nominal path.

Thus,

$$J_{T-(t+1)}^{\ast }\left(x_t\right)=J_{o,T-(t+1)}^{\ast }\left(x_t\right)+\Delta J_{T-(t+1)}^{\ast }\left(x_t\right)$$

(20)

and

$$\begin{array}{c}\Delta J_{T-(t+1)}^{\ast }\left(x_t\right)=\underset{u_{t+1}}{min}E\{\dots \underset{u_{T-1}}{min}E\{\sum _{j=t+1}^T(L_{jx}\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{L}_{jxx}\delta x_j\hfill \\ +\delta x_j^{{}^{\prime }}{L}_{jxu}\delta u_j+L_{ju}\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{juu}\delta u_j/P^{T-1}\}\dots /P^{t+1}\left)\right\}.\hfill \end{array}$$

(21)

Next, apply the Taylor expansion to the nonlinear system, i.e., linearize the system function f in (2) around the reference values:

$$x_t\approx f_t({\widehat{x}}_{t-1/t-1},u_t^{\ast },x_t)+F_{x_{t-1}}(x_{t-1}-{\widehat{x}}_{t-1/t-1})+F_u(u_t-u_t^{\ast })+F_{x_t}(x_t-{\widehat{x}}_{t/t})+{\epsilon }_t.$$

This can be rewritten in perturbation form:

$$\delta x_t=F_{x_{t-1}}\delta x_{t-1}+F_u\delta u_t+F_{x_t}\delta x_t+{\epsilon }_t$$

After transformation

$$\delta x_t={(I-F_{x_t})}^{-1}{F}_{x_{t-1}}\delta x_{t-1}+{(I-F_{x_t})}^{-1}{F}_{u_t}\delta u_t+{(I-F_{x_t})}^{-1}{\epsilon }_t.$$

(22)

Thus, (21) and (22) constitute the problem in perturbation form.

Next, we have to prove that the term (21) can be expressed in the quadratic form as a function of $\delta x_t$ only.

Assume

$$\Delta J_{T-(t+1)}^{\ast }\left(x_t\right)=g_{t+1}+E\{h_{t+1}^x\delta x_t+\frac{1}{2}\delta x_t^{{}^{\prime }}{H}_{t+1}\delta x_t/P^{t+1}\}$$

(23)

and apply the method of induction. By doing so the parameters g, $h^x$ and H are determined.

The next step is to prove that rule (23) is also true for $\Delta J_{T-j}^{\ast }\left(x_t\right)$$\forall j$.

Start with

$$J_{T-j}^{\ast }=\underset{u_j}{min}E\{L_j(x_j,u_j)+J_{T-(j+1)}^{\ast }\left(x_j\right)\}.$$

By second order expansion we get:

$$\begin{array}{cc}\hfill J_{T-j}^{\ast }=& \underset{u_j}{min}E\{L_j(x_{oj},u_{oj})+L_x\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{L}_{xx}\delta x_j+\delta x_j^{{}^{\prime }}{L}_{xu}\delta u_j+L_u\delta u_j\hfill \\ \hfill & +\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{uu}\delta u_j+(J_{o,T-(j+1)}^{\ast }+\Delta J_{T-(j+1)}^{\ast })\hfill \end{array}$$

$$\begin{array}{cc}\hfill J_{T-j}^{\ast }-L_j(x_{oj},u_{oj})-J_{o,T-(j+1)}^{\ast }=& \underset{u_j}{min}E\{L_x\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{L}_{xx}\delta x_j+\delta x_j^{{}^{\prime }}{L}_{xu}\delta u_j\hfill \\ \hfill & +L_u\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{uu}\delta u_j+\Delta J_{T-(j+1)}^{\ast }\}\hfill \end{array}$$

Use $L_j(x_{oj},u_{oj})+J_{o,T-(j+1)}^{\ast }=J_{o,T-j}^{\ast }$ and assumption (23)

$$\begin{array}{cc}\hfill J_{T-j}^{\ast }-J_{o,T-j}^{\ast }=& \underset{u_j}{min}E\{L_x\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{L}_{xx}\delta x_j+\delta x_j^{{}^{\prime }}{L}_{xu}\delta u_j+L_u\delta u_j\hfill \\ \hfill & +\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{uu}\delta u_j+g_{j+1}+E\{h_{j+1}^x\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}{H}_{j+1}\delta x_j/P^{j+1}/P^j\}\}\hfill \end{array}$$

Taking expectations over $P^{j+1}$ yields

$$\begin{array}{c}\Delta J_{T-j}^{\ast }:=J_{T-j}^{\ast }-J_{o,T-j}^{\ast }=\underset{u_j}{min}E\{(L_x+h_{j+1}^x)\delta x_j+\frac{1}{2}\delta x_j^{{}^{\prime }}(L_{xx}+H_{j+1})\delta x_j\\ \\ +\delta x_j^{{}^{\prime }}{L}_{xu}\delta u_j+L_u\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{uu}\delta u_j+g_{j+1}/P^j\}.\end{array}$$

(24)

Put (22) into (24) and set

$$L_x+h_{j+1}^x:=k_j^x,L_{xx}+H_{j+1}:=K_j,H_{T+1}=0.$$

(25)

Then by taking expectations over $P^j$

$$\begin{array}{cc}\hfill \Delta J_{T-j}^{\ast }=& \underset{u_j}{min}E\{k_j^x[{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\delta x_{j-1}+{(I-F_{x_j})}^{-1}{F}_u\delta u_j+{(I-F_{x_j})}^{-1}{\epsilon }_j]\hfill \\ \hfill & +\frac{1}{2}{[{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\delta x_{j-1}+{(I-F_{x_j})}^{-1}{F}_u\delta u_j+{(I-F_{x_j})}^{-1}{\epsilon }_j]}^{{}^{\prime }}{K}_j\hfill \\ \hfill & \times [{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\delta x_{j-1}+{(I-F_{x_j})}^{-1}{F}_u\delta u_j+{(I-F_{x_j})}^{-1}{\epsilon }_j]\hfill \\ \hfill & +{[{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\delta x_{j-1}+{(I-F_{x_j})}^{-1}{F}_u\delta u_j+{(I-F_{x_j})}^{-1}{\epsilon }_j]}^{{}^{\prime }}{L}_{xu}\delta u_j\hfill \\ \hfill & +L_u\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{L}_{uu}\delta u_j+g_{j+1}/P^{j-1}\}\hfill \end{array}$$

$$\begin{array}{cc}\hfill \Delta J_{T-j}^{\ast }=& \underset{u_j}{min}E\{k_j^x{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\delta x_{j-1}+[k_j^x{(I-F_{x_j})}^{-1}{F}_u+L_u]\delta u_j\hfill \\ \hfill & +\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}[F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}]\delta x_{j-1}\hfill \\ \hfill & +\delta x_{j-1}^{{}^{\prime }}[F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1}{F}_u+F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{L}_{xu}]\delta u_j\hfill \\ \hfill & +\frac{1}{2}\delta u_j^{{}^{\prime }}[F_u^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1})F_u+2F_u^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{L}_{xu}+L_{uu}]\delta u_j\hfill \\ \hfill & +\frac{1}{2}{\epsilon }_j^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1}{\epsilon }_j+g_{j+1}/P^{j-1}\}.\hfill \end{array}$$

The terms which are higher than second order are dropped here.

Then use the following notations:

$$\begin{array}{c}{\lambda }^x:=k_j^x[{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\hfill \\ {\lambda }^u:=k_j^x[{(I-F_{x_j})}^{-1}{F}_u+L_u\hfill \\ {\Lambda }^{xx}:=F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1}{F}_{x_{j-1}}\hfill \\ {\Lambda }^{xu}:=F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1}{F}_u+F_{x_{j-1}}^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{L}_{xu}\hfill \\ {\Lambda }^{uu}:=F_u^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{K}_j{(I-F_{x_j})}^{-1})F_u+2F_u^{{}^{\prime }}{({(I-F_{x_j})}^{-1})}^{{}^{\prime }}{L}_{xu}+Luu\hfill \\ {\xi }_j:={(I-F_{x_j})}^{-1}{\epsilon }_j\hfill \end{array}$$

(26)

and get

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=\underset{u_j}{min}E\{{\lambda }^x\delta x_{j-1}+{\lambda }^u\delta u_j+\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta x_{j-1}+\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xu}\delta u_j\\ \\ +\frac{1}{2}\delta u_j^{{}^{\prime }}{\Lambda }^{uu}\delta u_j+\frac{1}{2}{\xi }_j^{{}^{\prime }}{K}_j{\xi }_j+g_{j+1}/P^{j-1}\}.\end{array}$$

(27)

Apply the rule $E\left(x^{\prime }Ax\right)={\widehat{x}}^{\prime }A\widehat{x}+tr\left(A{\Sigma }^{xx}\right)$ and take expectations over $P^{j-1}$:

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=\underset{u_j}{min}E\{{\lambda }^x\delta {\widehat{x}}_{j-1/j-1}+{\lambda }^u\delta u_j+g_{j+1}+\frac{1}{2}\delta {\widehat{x}}_{j-1/j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta {\widehat{x}}_{j-1/j-1}\\ \\ +\delta {\widehat{x}}_{j-1/j-1}^{{}^{\prime }}{\Lambda }^{xu}\delta u_j+\frac{1}{2}\delta u_j^{{}^{\prime }}{\Lambda }^{uu}\delta u_j+\frac{1}{2}tr({\Lambda }^{xx}{\Sigma }_{j/j-1}^{xx})+\frac{1}{2}tr(K_j{\Sigma }^{\xi \xi })\},\end{array}$$

(28)

where $E\left(\xi \right)=0$.

Next, minimize the objective function $\Delta J_{T-j}^{\ast }$ with respect to $\delta u_j$:

$$\begin{array}{c}\frac{\Delta J_{T-j}^{\ast }}{\delta u_j}={\lambda }^u+\delta {\widehat{x}}_{j-1/j-1}^{{}^{\prime }}{\Lambda }^{xu}+\delta u_j^{{}^{\prime }}{\Lambda }^{uu}=0\\ \\ \delta u_j=-{\left({\Lambda }^{uu}\right)}^{-1}({\lambda }^u+{\Lambda }^{xu}\delta {\widehat{x}}_{j-1/j-1})\end{array}$$

(29)

Put rule (29) into (27):

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=E\{{\lambda }^x\delta x_{j-1}+{\lambda }^u[-{({\Lambda }^{uu})}^{-1}({\lambda }^u+{\Lambda }^{xu}\delta {\widehat{x}}_{j-1/j-1})]+\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta x_{j-1}\hfill \\ \\ +\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xu}[-{({\Lambda }^{uu})}^{-1}({\lambda }^u+{\Lambda }^{xu}\delta {\widehat{x}}_{j-1/j-1})]+\frac{1}{2}{[-{({\Lambda }^{uu})}^{-1}({\lambda }^u+{\Lambda }^{xu}\delta {\widehat{x}}_{j-1/j-1})]}^{{}^{\prime }}{\Lambda }^{uu}\hfill \\ \\ \times [-{({\Lambda }^{uu})}^{-1}({\lambda }^u+{\Lambda }^{xu}\delta {\widehat{x}}_{j-1/j-1})]+\frac{1}{2}{\xi }^{{}^{\prime }}{K}_j\xi +g_{j+1}\}\hfill \end{array}$$

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=E\{g_{j+1}+({\lambda }^x+{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\lambda }^u)\delta x_{j-1}-\frac{1}{2}{\lambda }^u{({\Lambda }^{uu})}^{-1}{\lambda }^u+\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta x_{j-1}\hfill \\ \\ +\frac{1}{2}{\widehat{x}}_{j-1/j-1}^{{}^{\prime }}{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\Lambda }^{ux}{\widehat{x}}_{j-1/j-1}-\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\Lambda }^{ux}\delta {\widehat{x}}_{j-1/j-1}+\frac{1}{2}{\xi }^{{}^{\prime }}{K}_j\xi \}\hfill \end{array}$$

Define

$${\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux}:={\Omega }_{xx}.$$

(30)

Note that $\frac{1}{2}{\xi }^{{}^{\prime }}{K}_j\xi =\frac{1}{2}[E\left(\xi \right)K_{j}E\left(\xi \right)+tr\left(K_j{\Sigma }^{\xi \xi }\right)]=\frac{1}{2}tr\left(K_j{\Sigma }^{\xi \xi }\right)$ because $E\left(\xi \right)=0$. Moreover $E\{\delta x_{j-1}^{{}^{\prime }}{\Omega }_{xx}\delta {\widehat{x}}_{j-1/j-1}\}=\delta {\widehat{x}}_{j-1/j-1}{\Omega }_{xx}\delta {\widehat{x}}_{j-1/j-1}$. Thus,

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=g_{j+1}-\frac{1}{2}{\lambda }^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }^u+\frac{1}{2}tr\left(K_j{\Sigma }^{\xi \xi }\right)+E\{({\lambda }^x+{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\lambda }^u)\delta x_{j-1}\hfill \\ \\ +\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta x_{j-1}\}-\frac{1}{2}\delta {\widehat{x}}_{j-1/j-1}{\Omega }_{xx}\delta {\widehat{x}}_{j-1/j-1}.\hfill \end{array}$$

Use the following rule:

$$\begin{array}{c}E\left\{\delta x_{j-1}{\Omega }_{xx}\delta x_{j-1}\right\}={\widehat{x}}_{j-1/j-1}{\Omega }_{xx}\delta {\widehat{x}}_{j-1/j-1}+tr\left({\Omega }_{xx}{\Sigma }^{xx}\right)\hfill \\ \Rightarrow {\widehat{x}}_{j-1/j-1}{\Omega }_{xx}\delta {\widehat{x}}_{j-1/j-1}=E\left\{\delta x_{j-1}{\Omega }_{xx}\delta x_{j-1}\right\}-tr\left({\Omega }_{xx}{\Sigma }^{xx}\right)\hfill \end{array}$$

and get

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=g_{j+1}-\frac{1}{2}{\lambda }^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }^u+\frac{1}{2}tr\left(K_j{\Sigma }^{\xi \xi }\right)+E\{({\lambda }^x+{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\lambda }^u)\delta x_{j-1}\hfill \\ \\ +\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{\Lambda }^{xx}\delta x_{j-1}\}-\frac{1}{2}(E\left\{\delta x_{j-1}{\Omega }_{xx}\delta x_{j-1}\right\}-tr\left({\Omega }_{xx}{\Sigma }^{xx}\right))\hfill \\ \\ =g_{j+1}-\frac{1}{2}{\lambda }^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }^u+\frac{1}{2}tr\left(K_j{\Sigma }^{\xi \xi }\right)+tr\left({\Omega }_{xx}{\Sigma }^{xx}\right)\hfill \\ \\ +E\{({\lambda }^x+{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\lambda }^u)\delta x_{j-1}+\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}({\Lambda }^{xx}-{\Omega }_{xx})\delta x_{j-1}\}.\hfill \end{array}$$

Defining

$$\begin{array}{cc}{h}_j^x:={\lambda }^x+{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }^u,\hfill & H_j:={\Lambda }^{xx}-{\Omega }_{xx}:={\Lambda }^{xx}-{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux}\hfill \end{array}$$

(31)

and

$$g_j=g_{j+1}-\frac{1}{2}{\lambda }_j^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }_j^u+\frac{1}{2}tr(K_j{\Sigma }_{j/j}^{\xi \xi }+{\Omega }_{xx}{\Sigma }_j^{xx}),$$

we get

$$\begin{array}{c}\Delta J_{T-j}^{\ast }=g_j+E\{h_j^x\delta x_{j-1}+\frac{1}{2}\delta x_{j-1}^{{}^{\prime }}{H}_j\delta x_{j-1}\}.\hfill \end{array}$$

(32)

This shows that $\Delta J_{T-j}^{\ast }$ is a quadratic function of $\delta x_{j-1}$.

$$$$

QED

Next, using the results of [31] we find the solution of the g recursion:

$$g_j=g_{j+1}-\frac{1}{2}{\lambda }_j^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }_j^u+\frac{1}{2}tr(K_j{\Sigma }_{j/j}^{\xi \xi }+{\Omega }_{xx}{\Sigma }_j^{xx}).$$

Let us define a deterministic term $M_j=\frac{1}{2}{\lambda }_j^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }_j^u$ and a stochastic term $N_j=\frac{1}{2}tr(K_j{\Sigma }_{j/j}^{\xi \xi }+{\Omega }_{xx}{\Sigma }_j^{xx})$, so that $g_j=g_{j+1}-M_j+N_j$. By working backwards from period T the following can be shown:

$$\begin{array}{ccc}{g}_{T-j}=g_{T-j+1}-\sum _{i=T-j}^T{M}_i+\sum _{s=T-j}^T{N}_i\hfill & and\hfill & g_{T+1}=0.\hfill \end{array}$$

(33)

Next, solve it partially for the stochastic term $N_j$. For that define ${\gamma }_j={\gamma }_{j+1}-M_j$ with ${\gamma }_{T+1}=0$.

Using again the backward recursion get the equation:

$${\gamma }_{T-j}={\gamma }_{T-j+1}-\sum _{i=T-j}^T{M}_i=-\sum _{i=T-j}^T{M}_i.$$

Put it into (33) and get

$$g_{T-j}={\gamma }_{T-j}+\sum _{i=T-j}^T{N}_i$$

or in general

$$g_{j+1}={\gamma }_{j+1}+\sum _{i=j+1}^T{N}_i.$$

Thus

$$g_{j+1}={\gamma }_{j+1}+\frac{1}{2}\sum _{i=j+1}^{T}tr(K_i{\Sigma }_{i-1/i-1}^{\xi \xi }+{\Omega }_{xx}{\Sigma }_{i-1}^{xx})$$

(34)

and

$${\gamma }_j={\gamma }_{j+1}-\frac{1}{2}{\lambda }_{j-1}^u{\left({\Lambda }^{uu}\right)}^{-1}{\lambda }_{j-1}^u.$$

Use the fact that $\Delta J_{T-(t+1)}^{\ast }$ and $\Delta J_{t+1}^{\ast }$ are equivalent; thus put (34) into (32) for $t+1$:

$$\Delta J_{t+1}^{\ast }={\gamma }_{t+1}+\frac{1}{2}\left(\sum _{i=t+1}^{T}tr(K_i{\Sigma }^{\xi \xi }+{\Omega }_{xx}{\Sigma }^{xx})\right)+E\{h_{t+1}^x\delta x_t+\frac{1}{2}\delta x_t^{{}^{\prime }}{H}_{t+1}\delta x_t/P^t\}.$$

(35)

Then

$$\begin{array}{c}{J}_{t+1}^{\ast }\left(x_t\right)=J_{o,t+1}^{\ast }+\Delta J_{t+1}^{\ast }=C_{o,t+1}+{\gamma }_{j+1}+\frac{1}{2}(\sum _{i=t+1}^{T}tr(K_i{\Sigma }^{\xi \xi }+{\Omega }_{xx}{\Sigma }^{xx}))\hfill \\ \\ +E\{h_{t+1}^x\delta x_t+\frac{1}{2}\delta x_t^{{}^{\prime }}{H}_{t+1}\delta x_t/P^t\},\hfill \end{array}$$

(36)

where $C_{o,t+1}={\sum }_{i=t+1}^T{L}_{t+1}(x_{o,t+1},u_{o,t+1})$.

$$\begin{array}{c}{J}_t^{\ast }=\underset{u_t}{min}{E}_{t-1}\{L_t(x_t,u_t)+J_{t+1}^{\ast }\left(x_t\right)\}\hfill \\ \\ =\underset{u_t}{min}E\{L_t(x_t,u_t)+C_{o,t+1}+{\gamma }_{t+1}+\frac{1}{2}(\sum _{i=t+1}^{T}tr(K_i{\Sigma }^{\xi \xi }+{\Omega }_{xx}{\Sigma }^{xx}))\hfill \\ \\ +E\{h_{t+1}^x\delta x_t+\frac{1}{2}\delta x_t^{{}^{\prime }}{H}_{t+1}\delta x_t/P^t/P^{t-1}\}\}\hfill \end{array}$$

(37)

$E\{h_{t+1}^x\delta x_t/P^{t-1}\}=0$ and

$E\{\frac{1}{2}\delta x_t^{{}^{\prime }}{H}_{t+1}\delta x_t/P^{t-1}\}=\frac{1}{2}\delta {\widehat{x}}_{t/t-1}^{{}^{\prime }}{H}_{t+1}\delta {\widehat{x}}_{t/t-1}+\frac{1}{2}tr(H_{t+1}{\Sigma }_{t/t-1}^{xx})=\frac{1}{2}tr(H_{t+1}{\Sigma }_{t/t-1}^{xx})$. Then

$$\begin{array}{c}{J}_t^{\ast }=:J_{t,d}^{\ast }=\underset{u_t}{min}{E}_{t-1}\{L_t(x_t,u_t)+C_{o,t+1}+{\gamma }_{t+1}\hfill \\ +\frac{1}{2}(\sum _{i=t+1}^{T}tr(K_i{\Sigma }_{i/i}^{\xi \xi }+{\Omega }_{xx}{\Sigma }_{i/i}^{xx}))+\frac{1}{2}tr(H_{t+1}{\Sigma }_{t/t-1}^{xx})\}.\hfill \end{array}$$

(38)

This objective function can be split into three parts:

deterministic:

$$J_{D,t}=L_t(x_t,u_t)+C_{o,t+1}+{\gamma }_{t+1},$$

(39)

cautionary:

$$J_{c,t}=\frac{1}{2}\left(\sum _{i=t}^{T}tr\left(K_i{\Sigma }_{i/i}^{\xi \xi }\right)\right)+\frac{1}{2}tr(H_{t+1}{\Sigma }_{t/t-1}^{xx}),$$

(40)

probing:

$$J_{p,t}=\frac{1}{2}\sum _{i=t+1}^{T}tr({\Omega }_{xx}{\Sigma }_{i/i}^{xx}).$$

(41)

In this way the optimal control problem is solved and the components of the objective function are found. However, these are not the final results. In the next step, the stochastic parameters have to be taken into account, i.e., the extension of the system has to be considered.

3.5. Extension of the System

Recall that the algorithm deals with a quadratic criterion function when the parameters of the system equations are unknown. One technique to incorporate the uncertain parameters is to treat the random parameters as additional state variables.

Thus, apply the equations of the objective function to the system

$${\Psi }_t=\left[\begin{array}{c}{x}_t\\ {\theta }_t\end{array}\right].$$

Then

$$\begin{gathered} L_t({\Psi }_t,u_t)=\frac{1}{2}[{(x_t-{\tilde{x}}_t)}^{\prime }⋮0]\left[\begin{array}{ccc}{W}^{xx}& ⋮& 0\\ \cdots & & \cdots \\ 0& ⋮& 0\end{array}\right]\left[\begin{array}{c}(x_t-{\tilde{x}}_t)\\ \cdots \\ 0\end{array}\right] \\ +[{(x_t-{\tilde{x}}_t)}^{\prime }⋮0]\left[\begin{array}{c}{W}^{xu}\\ \cdots \\ 0\end{array}\right][u_t-{\tilde{u}}_t]+\frac{1}{2}{[u_t-{\tilde{u}}_t]}^{\prime }{W}^{uu}[u_t-{\tilde{u}}_t] \end{gathered}$$

(42)

and

$$f_t({\Psi }_t,u_t)=\left[\begin{array}{c}{f}^x({\Psi }_t,u_t)\\ f^{\theta }({\Psi }_t,u_t)\end{array}\right]=\left[\begin{array}{c}f(x_t,x_{t-1},u_t,{\theta }_t)\\ I{\theta }_{t-1}\end{array}\right].$$

(43)

In order to calculate the adjusted components of the objective function (especially (40) and (41)) we need the terms H and $\Omega$.

First, we calculate the adapted ${\Omega }_{\Psi \Psi }$ using Definition (30):

$$\begin{array}{cc}\hfill {\Omega }_{\Psi \Psi }& ={\Lambda }^{\Psi u}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{u\Psi }=[F_{{\Psi }_{t-1}}{({(I-F_{{\Psi }_t})}^{-1})}^{\prime }{K}_t{(I-F_{{\Psi }_t})}^{-1}{F}_u\hfill \\ \hfill & +F_{{\Psi }_{t-1}}{({(I-F_{{\Psi }_t})}^{-1})}^{\prime }{L}_{\Psi u}]{\left({\Lambda }^{uu}\right)}^{-1}[F_u{({(I-F_{{\Psi }_t})}^{-1})}^{\prime }{K}_t{(I-F_{{\Psi }_t})}^{-1}{F}_{{\Psi }_{t-1}}\hfill \\ \hfill & +L_{\Psi u}^{\prime }{({(I-F_{{\Psi }_t})}^{-1})}^{\prime }{F}_{{\Psi }_{t-1}}]\hfill \\ \hfill & =(\left[\begin{array}{cc}{\left(F_{x_{t-1}}^x\right)}^{\prime }& F_{x_{t-1}}^{\theta }\\ {\left(F_{{\theta }_{t-1}}^x\right)}^{\prime }& F_{{\theta }_{t-1}}^{\theta }\end{array}\right]\left[\begin{array}{cc}{\left({(I-F_{x_t}^x)}^{-1}\right)}^{\prime }& 0\\ {\left({(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\right)}^{\prime }& I\end{array}\right]\left[\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right]\hfill \\ \hfill & \times \left[\begin{array}{cc}{\left({(I-F_{x_t}^x)}^{-1}\right)}^{\prime }& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right]\left[\begin{array}{c}{F}_u^x\\ F_u^{\theta }\end{array}\right]+\left[\begin{array}{cc}{F}_{x_{t-1}}^x& F_{x_{t-1}}^{\theta }\\ F_{{\theta }_{t-1}}^x& F_{{\theta }_{t-1}}^{\theta }\end{array}\right]\hfill \\ \hfill & \times \left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}& 0\\ {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x& I\end{array}\right]\left[\begin{array}{c}{L}_{xu}\\ 0\end{array}\right]\left){\left({\Lambda }^{uu}\right)}^{-1}\right(\left[\begin{array}{cc}{F}_u^{\prime }& 0\end{array}\right]\left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}& 0\\ {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x& I\end{array}\right]\hfill \\ \hfill & \times \left[\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right]\left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right]\left[\begin{array}{cc}{F}_{x_{t-1}}^x& F_{{\theta }_{t-1}}^x\\ F_{x_{t-1}}^{\theta }& F_{{\theta }_{t-1}}^{\theta }\end{array}\right]\hfill \\ \hfill & +\left[\begin{array}{cc}{L}_{ux}& 0\end{array}\right]\left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right]\left[\begin{array}{cc}{F}_{x_{t-1}}^x& F_{{\theta }_{t-1}}^x\\ F_{x_{t-1}}^{\theta }& F_{{\theta }_{t-1}}^{\theta }\end{array}\right]),\hfill \end{array}$$

(44)

where

$$\begin{array}{c}I-F_{{\Psi }_t}=\left(\begin{array}{cc}I& 0\\ 0& I\end{array}\right)-\left(\begin{array}{cc}{F}_{x_t}^x& F_{{\theta }_t}^x\\ F_{x_t}^{\theta }& F_{{\theta }_t}^{\theta }\end{array}\right)=\left(\begin{array}{cc}I& 0\\ 0& I\end{array}\right)-\left(\begin{array}{cc}{F}_{x_t}^x& F_{{\theta }_t}^x\\ 0& 0\end{array}\right)\hfill \\ \\ =\left(\begin{array}{cc}I-F_{x_t}^x& (-F_{{\theta }_t}^x)\\ 0& I\end{array}\right)\hfill \end{array}$$

(45)

and

$$\begin{array}{c}{(I-F_{{\Psi }_t})}^{-1}=\left(\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right).\hfill \end{array}$$

(46)

We know that $F_{x_{t-1}}^{\theta }=0$, $F_{{\theta }_{t-1}}^x=0$, $F_{{\theta }_{t-1}}^{\theta }=I$, and $L_{xu}=W^{xu}$. Then we get

$$\begin{array}{c}{\Omega }_{\Psi \Psi }=(\left[\begin{array}{cc}{\left(F_{x_{t-1}}^x\right)}^{\prime }{(I-F_{x_t}^x)}^{-1}& 0\\ {\left({(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\right)}^{\prime }& I\end{array}\right]\left[\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right]\left[\begin{array}{c}{(I-F_{x_t}^x)}^{-1}{F}_{u_t}^x\\ 0\end{array}\right]\hfill \\ +\left[\begin{array}{cc}{\left(F_{x_{t-1}}^x\right)}^{\prime }{(I-F_{x_t}^x)}^{-1}& 0\\ {\left({(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\right)}^{\prime }& I\end{array}\right]\left[\begin{array}{c}{W}^{xu}\\ 0\end{array}\right]){\left({\Lambda }^{uu}\right)}^{-1}\hfill \\ \times (\left[\begin{array}{cc}{\left(F_{u_t}^x\right)}^{\prime }{(I-F_{x_t}^x)}^{-1}\hfill & 0\hfill \end{array}\right]\left[\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right]\hfill \\ \times \left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}{F}_{x_{t-1}}^x& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right]\hfill \\ +\left[\begin{array}{cc}{W}^{ux}\hfill & 0\hfill \end{array}\right]\left[\begin{array}{cc}{(I-F_{x_t}^x)}^{-1}{F}_{x_{t-1}}^x& {(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x\\ 0& I\end{array}\right]).\hfill \end{array}$$

(47)

Define

$$D={(I-F_{x_t}^x)}^{-1}{F}_{{\theta }_t}^x.$$

(48)

Then using $A={(I-F_{x_t}^x)}^{-1}{F}_{x_{t-1}}^x$ and $B={(I-F_{x_t}^x)}^{-1}{F}_{u_t}^x$ we get the following:

$$\begin{array}{c}{\Omega }_{\Psi \Psi }=\left(\left[\begin{array}{cc}{A}^{\prime }\hfill & 0\hfill \\ D^{\prime }\hfill & I\hfill \end{array}\right]\left[\begin{array}{c}{K}^{xx}B\\ K^{\theta x}B\end{array}\right]+\left[\begin{array}{c}{A}^{\prime }{W}^{xu}\\ D^{\prime }{W}^{xu}\end{array}\right]\right){\left({\Lambda }^{uu}\right)}^{-1}\hfill \\ \times \left(\left[\begin{array}{cc}{B}^{\prime }{K}^{xx}& B{K}^{x\theta }\end{array}\right]\left[\begin{array}{cc}A& D\\ 0& I\end{array}\right]+\left[\begin{array}{c}{W}^{ux}A\\ W^{ux}D\end{array}\right]\right)\hfill \\ =\left[\begin{array}{c}(A^{\prime }{K}^{xx}B+A^{\prime }{W}^{xu})\\ (D^{\prime }{K}^{xx}B+K^{\theta x}B+D^{\prime }{W}^{xu})\end{array}\right]{\left({\Lambda }^{uu}\right)}^{-1}\hfill \\ \\ \times \left[\begin{array}{cc}(B^{\prime }{K}^{xx}A+W^{ux}A)& (B^{\prime }{K}^{xx}D+B^{\prime }{K}^{x\theta }+W^{ux}D)\end{array}\right]=:\left[\begin{array}{cc}{\psi }_1& {\psi }_2\\ {\psi }_3& {\psi }_4\end{array}\right].\hfill \end{array}$$

(49)

Use the definition ${\Lambda }^{xu}=A^{\prime }{K}^{xx}B+A^{\prime }{W}^{xu}$ and get the following:

$$\begin{array}{c}{\psi }_1={\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux},\hfill \\ {\psi }_2={\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }{K}^{xx}D+B^{\prime }{K}^{x\theta }+W^{ux}D),\hfill \\ {\psi }_3=(D^{\prime }{K}^{xx}B+K^{\theta x}B+D^{\prime }{W}^{xu}){\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux},\hfill \\ {\psi }_4=(D^{\prime }{K}^{xx}B+K^{\theta x}B+D^{\prime }{W}^{xu}){\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }{K}^{xx}D+B^{\prime }{K}^{x\theta }+W^{ux}D).\hfill \end{array}$$

For the calculation of H we need the term ${\Lambda }^{\Psi \Psi }$:

$$\begin{array}{c}{\Lambda }^{\Psi \Psi }={(F_{{\Psi }_{t-1}})}^{\prime }{({(I-F_{{\Psi }_t})}^{-1})}^{\prime }{K}_t{(I-F_{{\Psi }_t})}^{-1}{F}_{{\Psi }_t}\hfill \\ \\ =\left(\begin{array}{cc}{A}^{\prime }& 0\\ D^{\prime }& I\end{array}\right)\left(\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right)\left(\begin{array}{cc}A& D\\ 0& I\end{array}\right)\hfill \\ \\ =\left(\begin{array}{cc}{A}^{\prime }{K}^{xx}A& A^{\prime }(K^{xx}D+K^{x\theta })\\ (D^{\prime }{K}^{xx}+K^{\theta x})A& D^{\prime }(K^{xx}D+K^{x\theta })+(K^{\theta x}D+K^{\theta \theta })\end{array}\right).\hfill \end{array}$$

(50)

Now we can calculate the term H for the extended system:

$$H^{\Psi \Psi }={\Lambda }^{\Psi \Psi }-{\Lambda }^{\Psi u}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{u\Psi }={\Lambda }^{\Psi \Psi }-{\Omega }_{\Psi \Psi }$$

$$H_t^{\Psi \Psi }={\left(\begin{array}{cc}{H}^{xx}& H^{x\theta }\\ H^{\theta x}& H^{\theta \theta }\end{array}\right)}_t$$

(51)

Thus,

$$\begin{array}{c}{H}^{xx}={\Lambda }^{xx}-{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}{\Lambda }^{ux}\hfill \\ \\ H^{x\theta }=A^{\prime }(K^{xx}D+K^{x\theta })-{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }{K}^{xx}D+B^{\prime }{K}^{x\theta }+W^{ux}D)\hfill \\ \\ H^{\theta \theta }=D^{\prime }(K^{xx}D+K^{x\theta })+(K^{\theta x}D+K^{\theta \theta })\hfill \\ \\ -(D^{\prime }{K}^{xx}B+K^{\theta x}B+D^{\prime }{W}^{xu}){\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }{K}^{xx}D+B^{\prime }{K}^{x\theta }+W^{ux}D).\hfill \end{array}$$

(52)

After that we go back to the components of the objective function. According to (41) we need to calculate the term ${\Omega }_{\Psi \Psi }{\Sigma }_{t/t}^{\Psi \Psi }$.

$$\begin{array}{c}{\Omega }_{\Psi \Psi }{\Sigma }_{t/t}^{\Psi \Psi }=\left(\begin{array}{cc}{\psi }_1& {\psi }_2\\ {\psi }_3& {\psi }_4\end{array}\right){\left(\begin{array}{cc}{\Sigma }^{xx}& {\Sigma }^{x\theta }\\ {\Sigma }^{\theta x}& {\Sigma }^{\theta \theta }\end{array}\right)}_{t/t}\hfill \\ \\ =\left(\begin{array}{cc}{\psi }_1{\Sigma }_{t/t}^{xx}+{\psi }_2{\Sigma }_{t/t}^{\theta x}& {\psi }_1{\Sigma }_{t/t}^{x\theta }+{\psi }_2{\Sigma }_{t/t}^{\theta \theta }\\ {\psi }_3{\Sigma }_{t/t}^{xx}+{\psi }_4{\Sigma }_{t/t}^{\theta x}& {\psi }_3{\Sigma }_{t/t}^{x\theta }+{\psi }_4{\Sigma }_{t/t}^{\theta \theta }\end{array}\right).\hfill \end{array}$$

(53)

Thus,

$$\begin{array}{c}{J}_{p,t}=\frac{1}{2}\sum _{j=t+1}^{T}tr({\Omega }_{\Psi \Psi }{\Sigma }_{j/j}^{\Psi \Psi })=\frac{1}{2}(\sum _{j=t+1}^{T}tr[{\Lambda }^{xu}{({\Lambda }^{uu})}^{-1}{\Lambda }^{ux}{\Sigma }_{j/j}^{xx}])\hfill \\ +tr\left[{\Lambda }^{xu}{\left({\Lambda }^{uu}\right)}^{-1}(B^{\prime }(K^{xx}\overline{D}+K^{x\theta })+W^{ux}\overline{D}){\Sigma }_{j/j}^{\theta x}\right]\hfill \\ +\frac{1}{2}tr[((K^{\theta x}+{\left(D\right)}^{\prime }{K}^{xx})B+{\left(D\right)}^{\prime }{W}^{xu}){\left({\Lambda }^{uu}\right)}^{-1}\hfill \\ \times (B^{\prime }(K^{xx}D+K^{x\theta })+W^{ux}D){\Sigma }_{j/j}^{\theta \theta }].\hfill \end{array}$$

(54)

Next we calculate $J_{C,t}$. According to (40) we need to calculate $tr(K_j^{\Psi \Psi }{\Sigma }_{j/j-1}^{\xi \xi })$ and $tr(H_{t+1}^{\Psi \Psi }{\Sigma }_{t/t-1}^{xx})$:

$$tr\left(K_j^{\Psi \Psi }{\Sigma }_{j/j}^{\xi \xi }\right)=tr\left[{\left(\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right)}_j{\left(\begin{array}{cc}{\Sigma }^{\xi \xi }& 0\\ 0& 0\end{array}\right)}_{j/j}\right]=tr\left(K_j^{xx}{\Sigma }_{j/j}^{\xi \xi }\right)$$

(55)

$$\begin{array}{c}tr\left(H_{t+1}^{\Psi \Psi }{\Sigma }_{t/t-1}^{xx}\right)=tr\left[{\left(\begin{array}{cc}{H}^{xx}& H^{x\theta }\\ H^{\theta x}& H^{\theta \theta }\end{array}\right)}_{t+1}{\left(\begin{array}{c}{\Sigma }^{xx}{\Sigma }^{x\theta }\\ {\Sigma }^{\theta x}{\Sigma }^{\theta \theta }\end{array}\right)}_{t/t-1}\right]\hfill \\ \\ =tr(H_{t+1}^{xx}{\Sigma }_{t/t-1}^{xx}+H_{t+1}^{x\theta }{\Sigma }_{t/t-1}^{\theta x})+tr(H_{t+1}^{\theta x}{\Sigma }_{t/t-1}^{x\theta }+H_{t+1}^{\theta \theta }{\Sigma }_{t/t-1}^{\theta \theta })\hfill \\ \\ =tr\left(H_{t+1}^{xx}{\Sigma }_{t/t-1}^{xx}\right)+2tr\left(H_{t+1}^{\theta x}{\Sigma }_{t/t-1}^{x\theta }\right)+tr\left(H_{t+1}^{\theta \theta }{\Sigma }_{t/t-1}^{\theta \theta }\right).\hfill \end{array}$$

(56)

Thus,

$$J_{c,t}=\frac{1}{2}[\sum _{i=j}^{T}tr(K_i^{xx}{\Sigma }_{i/i}^{\xi \xi })]+\frac{1}{2}tr(H_{t+1}^{xx}{\Sigma }_{t/t-1}^{xx})+tr(H_{t+1}^{\theta x}{\Sigma }_{t/t-1}^{x\theta })+\frac{1}{2}tr(H_{t+1}^{\theta \theta }{\Sigma }_{t/t-1}^{\theta \theta }).$$

(57)

Under the assumption $\gamma =0$ (Kendrick (1981), Appendix J) the deterministic term is

$$\begin{array}{c}{J}_{D,t}=L_t({\Psi }_t,u_t)+C_{o,t+1}+{\gamma }_{j+1}=L_t(x_t,u_t)+\sum _{j=t+1}^T\left(L_j(x_{oj},u_{oj})\right)\hfill \\ \\ =\frac{1}{2}{(x_t-{\tilde{x}}_t)}^{\prime }{W}^{xx}(x_t-{\tilde{x}}_t)+{(x_t-{\tilde{x}}_t)}^{\prime }{W}^{xu}(u_t-{\tilde{u}}_t)\hfill \\ \\ +\frac{1}{2}{(u_t-{\tilde{u}}_t)}^{\prime }{W}^{uu}(u_t-{\tilde{u}}_t)+\sum _{j=t+1}^T[{(x_{ot}-{\tilde{x}}_t)}^{\prime }{W}^{xx}(x_{ot}-{\tilde{x}}_t)\hfill \\ \\ +{(x_{ot}-{\tilde{x}}_t)}^{\prime }{W}^{xu}(u_{ot}-{\tilde{u}}_t)+\frac{1}{2}{(u_{ot}-{\tilde{u}}_t)}^{\prime }{W}^{uu}(u_{ot}-{\tilde{u}}_t)],\hfill \end{array}$$

(58)

where $(x_{ot},u_{ot})=(x_t^{\ast },u_t^{\ast })$ is a nominal path.

In order to calculate the components of the objective function we need to determine one more term, $K_t^{\Psi \Psi }$.

Use $K_t^{\Psi \Psi }=L_{\Psi \Psi }+H_{t+1}^{\Psi \Psi }$

$$\begin{array}{cc}\hfill K_t^{\Psi \Psi }& ={\left(\begin{array}{cc}{K}^{xx}& K^{x\theta }\\ K^{\theta x}& K^{\theta \theta }\end{array}\right)}_t={\left(\begin{array}{cc}{L}_{xx}& L_{x\theta }\\ L_{\theta x}& L_{\theta \theta }\end{array}\right)}_t+{\left(\begin{array}{cc}{H}^{xx}& H^{x\theta }\\ H^{\theta x}& H^{\theta \theta }\end{array}\right)}_{t+1}\hfill \\ \hfill & ={\left(\begin{array}{cc}{W}^{xx}& 0\\ 0& 0\end{array}\right)}_t+{\left(\begin{array}{cc}{H}^{xx}& H^{x\theta }\\ H^{\theta x}& H^{\theta \theta }\end{array}\right)}_{t+1},\hfill \end{array}$$

$$\begin{array}{c}{K}_t^{xx}=W_t^{xx}+H_{t+1}^{xx},\hfill \\ K_t^{x\theta }=H_{t+1}^{x\theta },\hfill \\ K_t^{\theta x}=H_{t+1}^{\theta x},\hfill \\ K_t^{\theta \theta }=H_{t+1}^{\theta \theta },\hfill \end{array}$$

(59)

where $H_{T+1}^{xx}=O_{n\times n}$, $H_{T+1}^{x\theta }=O_{n\times p}$, $H_{T+1}^{\theta x}=O_{p\times n}$, $H_{T+1}^{\theta \theta }=O_{p\times p}$.

Use the linearized system, i.e., apply:

$A={(I-F_{x_t}^x)}^{-1}{F}_{x_{t-1}}^x$, $B={(I-F_{x_t}^x)}^{-1}{F}_{u_t}^x$, $L_u=x^{\prime }{W}^{xu}+u^{\prime }{W}^{uu}+{\left(w^u\right)}^{\prime }$

$L_{uu}=W_t^{uu}$ and $L_{xu}=W_t^{xu}$

to (26) and obtain variables as used in Step I-2B in Section 3.3:

$$\begin{array}{c}{\Lambda }_t^{xx}={\left(A_t\right)}^{\prime }{K}_t^{xx}{A}_t,\hfill \\ {\Lambda }_t^{ux}={\left(B_t\right)}^{\prime }{K}_t^{xx}{A}_t+W_t^{ux}{A}_t,\hfill \\ {\Lambda }_t^{xu}={\left({\Lambda }_t^{ux}\right)}^{\prime },\hfill \\ {\Lambda }_t^{uu}={\left(B_t\right)}^{\prime }{K}_t^{xx}{B}_t+2{\left(B_t\right)}^{\prime }{W}_t^{xu}+W_t^{uu},\hfill \end{array}$$

(60)

$$\begin{array}{c}{\lambda }_t^x=k_t^x{A}_t,\hfill \\ {\lambda }_t^u=k_t^x{B}_t+x_t^{\prime }{W}_t^{xu}+u_t^{\prime }{W}_t^{uu}+{\left(w^u\right)}^{\prime }.\hfill \end{array}$$

(61)

4. Computational Aspects

In this paper we concentrate on mathematical details of the OPTCON algorithm. However, it is important to mention the computational characteristics of this algorithm. In order to show computational time and accuracy we used two different models: the MacRae model and the ATOPT model. As it would go beyond the scope of the present paper, we do not discuss these models and their economic interpretation (see detailed descriptions in [18]) but provide computational details of applying different versions of the OPTCON algorithm. We compared four different optimization strategies: deterministic solution (det), stochastic open-loop solution (OL) as described in Section 3.1, stochastic passive learning solution (OLF) as described in Section 3.2, and stochastic active learning solution (AL) as described in Section 3.3. In the case of the active learning strategy, a grid search with 100 grid points for each control variable was applied (see parameter $\Pi$ in Step-I-[2] in Section 3.3). All calculations were performed on a Windows 10 computer with 16 GB RAM. The OPTCON algorithm is programmed in MATLAB (MATLAB R2020a); the MATLAB computer code is available on request.

The MacRae model is a simple linear model with one state and one control variable. One parameter is stochastic and the optimal control was calculated for two periods only.

Table 1. Computational aspects of optimal control for the MacRae model.

Method	Computational Time	Convergence 1
det	0.024879 sec.	2
OL	0.036867 sec.	2
OLF	0.134725 sec.	2
AL	0.771425 sec.	2

¹ The number of iterations for convergence is available for the deterministic and OL solution techniques. In the case of the OLF and AL solution techniques the number of iterations for S = 1 is given, as this is the optimization for the whole planing horizon.

summarizes the computational performance of the OPTCON algorithm for the MacRae model. In the case of a linear-quadratic deterministic optimal control framework convergence can be shown analytically. This is true for the MacRae model, at least for the deterministic and the OL case. Moreover, as a check for the special case of a linear system, we compared the results of OPTCON with those of the same model calculated by available software and found them to be the same apart from round-off errors.

The ATOPT model is a small nonlinear model consisting of three equations, one control variable, and two stochastic parameters. The optimal control is calculated for five periods. Table 1 summarizes the computational performance of the OPTCON algorithm for the MacRae model.

Table 2. Computational aspects of optimal control for the ATOPT model.

Method	Computational Time	Convergence
det	0.148106 sec.	15
OL	0.134021 sec.	7
OLF	0.265504 sec.	7
AL	15.726499 sec.	7

summarizes the computational performance of the OPTCON algorithm for the ATOPT model.

5. Concluding Remarks

In this paper, we described the OPTCON algorithm for the approximately optimal control of stochastic processes under a quadratic objective function. For systems with complete information, either deterministic or stochastic with known statistical characteristics of the disturbances, the open-loop version of OPTCON1 is suitable. OPTCON2 assumes partial information, in particular uncertain parameters of the system, with passive storage of information accruing during the control horizon, i.e., not used for control purposes. If active storage of information is possible, the dual control policy of OPTCON3 is appropriate. All three versions of OPTCON were programmed first in C# and then in MATLAB. We have applied OPTCON1 to various economic policy problems ([32,33], among others); Monte Carlo experiments with OPTCON2 and OPTCON3 have also yielded satisfactory results in terms of computing time. The results of approximately optimal policies were also as expected from the point of view of the economic problems under consideration. Due to the numerical nature of the algorithm, we cannot prove convergence in general; however, no problems of non-convergence have occurred so far. For future research, more applications are desirable to gain more insight into the different policies resulting from the various assumptions about the information structure of adaptive stochastic policy problems.