# Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Statements of Optimal Control Problems

**Problem**

**1**

**Problem**

**2**

**Problem**

**3**

#### 2.2. Self-Organizing Migrating Optimization Algorithms

#### 2.2.1. Optimization Problem Statement

#### 2.2.2. Self-Organizing Migrating Algorithm (SOMA)

#### 2.2.3. Modified Self-Organizing Migrating Algorithm

**Algorithm MSOMA for solving the optimization problem**

**Step 1.**Set the method parameters:

- Control parameter NStep, determining the number of steps until the end of the movement;
- Control parameter PRT, determining whether the individual will move along the chosen coordinate to the leader;
- Control parameter NP, determining the size of the population of individuals;
- Stop parameter Migration, determining the maximum number of iterations;
- Stop parameter MinDist, reflecting the value of the average deviation between the three leaders of the population in terms of the value of the function;
- Iteration counter MCount, necessary to stop the algorithm when it reaches the number Migration.

**Step 2.**Create an initial population.

**Step 3.**Realize a migration cycle regarding the three leaders.

- for the first leader:

- for the second leader:

- for the third leader:

- for the first leader:

- for the second leader:

- for the third leader:

**Step 4.**Checking the conditions for completing the search.

**Step 5.**Update the current population.

**Step 6.**Realize the intensively clarifying migration cycle.

Algorithm 1 MSOMA Algorithm |

1: Initialize the algorithm parameters NStep, PRT, NP, Migration, MinDist, MCount 2: Set MCount = 1 3: Create an initial population: 4: for all $j=1,\dots ,NP$ do 5: ${x}_{i}^{j}={a}_{i}+ran{d}_{i}\left[0,\text{}1\right]\left({b}_{i}-{a}_{i}\right),$ $i=1,\dots ,n;$ 6: end for 7: for all $j=1,\dots ,NP$ calculate the objective function value 8: end for 9: Select the first three individuals with the smallest values of the objective function from the current population 10: Realize a migration cycles regarding the three leaders: 11: for all $j=1,\dots ,NP$ create two “clones” as 12: ${x}_{i}^{Np+j}={x}_{i}^{j},\text{}{x}_{i}^{2Np+j}={x}_{i}^{j},\text{}i=1,\dots ,n;$ 13: end for 14: Implement the movement of individuals to the first leader: 15: for all $j=1,\dots ,NP$ do 16: for all $i=1,\dots ,n$ do 17: generate a random number $rn{d}_{i}$ on the segment $[0,1]$; 18: $\mathrm{if}\text{}rn{d}_{i}PRT,\text{}\mathrm{then}$ 19: $PRTVecto{r}_{i}=1$ 20: $\mathrm{else}$ 21: $PRTVecto{r}_{i}=0$ 22: end if 23: end for 24: for ${m}_{1}=0,\text{}1,\dots ,4NStep$ do 25: ${x}^{j,{m}_{1}}={x}^{j}+{m}_{1}\frac{\left({x}^{1}-{x}^{j}\right)}{2NStep}\otimes PRTVector$; 26: end for 27: Calculate ${x}^{j,New}=\underset{{m}_{1}=0,1,\dots ,4NStep}{\mathrm{argmin}}f\left({x}^{j,{m}_{1}}\right)$ 28: end for 29: Implement the movement of individuals to the second leader: 30: for all $j=\left(NP+1\right),\dots ,2NP$ do 31: for all $i=1,\dots ,n$, do 32: generate a random number $rn{d}_{i}$ on the segment $[0,1]$; 33: $\mathrm{if}\text{}rn{d}_{i}PRT,\text{}\mathrm{then}$ 34: $PRTVecto{r}_{i}=1$ 35: $\mathrm{else}$ 36: $PRTVecto{r}_{i}=0$ 37: end if 38: end for 39: for ${m}_{2}=0,\text{}1,\dots ,2NStep$ do 40: ${x}^{j,{m}_{2}}={x}^{j}+{m}_{2}\frac{\left({x}^{2}-{x}^{j}\right)}{Nstep}\otimes PRTVector$; 41: end for 42: Calculate ${x}^{j,New}=\underset{{m}_{2}=0,1,\dots ,2NStep}{\mathrm{argmin}}f\left({x}^{j,{m}_{2}}\right)$ 43: end for 44: Implement the movement of individuals to the third leader: 45: for all $j=\left(2NP+1\right),\dots ,4NP$ do 46: for all $i=1,\dots ,n$, do 47: generate a random number $rn{d}_{i}$ on the segment $[0,1]$; 48: $\mathrm{if}\text{}rn{d}_{i}PRT,\text{}\mathrm{then}$ 49: $PRTVecto{r}_{i}=1$ 50: $\mathrm{else}$ 51: $PRTVecto{r}_{i}=0$ 52: end if 53: end for 54: for ${m}_{3}=0,\text{}1,\dots ,NStep$ do 55: ${x}^{j,{m}_{3}}={x}^{j}+{m}_{3}\frac{\left({x}^{3}-{x}^{j}\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector$; 56: end for 57: Calculate ${x}^{j,New}=\underset{{m}_{3}=0,1,\dots ,NStep}{\mathrm{argmin}}f\left({x}^{j,{m}_{3}}\right)$ 58: end for 59: Sort all individuals in non-decreasing order of the objective function value 60: Check the conditions for completing the search 61: if $\sqrt{\frac{1}{2}{\displaystyle \sum _{j=2}^{3}}{\left[f\left({x}^{j}\right)-f\left({x}^{1}\right)\right]}^{2}}<MinDist$ ˅ Mcount = Migration, then 62: go to 75 63: else 64: go to 66 65: end if 66: Update the current population: 67: Sort all individuals in non-decreasing objective function value: $\{{x}^{1},\text{}\dots ,{x}^{3NP}|f({x}^{k})\le f\left({x}^{k+1}\right)\},k=1,\dots ,3NP-1$ 68: Delete last $2NP+\left[\frac{1}{3}NP\right]$ individuals and create $\left[\frac{1}{3}NP\right]$ new individuals: 69: for $j=\left[\frac{2}{3}NP\right]+2,\dots ,NP$ do 70: for $i=1,\dots ,n$ do 71: ${x}_{i}^{j}={a}_{i}+ran{d}_{i}\left[0,\text{}1\right]\xb7\left({b}_{i}-{a}_{i}\right)$ 72: end for 73: end for 74: Set $MCount=MCount+1$ and go to 10 75: Realize the intensively clarifying migration cycle 76: Set $NStep=10\xb7NStep$ 77: for $j=1,2,3$ do 78: for all $i=1,\dots ,n$, do 79: generate a random number $rn{d}_{i}$ on the segment $[0,1]$; 80: $\mathrm{if}\text{}rn{d}_{i}PRT,\text{}\mathrm{then}$ 81: $PRTVecto{r}_{i}=1$ 82: $\mathrm{else}$ 83: $PRTVecto{r}_{i}=0$ 84: end if 85: end for 86: for $m=0,1,\dots ,NStep$ do 87: ${x}^{j,m}={x}^{j}+m\frac{\left({x}^{1}-{x}^{j}\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector$; 88: end for 89: Calculate ${x}^{j,New}=\underset{m=0,1,\dots ,NStep}{\mathrm{argmin}}f\left({x}^{j,m}\right)$ 90: end for 91: Return the best solution ${x}^{ans}=\underset{j=1,2,3}{\mathrm{argmin}}\text{}f\left({x}^{j}\right)$ |

## 3. Model Examples

#### 3.1. Optimal Open-Loop Control of a Stochastic Discrete Systems

**Example**

**1.**

**Example**

**2.**

#### 3.2. Optimal Open-Loop Control of Bundles of Trajectories

**Example**

**3.**

**Example**

**4.**

#### 3.3. Optimal Open-Loop Control of Individual Trajectories

**Example**

**5.**

**Example**

**6.**

**Example**

**7.**

## 4. Discussion

## 5. Conclusions and Future Work

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Bertsekas, D.P. Dynamic Programming and Optimal Control; Athena Scientific: Belmont, MA, USA, 1995; Volume 1. [Google Scholar]
- Bertsekas, D.P.; Shreve, S.E. Stochastic Optimal Control: The Discrete Time Case; Academic: New York, NY, USA, 2004. [Google Scholar]
- Luus, R. Iterative Dynamic Programming, 1st ed.; Chapman & Hall/CRC: London, UK, 2000. [Google Scholar]
- Salinelli, E.; Tomarelli, F. Discrete Dynamical Models; Springer: Milan, Italy, 2014. [Google Scholar]
- Propoi, A.I. Problems of discrete control with phase constraints. USSR Comput. Math. Math. Phys.
**1972**, 12, 53–74. [Google Scholar] [CrossRef] - Kwakernaak, H.; Sivan, R. Linear Optimal Control Systems; Wiley–Interscience: Hoboken, NJ, USA, 1972. [Google Scholar]
- Fleming, W.H.; Soner, H.M. Controlled Markov Processes and Viscosity Solutions; Springer: New York, NY, USA; Berlin/Heidelberg, Germany, 1993. [Google Scholar]
- Kushner, H.J. Approximation and Weak Convergence Methods for Random Processes with Applications to Stochastic Systems Theory; MIT Press: Cambridge, MA, USA, 1984. [Google Scholar]
- Kushner, H.J.; Schweppe, F.C. A maximum principle for stochastic control systems. J. Math. Anal. Appl.
**1964**, 8, 287–302. [Google Scholar] [CrossRef] - Peng, S. A general stochastic maximum principle for optimal control problems. SIAM J. Control. Optim.
**1990**, 28, 966–979. [Google Scholar] [CrossRef] - Lin, X.; Zhang, W. A maximum principle for optimal control of discrete-time stochastic systems with multiplicative noise. IEEE Trans. Autom. Control
**2015**, 60, 1121–1126. [Google Scholar] [CrossRef] - Golovkina, A.; Ovsyannikov, D. Optimal control of trajectories ensemble for a class of discrete dynamic systems. IFAC-PapersOnLine
**2020**, 53, 6893–6898. [Google Scholar] [CrossRef] - Kotina, E.D.; Ovsyannikov, D.A. Mathematical model of joint optimization of programmed and perturbed motions in discrete systems. In Prikladnaya Matematika, Informatika, Protsessy Upravleniya; Vestnik Sankt-Peterburgskogo Universiteta: St. Petersburg, FL, USA, 2021; Volume 17, pp. 213–224. [Google Scholar] [CrossRef]
- Kotina, E.D. Discrete optimization problem in beam dynamics. Nucl. Instrum. Methods Phys. Res. A
**2006**, 558, 292–294. [Google Scholar] [CrossRef] - Floudas, C.; Pardalos, P. Encyclopedia of Optimization, 2nd ed.; Springer: New York, NY, USA, 2009. [Google Scholar]
- Gendreau, M. Handbook of Metaheuristics, 2nd ed.; Springer: New York, NY, USA, 2010. [Google Scholar]
- Panteleev, A.V.; Kolessa, A.A. Application of the tomtit flock metaheuristic optimization algorithm to the optimal discrete time deterministic dynamical control problem. Algorithms
**2022**, 15, 301. [Google Scholar] [CrossRef] - Panteleev, A.V.; Kolessa, A.A. Optimal open-loop control of discrete deterministic systems by application of the perch school metaheuristic optimization algorithm. Algorithms
**2022**, 15, 157. [Google Scholar] [CrossRef] - Davendra, D.; Zelinka, I. Self-Organizing Migrating Algorithm, 1st ed.; Methodology and Implementation; Studies in Computational Intelligence, 626; Springer: New York, NY, USA, 2016. [Google Scholar]
- Zelinka, I.; Lampinen, J. SOMA—Self-organizing migrating algorithm. In Proceedings of the International Conference on Soft Computing (Mendel 2000), Brno, Czech Republic, 10–12 July 2000; pp. 177–187. [Google Scholar]
- Panteleev, A.V.; Rakitianskii, V.M. Application of the modified self-organizing migration algorithm MSOMA in optimal open-loop control problems. J. Phys. Conf. Ser.
**2021**, 1925, 012017. [Google Scholar] [CrossRef] - Horn, R.A.; Johnson, C.R. Matrix Analysis; Cambridge University Press: Cambridge, UK, 2012. [Google Scholar]

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDist$ |

1500 | 100 | 0.4 | 100 | ${10}^{-5}$ |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDiv$ |

600 | 40 | 0.4 | 50 | ${10}^{-10}$ |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDist$ |

900 | 70 | 0.3 | 100 | ${10}^{-5}$ |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDiv$ |

600 | 50 | 0.3 | 50 | ${10}^{-10}$ |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDist$ |

1000 | 100 | 0.3 | 70 | 0.001 |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDiv$ |

1000 | 100 | 0.3 | 150 | 0.001 |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDist$ |

500 | 50 | 0.4 | 100 | ${10}^{-10}$ |

Parameters | ||||
---|---|---|---|---|

$NP$ | $Nstep$ | $PRT$ | $Migrations$ | $MinDiv$ |

100 | 50 | 0.3 | 100 | 0.0001 |

$\mathit{t}$ | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
---|---|---|---|---|---|---|---|

${u}^{*}(t)$ | −1.83466 | −0.68896 | −0.33061 | −0.10169 | 0.01892 | −0.05302 | −0.05551 |

${x}^{*}(t)$ | 3.00000 | 1.16534 | 0.47638 | 0.14576 | 0.04407 | 0.06299 | 0.00997 |

t | 7 | 8 | 9 | 10 | |||

${u}^{*}(t)$ | 0.05713 | −0.04223 | −0.04906 | — | |||

${x}^{*}(t)$ | −0.04554 | 0.01159 | −0.03064 | −0.07970 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Panteleev, A.; Rakitianskii, V.
Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems. *Axioms* **2023**, *12*, 1014.
https://doi.org/10.3390/axioms12111014

**AMA Style**

Panteleev A, Rakitianskii V.
Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems. *Axioms*. 2023; 12(11):1014.
https://doi.org/10.3390/axioms12111014

**Chicago/Turabian Style**

Panteleev, Andrei, and Vladislav Rakitianskii.
2023. "Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems" *Axioms* 12, no. 11: 1014.
https://doi.org/10.3390/axioms12111014