On Energy–Information Balance in Automatic Control Systems Revisited

Abstract

We revise and slightly generalize some variational problems related to the “informational approach” in the classical optimization problem for automatic control systems which was popular from 1970–1990. We find extremals for various degenerated (derivative independent) functionals and propose some interpretations of obtained minimax relations. The main example of such functionals is given by the Gelfand–Pinsker–Yaglom formula for the information quantity contained in one random process in another one. We find some balance relations in a linear stationary one-dimensional system with Gaussian signal and interpret them in terms of Legendre duality.

1. Introduction

An informational approach to an opimality criteria for a control system synthesis was proposed by Bukanov in [1]. He has considered the following one-dimensional, control system.

Remark 1.

The “subtraction” and “addition” notation of the diagram nodes means that f.e. the input function $X(t)=U(t)-P(t)$ and $Z(t)=Y(t)-Y_0(t)$ while $V(t)=Z(t)+N(t).$

His choice of an informational criterion was motivated by a straightforward engineering application: studies of the automatic board control system of the passenger jet TU-154. Mathematically, he had used the classical Gelfand–Pinsker–Yaglom functional formula [2]

$$J_{U,Z}=-\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(1-{\mathcal{R}}_{UZ}^2)d\omega$$

for the amount of information about the one random process (the defect or the error signal Z) contained in another such process (the control signal U), where ${\mathcal{R}}_{UZ}$ denotes the mutual correlation function for these processes.

Pinsker had re-written this functional in the following form:

$$J_{U,Z}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_U{s}_Z}{s_U{s}_Z-{|s_{UZ}|}^2}d\omega .$$

(1)

Here, $S_U,S_Z$ are spectral densities of the powers of the control signal U and the defect signal Z, $S_{UZ}$ -is the mutual spectral density.

Bukanov has expressed the functional variables of (1) in terms of the object-regulator transfer functions and spectral densities of the perturbations and has obtained the following explicit integral:

$$J_{U,Z}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{{({\mathcal{R}}_0{s}_P)}^2+{({\mathcal{R}}_r{s}_N)}^2+(1+{\mathcal{R}}_0^2{\mathcal{R}}_r^2)s_N{s}_P}{(1+{\mathcal{R}}_0^2{\mathcal{R}}_r^2+2{\mathcal{R}}_0{\mathcal{R}}_{r}cos({\varphi }_0+{\varphi }_r))s_N{s}_P}d\omega .$$

(2)

The transfer functions of the object and the regulator are represented by the frequency responses $K_0(j\omega )={\mathcal{R}}_{0}exp(j\omega )$ and $K_r(j\omega )={\mathcal{R}}_{r}exp(j\omega ).$ (We shall describe below the detailed reduction of the Pinsker Formula (1) to the Bukanov’s expression (2) in a slightly general context.)

The aim of this paper to revise and to generalize this optimal regulator problem (for a class of linear stationary systems with non-zero program ($Y_0$)) as the classical optimization problem for automatic control system of Figure 1. We get the following results:

Figure 1. This is a figure from [1]. Here, $K_0$ denotes the operator object which transform the input function $X(t)$ to the output function $Y(t)$ which should be compared with the given function $Y_0$ (the program function). The “defect” signal Z together with a measurement “noise” N forms the input of a regulator given by an operator $K_r$. The resulting control signal U with an external perturbation P goes back to the input of the object operator $K_0$.

• find extremals for a generalization of (1) and for some other degenerated (derivative independent) “informational” functionals (the main example of such functionals is given by the Pinsker formula for the entropy quantity of one random process in another.)
• obtained some minimax relations and propose its new interpretations as an “energy–balance equilibrium” in a spirit of the Brillouin’s and Schrodinger’s ideas of “negentropy” (we remind briefly this notion in the Discussion section).
• write our balance relations in a linear stationary one-dimensional system with Gaussian signals and interpret them in terms of Legendre duality.

We would like to stress and warn a mathematical purist reader that this manuscript is not a “mathematical paper” in a formal sense of this notion. We (almost) never discuss and almost never precise any “existence and finitude conditions”. We would rather prefer formal manipulations whenever they exist. Therefore, we decided not to overload these small notes with probability and measure theory precise statements, technicalities, and terminology. We refer all interested readers to the book [2] for all rigorous conditions and other pure mathematical details and statements.

1.1. The Origins

The famous Shannon’s formula [3] provides a facility to measure the capacity of a communication channel in a presence not only of white noise but also in the case when the transmission is perturbed by any Gaussian noise with the power spectrum function $ϵ(\omega )$ proportional to the square module of a filter transfer function $y(\omega )$:

$$ϵ(\omega )={\kappa |y(\omega )|}^2.$$

We recall it because it is probably the first when implicitly the informational characteristics are stipulated by some “energy” constraint. Namely, let the power of the “transmitter” is bounded by some (positive) quantity $A^2$ such the spectral function of the transmitted signal is $s(\omega )$:

$${\int }_{{\omega }_0}^{{\omega }_1}s(\omega )d\omega =A^2.$$

(3)

The Shannon formula we have mentioned above for the total capacity of $s(\omega )$ reads for the frequency bandwidth $[{\omega }_0,{\omega }_1]$ as

$$C_{s,ϵ}={\int }_{{\omega }_0}^{{\omega }_1}log(1+\frac{s(\omega )}{ϵ(\omega )})d\omega .$$

Now, it is easy to give a mathematical answer to the natural question: what should we do to get a maximal transmission rate for the given constraint (3)? Shannon proposes to consider the variational problem with the Lagrange multiplier:

$$max\{{\int }_{{\omega }_0}^{{\omega }_1}[log(1+\frac{s(\omega )}{ϵ(\omega )})+\lambda s(\omega )]d\omega \}$$

(4)

which gives the condition

$$\lambda +\frac{1}{s(\omega )+ϵ(\omega )}=0$$

and implies that $s(\omega )+ϵ(\omega )$ is a constant. Going further, we put $s(\omega )=-\frac{1}{\lambda }-ϵ(\omega )$ in (3) and obtain

$$\lambda =-\frac{{\omega }_1-{\omega }_0}{A^2+B^2},$$

where $B^2={\int }_{{\omega }_0}^{{\omega }_1}ϵ(\omega )d\omega$ and the extremal $s_{max}(\omega )=\frac{A^2+B^2}{{\omega }_1-{\omega }_0}-ϵ(\omega )$, while

$$C_{s,ϵ}^{max}=({\omega }_1-{\omega }_0)log\frac{A^2+B^2}{{\omega }_1-{\omega }_0}-{\int }_{{\omega }_0}^{{\omega }_1}ϵ(\omega ))d\omega .$$

In this paper, we are going to apply these beautiful observations of the founding father of Theory of Information to some (degenerated) variational problems with power constraints arising in the optimal control of automatic systems.

1.2. Review of Some Previously Known Results

The informational criteria for a control system synthesis were very popular in applied automatic control and measure systems in 1970–1980. We should mention here relevant to our interests works of academician Petrov’s school researchers [4]. They were concentrated on the questions of correctness and regularization (in sense of Tikhonov) the control and statistical optimization problems. Their most important and most interesting (to our aims ) input has concluded in a study of a connection between the regularization and a correctness of problems in a one side and an approach motivated by the above Shannon channel capacity formulae in the other. Mathematically, their main tool was based on the Wiener filter theory and related integral equations of the Kolmogorov–Wiener type (see, for example, [5]).

It seems that the system considered by N. Bukanov was ideologically similar to models elaborated by Petrov’s school researchers [5,6]. In the same time, one should stress that Bukanov’s choice of the informational criterion has some advantages. Its mathematical toolbox does not appeal to integral equations and was strongly motivated as we have mentioned by engineering applications to numerical computations related with an optimization of the jet TU-154 automatic board control system.

We shall compare both approaches—[1,5,6] for some basic examples of control linear systems.

We remind that, in [1], a solution was proposed for the variational problem in the class of linear stationary systems with zero program ($Y_0=0$) and Gaussian perturbation signal functions. Namely, Formula (1) makes sense exactly in this class of random functions admissible in [2].

More precisely, the main result of [1] is that the (degenerated) variational problem for the functional (2) has the following extremals:

$${\mathcal{R}}_r={\mathcal{R}}_0\frac{s_P}{s_N},{\varphi }_r=-{\varphi }_0,$$

(5)

which are minimized at the same time as the functional of the error signal dispersion:

$$D_{Z,Z}={\int }_{-\infty }^{+\infty }\frac{{\mathcal{R}}_0^2(s_P+{\mathcal{R}}_r^2{s}_N)}{1+{\mathcal{R}}_0^2{\mathcal{R}}_r^2+2{\mathcal{R}}_0{\mathcal{R}}_{r}cos({\varphi }_0+{\varphi }_r)}d\omega$$

(6)

and the functional of the control signal power dispersion:

$$D_{U,U}={\int }_{-\infty }^{+\infty }\frac{{\mathcal{R}}_r^2(s_N+{\mathcal{R}}_0^2{s}_P)}{1+{\mathcal{R}}_0^2{\mathcal{R}}_r^2+2{\mathcal{R}}_0{\mathcal{R}}_{r}cos({\varphi }_0+{\varphi }_r)}d\omega .$$

(7)

One can conclude that the extremal relations (5) between the object–regulator characteristics guarantee that the optimal regulator provides the minimal power to the object of control. Another interesting observation made in [1] is that, in spite of the functional (2) minimal value ${(J_{U,Z})}_{min}=0$, it does not mean that the regulator does not use the information at all—it means only that there is a certain “informational balance”:

$$J_{U,Z}=[C_{0,P,N}-Q_0]+[C_{r,P,N}-Q_r]$$

(8)

between the channel capacity

$$C_{0,P,N}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(1+{\mathcal{R}}_0^2\frac{s_P}{s_N})d\omega .$$

of the object channel in the presence of perturbations P and N, the channel capacity

$$C_{r,P,N}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(1+{\mathcal{R}}_r^2\frac{s_N}{s_P})d\omega .$$

of the regulator channel in the presence of perturbations P and N, and the (half-)differences

$$Q_0=\frac{1}{2}(Q_0^{\prime }-Q_0″),Q_r=\frac{1}{2}(Q_r^{\prime }-Q_r″)$$

of the informational performances $Q_0,Q_r$ of the output of the object and the regulator channels with open ($Q_0^{\prime },Q_r^{\prime }$) and closed ($Q_0″,Q_r″$) loop–controllers:

$$Q_0^{\prime }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log({\mathcal{R}}_0^2\frac{s_P}{s_N})d\omega ,Q_r^{\prime }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log({\mathcal{R}}_r^2\frac{s_N}{s_P})d\omega ,$$

$$Q_0″=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(|W_{ZP}{|}^2\frac{s_P}{s_N})d\omega ,Q_r″=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(|W_{UN}{|}^2\frac{s_N}{s_P})d\omega .$$

Here, $W_{ZP}$ and $W_{UN}$ are corresponding complex frequency responses (see details in [1]).

1.3. Further Generalizations and the Subject of the Paper

Further studies of this type optimal linear control systems were concerned to a natural generalization for the systems with a non-trivial program ($Y_0\ne 0$ ). At the same time, it was an interesting and mathematically natural question to study similar (degenerated) variational problems for various informational functionals similar to (1) with a “power” constraint of type (3).

The first steps in this direction were done by the author (in a collaboration with N. Bukanov and V. Kublanov). We have reported our partial results in the paper [7] which was, in fact, a short conference proceedings announcement. No details of those results were ever published (to the best of my knowledge). It should be mentioned that some results and the intrinsic ideology—to consider an “entropy criterion” of optimal regulation proposed in our paper [7] were rediscovered in 1986 in the announcement [8].

One of the aims of this manuscript is to revise and to generalize the results reported in [7] and to consider them in a general context of variational problems with Legendre–Fenchel dual functionals.

The paper is organized as follows: Section 2 contains a reminder and a generalization of basic results from [1]. We introduce the notations and remind main definitions of all important notions, objects, and formulas used in the manuscript—among them are: the Pinsker functionals of entropy and informational rates of one stationary Gaussian signal with respect to another, signal spectral densities, and transfer functions.

First, we remind readers about (and generalize in the case of a non-trivial Gaussian stationary program signal) the variational problem for the Pinsker functional in the form proposed by Bukanov. We observe and discuss (Proposition 1) an important formula that represents the Pinsker informational rate functional associated with the linear system in Figure 1 in a Shannon-like channel capacity form.

Theorem 2 in Section 3 solves two degenerate variational problems for channel capacities functionals related with the program channel and with the error signal perturbation channel. We compare the expressions for both channel capacities with the Petrov–Uskov Formula (6) from [6]. Theorem 3 (essentially reported in [7]) gives the main balance relations for the system on Figure 1. We want to stress that the observed similarity of these relations with the famous Brilliouin and Schrodinger “negentropy” notion appeared due to my numerous discussions with Bukanov, who also indicated a close connection with the information gain, or the Kullback–Leibler divergence (see, for example [9], for a modern geometric approach to “informational thermodynamics”). In Section 4, we consider another informational functionals with or without necessary connections with optimization of linear automatic control systems. Our main observation here is Theorem 4 which remarks that the formula for the second Pinsker main functional—the entropy rate of a random process relatively to another random process—can be interpreted like a Legendre duality relation for the first Pinsker informational rate functional.

2. Optimization of the Linear Control System with a Non-Trivial Stationary Gaussian Program

The results of the section (slightly) generalize and extend the material of the paper [1] and give a new interpretation of it. It is worth noting that our notations are different from the original and our motivations and reasonings do not base on possible engineering applications and are purely theoretical.

2.1. Useful Formulas and Notations

We collect here (for the convenience of readers) all basic theoretical formulas which we shall use throughout the text. We clarify (once and forever in the paper) that all random variables and random processes (discrete or continuous) we shall understand in the same sense as in the Pinsker book [2], and we refer a “mathematically oriented” reader to this book for precise definitions, description, and properties.

2.1.1. Formulae

In what follows, we shall use the following important formulas. Let $\xi ,\eta$ be two random functions (= stationary (generalized) Gaussian random processes).

• The Gelfand–Pinsker–Yaglom functional (Formula (2.58) from [10]) for the amount of information about a random function contained in another such function:

  $$J_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{\xi }{s}_{\eta }}{s_{\xi }{s}_{\eta }-{|s_{\xi \eta }|}^2}d\omega =-\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(1-{\mathcal{R}}_{\xi \eta }^2)d\omega$$

  where $s_{\xi }(\omega )$ and $s_{\eta }(\omega )$ are respectively the spectral densities of $\xi$ and $\eta$ and $s_{\xi \eta }(\omega )$—the mutual spectral density of $\xi$ and $\eta .$ Here,

  $${\mathcal{R}}_{\xi \eta }^2:=\frac{|s_{\xi \eta }{|}^2}{s_{\xi }{s}_{\eta }}$$

  is the common correlation function for the processes $s_{\xi }$ and $s_{\eta }$ (Formula (10.21) from [2]).
• Gelfand–Pinsker–Yaglom functional for Gaussian $\xi ,\eta =\xi +\nu$ such that $\xi$ and $\nu$ are non-correlated (Formula (2.61) from [10]):

  $$J_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(1+\frac{s_{\nu }}{s_{\xi }})d\omega$$
• The entropy rate of a random variable $\xi$ with respect to $\eta$ was defined by Pinsker [2]. For two one-dimensional Gaussian stationary (in the wide sense) stochastic processes $\xi$ and $\eta$, the formula for the entropy rate (Formula (10.5.7) from [2]) reads

  $$H_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }(\frac{s_{\xi }}{s_{\eta }}-1-log\frac{s_{\xi }}{s_{\eta }})d\omega$$

  (9)
• For Gaussian $\xi ,\eta =\xi +\nu$ such that $\xi$ and $\nu$ are non-correlated:

  $$H_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }(\frac{s_{\nu }}{s_{\xi }}-log(1+\frac{s_{\nu }}{s_{\xi }}))d\omega$$

2.1.2. Generalized Variational Problem

We suppose now that the stationary Gaussian program $Y_0\ne 0.$ Then, the following system of linear operator equations with respect to the variables U and Z in $p-$coordinates of the Laplace transform for our control system in Figure 1 takes place:

$$\left\{\begin{array}{c}{K}_r(p)Z(p)-U(p)=K_r(p)N(p),\hfill \\ Z(p)+K_0(p)U(p)=K_0(p)P(p)-Y_0(p).\hfill \end{array}\right.$$

(10)

The main determinant of the system

$$\mathsf{\Delta }=\left|\begin{array}{cc}{K}_r& -1\\ 1& K_0\end{array}\right|=1+K_0{K}_r\ne 0.$$

We remind readers about frequency representations $K_0(j\omega ):={\mathcal{R}}_0(\omega )exp(j{\varphi }_0(\omega ))$ and $K_r(j\omega ):={\mathcal{R}}_r(\omega )exp(j{\varphi }_r(\omega ))$

2.1.3. Notations

For the automatic control system in Figure 1, we introduce the following notations:

• $${\mathcal{S}}_{0,P,N}:={\mathcal{R}}_0^2{s}_P+s_N$$
• $${\mathcal{S}}_{r,N,P}:={\mathcal{R}}_r^2{s}_N+s_P$$
• $${\mathcal{S}}_{0,r,\varphi }:=(1+2{\mathcal{R}}_0{\mathcal{R}}_{r}cos({\varphi }_0+{\varphi }_r)+{\mathcal{R}}_0^2{\mathcal{R}}_r^2)s_P{s}_N$$

In these notations, the functional (2) $J_{\xi ,\eta }$ for the processes $\xi =U$ and $\eta =Z$ is written as

$$J_{U,Z}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{{\mathcal{S}}_{0,P,N}{\mathcal{S}}_{r,N,P}}{{\mathcal{S}}_{0,r,\varphi }}d\omega$$

(11)

Solving the system (10) and calculating the spectral density of signals $U,Z$ and denoting the program signal $s_{Y_0}$, we obtain

$$\left\{\begin{array}{c}{s}_Z(\omega )=|W_{Z{Y}_0}{|}^2(j\omega )s_{Y_0}(\omega )+|W_{ZN}{|}^2(j\omega )s_N(\omega )+{|W_{ZP}|}^2(j\omega )s_P(\omega ),\hfill \\ s_U(\omega )=|W_{U{Y}_0}{|}^2(j\omega )s_{Y_0}(\omega )+|W_{UN}{|}^2(j\omega )s_N(\omega )+{|W_{UP}|}^2(j\omega )s_P(\omega ).\hfill \end{array}\right.$$

(12)

One can find the explicit expressions for the transfer functions $W(j\omega )$ in Appendix B. We have computed the full functional generalizing functional (2):

$$J_{U,Z}^{gen}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P[(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})(s_{Y_0}+{\mathcal{S}}_{0,P,N})]}{{\mathcal{S}}_{0,r,\varphi }(s_{Y_0}+{\mathcal{R}}_0^2{s}_P)}d\omega$$

(13)

We see that $Y_0=0$ (13) and (11) coincide.

Theorem 1.

The functional $J_{U,Z}^{gen}$ has the following extremals:

$$\left\{\begin{array}{c}{\varphi }_r=\widehat{{\varphi }_r}:=-{\varphi }_0,\hfill \\ {\mathcal{R}}_r=\widehat{{\mathcal{R}}_r}:={\mathcal{R}}_0\frac{s_P}{s_N}[1+\frac{s_{Y_0}}{{\mathcal{R}}_0^2{s}_p}]\hfill \end{array}\right.$$

(14)

Proof of Theorem 1.

We denote by $\mathcal{L}$ the “Lagrangian” density for the functional (13):

$$\mathcal{L}({\varphi }_r(\omega ),{\mathcal{R}}_r(\omega ))=log\frac{s_P[(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})(s_{Y_0}+{\mathcal{S}}_{0,P,N})]}{{\mathcal{S}}_{0,r,\varphi }(s_{Y_0}+{\mathcal{R}}_0^2{s}_P)}$$

The “degeneracy” of (13) means that the Lagrangian density does not depend on derivatives of ${\varphi }_r$ and ${\mathcal{R}}_r$ and the Euler–Lagrange equations are replaced by easy and straightforward computations from the system of variations

$$\left\{\begin{array}{c}\frac{\delta J_{U,Z}^{gen}}{\delta {\varphi }_r}=0,\hfill \\ \frac{\delta J_{U,Z}^{gen}}{\delta {\mathcal{R}}_r}=0\hfill \end{array}\right.$$

(15)

(due to the basic lemma of the calculus of variations) by the usual function minimax conditions. The first variation gives

$$\frac{\delta J_{U,Z}^{gen}}{\delta {\varphi }_r}=0\Rightarrow -\frac{2{\mathcal{R}}_0{\mathcal{R}}_{r}sin({\varphi }_0+{\varphi }_r)}{{\mathcal{S}}_{0,r,\varphi }({\mathcal{R}}_0^2{s}_P+s_{Y_0})}=0$$

and we obtain the first condition in Theorem 1. The second condition is verifying with a bit more cumbersome computation and we put it in Appendix A. □

Corollary 1.

• Taking $Y_0=0$ immediately obtains the extremals (5) of the functional (2);
• The extremal minimal value of the functional (13) is

  $$\widehat{J_{U,Z}^{gen}}=0,$$
  but, exactly like in the case of the zero program (see (15) in [1]), it does not mean that the regulator of the control system does not use the information. It just means that there is a balance of various channel capacities generalizing (8). We shall discuss and interpret these generalizations in the next sections.
• The same extremals (14) minimize the (generalized) functional of “power” error signal:

  $$D_{Z,Z}^{gen}={\int }_{-\infty }^{+\infty }\frac{s_P{s}_N(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})}{{\mathcal{S}}_{0.r,\varphi }}d\omega .$$

  (16)
• This extremal value easily computed

  $$\widehat{D_{Z,Z}^{gen}}={({\sigma }_Z^2)}_{min}={\int }_{-\infty }^{+\infty }\frac{s_N(s_{Y_0}+{\mathcal{R}}_0^2{s}_P)}{s_{Y_0}+{\mathcal{S}}_{0,P,N}}d\omega .$$
  and coincides with the minimal value of $D_{Z,Z}$ (14.1) in [1] for zero–program case.
• The same is true for the similar generalization of the functional (7) for the power of the regulator. We omit the corresponding evident formulas generalizing (14.2) in [1].

We shall obtain another remarkable representation for the functional (13) using the spectral densities of signals Z and $\widehat{Z}$:

Proposition 1.

$$J_{U,Z}^{gen}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_Z}{s_{\widehat{Z}}}d\omega$$

(17)

Proof. 

We shall compute the spectral density $s_Z$ from the first equation of (12):

$$s_Z(\omega )=\frac{(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})s_P{s}_N}{{\mathcal{S}}_{0,r,\varphi }},$$

the spectral density $s_{\widehat{Z}}$ putting the extremals (14) in the above formula and substituting both spectral densities in (17). □

This formula will be very useful in further interpretations in Section 4.

3. Capacity of Channels and Balance Theorems for the Linear Stationary Control System

Now, using the generalization for $Y_0\ne 0$, we can also generalize and extend the relations which we have reported in [7]. In the same time, we shall propose some new interpretations of these relations. We shall assume throughout this section that all signals are stationary and Gaussian.

3.1. Capacity of Channels via Differential Entropy

We recall that, for the capacity of the channel for the linear control system in Figure 1, the given program $Y_0$ can be computed via differential entropies (“reduced to a degree of freedom”) of this program and the error (defect) Z (see, for example, (ch. IV (12.5) in [11]):

$${\mathcal{C}}_{Y_0}=\mathcal{H}(Y_0)-\mathcal{H}(Z)$$

(18)

where

$$\left\{\begin{array}{c}\mathcal{H}(Y_0)=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(2\pi e)s_{Y_0}(\omega )d\omega ,\hfill \\ \mathcal{H}(Z)=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(2\pi e)s_Z(\omega )d\omega \hfill \end{array}\right.$$

(19)

and the expression (18) is written as

$${\mathcal{C}}_{Y_0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}(\omega )}{s_Z(\omega )}d\omega .$$

(20)

Analogously, one can obtain the capacity of the channel for the linear control system relatively the perturbations P and N:

$${\mathcal{C}}_{P,N}=\mathcal{H}(Z)-\mathcal{H}(P,N)$$

(21)

and

$$\mathcal{H}(P,N)=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(2\pi e){\mathcal{S}}_{0,P,N}(\omega )d\omega$$

such that

$${\mathcal{C}}_{P,N}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_Z(\omega )}{{\mathcal{S}}_{0,P,N}(\omega )}d\omega .$$

(22)

Remark 2.

It would be interesting to compare Formulas (20) and (22) with Formula (16) from [6]:

$$\mathcal{C}={\int }_{W}log\frac{s_m(\omega )}{s_{ϵ}(\omega )}d\omega .$$

(23)

for a channel capacity of the linear system in Figure 2.

Figure 2. This is a figure from [5]. Here, $W(p)$ denotes the transfer function of the operator object. The signal $m(t)$ with the perturbation signal $n(t)$ form a function $\varphi (t)$ on the input of a regulator given by an operator $W_k(p)$. The system should reproduce the “useful” signal $m(t)$ as an output.

Remark 3.

Here the “cross” notation of the diagram nodes means the same as the “addition” nodes notation in Figure 1.

Here, $s_m(\omega )$ and $s_{ϵ}$ are spectral densities of the “useful” signal $m(t)$ and the “error” signal $ϵ(t)$ correspondingly.

The integral should be calculated over the total frequency band W. In particular, if $u(t)=0$ and m and n are non-correlated, then

$$\mathcal{C}={\int }_{W}log\frac{s_m(\omega )+s_n(\omega )}{s_n(\omega )}d\omega .$$

(24)

We assume that the perturbation $n(t)$ has the spectral density with an additional “white noise” spectral density: $s_n(\omega )={\tilde{s}}_n(\omega )+c^2.$ We have from (24)

$$\mathcal{C}={\int }_{W}log\frac{s_m(\omega )+{\tilde{s}}_n(\omega )+c^2}{{\tilde{s}}_n(\omega )+c^2}d\omega .$$

(25)

The analog of the the balance relations (18) and (21) reads

$$\mathcal{H}(n,m)-\mathcal{H}(n,ϵ)=\mathcal{C}=\frac{1}{W}{\int }_{W}log\frac{s_m(\omega )+{\tilde{s}}_n(\omega )+c^2}{{\tilde{s}}_n(\omega )+c^2}d\omega .$$

The authors of [5] argue that the entropy quantity $\mathcal{H}(n,ϵ)$ is a characteristic of information lost while the signal goes through the system of Figure 2, and the minimum of this loss is achieved when $c\equiv 0$. When $c\to \infty$, the entropy $\mathcal{H}(n,ϵ)\to \mathcal{H}(n,m)$ i.e., the “error” entropy tends to the entropy of the “useful” signal and the channel capacity $\mathcal{C}$ goes to $0.$

Remark 4.

We should stress here that:

• all signals are supposed to be stationary random Gaussian with zero expectation value;
• the “optimality” solution in [5] was founded with the type “energy” constraint (3), which means in the context the minimality condition of the second momentum (“dispersion”) for the error signal $s_{ϵ}:$

  $${\int }_W{s}_{ϵ}d\omega ={\int }_W\frac{s_m(\widetilde{s_n}+c^2)}{s_m+\widetilde{s_n}+c^2}d\omega ={\sigma }_{ϵ}^2$$
• The minimal value $min{\sigma }_{ϵ}^2$ is achieved when $c=0$ and this condition shows that the “energy” and informational criteria give the same result (compare with the similar conclusion for the case of the system in Figure 1)
• The independency condition (the signals $m(t)$ and $n(t)$ are non-correlated) can be released, and this leads to the following generalization of capacity channel formula (24):

  $$\mathcal{C}={\int }_{W}log\frac{s_m(s_m+s_n)}{s_m{s}_n+{|s_{mn}|}^2}d\omega .$$

  (26)

3.2. Variational Problems for the Channel Capacity Functionals

Now, we need to substitute the spectral densities of all signals in Formulas (20) and (22) and, as a result, obtain the following two channel capacity functionals:

$${\mathcal{C}}_{Y_0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}{\mathcal{S}}_{0,r,\varphi }}{s_P{s}_N(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})}d\omega$$

(27)

and

$${\mathcal{C}}_{P,N}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P{s}_N(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})}{{\mathcal{S}}_{0,P,N}{\mathcal{S}}_{0,r,\varphi }}d\omega$$

(28)

Remark 5.

It is evident that the functional (28) generalizes the similar thing for the $Y_0=0$ case

$${\mathcal{C}}_{P,N}{|}_{Y_0=0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P{s}_N{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P}}{{\mathcal{S}}_{0,P,N}{\mathcal{S}}_{0,r,\varphi }}d\omega$$

(29)

whose extremum under conditions (5) are

$${\mathcal{C}}_{P,N}^{min}{|}_{Y_0=0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P{s}_N{\mathcal{R}}_0^2}{{\mathcal{S}}_{0,P,N}^2}d\omega$$

(30)

Remark 6.

(One can straightforwardly check that this extremal value is a sum of the informational performances $(Q{″}_0,Q{″}_r)$ of the object and the regulator channels with closed loop–controllers outputs under the conditions (5):

$${\mathcal{C}}_{P,N}^{min,0}{{|}_{Y_0=0}=(Q{″}_0+Q{″}_r)|}_{\varphi ={\varphi }_0,{\mathcal{R}}_r={\mathcal{R}}_0\frac{s_P}{S_N}}.$$

These remarks definitely indicate that there are more general information–performance balance relations and we start to discuss them.

Theorem 2.

The functional (27) has the maximal value

$${\mathcal{C}}_{Y_0}^{max}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}}{s_N}\left[1+\frac{s_N}{s_{Y_0}+{\mathcal{R}}_0^2{s}_P}\right]d\omega$$

(31)

and the functional (28) has the minimal value

$${\mathcal{C}}_{P,N}^{min}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{(s_{Y_0}+{\mathcal{R}}_0^2{s}_P)s_N}{(s_{Y_0}+{\mathcal{S}}_{0,P,N}){\mathcal{S}}_{0,P,N}}d\omega$$

(32)

for the same set of extremals (14):

$$\left\{\begin{array}{c}{\varphi }_r=\widehat{{\varphi }_r}:=-{\varphi }_0,\hfill \\ {\mathcal{R}}_r=\widehat{{\mathcal{R}}_r}:={\mathcal{R}}_0\frac{s_P}{s_N}[1+\frac{s_{Y_0}}{{\mathcal{R}}_0^2{s}_p}]\hfill \end{array}\right.$$

Proof of Theorem 2.

Both functionals are degenerated in the same sense as above. Therefore, their extremals are straightforwardly reduced to their Lagrangian densities’ usual minimax computations. Similarly, to determine a character of obtained extremums, it is enough to verify the Hessian matrices definiteness for these densities instead of computations of the second variation ${\delta }^2({\mathcal{C}}_{Y_0})$ and ${\delta }^2({\mathcal{C}}_{N,P})$. We omit these routine but tedious computations which are absolutely similar to those in Theorem 1. □

3.3. Informational Balance in the Linear Control System

The following theorem was obtained in a collaboration with Bukanov and Kublanov and was announced in [7].

Theorem 3.

There are two balance relations between the channel capacities:

• For the channel capacity of the program $Y_0$ and the informational rate in the control signal U in the “defect" signal Z:

  $${\mathcal{C}}_{Y_0}^{max}-{\mathcal{C}}_{Y_0}=J_{U,Z}^{gen}.$$

  (33)
• For the channel capacity of perturbations P and N and the above informational rate:

  $${\mathcal{C}}_{P,N}-{{\mathcal{C}}^{min}}_{P,N}=J_{U,Z}^{gen}.$$

  (34)

Proof. 

Both identities are easily checked by manipulations with definitions of the corresponding channel capacities and computed extrema. Let us verify the first one:

$${\mathcal{C}}_{Y_0}^{max}-{\mathcal{C}}_{Y_0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}}{s_N}\left[1+\frac{s_N}{s_{Y_0}+{\mathcal{R}}_0^2{s}_P()}\right]d\omega -\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}{\mathcal{S}}_{0,r,\varphi }}{s_P{s}_N(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})}d\omega$$

$$=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P(s_{Y_0}+{\mathcal{R}}_0^2{s}_P+s_N)(s_{Y_0}+{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P})}{{\mathcal{S}}_{0,r,\varphi }(s_{Y_0}+{\mathcal{R}}_0^2{s}_P)}d\omega =J_{U,Z}^{gen}.$$

 □

Remark 7.

Using (20), one can recast Formula (17) re-writing the first balance relation (33) as

$$J_{U,Z}^{gen}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}(\omega )}{s_{\widehat{Z}}(\omega )}d\omega -\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_{Y_0}(\omega )}{s_Z(\omega )}d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_Z(\omega )}{s_{\widehat{Z}}(\omega )}d\omega .$$

On other hand, it means that

$$J_{U,Z}^{gen}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(2\pi e)s_Z(\omega )d\omega -\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(2\pi e)s_{\widehat{Z}(\omega )}d\omega =\mathcal{H}(Z)-\mathcal{H}(\widehat{Z})$$

which shows a similarity with the Brillouin “negentropy" principle which says that the system loses information during a transition from a state with low entropy to a state with higher entropy. (We include a brief description of this notion in the Discussion section below).

Naively speaking, the entropy $\mathcal{H}(Z)$ of the error signal of the linear system with any regulator is bigger (“more chaos") than the error signal entropy $\mathcal{H}(\widehat{Z})$ for the case of the system with optimal regulator:

$$\mathcal{H}(\widehat{Z})=\mathcal{H}(Z)-J_{U,Z}^{gen}\ge 0.$$

Similarly, the same result can be obtained from the second balance relation

$${\mathcal{C}}_{P,N}-{\mathcal{C}}_{P,N}^{min}=\mathcal{H}(Z)-\mathcal{H}(P,N)-\mathcal{H}(\widehat{Z})+\mathcal{H}(P,N)=J_{U,Z}^{gen}.$$

One can imagine here the informational rate $J_{U,Z}^{gen}$ as an “informational difference” between Z and $\widehat{Z}$ or as a “Brillouin–Schrödinger negentropy” [12].

Remark 8.

We should admit that the first balance relation (33) does not exist in the absence of the program signal ($Y_0=0$), while the second is a generalization of the previously described informational balance relation (8): The LHS of the identity (34) under condition $Y_0=0$ is:

$$({\mathcal{C}}_{P,N}-{{\mathcal{C}}^{min}}_{P,N}){|}_{Y_0=0}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P{s}_N{\mathcal{R}}_0^2{\mathcal{S}}_{r,N,P}}{{\mathcal{S}}_{0,P,N}{\mathcal{S}}_{0,r,\varphi }}d\omega -\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{s_P{s}_N{\mathcal{R}}_0^2}{{\mathcal{S}}_{0,P,N}^2}d\omega$$

$$\begin{array}{c}\hfill =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{{\mathcal{S}}_{0,P,N}{\mathcal{S}}_{r,N,P}}{{\mathcal{S}}_{0,r,\varphi }}d\omega =J_{UZ}=(J_{UZ}^{gen}){|}_{Y_0=0.}\end{array}$$

(35)

which is exactly the RHS of (34) in the same condition.

4. Entropy Rate Functional for the Linear Stationary Control System

Now, we shall discuss the entropy rate functional (9) for the pair of Gaussian stationary error signal (Z) and the error signal with optimal regulator ($\widehat{Z}$) in the linear control system above:

$$H_{Z,\widehat{Z}}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }(\frac{s_Z}{s_{\widehat{Z}}}-1-log\frac{s_Z}{s_{\widehat{Z}}})d\omega .$$

(36)

Introduce the function $F(X):=log(X)-1+\frac{1}{X}$ of a positive argument $X>0$ and study its behavior.

4.1. Properties of the Function $F(X)$ and Legendre–Fenchel Transformation

Lemma 1.

The function $F(X)$ has the following (almost) evident properties:

• $F(X)\ge 0$ and $F(1)=0.$
• Let $X=\frac{u}{v}$ such that $u>0;v>0$, then

  $$log(\frac{u}{v})\ge (1-\frac{v}{u});$$
• If $u=u(\omega )$ and $v=v(\omega )$ are integrable on $(\mathbb{R},d\mu (\omega ))$ with some measure $\mu (\omega )$ and

  $${\int }_{\mathbb{R},\mu }ud\mu (\omega )={\int }_{\mathbb{R},\mu }vd\mu (\omega )$$
  then

  $${\int }_{\mathbb{R},\mu }ulog(\frac{u}{v})d\mu (\omega )\ge 0;$$
• Let $Y=\frac{1}{X}$ and $\mathcal{F}(Y):=F(\frac{1}{X})$; then, the function $\mathcal{F}(Y)=Y-1-log(Y)$ admits the Legendre–Fenchel transformation ${\mathcal{F}}^{*}(p)=-log(1-p).$

Proof. 

• To prove the first positivity property of $F(X)$, we split the half line $\{X\ge 0\}$ in two subsets: $\{0\le X<1\}$ and $\{X\ge 1\}.$ Start with the second subset: for $1\le t\le X$, we have $\frac{1}{t}\ge \frac{1}{X}$ and

  $$logX={\int }_1^X\frac{dt}{t}\ge {\int }_1^X\frac{dt}{X}=\frac{1}{X}{\int }_1^{X}dt=\frac{X-1}{X}=1-\frac{1}{X}.$$
  In the first subset, one has similarly $-\frac{1}{t}\ge -\frac{1}{X}$ if $X\le t<1$ and

  $$logX=-{\int }_X^1\frac{dt}{t}\ge -{\int }_X^1\frac{dt}{X}=1-\frac{1}{X}.$$
• Tautological corollary of the computations in (1).
• Using the inequality (2), we obtain:

  $${\int }_{\mathbb{R},\mu }ulog(\frac{u}{v})d\mu (\omega )\ge {\int }_{\mathbb{R},\mu }u(1-\frac{v}{u})d\mu (\omega )=0;$$
• The Legendre–Fenchel transformation ${\mathcal{F}}^{*}(p)$ of the function $\mathcal{F}(Y)$ exists because the function is smooth and convex for $Y>0.$ The only critical point $Y_{crit}=1$ with the critical value $\mathcal{F}(Y_{crit})=0$ is obtained from $1-\frac{1}{Y}=0$. By the definition (see, f.e. [13], we put $p:=1-\frac{1}{Y}$ and find $Y(p)=\frac{1}{1-p}$. Then, we find the Legendre-dual function from $\frac{d}{dp}({\mathcal{F}}^{*}(p))=\frac{1}{1-p},{\mathcal{F}}^{*}(p)=-log(1-p).$ We shall check that it satisfies the duality relation

  $${\mathcal{F}}^{*}(p)=pY(p)-\mathcal{F}(Y(p)):$$

  $$pY(p)-\mathcal{F}(Y(p))=p\frac{1}{1-p}-(\frac{1}{1-p}-1-log(\frac{1}{1-p}))=-log(1-p)={\mathcal{F}}^{*}(p).$$

 □

We note the famous Fenchel–Young inequality which reads in this case as

$$\mathcal{F}(Y)+{\mathcal{F}}^{*}(p)\ge Yp$$

which gives the evident inequality

$$1+log[Y(1-p)]\le Y(1-p).$$

4.2. Legendre Transformation Analogy for the Control System Functionals

We can observe that the Pinsker entropy rate functional (9) can be written in the form

$$H_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }(\frac{s_{\xi }}{s_{\eta }}-1-log\frac{s_{\xi }}{s_{\eta }})d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\mathcal{F}(Y(\omega ))d\omega ,$$

where $Y(\omega ):=\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )}.$ Then, the Legendre–Fenchel dual functional should be

$$\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }{\mathcal{F}}^{*}(p(\omega ))d\omega ,$$

with $p(\omega )=1-\frac{1}{Y(\omega }=1-\frac{s_{\eta }(\omega )}{s_{\xi }(\omega )}.$ Using the part 4 of the Lemma 1, one can compute

$${\mathcal{F}}^{*}(p(\omega ))=-log(1-p(\omega ))=-log(\frac{s_{\eta }(\omega )}{s_{\xi }(\omega )})=log(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )})=logY(\omega ).$$

Thus, we have obtained the following Legendre duality relation:

$$\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\mathcal{F}(Y(\omega ))d\omega +\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }{\mathcal{F}}^{*}(p(\omega ))d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }p(\omega )Y(\omega )d\omega .$$

Coming back to the Pinsker entropy rate functional Formula (9), one observes that the duality implies

$$H_{\xi ,\eta }+\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\left(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )}\right)d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\left(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )}-1\right)d\omega .$$

(37)

Formula (37) expressed the duality relation between the entropy rate and the functional $\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )})d\omega$ for any pair of stationary Gaussian signals. Let us consider the case of the Pinsker entropy rate for our control system (36), i.e., $s_{\xi }(\omega )=s_Z(\omega )$ and $s_{\eta }(\omega )=s_{\widehat{Z}}(\omega )$. In this case, (37) and (36) give us the Legendre duality condition between the functionals:

$$H_{Z,\widehat{Z}}+\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\left(\frac{s_Z(\omega )}{s_{\widehat{Z}}(\omega )}\right)d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\left(\frac{s_Z(\omega )}{s_{\widehat{Z}}(\omega )}-1\right)d\omega .$$

(38)

Summing up the discussion in Section 4.2, one can formulate the following theorem:

Theorem 4.

The Pinsker functional of the entropy rate of the error signal in the stationary linear control system with respect to the error signal of this control system with the optimal by the informational criterion regulator and the informational rate functional for these signals are in Legendre duality:

$$H_{Z,\widehat{Z}}+J_{U,Z}^{gen}=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\left(\frac{s_Z(\omega )}{s_{\widehat{Z}}(\omega )}-1\right)d\omega .$$

(39)

Remark 9.

We can also check that the the duality relation (37) can be easily generalized to the multidimensional case for two $n-$ dimensional stationary Gaussian processes ξ and η such that $\xi =({\xi }_1,\dots ,{\xi }_n),\eta =({\eta }_1,\dots ,{\eta }_n)$ and such that the pairs $({\xi }_j,{\eta }_j)$ are pair-wise uncorrelated. Then, one can define $n-$ dimensional analogues of entropy rate and informational rate functionals:

$$H_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\left[\left(\sum _{j,k=1}^n{s}_{{\xi }_j{\xi }_k}(\omega )s_{{\eta }_j{\eta }_k}^{-1}(\omega )\right)-n-log\frac{det\parallel s_{{\xi }_j{\xi }_k}(\omega )\parallel }{det\parallel s_{{\eta }_j{\eta }_k}(\omega )\parallel }\right]d\omega$$

which can be simplified (because of the mutual independence of the signals ξ and η ) to

$$H_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\sum _{j=1}^n\left(\frac{s_{{\xi }_j{\xi }_j}(\omega )}{s_{{\eta }_j{\eta }_j}(\omega )}-1-log\frac{s_{{\xi }_j{\xi }_j}(\omega )}{s_{{\eta }_j{\eta }_j}(\omega )}\right)d\omega ,$$

(40)

where $\parallel s_{{\xi }_j{\xi }_k}(\omega )\parallel$ (resp. $\parallel s_{{\eta }_j{\eta }_k}(\omega )\parallel$) is the mutual spectral densities matrix of signal-components of ξ (resp.η) and $s_{{\eta }_j{\eta }_k}^{-1}(\omega )$ denotes the $(jk)$ entry of the matrix $\parallel s_{{\eta }_j{\eta }_k}(\omega )\parallel .$

We see that one can apply the Legendre duality Formula (37) to each summand of (40) and re-write it as

$$H_{\xi ,\eta }+L_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\sum _{j=1}^n\left(\frac{s_{{\xi }_j{\xi }_j}(\omega )}{s_{{\eta }_j{\eta }_j}(\omega )}-1\right)d\omega ,$$

(41)

where the functional

$$L_{\xi ,\eta }=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\frac{det\parallel s_{{\xi }_j{\xi }_k}(\omega )\parallel }{det\parallel s_{{\eta }_j{\eta }_k}(\omega )\parallel }d\omega =\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }\sum _{j=1}^{n}log\frac{s_{{\xi }_j{\xi }_j}(\omega )}{s_{{\eta }_j{\eta }_j}(\omega )}d\omega$$

plays the role of the Pinsker informational rate functional (see chapters 10 and 11 in [2]) and for $n=1$ coincides with $\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )})d\omega .$

We want to stress that our multidimensional considerations are purely formal. We do not discuss the conditions of existence and finiteness for all formulas above and refer to [2,14] for all necessary mathematical details. We also do not know if these multidimensional generalizations can immediately apply to an optimal automatic control system problem, but we hope that such functional might be useful in comparison with the method of multidimensional optimization where the minimum of the error expectation value square and maximum of the information mean value criteria were used [15].

5. Discussion

The relationship between entropy and information was first discovered in the seminal work of Szilard [16]. Later, in the works of Brillouin [17], the negentropy principle of information was formulated, generalizing the second law of thermodynamics. According to this principle, both entropy and information should be considered jointly and cannot be interpreted separately. Brillouin had shown [18] that the definition of “information” leads to a direct connection between information and the negative of entropy (here is the abbreviation “negentropy”). Every experiment consumes negentropy (increases entropy) and yields information. The negentropy principle of information is a generalization of Carnot’s principle in thermodynamics that Brillouin had formulated in the following form: the amount of information contained in a physical system must be considered as a negative term in the total entropy of this system. If a system is isolated, it fulfills Carnot’s principle in its generalized form. According to this principle, the total entropy of the system does not decrease. Our Theorem 4 is a good illustration of these energy-information principle.

The functional $L_{\xi ,\eta }:=\frac{1}{4\pi }{\int }_{-\infty }^{+\infty }log\left(\frac{s_{\xi }(\omega )}{s_{\eta }(\omega )}\right)d\omega$ enters in the Legendre duality relation (37) with the entropy rate functional $H_{\xi ,\eta }$ and has explicit interpretation in terms of the informational rate $J_{Z,\widehat{Z}}$ in the case of the control system. The Legendre balance relation predicts some thermodynamical allusions going back to Gibbs thermodynamical potential relation:

$$S(E)+F(\beta )=\beta E,$$

which relates the Helmholtz free energy F to the entropy S, total energy E, and $\beta =\frac{1}{T}$ is inverse to the temperature (the Boltzman constant is supposed to be 1 here). If we shall follow the initial idea of Bukanov about a “similarity” of the informational rate functional $L_{\xi ,\eta }$ with the Brillouin negentropy $\mathcal{N}(\rho )$ and the information gain or Kullback–Leibler divergence $I(p,q)=\int \rho log\rho dq$ which is “similar (up to sign) to the differential entropy expression”:

$$\mathcal{N}(\rho )=I(\xi )+I(p,q)=-\int \xi log\xi dq+I(p,q)=\int log\frac{\rho }{\xi }dq$$

for Gaussian $\xi$, which has the same mean value as $\rho$. (notations and the statement from p. 15 of [9]), then we should conclude that, in our case, the informational rate functional $J_{\xi ,\eta }$ is rather a Legendre dual to the entropy rate $H_{\xi ,\eta }$, though it plays a “role of Hamiltonian” while $H_{\xi ,\eta }$ figured here as a “Lagrangian”. Following the “mechanical and thermodynamical” allusions of [9], we could interpret the results of our Theorems 1 and 2 as a (specified) example of the “principle of minimal gain information” and the balance relations in Theorem 3 show the connection with Jaynes principle of maximum entropy. It would interesting to give further interpretations in this line and to understand what the role of the “partition function” $\mathcal{Z}$ is, which, in our context, should naively read

$$L_{\xi ,\eta }=\int log\frac{s_{\xi }}{s_{\eta }}d\omega =-\int log\mathcal{Z}d\omega .$$

This question, together with more serious comprehension of appropriate Lagrangian or canonical coordinates in these application approaches, would open a door to powerful invariant methods in the theory of information control systems.

There are many other possible developments to study, but all of them look more traditional. We could almost straightforwardly generalize our results using slightly more complicated cases of the capacity channel formulas in comparison with the results of Petrov et al. for signals with correlations, multidimensional systems generalizing the system in Figure 2, etc. The “physical realization” conditions are also beyond the scope of this paper.