Lyapunov Functions to Caputo Fractional Neural Networks with Time-Varying Delays

One of the main properties of solutions of nonlinear Caputo fractional neural networks is stability and often the direct Lyapunov method is used to study stability properties (usually these Lyapunov functions do not depend on the time variable). In connection with the Lyapunov fractional method we present a brief overview of the most popular fractional order derivatives of Lyapunov functions among Caputo fractional delay differential equations. These derivatives are applied to various types of neural networks with variable coefficients and time-varying delays. We show that quadratic Lyapunov functions and their Caputo fractional derivatives are not applicable in some cases when one studies stability properties. Some sufficient conditions for stability of equilibrium of nonlinear Caputo fractional neural networks with time dependent transmission delays, time varying self-regulating parameters of all units and time varying functions of the connection between two neurons in the network are obtained. The cases of time varying Lipschitz coefficients as well as nonLipschitz activation functions are studied. We illustrate our theory on particular nonlinear Caputo fractional neural networks.


Introduction
Neural networks have a wide range of applications in pattern recognition, associative memory, combinatorial optimization, etc. Delays are incorporated into the model equations of the networks because of the finite speeds of the switching and transmission of signals in a network and this leads to time delays in a working network.It was observed both experimentally and numerically in [1] that time delay could induce instability, causing sustained oscillations which may be harmful to a system.Additionally, when one considers memory and hereditary properties of various materials and processes ( [2]), it is natural to consider fractional derivatives in the model.Neural networks in biology, coupled lasers, wireless communication and power-grid networks in physics and engineering ( [3][4][5][6]) are modeled by fractional order differential equations.
In controlling nonlinear systems, the Lyapunov second method provides a way to analyze the stability of the system without explicitly solving the differential equations.Stability results concerning integer-order neural networks can be found in [7][8][9] and recently Lyapunov stability theory for fractional order systems was discussed (see [10,11]).Fractional order Lyapunov stability theory was applied for various types of fractional neural networks using quadratic Lyapunov functions (see [2,[12][13][14]) and stability analysis of fractional-order delay neural networks can be found in [15][16][17][18][19], for example.Finite-time stability of fractional-order neural networks with constant transmission delay and constant self-regulating parameters is studied in [20,21] and for distributed delays and constant self-regulating parameters see [22].
The main goal of this paper is to study stability properties of Caputo fractional order neural networks with delays.We study the general case of time varying transmission delays as well as time varying self-regulating parameters of all units and time varying functions of the connection between two neurons in the network.Note that usually in the literature the activation functions in the models are Lipschitz with constant Lipschitz coefficients.In this paper, the cases of time varying Lipschitz coefficients as well as nonLipschitz activation functions are studied.The investigation is based on the fractional Lyapunov method.It is worth pointing out that the Lyapunov functional method can not be easily generalized to fractional order systems.It is mainly connected with the derivative taken in the given fractional system.Taking these factors into consideration, we present a brief overview of the existing literature on the various definitions of fractional order derivatives of Lyapunov functions among Caputo fractional differential equations with variable delays and we compare them in several examples, discussing their advantages and disadvantages.Using the appropriate fractional derivative of Lyapunov functions, some stability sufficient criteria are provided and illustrated with examples.

Lyapunov Functions and Their Derivatives among the Delay Fractional Differential Equation
We first consider the derivative of Lyapunov functions among the delay fractional differential equation.
In this paper we consider Caputo fractional derivatives with order q ∈ (0, 1) defined by (see, for example [23]) where Γ(.) denotes the Gamma function.
Consider the initial value problem (IVP) for the nonlinear Caputo fractional delay differential equations (FrDDE) where , be a solution of the IVP for the FrDDE (1) and let V(t, x) be a Lyapunov function, i.e., V(t, x) : [t 0 − r, t 0 + T) × ∆ → R + is continuous and locally Lipschitzian with respect to its second argument, where ∆ ⊂ R n , 0 ∈ ∆.
In the literature there are three types of derivatives of Lyapunov functions among solutions of fractional differential equations used to study stability properties: first type-Let x(t) be a solution of IVP for FrDDE (1).Then the Caputo fractional derivative of the Lyapunov function V(t, x(t)) is defined by This type of derivative is applicable for continuously differentiable Lyapunov functions.It is used mainly for quadratic Lyapunov functions to study several stability properties of fractional differential equations (see, for example [10]).-second type-this type of derivative of V(t, x) among FrDDE (1) was introduced by I. Stamova, G. Stamov (Definition 1.12 [24]): where Note that the operator defined by (3) has no memory (the memory is typical for fractional derivatives) (see Case 2.1 of Example 1 and Case 2.2 of Example 2).
) where x(t) is a solution of (1).
Following this notation the fractional derivative of the Lyapunov function is defined by The derivative (4) has memory and it depends on the initial time t 0 .It is closer to both the Grunwald-Letnikov fractional derivative and the Riemann-Liouville fractional derivative, than to the Caputo fractional derivative of a function.We will call the derivative (4) the Dini fractional derivative of the Lyapunov function.The Dini fractional derivative is applicable for continuous Lyapunov functions.
third type -for any φ ∈ C([−r, 0], R n ) the derivative of the Lyapunov function V(t, x) among IVP for FrDDE (1) with initial point t 0 and initial function or its equivalent The derivative (6) depends significantly on both the fractional order q and the initial data (t 0 , ψ 0 ) of IVP for FrDDE (1) and this type of derivative is close to the idea of the Caputo fractional derivative of a function.
We call the derivative given by ( 5) or its equivalent (6) the Caputo fractional Dini derivative.This type of derivative is applicable for continuous Lyapunov functions (see, for example [26][27][28]).
This condition is called the Razumikhin condition.
We will illustrate the application of the above given derivatives on the quadratic Lyapunov function (used in stability of neural networks [2,[12][13][14]).
Case 1.First type of derivative.Let x(t) be a solution of the IVP for FrDDE (1).According to [29] we get Case 2. Second type of derivative.
Case 2.1: Let the function φ ∈ C([−r, 0], R n ).From Formula (3) we obtain It is seen that the derivative D + (1) V(t, φ(0)) does not depend on any initial data of the fractional equation.
Case 2.2: Dini fractional derivative.Let the function φ ∈ C([−r, 0], R n ).From Formula (4) we obtain The Dini fractional derivative depends on both the fractional order q and initial time.Case 3. Third type of derivative.Caputo fractional Dini derivative .According to Remark 4 and Case 2.2 the inequality c holds.
The Caputo fractional Dini derivative depends on both the fractional order q and initial data (t 0 , ψ 0 ).
From the literature we note that one of the sufficient conditions for stability is connected with the sign of the derivative of the Lyapunov function of the equation.

Example 2. Consider the IVP for the scalar linear FrDDE
where q ∈ (0, 1), g(t) = −0.5 Case 1.Consider the quadratic Lyapunov function [12][13][14]) i.e., V(t, x) = x 2 .Let x(t) be a solution of IVP for FrDDE (14) and t > 0 be such that V(t, x(t Then according to [29] we get The sign of C 0 D q t V(t, x(t)) is changeable (the graph of the function g(t) + 0.1 for various values of q is given on Figure 1).
Case 2. Consider the function V(t, x) = (sin 2 (t) + 0.1)x 2 .Case 2.1: Caputo fractional derivative.Let x(t) be a solution of IVP for FrDDE (14).From Equation (2) we obtain The fractional derivative of this function V is difficult to obtain so it is difficult to discuss its sign.Case 2.2: Use Formula (3) and for any function φ ∈ C([−π, 0], R) and any point t > 0 such that i.e., the sign of the derivative D we apply Formula (4) and obtain Therefore, for (14) both the Dini fractional derivative and the Caputo fractional Dini derivative seem to be more applicable than the Caputo fractional derivative of Lyapunov function.Remark 5.The above example notes that the quadratic function for studying stability properties of neural network might not be successful (especially when the right hand side depends directly on the time variable).Formula (3) is not appropriate for applications to fractional equations.The most general derivatives for non-homogenous fractional differential equations are Dini fractional derivatives and Caputo fractional Dini derivatives.

Remarks on Some Applications of Derivatives of Lyapunov Functions
Some authors use the derivative defined by (3) to study stability properties of delay fractional differential equations ( [24,30]) and delayed reaction-diffusion cellular neural networks of fractional order ( [31]).The proofs are based on the following result: The maximal solution u + (t) of the IVP for scalar Caputo fractional differential equation c t 0 D q t u(t) = g(t, u(t)), t > t 0 , u(t 0 ) = 1 exists on [t 0 , ∞).

3.
The Lipschitz function V(t, x) is such that for t ≥ t 0 , φ ∈ C([−r, 0], ∆) the inequality The claim of Lemma 1 is not true.We will give a counterexample.
Consider the Lyapunov function V(t, x) = x 2 .According to Example 1, Case 2.1, Equation (11) we get The function u(t) = cos(t) is a solution of the IVP for the comparison scalar fractional differential equation does not hold for t ≥ 0.
The application of the derivative D + (1) V(t, φ(0)) defined by ( 3) is not appropriate in studying stability properties of neural networks of fractional order (as presented in [31]).We will consider fractional neural networks with delays and using Lyapunov functions and their Caputo fractional derivative, Dini fractional derivative or Caputo fractional Dini derivative (and without applying Lemma 1) we study stability.

System Description
Consider the general model of fractional order neural networks with time varying delay (FODNN) where q ∈ (0, 1), n represents the number of units in the network, x i (t) is the pseudostate variable of the i-th unit of master system, is the self-regulating parameters of the i-th unit, , a ij (t), b ij (t), i, j = 1, 2, . . ., n, correspond to the connection of the i-th neuron to the j-th neuron at times t and t − τ(t) respectively, f j (x j (t)) and g j (x j (t − τ(t))) denote the activation functions of the neurons at time t and t − τ(t), respectively, ), g 2 (x 2 ), . . ., g n (x n )] T and I = [I 1 , I 2 , . . ., I n ] T is an external bias vector, τ(t) ∈ C(R + , [0, r]) is the transmission delay.
The initial conditions of the given system (15) are listed as where Remark 6.The problem of existence and uniqueness of equilibrium states of fractional-order neural networks was investigated by several authors; for sufficient conditions which guarantee the existence and uniqueness of solutions for FODNN with constant delays we refer the reader to [15].
We will discuss the equilibrium points on some FODNN with various activation functions.It will be useful for stability analysis.
The point x * = 0.5 is an equilibrium point of Caputo FODNN (17) because −c0.5 + a(t) f (0.5) + b(t)g(0.5)+ 0.5c = 0 for all t > 0. If Assumption 1 is satisfied then we can shift the equilibrium point x * of system (15) to the origin.The transformation y(t) = x(t) − x * is used to put system (15) in the following form: where We will give the Dini fractional derivative of a Lyapunov function depending directly on the time t among the FODNN (18) which will be used in the proofs of some of the main results.Example 6.Consider the IVP for FODNN (18).
Consider the function V(t, x) = p(t) Then from Formula (4) we obtain for the Dini fractional derivative:

Some Results for Caputo Fractional Differential Equations
We will give some results for fractional derivatives of Lyapunov functions among Caputo fractional differential equations which will be used for our main results concerning the stability of FODNN.

Lemma 2. ([29]
).Let P ∈ R n×n be constant, symmetric and positive definite matrix and X(t) : R + → R n be a function with the Caputo fractional derivative existing.
Then the equilibrium point of ( 1) is uniformly stable.

Stability Analysis
We will study stability properties of several different types of FODNN (1) using different types of Lyapunov functions and their fractional derivatives given in Section 2.

Lipschitz Activation Functions and Quadratic Lyapunov Functions
The stability analysis of the fractional-order neural networks with constant delays and Lipschitz active functions was studied in [35] and the argument was based on topological degree theory, nonsmooth analysis and a nonlinear measure method.We will apply the Lyapunov method to derive some sufficient conditions for stability in the case of variable delays.
We assume the following: Assumption 2. The neuron activation functions are Lipschitz, i.e., there exist positive numbers L i , Assumption 4. The inequality holds.
The case of multiple time constant delays and the functions of the connection of the i-th neuron to the j-th neuron at time t in FODNN (15) are constants studied using the quadratic Lyapunov function in [14].We will study the case of variable bounded functions of connection.
Then the equilibrium point x * of Caputo FODNN ( 15) is uniformly stable.
Proof.Consider the quadratic functions Inequality (20) and Lemma 3 prove the claim.

Non-Lipschitz Activation Functions and Quadratic Lyapunov Functions
There are many types of activation functions which are not Lipschitz (see Example 2, Cases 2 and 3).In this case we assume: Assumption 5.There exists a function ξ ∈ C(R + , R) such that for any solution x(t) of the FODNN (18) and any point t > 0 : ∑ n j=1 x 2 j (t) = sup s∈[−r,0] ∑ n x 2 j (s) and i = 1, 2, . . ., n the inequality holds.
Then the equilibrium point x * of Caputo FODNN ( 15) is uniformly stable.

Non-Lipschitz Activation Functions and Time-Depended Lyapunov Functions
In the case the function η(t) in Assumption 6 not being large enough, we introduce Assumption 7.There exists a continuous positive function p(t) ∈ C([0, ∞), R + ) such that 0 < α ≤ p(t) ≤ β and the fractional derivative RL 0 D q t p(t) exists for t > 0.
In this case, Assumption 5 could be weakened to Assumption 8.There exists a function ξ ∈ C(R + , R) such that for any function φ ∈ C([−r, 0], R n ) and t > 0 be such that p(t) holds.
Theorem 3. Let the Assumptions 1, 6-8 be satisfied and Then the equilibrium point x * of Caputo FODNN ( 15) is uniformly stable.

Conclusions
Several sufficient conditions are given for stability of the equilibrium point of Caputo fractional order neural networks with delays of order 0 < q < 1.We study the general case when the transmission delays are time variables; -the self-regulating parameters of all units are variable in time; -the functions of the connection between two neurons in the network are variable in time bounded functions; -the activation function is nonLipschitz (usually in the literature only the case of Lipschitz activation functions is studied).
The investigation is based on applying the fractional Lyapunov approach and various types of Lyapunov functions are discussed (not only the quadratic one as is usually considered in the literature).The presence of the fractional derivative in the model requires an appropriate defined and used derivative of Lyapunov functions among the fractional model.A brief overview of the existing literature and the main types of derivatives among Caputo fractional equations are presented.These types of derivatives are used as an apparatus to study stability properties of the considered Caputo fractional order neural networks for different types of activation functions.Many examples with different types of activation functions such as tanh function, Logit function, continuous Tan-Sigmoid function, are provided to illustrate the practical application of the obtained sufficient conditions.

Figure 1 .
Figure 1.Example 2. Graph of the function g(t) + 0.1 for various values of q.

Example 9 .
ii) of Lemma 4 is satisfied.Applying Lemma 4 we prove the claim.Let n = 1 and consider the scalar FODNN (17) with c(t) = 0.55