Some Remarks on a Variational Method for Stiff Differential Equations

Abstract

We have recently proposed a variational framework for the approximation of systems of differential equations. We associated, in a natural way, with the original problem, a certain error functional. The discretization is based on standard descent schemes, and we can use a variable-step implementation. The minimization problem has a unique solution, and the approach has a global convergence. The use of our error-functional strategy was considered by other authors, but using a completely different way to derive the discretization. Their technique was based on the use of an integral form of the Euler equation for a related optimal control problem, combined with an adapted version of the shooting method, and the cyclic coordinate descent method. In this note, we illustrate and compare our strategy to theirs from a numerical point of view.

1. Introduction

Ordinary differential equations (ODEs) can be used to model problems coming from all of science and engineering. The numerical approximation of their solutions is one of the fields within mathematics that has received more attention over the years. It is virtually impossible to solve in an exact way even the most innocent-looking examples. New studies arise because of new properties of systems of ODEs appear. For instance, when the problem is stiff, one should use, as a general rule, implicit integration schemes. For more details, we propose this incomplete list of references [1,2,3,4,5,6].

Several years ago, we started a new approach for the approximation and analysis of ODEs [7]. We start with a simple observation saying that the solution of the Cauchy problem:

$${\mathbf{x}}^{\prime }\left(t\right)=\mathbf{F}\left(\mathbf{x}\left(t\right)\right)\mathrm{in}(0,T),\mathbf{x}\left(0\right)={\mathbf{x}}_0,$$

(1)

where $\mathbf{x}\left(t\right):[0,T]\to {\mathbf{R}}^N$, $\mathbf{F}\left(\mathbf{x}\right):{\mathbf{R}}^N\to {\mathbf{R}}^N$, ${\mathbf{x}}_0\in {\mathbf{R}}^N$, and $T>0$, should be understood as the unique optimal solution $\mathbf{x}\left(t\right)$ of the variational problem:

$$\mathrm{Minimize}\mathrm{in}\mathbf{z}\left(t\right):\frac{1}{2}{\int }_0^T{|{\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right)|}^{2}dt$$

(2)

under the initial condition $\mathbf{z}\left(0\right)={\mathbf{x}}_0$.

This initial article inspired [8] to set up an approximation scheme based on this optimization perspective. The authors of that contribution utilized this approach to reformulate it as an optimal control problem and then proposed an approximation technique for this related problem. They reported on some encouraging numerical experiments.

Later, including applications where implicit methods are used, we expanded and developed our variational technique in various directions [9,10,11,12,13,14]. In particular, we have paid special attention to stiff problems. The numerical scheme that stems from (2) is based on a typical descent procedure. Let us look at the variation of the functional:

$$E\left(\mathbf{z}\right)=\frac{1}{2}{\int }_0^T{|{\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right)|}^{2}dt,$$

when we perturb a given feasible path $\mathbf{z}$ by some $\mathbf{Z}$ with $\mathbf{Z}\left(0\right)=\mathbf{0}$ so as to not change the initial condition $\mathbf{z}\left(t\right)={\mathbf{x}}_0$. We compute:

$${\left.\frac{dE(\mathbf{z}+ϵ\mathbf{Z})}{dϵ}\right|}_{ϵ=0}={\int }_0^T({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))·({\mathbf{Z}}^{\prime }\left(t\right)-\nabla \mathbf{F}\left(\mathbf{z}\left(t\right)\right)\mathbf{Z}\left(t\right))dt.$$

(3)

We are interested to finding a procedure to produce $\mathbf{Z}$ such that the above derivative is negative:

$${\left.\frac{dE(\mathbf{z}+ϵ\mathbf{Z})}{dϵ}\right|}_{ϵ=0}<0$$

except when $\mathbf{z}$ is the solution of (1), and in that case, the derivative ought to vanish.

Using steepest-descent ideas, one first possibility is to select $\mathbf{Z}$ as the optimal solution of the variational problem:

$$\mathrm{Minimize}\mathrm{in}\mathbf{Z}:{\int }_0^T\left[\frac{1}{2}{\left|{\mathbf{Z}}^{\prime }\left(t\right)\right|}^2+({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))·({\mathbf{Z}}^{\prime }\left(t\right)-\nabla \mathbf{F}\left(\mathbf{z}\left(t\right)\right)\mathbf{Z}\left(t\right))\right]dt$$

under $\mathbf{Z}\left(0\right)=\mathbf{0}$. Since it is a problem for $\mathbf{Z}$ ($\mathbf{z}$ is fixed here), the unique optimal solution of this problem is the unique solution of the optimality system, namely,

$$\begin{array}{c}-{\left[{\mathbf{Z}}^{\prime }\left(t\right)+({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))\right]}^{\prime }-({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))\nabla \mathbf{F}\left(\mathbf{z}\left(t\right)\right)=\mathbf{0}\mathrm{in}(0,T),\\ \mathbf{Z}\left(0\right)=\mathbf{0},{\left.{\mathbf{Z}}^{\prime }\left(t\right)+({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))\right|}_{t=T}=\mathbf{0}.\end{array}$$

The condition at the right-end point T is the transversality condition. It is important to point out that the solution $\mathbf{Z}$ of this problem can be given in closed form in terms of several integrals, as we can find in [14].

Another possibility to find a good descent direction is to consider the linear, non-constant coefficient system:

$${\mathbf{Z}}^{\prime }\left(t\right)-\nabla \mathbf{F}\left(\mathbf{z}\left(t\right)\right)\mathbf{Z}\left(t\right)=-({\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right))\mathrm{in}(0,T),$$

again with $\mathbf{Z}\left(0\right)=\mathbf{0}$. With this choice of $\mathbf{Z}$, we have that:

$$-{\int }_0^T{|{\mathbf{z}}^{\prime }\left(t\right)-\mathbf{F}\left(\mathbf{z}\left(t\right)\right)|}^{2}dt,$$

a multiple of the value of the error functional E at $\mathbf{z}$. Thus, for the implementation of this second possibility, we will approximate only linear problems. For these linear problems, we can use robust implicit integrators like Runge–Kutta methods and apply their theoretical results avoiding the problem of the local convergence of Newton-type methods.

This iterative approach should be used for problems where implicit schemes are required, for instance: Stiff problems, algebraic-differential equations or Hamiltonian systems. In the classical implementation of the implicit schemes, we need to approximate systems of nonlinear equations via Newton-type methods. In particular, the computational costs of both approaches are similar. The advantage of our method is that we have global convergence. Thus, the selection of a good initial guess like Newton-type methods having only local convergence is not necessary. If for a given problem, we are able to use explicit schemes (without associated systems of equations to approximate), our approach makes no sense, since it will be clearly more expensive.

This note was motivated by [8], where inspired by our paper [7], the use of our error-functional strategy was considered but using a completely different way to derive the discretization. The particular transformation that the authors utilized to transform the variational problem into a control problem, in order to approximate the solution of the differential equation or system, seems a bit artificial, and nothing is missed if one deals directly with the variational problem to approximate the global minimizer numerically, i.e., the solution of the differential system (1). We present some comparisons in Section 3.

2. Preliminaries

We start with any initial approximation ${\mathbf{x}}_{\left(0\right)}(\equiv \mathbf{x})$ to the solution of our problem (1). In order to improve this approximation, we will decrease the error in going from ${\mathbf{x}}_{\left(j\right)}$ to ${\mathbf{x}}_{(j+1)}$. We find the next iterate ${\mathbf{x}}_{(j+1)}$ by adding to ${\mathbf{x}}_{\left(j\right)}$ the solution $\mathbf{z}$ of the linear problem:

$${\mathbf{z}}^{\prime }\left(t\right)-\nabla \mathbf{F}\left(\mathbf{x}\left(t\right)\right)\mathbf{z}\left(t\right)=\mathbf{F}\left(\mathbf{x}\left(t\right)\right)-{\mathbf{x}}^{\prime }\left(t\right)\mathrm{in}(0,T),\mathbf{z}\left(0\right)=\mathbf{0}.$$

The following result, taken directly from [9], very clearly supports this strategy.

Theorem 1.

The iterative procedure ${\mathbf{x}}_{\left(j\right)}={\mathbf{x}}_{(j-1)}+{\mathbf{z}}_{\left(j\right)}$, starting from any approximation ${\mathbf{x}}_{\left(0\right)}$, where:

$$\begin{array}{c}{\left({\mathbf{z}}_{\left(j\right)}\right)}^{\prime }\left(t\right)-\nabla \mathbf{F}\left({\mathbf{x}}_{(j-1)}\left(t\right)\right){\mathbf{z}}_{\left(j\right)}\left(t\right)=\mathbf{F}\left({\mathbf{x}}_{(j-1)}\left(t\right)\right)-{\left({\mathbf{x}}_{(j-1)}\right)}^{\prime }\left(t\right)\mathrm{in}(0,T),\\ {\mathbf{z}}^{\left(j\right)}\left(0\right)=\mathbf{0},\end{array}$$

converges strongly, in $L^{\infty }(0,T;{\mathbf{R}}^N)$ and in $H^1(0,T;{\mathbf{R}}^N)$, to the unique solution of the Cauchy problem (1).

An implicit fixed-step schemes computes the new state as a function of the state at the current time step. However, any variable-step solver dynamically varies the step size during the approximation. Using its local error control to achieve the tolerance that one specifies, these solvers increase or reduce the step size. The computation of the step size at each time step adds to the computational overhead, but can reduce the simulation time required to maintain a specified level of accuracy, reducing the number of steps.

The solution of a stiff differential equation should change on a time scale smaller in comparison to the discretization considered. Moreover, if we take a sufficiently small discretization parameter, the method will have a prohibitive computational cost. Thus, in general, step-variable implementations are necessary.

When we approximate a differential equation using a method of order p, the error $e_n$ can be written as:

$$\Vert e_n\Vert =\Vert \mathbf{y}(t_n,t_{n-1},{\mathbf{y}}_{n-1})-{\mathbf{y}}_n\Vert =C{h}^{p+1}+O\left(h^{p+2}\right),$$

where $\mathbf{y}(t_n,t_{n-1},{\mathbf{y}}_{n-1})$ denotes the solution.

We can estimate the constant C using extrapolations methods. We should also consider schemes ${\tilde{\mathbf{y}}}_n$ with bigger order q. In these cases, we have:

$$\Vert {\tilde{e}}_n\Vert =\Vert \mathbf{y}(t_n,t_{n-1},{\mathbf{y}}_{n-1})-{\tilde{\mathbf{y}}}_n\Vert =\tilde{C}{h}^{q+1}+O\left(h^{q+2}\right),$$

and then:

$$\Vert e_n\Vert =\Vert {\mathbf{y}}_n-{\tilde{\mathbf{y}}}_n\Vert +O\left(h^{p+2}\right).$$

The embedded methods are constructed having the same Butcher tableau. Thus, the sub-stages of the lower order scheme are used in the higher order scheme. In particular, the extra computational cost derived by the use of two methods is reduced. Of course, we need to compute the rest of the functional evaluations used in the higher order scheme.

However, in our approach, we can use, as an estimation of the error, the norm of the direction ${\mathbf{z}}_{\left(j\right)}$. These numbers have a relation with the local error. We use this norm in the stop criterion; in particular, our variable step implementation can be implemented without any extra computation.

The variable step codes select the new step in such a way that the error is smaller than a prescribed tolerance $TOL$, with any estimate of the error.

The desired error associated with a $h_{*}$ will be:

$$\Vert e_n\left(h^{{}^{\prime }}\right)\Vert =C{h}_{*}^{p+1}=TOL,$$

and the real error:

$$\Vert e_n\left(h\right)\Vert =C{h}^{p+1}=\Vert {\mathbf{y}}_n-{\tilde{\mathbf{y}}}_n\Vert .$$

Thus, we compute:

$$h_{*}=h\sqrt[p+1]{\frac{TOL}{\Vert {\mathbf{y}}_n-{\tilde{\mathbf{y}}}_n\Vert }}.$$

For more details concerning the analytical aspect of our approach on several applications, we refer to our previous works [7,9,10,11,12].

3. Numerical Comparison

In this section, we compare our strategy using a fixed step implementation in the implicit Euler method, which is the simplest choice that we can consider, and the strategy used in [8]. We present the result of two examples studied in [8] (with a discretization of 1000 nodes), but similar conclusions are obtained for the other problems. At the end, we include an example to emphasize the additional advantages of our variable step implementation. The CPU time in seconds is calculated using the average of 50 computations in order to reduce the influence of other processes.

• Let us consider the following problem:

  $$x^{″}\left(t\right)+t^{2}x\left(t\right)=0;x\left(0\right)=0,x^{\prime }\left(0\right)=0.1.$$
  We are interested in approximating the value $x\left(1\right)$. See

  Table 1. Comparison of our approach using the implicit Euler method and the approach proposed in [8].

  First Problem	Our Approach	The Approach of [8]
  Error functional	$7.55\times {10}^{-17}$	$4.8\times {10}^{-13}$
  CPU	$0.004678$	$1.46$

  for an explicit comparison of our iterative scheme versus the approximation suggested in [8].

• For the problem:

  $$x^{″}\left(t\right)-sin{\left(t\right)}^2{x}^{\prime }\left(t\right)+2x\left(t\right)=0;x\left(0\right)=0,x^{\prime }\left(0\right)=0.1,$$

  we are also interested in approximating the value $x\left(1\right)$. In

  Table 2. Comparison of our approach using the implicit Euler method and the approach proposed in [8].

  Second Problem	Our Approach	The Approach of [8]
  Error functional	$2.4\times {10}^{-17}$	$1.6\times {10}^{-8}$

  , we can see the performance of both methods.

• Finally, we would like to emphasize that our approach can be implemented in an easy way using a variable step strategy. We include here an example to see the advantages of this adaptive implementation.
  We are interested in approximating $x\left(T\right)$, for different values of T, for the problem:

  $$\begin{array}{ccc}\hfill x^{\prime }\left(t\right)& =& \lambda x\left(t\right)+x^2\left(t\right),\hfill \\ \hfill x\left(0\right)& =& 1.\hfill \end{array}$$
  We consider the trapezoid method:

  $$x_{n+1}=x_n+\frac{h}{2}(f\left(x_n\right)+f\left(x_{n+1}\right)).$$

  (4)
  In

  Table 3. Comparison of the CPU-time in our approach, for different values of T and for $T/N=1/100$, $\lambda =-100$.

  T	Fixed Step	Variable Step
  10	$0.26$	$1.17$
  ${10}^2$	$2.96$	$2.04$
  ${10}^3$	$619.16$	$4.23$
  ${10}^4$	$7.86\times {10}^4$	$55.91$

  , we can see the advantages of the step-variable approach.

We refer to our paper [13] for more details.