Bifurcation Analysis Software and Chaotic Dynamics for Some Problems in Fluid Dynamics Laminar–Turbulent Transition

Abstract

The analysis of bifurcations and chaotic dynamics for nonlinear systems of a large size is a difficult problem. Analytical and numerical approaches must be used to deal with this problem. Numerical methods include solving some of the hardest problems in computational mathematics, which include system spectral and algebraic problems, specific nonlinear numerical methods, and computational implementation on parallel architectures. The software structure that is required to perform numerical bifurcation analysis for large-scale systems was considered in the paper. The software structure, specific features that are used for successful bifurcation analysis, globalization strategies, stabilization, and high-precision implementations are discussed. We considered the bifurcation analysis in the initial boundary value problem for a system of partial differential equations that describes the dynamics of incompressible ABC flow (3D Navier–Stokes equations). The initial stationary solution is characterized by the stability and connectivity to the main solutions branches. Periodic solutions were considered in view of instability transition problems. Finally, some questions of higher dimensional attractors and chaotic regimes are discussed.

1. Introduction

Let us consider the bifurcation analysis for a fluid dynamics problem. The problem can be described by a system of equations in the general form:

$$\mathcal{F}(\mathbf{u},R)=\mathbf{0},$$

(1)

where B is a Banach space, $\mathbf{u}\in B$, $R\in {\mathbb{R}}^k$ is a vector of k-parameters, and $\mathcal{F}:B\to B$ is a nonlinear operator (which can include temporal and spatial operators). For each specific nonlinear system under consideration, one formulates more-specific properties of space B and a nonlinear operator $\mathcal{F}$. Notice that a majority of different models, described by the system of nonlinear equations, can be recast into (1)’s format, so this setting is rather general.

Some standard methods of analytical and semi-analytical analysis exist that allow one to obtain first or even secondary bifurcations [1]. These methods include Lyapunov–Schmidt reduction [2,3], central manifold reduction [4,5], equivariant unfolding theory [6], etc. Some examples from the area of fluid dynamics are presented below. Lyapunov–Schmidt reduction was applied to 1D spatial Euler equations in [7] to obtain periodic solutions; the classical results of Youdovich with these approaches were obtained in [8,9,10], as well as other results in the field of fluid dynamics and Navier–Stokes equations [11,12,13]. Equivariant unfolding theory together with the Lyapunov–Schmidt reduction was applied to the Taylor–Couette flow in [6] by reducing $\mathbf{O}\left(\mathbf{2}\right)\times \mathbf{SO}\left(\mathbf{2}\right)$ group symmetries in the system. Unfortunately, even such sophisticated methods cannot always predict bifurcations for substantial variation of parameters and are not related to chaotic regimes. These methods can be used in particular cases, as described in the cited papers. For a general analysis, one needs to apply numerical methods that take into account the properties of the underlying system (1), as well as being efficient and obtaining solutions with satisfactory numerical precision. Under these conditions, one needs efficient software capable of running on modern hardware that can be executed on any computational system, from a desktop to a high-performance cluster.

The finite-dimensional system of equations is derived from (1) by applying some consistent method (for example, the Galerkin method). It is assumed that the obtained system represents the original system (1) with enough Degrees Of Freedom (DOFs) to approximate all solutions in the range of parameter variations. The new finite-dimensional discrete system can be presented as:

$$\mathtt{M}{\mathbf{u}}_t=\mathbf{F}(\mathbf{u},R).$$

(2)

Here, $\mathbf{u}\left(t\right)\in {\mathbb{R}}^N$ is a discrete solution of (2) that approximates the solution of the original system, t is a temporal variable, N is the number of DOFs (typically from $1·{10}^6$ to $1·{10}^9$), $\mathtt{M}$ is a linear operator, which can be an identity ($\mathtt{E}$), and $\mathbf{F}$ is a discrete, generally nonlinear, operator. Boundary conditions are included in the $\mathtt{M}$ and $\mathbf{F}$ operators. These systems (1) and (2) are the basic setup on which one can consider many different problems (in this paper, for example, System (2) is represented by the Bubnov–Galerkin representation of the incompressible Navier–Stokes equations on a periodic domain). In order to solve (2) with high precision on all necessary DOFs, one must utilize large computational clusters and use High-Performance Computations (HPCs). This difficulty is superimposed on advanced methods required for bifurcation analysis that include the stabilization method for stationary, periodic, or quasi-periodic solutions, continuation methods, and leading eigenvalue solvers. Assume that, for some $R_0$, the solution ${\mathbf{u}}_0$ of (2) is stable. The process of bifurcation analysis can be described as follows:

• Find a solution of (2) that can be traced from ${\mathbf{u}}_0$ for some variation of $\delta R$ to obtain ${\mathbf{u}}_{*}$ using the continuation process, and apply stabilization methods if the solution is linear unstable.
• Obtain the linearization of the system (2) at ${\mathbf{u}}_{*}$ and find the leading eigenvalues of the linearized system (these can be eigenvalues with the largest real parts for stationary solutions or the largest magnitude eigenvalues for periodic solutions).
• Find $R_{*}$ where the leading eigenvalues cross an imaginary axis or cross the unit circle for exponential mapping.
• Identify the bifurcation type.
• Check if there are other solutions that exist for the system at $R_0$, then set such solutions as ${\mathbf{u}}_0$, and go to Step 1.

These processes can be automated for stationary and periodic solutions. For quasi-periodic solutions, such processes can only be partially automated; hence, the manual execution of the above-listed steps is required.

The following set of numerical methods is required to perform the above-listed steps:

• Deflation of the solutions.
• Continuation of the solutions through bifurcation points.
• Stabilization of the stationary points of the original system (stationary solutions).
• Stabilization of periodic orbits (for example, by the stabilization of the fixed points in the Poincare mapping).
• Obtaining the leading eigenvalues of either the original system or of the return Poincare mapping.

Let us consider the methods that have already been developed and applied to the bifurcation analysis of (2), where $\mathbf{F}$ is the discrete representation of the Navier–Stokes operator, i.e., conservation laws for incompressible flow. One the most-famous approaches to partially automated bifurcation analysis for incompressible fluid dynamics problems was suggested by L.S. Tuckerman and co-authors. The idea is the utilization of already-available time stepper codes (Direct Numerical Simulation (DNS) methods); see [14,15,16,17]. The design of the eigensolvers for (2) in the case of incompressible flow is based on the exponential preconditioner [18] or shift and inverse preconditioner [15] together with available time steppers. An example of the application of these approaches was presented in [19,20], where the bifurcations of steady-state solutions in cylindrical Rayleigh–Benard instability were investigated. An excellent review on the question of bifurcation analysis using this approach for fluid dynamics is presented in [21]. Another successful approach was developed by Sanchez and co-authors; see [22,23,24]. Many other papers exist that deal with the instability analysis of the Navier–Stokes equations in the linear and nonlinear cases. However, these papers are focused on the detection of the first instability (first bifurcation) and cannot be qualified as bifurcation-analysis-oriented.

Some attempts to apply machine learning with a reduced models approach [25] to (2) were made; for example, see [26]. However, all these problems are limited to 2D spatial flows only.

Another important property of bifurcation analysis is the identification of distinct, disconnected solutions. Such a procedure is called the deflation of solutions. This was suggested in the work of Farrell et al. [27] and earlier works of Kalantonis et al. [28], which were both based on the work of Davidenko [29]. For large-scale systems on the HPC architecture, the approach was suggested in [30], and the algorithm for the deflation of periodic solutions of large-scale systems was suggested in [31]. Such approaches allow one to find disconnected solutions, as well as construct bifurcation diagrams. For example, the results for the disconnected solutions in fluid dynamics were presented in [32,33] for 2D flows and in [34] for 3D flows.

On the other hand, it is impossible to trace the complexity of solutions further than 2D quasi-periodic using automatic bifurcation software. In this case, direct simulation is in order since it is possible to use DNS and Poincare section analysis to study higher-dimensional attractors, including N-dimensional tori. The stabilization of periodic solutions embedded in these higher-order attractors is also essential; it allows one to study the dimension of the unstable manifold and its eigenvectors, which decode the dynamics around the stabilized periodic orbits. This approach was used in many studies by the authors, including 2D and 3D Kolmogorov flows [35,36,37], Rayleigh–Benard convection [38], and Rayleigh–Taylor and Kelvin–Helmholtz instabilities [39]. In these papers, the authors were able to unravel some universal patterns, including bifurcations in limited periodic orbits and 2D and 3D invariant tori. This was possible only due to the use of high-precision operations in the base floating-point standards.

Our goal was to develop a rather universal software that allows one to perform not only bifurcation analysis, but the analysis of chaotic regimes in large-scale nonlinear problems, defined by (2). This paper gives more details on some parts of our software (deflation that takes discrete symmetry into account and the compensated variant of the time-stepping method for the floating-point arithmetic), as well as demonstrates bifurcation and chaotic analysis. We present a survey of the automatic bifurcation analysis software used for the problems of computational fluid dynamics, as well as the results obtained with such methods. In the next section, a review of the method and some specific features are considered. Next, some new results of the automated bifurcation analysis are presented for the ABC flow problem as a demonstration of the developed software. The paper is finalized by a discussion.

2. Numerical Methods and Software

There are several libraries available that allow one to automate bifurcation analysis. However, most of these software packages are aimed at small or medium-sized maps or systems of equations that allow one to compute direct matrix factorization (LU or QR) and store dense Jacobi and Hessian matrices. Examples of such methods are AUTO [40], MATCONT [41], DSTOOL [42], MANLAB [43,44,45], etc. For the continuation of time periodic problems for Partial Differential Equations (PDEs), a PDECONT [46] is available that performs time-stepper-based continuation, but only for middle-sized problems. Among others, one can outline a bifurcation package [47] implemented in Julia that allows one to find bifurcations even for large-scale problems. It requires recasting the system in the Julia universe and can be used if the system can be efficiently implemented in the “Julia way”. Unfortunately, this is not always possible; hence, a standard massively parallel C++ implementation, though much more time-consuming, is still, in the final accord, more efficient.

The structure of the current bifurcation analysis and simulation software is outlined in Figure 1 and Figure 2.

The implementation of the software was performed using template metaprogramming in C++ and CUDA C++. Parallelization was implemented using the Message-Passing Interface (MPI) for distributed memory, Open Multi-Processing (OpenMP), and the NVIDIA Compute Unified Device Architecture (CUDA) for central and Graphics Processing Units (GPUs). Each block of a different color depicts a different namespace. The software, developed by the authors, is based on a set of publications: some were published, and some are to be published and are referenced in the namespaces’ descriptions. The namespaces are distributed between a user-provided problem description (discrete models of fluid dynamics problems) [39,48,49], methods that perform automatic bifurcation analysis [30,31], methods that solve nonlinear problems, methods that solve eigenvalue problems [50,51], methods that remove continuous and discrete symmetries, methods that solve large-scale linear problems and perform basic operations on linear spaces [52,53,54,55], and the solutions’ storage database.

Next, we shall consider different parts of the designed software that were not discussed before and show some specific features that we used to perform bifurcation analysis.

2.1. Time Marching Integration

The time-marching algorithm is one of the main parts of the simulation process, and it is also used to perform the Poincare shooting method in the deflation and continuation of periodic orbits. The simulation of solutions that correspond to the invariant s-dimensional tori also requires time marching with millions of time steps. In this subsection, we shall analyze the behavior of the time-stepping algorithms on inertial manifolds and discuss possible increases in precision for the standard time-marching methods.

The first important property of our implementation is the usage of Diagonally Implicit (DIRK) or Semi-Implicit (IMEX) Runge–Kutta (RK) methods, which are L- and B-stable in their implicit part; see [56,57,58]. Let $g\left(z\right)$ be a stability function of an RK method. The method is stable for such $z\in \mathbb{C}$, so that $\left|g\right(z\left)\right|<1$. The last inequality generates a region $\mathcal{S}\subset \mathbb{C}$ for which the method is linearly stable, i.e.,

$$\mathcal{S}\left(z\right):=\left\{z\in \mathbb{C}:g\left(z\right)=exp\left(i\theta \right),\forall \theta \in [0,2\pi ]\right\}.$$

An important property of these methods (and any methods that we applied in our simulations) is that a segment of the imaginary axis (or the whole axis) is enclosed inside $\mathcal{S}$. In this case, the following estimate is known. Assume that the Jacobi matrix of (2) $J\left(\mathbf{u}\right)$ has the following property: for some set U that represents a set of solutions, the symmetric part $A\left(\mathbf{u}\right):=(J\left(\mathbf{u}\right)+J{\left(\mathbf{u}\right)}^{*})/2$ of the matrix is such that $\left(A\right(u)\xi ,\xi )\le 0,\forall \xi ,\mathbf{u}\in U$. The system (2) can be written in the following form for any RK method on the n-th step:

$$\begin{array}{c}\hfill {\mathbf{u}}_{n+1}={\widehat{\mathbf{F}}}_k(\tau ,{\mathbf{u}}_n,{\mathbf{u}}_{n+1},\mathbf{F}),{\mathbf{u}}_0={\mathbf{u}}_{\mathbf{0}}\end{array}$$

(3)

$$\begin{array}{c}\hfill {\mathbf{u}}_{n+1}^{\prime }=\tau {\widehat{\mathbf{F}}}_k({\mathbf{u}}_n^{\prime },{\mathbf{u}}_{n+1}^{\prime },\mathbf{F})+\epsilon ,{\mathbf{u}}_0^{\prime }={\mathbf{u}}_{\mathbf{0}}+\epsilon ,\end{array}$$

(4)

where ${\widehat{\mathbf{F}}}_k$ is the RK operator or order k, which is built around the current and previous steps, time step $\tau$, $\mathbf{F}$, and the Butcher tableau; ${\mathbf{u}}^{\prime }$ is a perturbed solution with the small perturbation $\epsilon$. The error is given as ${\mathbf{e}}_n={\mathbf{u}}_n-{\mathbf{u}}_n^{\prime }$. If the above-listed property of the Jacobi matrix is true, then:

$$\parallel {\mathbf{e}}_n\parallel \le exp\left(C_1\right)\parallel {\mathbf{e}}_0\parallel +\mathcal{O}\left({\tau }^{k-1}\right)exp\left(C_2\right),$$

where constants $C_1$ and $C_2$ are independent of n and $\tau$. Hence, for a periodic orbit, one usually obtains satisfactory simulation results. Such an estimate cannot hold for other types of attracting sets, such as quasi-periodic orbits or chaotic attractors. However, if one assumes the existence of an Inertial Manifold (IM), then the estimate can be extended. Define an IM [5]. Let $B:=\mathbb{H}$ be a Hilbert space of the original system (1) with the distance function $\mathrm{dist}(·,·)$; this system can be rewritten in terms of the semi-group translation form $u\left(t\right):=\phi {\left(\mathcal{F}\right)}^t\mathbf{u}\left(0\right)$, with the t state a continuous or discrete time and $\mathbf{u}\left(0\right)$ an allowed initial condition.

Definition 1.

Let the manifold $\mathcal{M}$ enjoy the following properties:

• $\dim \left(\mathcal{M}\right)=m<\infty$;
• $\forall t>0:\phi \left(\mathcal{F}\right)\mathcal{M}\subseteq \mathcal{M}$;
• $\forall t>0$ and $\mathbf{u}\left(0\right)\in \mathbb{H}$, there exist constants $c_1>0$ and $c_2>0$ s.t. $\mathrm{dist}(\phi {\left(\mathcal{F}\right)}^t\mathbf{u}\left(0\right),\mathcal{M})\le c_{1}exp(-c_{2}t)$.

Such a manifold $\mathcal{M}$ is called an inertial manifold.

Notice that the existence of the IM for a particular problem (1) needs to be proven and requires some strong conditions on the operator spectrum (spectral gap condition; see [59,60]). Since we are interested in the finite-dimensional representation (2) of the original system (1), we need to assume that the number of Degrees Of Freedom (DOFs) is sufficient to represent the dynamics of the system on the inertial manifold, say using the Galerkin method. Approaches to define these DOFs for different dissipative dynamical systems can be considered in [5,60,61] and, in particular, in [62] for the application in our research related to the problems in fluid dynamics.

Remark 1.

The number (n) of DOFs in the approximation (2) is not directly related to the dimension of the IM (m) given in Definition 1. Sufficient conditions should be provided for the value of n that can represent the dynamics on the IM, for example for space periodic boundary conditions; see [61], Inequality (1.13).

If such conditions are met and withthe system (2), then one can see from the properties in the Definition 1 that the RK methods applied to such systems can simulate dynamics on the approximate inertial manifold if the selected RK methods satisfy the stability conditions. If the condition on the existence of the IM is not met, one can assume the existence of the absorbing manifold that attracts trajectories algebraically quickly.

Nevertheless, for the IM and general absorbing and attracting manifolds, it is desirable to increase the precision of the RK methods and reduce the influence of rounding-off errors because of the usage of floating-point arithmetic. A general pessimistic estimate for the floating-point representation of Runge–Kutta methods is well known; for example, see [63]. Let us designate the floating-point representation of ${\mathbf{u}}_n$ on step n as $\widetilde{{\mathbf{u}}_n}$. Then, the floating-point error ${\tilde{e}}_n={\mathbf{u}}_n-\widetilde{{\mathbf{u}}_n}$ can be estimated as:

$$\parallel {\tilde{e}}_n\parallel <C{\gamma }_n/\tau ,$$

(5)

where C is a constant (encompassing a maximum from a solution value at different steps that do not depend on the time step values), $\tau$ is the time step floating-point representation, and ${\gamma }_n$ is the quantity of the rounding error estimates, given as:

$${\gamma }_n=\frac{n\epsilon }{1-n\epsilon },$$

when $n\epsilon <1$ and $\epsilon$ is the unit round-off (or macheps) of the selected floating-point arithmetic, usually represented as $\epsilon =2^{-p}$, where p is the number of significant bits (for example, $\epsilon =2^{-53}$ for double-precision in the IEEE 754 standard). One can observe that, for many time steps and a small value of ${\tau }_n$, the round-off floating-point error becomes significant. Notice that a refined estimate is provided in [64].

The improvement of the estimate (5) can be made by either using extended precision floating-point arithmetic or by using the compensated summation algorithm. The first approach can be achieved by using standard libraries for CPUs or custom implementations for GPUs, for example [65]. However, it is known that, for GPUs, such an approach greatly reduces performance. In our research, we used the second approach, which is discussed in detail in [54]. We shall indicate any operation that is executed using floating-point arithmetic as $\mathrm{fl}(·)$, which includes all operations inside the brackets. The floating-point-compensated operations that are to be used are based on error-free operations, presented in Algorithms 1 and 2; for more information, see [54].

Algorithm 1 Error-free addition s.t. $s+c=a+b$, $s=\mathrm{fl}(a+b)$, and $c\le \epsilon (a+b)$.

1: function add(a, b) 2:     $s=\mathrm{fl}(a+b)$; 3:     $z=\mathrm{fl}(s-a)$; 4:     $c=\mathrm{fl}\left(\right(a-(s-z))+(b-z\left)\right)$; 5:     return (s, c);

Algorithm 2 Error-free multiplication s.t.: $s+c=ab$, $s=\mathrm{fl}\left(ab\right)$, and $c\le \epsilon \left|ab\right|$

1: function MUL(a, b) 2:     $s=\mathrm{fl}\left(ab\right)$; 3:     $c=\mathrm{FMA}(a,b,-s)$; 4:     return (s, c);

In light of the RK methods, one can implement such an approach as follows. For simplicity, we shall consider the autonomous variant of the method. Let some s-stage RK-method be given in the Butcher’s table in the form of a matrix $\mathbf{A}$ and a vector $\mathbf{b}$. As already mentioned, the methods are either SSP-stable (for explicit methods) or B-stable for diagonally implicit methods. For explicit RK methods with the provided general nonlinear function rhs, the system of equations between steps n and $n+1$ with summation and product compensations can be written in Algorithm 3. Implicit diagonal RK methods can be constructed in the same fashion, with the additional stage of using Newton’s method to obtain the diagonal values of $\mathbf{k}$. One can formulate the following:

Algorithm 3 Compensated explicit autonomous Runge–Kutta method single-step from n to $n+1$. All variables are presented in the base floating-point precision.

1: function erk_internal($\tau$, $(\mathbf{u},\mathbf{r})$, w, $\mathbf{k}$)   2:     $(d,p_1)\leftarrow \mathrm{MUL}(\tau ,w)$;   3:     $(\mathbf{d},{\mathbf{p}}_2)\leftarrow \mathrm{MUL}(d,\mathbf{k})$;   4:     $(\mathbf{u},{\mathbf{p}}_3)\leftarrow \mathrm{ADD}(\mathbf{u},\mathbf{d})$;   5:     $\mathbf{r}\leftarrow \mathbf{r}+p_1+{\mathbf{p}}_2+{\mathbf{p}}_3$;   6:     return ($\mathbf{u}$, $\mathbf{r}$)   7: function erk(${(\mathbf{u},\mathbf{r})}_{n+1}$, ${(\mathbf{u},\mathbf{r})}_n$, ${\tau }_n$, rhs, $\mathbf{A}$, $\mathbf{b}$)   8:     ${\mathbf{k}}_{j=1,..,s}\leftarrow \mathbf{0}$; $s\leftarrow \mathrm{LEN}\left(\mathbf{b}\right)$;   9:     for $j=1,\cdots ,s$ do 10:         $\mathbf{u}\leftarrow {\mathbf{u}}_n+{\mathbf{r}}_n$; $\mathbf{q}\leftarrow \mathbf{0}$; 11:         for $l=1,\cdots ,s$ do 12:            $(\mathbf{u},\mathbf{q})\leftarrow \mathrm{ERK}\_\mathrm{INTERNAL}({\tau }_n,(\mathbf{u},\mathbf{q}),A_{j,l},{\mathbf{k}}_l)$; 13:         ${\mathbf{k}}_j\leftarrow \mathrm{RHS}(\mathbf{u}+\mathbf{q})$; 14:     ${(\mathbf{u},\mathbf{r})}_{n+1}\leftarrow {(\mathbf{u},\mathbf{r})}_n$; 15:     for $j=1,\cdots ,s$ do 17:         ${(\mathbf{u},\mathbf{r})}_{n+1}\leftarrow \mathrm{ERK}\_\mathrm{INTERNAL}({\tau }_n,{(\mathbf{u},\mathbf{r})}_{n+1},b_j,{\mathbf{k}}_j)$; 17:     return ${(\mathbf{u},\mathbf{r})}_{n+1}$;

Theorem 1.

Let the ODE system (2) be given with an identity matrix for $\mathtt{M}$ and Lipschitz continuous nonlinear function $\mathbf{F}$. Let the provided initial conditions be consistent with respect to the function $\mathbf{F}$. Let the basic floating-point arithmetic (according to the IEEE 754 standard) be used, and the compensated explicit Runge–Kutta method in the Algorithm 3 is applied to the system (2) with such a sequence of time steps $\left\{{\tau }_n\right\}$, such that $\forall n:\left|g\right({\tau }_n,\mathbf{A},\mathbf{b},{\mathbf{u}}_n,\mathbf{F}\left)\right|<1$, and now, the underflow of floating-point representation occurs; g is an RK method stability function. Then, the following estimate for the round-off error is true:

$$\parallel {\tilde{e}}_n\parallel <\frac{C{\gamma }_n^2}{\tau },$$

(6)

where τ is the minimum accepted time step value and C is a constant that does not depend on the time step.

Proof. 

First, let us obtain an estimate (5). A single step of an S-stage explicit RK method with Butcher’s table coefficients $\mathbf{A}$, $\mathbf{b}$ is given as:

$$\begin{array}{c}{\mathbf{K}}_j=\mathbf{F}({\mathbf{u}}_n+{\tau }_n\sum _{l=1}^{j-1}{a}_{j,l}{\mathbf{K}}_l),j=1,\cdots ,S\\ {\mathbf{u}}_{n+1}={\mathbf{u}}_n+{\tau }_n\sum _{j=1}^S{b}_j{\mathbf{K}}_j,\end{array}$$

Let us consider a single-step in floating-point operations.

$$\begin{array}{c}\mathrm{fl}\left({\mathbf{K}}_j=\mathbf{F}({\mathbf{u}}_n+{\tau }_n\sum _{l=1}^{j-1}{a}_{j,l}{\mathbf{K}}_l)\right),j=\{1,\cdots ,S\}\\ \mathrm{fl}\left({\mathbf{u}}_{n+1}={\mathbf{u}}_n+{\tau }_n\sum _{j=1}^S{b}_j{\mathbf{K}}_j\right),\end{array}$$

We used the estimates from [54], Appendix A (Proofs of lemmas and propositions), the Lipschitz continuity of $\mathbf{F}$ with Lipschitz constant L, and the stability condition for g, under which we assume that ${\tau }_{n}L\le 1$. Then,

$$\parallel {\mathbf{K}}_j-\mathrm{fl}\left({\mathbf{K}}_j\right)\parallel <(L\epsilon +4j\epsilon )\parallel {\mathbf{u}}_n\parallel ,j=\{1,\cdots ,S\}.$$

Then, for the first step, we have:

$$\parallel \mathrm{fl}\left({\mathbf{u}}_1\right)-{\mathbf{u}}_1\parallel <(2\epsilon +4{\tau }_1(3S^2+S)/2\epsilon )\parallel {\mathbf{u}}_0\parallel .$$

By using induction and maximizing the time steps ($\tau ={max}_{k\in [1\cdots n]}{\tau }_n$), we arrive at:

$$\parallel \mathrm{fl}\left({\mathbf{u}}_{n+1}\right)-{\mathbf{u}}_{n+1}\parallel <(2n\epsilon +2\tau n(3S^2+S)\epsilon )\underset{k\in [1\cdots n]}{max}\parallel {\mathbf{u}}_k\parallel .$$

The latter can be bounded to obtain Inequality (5):

$${\tilde{e}}_n=\parallel {\tilde{\mathbf{u}}}_n-{\mathbf{u}}_n\parallel <\tau n(2/\tau +6S^2+2S)\epsilon C_1\lesssim C{\gamma }_n/\tau .$$

Now, we considered Algorithm 3. The same manipulations can be performed, taking into account the compensated algorithms and using prefix induction; see Lemma 1 from [54]. Then:

$$\parallel {\mathbf{K}}_j-\mathrm{fl}\left({\mathbf{K}}_j\right)\parallel <(L{\epsilon }^2+6j{\epsilon }^2)\parallel {\mathbf{u}}_n\parallel ,j=\{1,\cdots ,S\},$$

$$\parallel \mathrm{fl}\left({\mathbf{u}}_{n+1}\right)-{\mathbf{u}}_{n+1}\parallel <(2n{\epsilon }^2+3\tau n(9S^2+3S)/2{\epsilon }^2)\underset{k\in [1\cdots n]}{max}\parallel {\mathbf{u}}_k\parallel .$$

The latter can be bounded to obtain the results of the theorem as:

$${\tilde{e}}_n=\parallel {\tilde{\mathbf{u}}}_n-{\mathbf{u}}_n\parallel <\tau n(2/\tau +12S^2+6S){\epsilon }^2{C}_1\lesssim C{\gamma }_n^2/\tau .$$

□

Remark 2.

In actual computations, however, the estimate (6) is very pessimistic. Many floating-point errors are compensated and are not essential during long-time calculations.

Let us demonstrate that this approach not only allows one to decrease the time step close to $\epsilon$, but also to simulate linearly unstable solutions for a substantial simulation time with no loss of accuracy. We considered the Dahlquist equation $u_t=u$ subject to the initial conditions $u\left(0\right)=1$ with an exact unstable solution $u\left(t\right)=exp\left(t\right)$. One can observe the error ${\tilde{e}}_n$ between the exact solution and approximate floating-point solutions in Figure 3.

The results demonstrate that the suggested approach works even better than the application of quadruple-precision floating-points by a factor of 5. Since the base macheps $\epsilon =2^{-53}\sim 1.11\times {10}^{-16}$, one can safely decrease the time step to the order of several magnitudes of $\epsilon$, say about $1.0\times {10}^{-12}$, with no worries about losing precision due to the round-off errors. This allows one to select small time steps in the adaptive time step control strategy for general nonlinear $\mathbf{F}$ for chaotic attractors near stiff regimes. This approach allowed the authorsto obtain very sensitive attractors of nonlinear systems, such as 3D invariant tori, for example [37].

2.2. Deflation with Discrete Symmetry

In order to reduce the complexity of the problem, it is beneficial to reduce the symmetries of the system if those are known. The symmetries are saved in the system (2) in the case that a correct spatial discretization method is used. One of the most-important reductions is the reduction of continuous symmetries related to the $SO\left(n\right)$ group. If such a reduction cannot be performed on a discrete level, then it becomes a difficult problem and will be discussed elsewhere. The reduction of discrete symmetries, however, can be performed more easily if the deflated Newton method is used. The discussion here is based on the definitions from [27,30]. First, we consider stationary solutions:

$$\mathbf{F}\left(\mathbf{u}\right)=\mathbf{0},$$

(7)

for a fixed value of the parameter R, which is omitted. Recap the following definition from [27]:

Definition 2.

Let B, C, and D be Banach spaces and U be an open subset of B. Let $\mathbf{F}:U\subset B\to C$ be a Frechet differentiable operator with derivative ${\mathbf{F}}^{\prime }$. For each $\mathbf{r}\in U$, $\mathbf{u}\in U\setminus \left\{\mathbf{r}\right\}$, let $M(\mathbf{u};\mathbf{r}):C\to D$ be an invertible linear operator. We say that M is a deflation operator if, for any $\mathbf{F}$ s.t. $\mathbf{F}\left(\mathbf{r}\right)=0$ and ${\mathbf{F}}^{\prime }\left(\mathbf{r}\right)$ nonsingular, we have:

$${\displaystyle \underset{j\to \infty }{lim}inf{\parallel M(\mathbf{u};\mathbf{r})\mathbf{F}\left({\mathbf{u}}_j\right)\parallel }_D>0,}$$

(8)

for any sequence $\left\{{\mathbf{u}}_j\right\}$ converging to $\mathbf{r}$, where ${\mathbf{u}}_j\in U\setminus \left\{\mathbf{r}\right\}$.

Let the system (1) be recast into the deflated system by using this definition with the multiple deflation operator G:

$$G(\mathbf{u},S)\mathbf{F}\left(\mathbf{u}\right)=\mathbf{0},$$

(9)

where $S:=\left\{{\mathbf{r}}_j\right|\forall j=1,\cdots ,K:\mathbf{F}\left({\mathbf{r}}_j\right)=0\}$ is the set that contains all K known or previously found solutions. We shall use a particular form of the deflation operator from [30]:

$$G\left(\mathbf{u},S\right)=\frac{1}{K}\mathrm{diag}\left(\sum _{j=1}^K\left(\parallel \mathbf{u}-{\mathbf{u}}_j{\parallel }_2^{-p}+{\alpha }_j\right)\right),$$

(10)

where ${\alpha }_k$ is the shift value, which can be chosen in order to improve the convergence of iterative methods.

Let ${\sigma }_k$ be a non-trivial element of some discrete finite group $\sigma$ acting on the solution of the system (7). The nonlinear operator $\mathbf{F}$ is invariant under the action of the elements of this group, i.e., if $\mathbf{u}$ is a solution of (7), then $\mathbf{F}\left({\sigma }_k\mathbf{u}\right)={\sigma }_k\mathbf{F}\left(\mathbf{u}\right)=\mathbf{F}\left(\mathbf{u}\right)=0$. However, the deflated system (9) will not be able to take this group element into account (since $\parallel {\sigma }_k\mathbf{u}-\mathbf{u}\parallel >0$), and Newton’s method will converge to the solution. One way to deal with this problem is to wait until all such solutions are found. However, such an approach may require too much memory to store all those solutions. The other approach that we adopted here is to use a modified deflated operator s.t. taking known discrete symmetry elements into account. Such a deflation operator we shall define as a discrete symmetry-invariant deflation operator.

Definition 3.

Let B, C, and D be Banach spaces and U be an open subset of B. Let ${\sigma }_k$ be an element of a discrete finite group σ. Let $\mathbf{F}:U\subset B\to C$ be invariant under the action of elements from the σ Frechet differentiable operator with derivative ${\mathbf{F}}^{\prime }$. For each ${\sigma }_k\mathbf{r}\in U$, $\mathbf{u}\in U\setminus {\cup }_k\left\{{\sigma }_k\mathbf{r}\right\}$, let $M(\mathbf{u};\mathbf{r}):C\to D$ be an invertible linear operator. We say that M is a discrete symmetry invariant deflation operator or D-deflation operator if, for any $\mathbf{F}$ and ${\sigma }_k\in \sigma$, such that $\mathbf{F}\left({\sigma }_k\mathbf{r}\right)=0$ and ${\mathbf{F}}^{\prime }\left({\sigma }_k\mathbf{r}\right)$ nonsingular, we have:

$${\displaystyle \underset{j\to \infty }{lim}inf{\parallel M(\mathbf{u};\mathbf{r})\mathbf{F}\left({\mathbf{u}}_j\right)\parallel }_D>0,}$$

(11)

for any sequence $\left\{{\mathbf{u}}_j\right\}$ converging to ${\sigma }_k\mathbf{r}$, where ${\mathbf{u}}_j\in U\setminus {\cup }_k\left\{{\sigma }_k\mathbf{r}\right\}$.

The actual implementation of such a D-deflation operator can be performed in different ways. In the software implementation, we propose the following modification of (10):

$$G\left(\mathbf{u},S,Q\right)=\frac{1}{K+\left|Q\right|}\mathrm{diag}\left(\sum _{j=1}^K\left(\sum _{k=1}^{\left|Q\right|}{\parallel {\sigma }_k\mathbf{u}-{\mathbf{u}}_j\parallel }_2^{-p}\right)+{\alpha }_j\right),$$

(12)

where the set Q contains the description of all elements of known discrete groups in terms of the action of this group on $\mathbf{u}$. ${\sigma }_k\in Q$. The whole nonlinear operator now becomes:

$$G(\mathbf{u},S,Q)\mathbf{F}\left(\mathbf{u}\right)=\mathbf{0},$$

(13)

This form of the proposed operator (13) retains the rather simple form of the original Jacobian operator that was derived in [30].

3. Bifurcation Analysis and Chaotic Dynamics in ABC Flow

In this section, we provide an example of the bifurcation and chaotic analysis of a fluid dynamics problem: a well-known problem of the dynamics in the ABC flow [66] is considered. Although it has been studied intensively, the transition to chaos in this problem is still not fully understood. This example is mainly a demonstration of the approach used by our methodology and software applied to large-scale problems in fluid dynamics. However, some new results were obtained.

The governing equations are given by the incompressible Navier–Stokes equations:

$$\begin{array}{c}\hfill {\mathbf{u}}_t+(\mathbf{u},\nabla )\mathbf{u}+\nabla p-\frac{1}{R}△\mathbf{u}-\frac{1}{R}\mathbf{f}=0,\\ \hfill \nabla ·\mathbf{u}=0,\end{array}$$

(14)

on the periodic domain ${\mathbb{T}}^3:=[-\pi ;\pi ]\times [-\pi ;\pi ]\times [-\pi ;\pi ]$, and the forcing term is given as:

$$\mathbf{f}:={(Asin\left(x_3\right)+Ccos\left(x_2\right);Bsin\left(x_1\right)+Acos\left(x_3\right);Csin\left(x_2\right)+Bcos\left(x_1\right))}^{\mathrm{T}},$$

(15)

where $A,B,C$ are positive whole numbers and R is the positive real number (Reynolds number) that is used as the bifurcation parameter. Note that $\mathbf{f}$ is divergence-free. Equations (14) and (15) formulate the problem (1). The problem has a trivial base solution:

$$\mathbf{u}=\mathbf{f}.$$

(16)

A finite-dimensional problem is formulated by expanding the solution using a finite set of trigonometric polynomials and applying Bubnov–Galerkin and divergence-free projections; we arrive at the following ODE system:

$${\widehat{\mathbf{u}}}_t+\mathtt{P}\mathcal{B}(\widehat{\mathbf{u}},\widehat{\mathbf{u}})-R^{-1}\mathtt{C}\widehat{\mathbf{u}}-R^{-1}\widehat{\mathbf{f}}=\mathbf{0},$$

(17)

where $\widehat{\mathbf{u}}$ is the vector of expansion coefficients (DOFs), $\mathtt{E}$ is the identity matrix, $\mathtt{G}$ is the matrix that represents the gradient operator, ${\mathtt{G}}^T$ is the matrix that represents the divergence operator, $\mathtt{C}$ is the diagonal matrix that represents the Laplace operator, and $\mathtt{P}=\mathtt{E}-\mathtt{G}{\mathtt{C}}^{-1}{\mathtt{G}}^T$ is the Leray–Helmholtz orthogonal projector. The convolution term (nonlinearity) is designated as $\mathcal{B}$, and $\widehat{\mathbf{f}}$ is the ABC forcing term (15) projected to the approximating space of polynomials. The system (17) is basically the problem (2) with a unit mass matrix. In order to accelerate the convergence of Newton’s method, we applied the preconditioned gradient descent method before each Newton method iteration.

A detailed analysis of this problem was performed in [67,68,69] by O.Podvigina and co-authors. In this section, we are interested in diving into more details on the initial stage of transition to chaos and constructing bifurcation diagrams to unravel the interaction between stationary, periodic solutions and chaotic regimes. Here, we only considered the case $A=B=C=1$ and $1\le R\le 20$. The system (17) is invariant under the action of 24 elements isomorphic to O (octahedral symmetry) and $S_4$ (permutations of four elements) group elements; see [70] for more details. All elements of these groups were added to the set Q in (12). For the visualization of the bifurcation diagrams, we used the following set of functions, depicted on the figure axes:

$$\begin{array}{c}\mathrm{L}2\mathrm{ph}:={\parallel \mathbf{u}\parallel }_{L2}:=\sqrt{{\left(8{\pi }^3\right)}^{-1}{\int }_{{\mathbb{T}}^3}{\left|\mathbf{u}\left(\mathbf{x}\right)\right|}^{2}d\mathbf{x}},\\ \mathrm{L}2:=\parallel \widehat{\mathbf{u}}{\parallel }_{l2}:=\sqrt{N^{-1}{\sum }_{\mathbf{j}=0}^N{\widehat{u}}_{\mathbf{j}}{\widehat{u}}_{\mathbf{j}}^{*}},\\ \mathrm{a}:=u_x\left(\frac{4\pi }{3},\frac{2\pi }{3},\frac{2\pi }{3}\right),\mathrm{b}:=u_x\left(\frac{4\pi }{5},\frac{4\pi }{3},\frac{2\pi }{3}\right),\\ \mathrm{c}:=u_x\left(\pi ,\frac{2\pi }{5},\frac{2\pi }{5}\right),{\mathrm{c}}^{\prime }:=u_y\left(\pi ,\frac{2\pi }{5},\frac{2\pi }{5}\right),\mathrm{d}:=u_x\left(2\pi ,\pi ,\pi \right).\end{array}$$

(18)

We applied the discretization of ${128}^3$ for this problem; hence, the number of DOFs is $N=6\times 128\times 128\times (128/2+1)-1-128/2-128/2-2(128\times 128)/2\simeq 6.37\times {10}^6$ due to the reality condition on the functions. We used two Nvidia GTX TITAN BLACK GPUs to perform the calculations, the base double-precision with error compensation on the blas1 operations, and the compensated ERK fifth-order method (Algorithm 3) for the time-dependent solutions.

3.1. Bifurcation Analysis

First, we performed the deflation and continuation of all stationary solutions. This was performed by first tracing the solution (16), adding solutions at knot points to the deflation operator (for more information; see [30]), and then, using the deflation algorithm on the set of random vectors; each random vector was initialized by divergence-free perturbations. After a new solution was found, it was traced by the pseudo-arc length continuation algorithm. The cycle was repeated until no more solutions were found on the deflation step. For this procedure, we called bifurcation_analysis::deflation_operations and bifurcation_analysis::pseudo_arclength_continuation classes, presented in Figure 2. These methods use the concepts and classes that are provided in Figure 1.

A bifurcation diagram for stationary solutions is presented in Figure 4. Solutions of the problem in physical space are represented at some points by using the iso-surfaces of velocity vector magnitude. Bellow, we shall demonstrate that these stationary solutions are essential for understanding the process of the transition to chaos.

Next, the stability analysis is executed, and bifurcation points are located in the stationary solutions. These bifurcation points are represented in Figure 5 by blue crosses. For the trivial solution branch, all bifurcation points are shown, while only the first loss of stability is shown on secondary branches. The dimension of the unstable manifold is 35 by the last presented bifurcation at $R=15.006$ on the main branch, so we can expect the fast growth of instabilities. The existence of multistable stationary solutions can be seen for $7.8753\le R\le 13.0059$ between the main trivial branch and two disconnected branches. Additionally, one can see the disconnected nature of these branches in the 3D Figure 6. There are three independent Hopf bifurcations on each branch that form periodic orbits: at $R=13.0059$ for the trivial branch and at $R=13.8922$ at secondary branches.

The presented branches of periodic orbits were obtained using the reduction of the symmetry of the problem; otherwise, multiple branches would cover the bifurcation diagram entirely for non-$L_2$-norms after the first Hopf bifurcation at $R=13.09$ and would coincide for $L_2$-norms. The periodic orbits that emanated from the secondary branches have a smaller period, with a ratio of about $1/3$ to the periodic orbits on the main branch.

Next, we considered the stabilization of the periodic orbits that bifurcate from the Hopf bifurcation points. The bifurcation diagram is presented in Figure 7 and more projections in different coordinates in Figure 8 and Figure 9.

The periodic orbits that emanated from the main branch lose stability almost immediately by the second Hopf bifurcation at $R=13.09$ and, then, the third Hopf bifurcation at $R=13.122$; stable invariant 2D and 3D tori are presented in Figure 10 and Figure 11, respectively, along with stabilized periodic orbits. At $R=13.128$, the 3D torus is transformed into the 2D torus given in Figure 12 by phase locking. This transition is equivalent to the bifurcation scenario outlined in [69].

3.2. Analysis of Chaotic Regimes

Five leading Lyapunov exponents were calculated for the considered chaotic regimes to indirectly indicate chaotic behavior (in general, we believe that such an indication is prone to rounding errors and inaccuracy; see [71] for more details). The procedure of calculating Lyapunov exponents is Jacobian-based with numerically stable reorthogonalization (which is executed as soon as the iteration time is reached or the norm of perturbations increases by 100-times). Each iteration consists of $1\times {10}^5$ time steps; for more information on calculating Lyapunov exponents for large-scale problems see [72].

For $R>13.14$, we observe a chaotic behavior; an example of such behavior is shown in Figure 13. The chaotic attractor is located around an unstable periodic orbit, thus producing low-dimensional chaos. The eigenvalues of the monodromy matrix, minus the value at $(1,0)$, for the given periodic orbit indicate at most a three-dimensional unstable manifold, partially confirmed by the Lyapunov exponents in Figure 14, left. This approach is characterized by low-dimensional chaotic behavior. In this problem, we observed the transition to chaos through 2D and 3D stable invariant tori, followed by phase locking.

Now, we turn our attention to the chaotic regimes of the considered problem beyond low-dimensional chaos. Starting from $R>13.9$, an intermittent pattern is formulated, which cannot be explained by the stabilized periodic orbits; see Figure 15, where an intermittent pattern with stabilized periodic orbits on this pattern is provided.

The dimension of the unstable manifold with respect to the stabilized periodic orbits is much larger, and 20 leading eigenvalues of the monodromy matrix for both periodic orbits are complex-conjugate pairs located outside the unit circle; see Figure 16. At this stage, we already obtained a set of bifurcations on the unstable stationary solution; see Figure 5. Such a situation usually indicates the increase of the dimension of the unstable manifold. This is partially confirmed by the Lyapunov exponents in Figure 14, right, where all five leading exponents are larger than zero.

The chaotic dynamics is revealed when stationary solutions are considered. A zoomed area is presented in Figure 17.

The trajectory is governed by the stable and unstable manifolds around stationary solutions, represented by points in the plots (cut of bifurcation diagram in Figure 4 at $R=15$). The solutions are controlled by the inherited symmetry of the problem. The stationary unstable solutions are likely to form heteroclinic connections around which the chaotic intermittency is developed. A more-probable clustering of trajectories is presented around new branches of unstable solutions presented in Figure 17 (left), and a less-probable clustering of trajectories is presented around a base flow branch shown in Figure 17 (right). This chaotic intermittent behavior continues at least until the maximum $R=20$, while more symmetries are evolved and $R=20$, almost fully developed chaos is formed; see Figure 18 and Figure 19. An additional unstable stationary branch is obtained by the action of the $S_4$ symmetry element on the second branch; this symmetric branch solution’s $L2$- and $L2ph$-norms coincide.

4. Discussion

In this paper, we presented a methodology and software for bifurcation and chaotic analysis of abstract dynamic systems given by the formal Equations (1). The software assumes that the user side provides the correct and an efficient implementation of the solution methods for the abstract system, which is listed in Figure 1. The provided approach for the compensated evolution of Runge–Kutta integration for floating-point arithmetic allowed us to trace attractors of higher dimensions (such as a 3D invariant torus). It is interesting to note that the structure of the bifurcation analysis software allows one to add new methods and computational technologies without modifying the underlying algorithms. Still, such an approach may not be optimal, as it is based on the heavy use of C++ template metaprogramming with parallel heterogeneous architectures. Hence, the user experience must be quite high, and it is possible that modern programming languages such as Python and Julia can be more user-friendly for medium-sized problems.

In this research, we presented the analysis of the ABC flow problem, mostly for demonstration purposes; we used only the reviewed software for the analysis. Most of the work, such as the deflation and stability analysis of stationary solutions and the deflation and tracing of periodic solutions, was performed in an automated manner. However, one must point out that the numerical analysis of the quasi-periodic solutions is still a challenging task requiring manual labor. Some papers are known to deal with this problem, for example [73], but additional research is required to determine whether such methods can be used for large-scale problems and can be incorporated into the automated bifurcation analysis software.

The ABC flow that was analyzed in the previous section is of separate interest on its own. The initial stage of the transition to chaos is almost identical to the results from [69]. In addition to the cited paper, we presented detailed bifurcation diagrams of stationary and periodic solutions. This allowed us to reveal more information about the chaotic regime, where chaos is being developed around unstable stationary solutions rather than unstable periodic orbits. This gives us more information on the structure of chaos; it also forces one to study not only unstable periodic orbits of the Navier–Stokes flow (for example, see [74]), but also unstable stationary branches. Our future work will focus on perfecting the software and performing a detailed bifurcation analysis of different fluid dynamics problems that are currently beyond the capabilities of other software.