Piecewise Neural Network Method for Solving Large Interval Solutions to Initial Value Problems of Ordinary Differential Equations

Abstract

Traditional numerical methods often provide local solutions for initial value problems of differential equations, even though these problems may have solutions over larger intervals. Current neural network algorithms and deep learning methods also struggle to ensure solutions across these broader intervals. This paper introduces a novel approach employing piecewise neural networks to address this issue. The method involves dividing the solution interval into smaller segments and utilizing neural networks with a uniform structure to solve sub-problems within each segment. These solutions are then combined to form a piecewise expression representing the overall solution. The approach guarantees continuous differentiability of the obtained solution over the entire interval, except for finite end points of those sub-intervals.To enhance accuracy, parameter transfer and multiple rounds of pre-training are employed. Importantly, this method maintains a consistent network size and training data scale across sub-domains, unlike existing neural network algorithms. Numerical experiments validate the efficiency of the proposed algorithm.

1. Introduction

As we know, constrained by convergence requirements, various traditional numerical solution methods, such as Picard approximation and Runge–Kutta methods, for initial value problems (IVPs) of differential equations or dynamic systems (ODEs) are typically suitable only for local solutions within a neighborhood interval of the initial value point, even if the problem possesses larger interval (global) solutions. Consequently, extending a local solution to a larger interval remains a fundamental challenge in the field of numerical solutions to ODE problems, which has yet to be fully addressed [1,2].

Currently, the artificial neural network (ANN) algorithm, as a novel numerical solution method for solving differential equation problems, is rapidly evolving and attracting attention and exploration from many researchers [3]. Although the ANN algorithm, like traditional methods, performs well mainly in the neighborhood of the initial value point and lacks inherent learning capability on large intervals, it offers numerous advantages in solving IVPs of ODEs. For instance, the neural network-based solution to a differential equation is differentiable and presented in closed analytic form, suitable for subsequent calculations. It is independent of discrete schemes and the shape of the variable domain. Therefore, it is anticipated that the aforementioned solution extension problem can be effectively addressed through the enhancement and efficient utilization of the ANN algorithm. In this paper, we propose a piecewise neural network (PWNN) method to tackle this problem.

Before describing our algorithm, let us briefly review the development profile of ANN algorithms for solving differential equation problems.

The theoretical foundation supporting the use of ANN algorithms for solving differential equation problems lies in the general approximation theorems provided by Hornik, Womik, and Li X et al. [4,5,6], which theoretically establish that any continuous function can be approximated by a neural network. Subsequently, a widely utilized ANN algorithm for solving differential equations with initial boundary value conditions was introduced by Lagaris et al. [7]. However, for problems involving more complex initial boundary values, this method faces difficulties in constructing the necessary trial solutions for the underlying problems.

Another significant development in ANN algorithms for solving differential equation problems is the implementation of automatic differentiation technology [8,9,10,11], which has led to the proliferation of ANN methods for solving differential equation problems and their broader application. Building upon this technique, Justin Sirignano and Konstantinos Spiliopoulos proposed the Deep Galerkin Method (DGM), a deep neural network method for solving (partial) differential equation problems in higher dimensions [12]. Anitescu et al. introduced an ANN method utilizing adaptive configuration strategies to enhance method robustness and reduce computational costs in solving boundary value problems of differential equations [13].

A landmark advancement in ANN algorithms for solving differential equation problems was the introduction of physical information neural networks (PINN) by Raissi et al. [14]. In this method, the initial boundary conditions of the differential equation are incorporated into the loss function, allowing the ANN to directly express an approximate solution to the differential equation. This eliminates the need to construct a trial solution according to equations and initial boundary value conditions, as required by Lagaris’s method. Subsequently, numerous ANN algorithms based on PINN have been developed rapidly. For example, Lei Yuan and Ni et al. introduced the Auxiliary PINN (A-PINN), capable of bypassing limitations in integral discretization and solving forward and inverse problems of nonlinear integral differential equations [15]. Chiu et al. proposed novel PINN methods for coupling neighboring support points and their derivative terms obtained by automatic differentiation [16]. Huang et al. combined PINN with the homotopy continuation method, proposing Homotopy PINN (HomPINN) for solving multiple solutions of nonlinear elliptic differential equations, overcoming the limitation of PINN in finding only the flattest solution in most cases [17]. Yin et al. utilized PINN to address a range of femtosecond optical soliton solutions pertaining to the high-order nonlinear Schr$\ddot{\mathrm{o}}$dinger equation [18]. Based on the PINN method, Yuexing et al. proposed an enhanced version of PINN called IPINN, introducing localized adaptive activation functions to improve performance, successfully applying the method to solve several differential equation models in finance [19]. Meng et al. introduced a modified PINN method called PPINN, dividing a long-period problem into a series of short-period ones to accelerate the training of ANN algorithms [20].

Moreover, there are numerous other analogs of PINN-based studies and various types of ANN methods. For instance, convolutional ANN methods [21,22,23] and theoretically guided ANN methods [24,25] have also been employed to explore new solution methods for (partial) differential equations. Particularly noteworthy is the work of Run-Fa Zhang and Sudao Bilige, who proposed bilinear neural networks, the first attempt to obtain analytical solutions to nonlinear partial differential equations using the ANN method [26,27].

Importantly, a general observation from this literature is that ANN algorithms for solving IVPs of differential equations perform well in relatively small domains near the initial value point but sometimes exhibit poor convergence over large intervals. To obtain a large interval solution, it is often necessary to enhance the training data and expand the network size (depth and width). However, doing so not only reduces computational efficiency but also makes it challenging to guarantee obtaining a large interval solution to the problem. Therefore, alternative strategies are being explored to enhance the efficiency of ANN algorithms. For example, various adaptive activation functions [19,28,29,30] and adaptive weights [31,32] have been proposed. The key feature of these studies is the introduction of hyperparameters into traditional activation functions to adjust ANN convergence. Studies by Jaftap et al. have shown that these methods are primarily effective in the early stages of network training [33]. Adaptive weights leverage gradient statistics to optimize the interplay between various components in the loss function by incorporating additional weighting during the training process. While these solutions alleviate the convergence difficulties of neural networks in complex problems from different perspectives, there is insufficient evidence to demonstrate that simultaneously using different optimization schemes can still reduce network training difficulty and achieve large interval solutions. To solve the IVP of dynamic systems, Wen et al. proposed an ANN algorithm based on the Lie symmetry of differential equations [34,35,36,37]. This novel approach combines Lie group theory and neural network methods. Fabiani et al. proposed a numerical method [38] for IVPs of stiff ODEs and index-1 differential algebraic equations (DAEs) using random projections with Gaussian kernels and PINNs, demonstrating its efficiency against standard solvers and deep learning approaches.

Given the aforementioned challenges, this paper proposes an innovative approach for obtaining large interval approximate solutions to IVPs of ODEs. In this algorithm, the interval is divided into several small compartments, and a neural network solution is learned on each compartment using PINN. Consequently, the trained ANNs generate fragments of the large interval solution to the IVP of an ODE on those sub-intervals, respectively. For each specific sub-interval, the ANN training requires neither complex structure nor dependence on large-scale training data, significantly reducing computational overhead. Subsequently, by assembling these neural network solutions, a piecewise expression of the large interval solution to the problem is constructed, termed the piecewise neural network solution. The compatibility of these sub-interval network solutions and the continuity and differentiability of the piecewise solution over the entire interval are investigated theoretically. Transfer learning techniques of network parameters between adjacent neural networks and multiple rounds of pre-training approaches in the training procedure are utilized.

The innovative contributions of this paper are threefold. First, a new ANN method for solving the extension problem of the local solution to the initial value problem of differential equations is presented. Second, a novel method of generalizing ANN and its application is introduced. Third, a new approach is explored in which modern algorithms (artificial neural networks, deep learning) are used to overcome the shortcomings of traditional methods for solving numerical solutions to differential equation problems. Our work aims to provide a more comprehensive understanding of the ANN method to solve ODE problems in both theoretical and practical aspects.

The structure of this paper is organized as follows. For completeness, Section 2 briefly reviews the PINN method for IVPs of ODE systems; Section 3 details the PWNN algorithm proposed in this paper; Section 4 presents the theoretical analysis of PWNN solutions and implementation of the proposed method; and Section 5 demonstrates the applications of the proposed method in solving several specific IVPs of ODEs. Concurrently, comparisons between the results of our method and those of the PINN method and the Runge–Kutta method are presented to demonstrate the efficiency of the proposed algorithm. Finally, Section 6 summarizes the work.

2. A Brief Recall of PINN

Consider an IVP of a system of differential equations

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}y}{\mathrm{d}x}}=f(x,y),x\in I=(0,T),\hfill \\ y\left(0\right)=y^0,\hfill \end{array}\right.$$

(1)

with independent variable $x\in R$, dependent variable $y=(y_1,y_2,\cdots ,y_n)\in R^n$, and initial value (or initial condition, IC) $y^0=(y_1^0,y_2^0,\cdots ,y_n^0)\in R^n$. We assume through the article that the right-hand-side function $f(x,y)=(f_1(x,y),f_2(x,y),\cdots ,f_n(x,y)$ satisfies the condition [1]

(H₁): 

$f(x,y)$is continuous in $(x,y)\in \Omega \subset R\times R^n$ for a domain Ω and Lipschitz continuous on y.

The condition guarantees the existence of a unique local solution to the problem (1).

In addition, since the present article works on the large interval solution to IVP (1), we also use the default assumption below:

(H₂): 

The solution to IVP (1) exists on large interval I of x (called large interval solution).

Thus, the PINN is an ANN algorithm for solving the solution to problem (1) with a fully connected structure that consists of one-dimensional input layer (0the layer), M hidden layers, $s_i$ nonlinear active neurons for ith layer with $i=0,1,\cdots ,M+1$ and $s_0=1,s_{M+1}=n$, and output layer ($(M+1)$th layer) with n linear output neurons. The output of the network is denoted as

$$N\left(x\right)=(N_1\left(x\right),N_2\left(x\right),\cdots ,N_n\left(x\right)),$$

for an input sample $x\in I$, which mathematically has the form of

$$N\left(x\right)={\sigma }_M({\sigma }_{M-1}(\cdots ({\sigma }_1(W^{1}t+b^1)W^2+b^2)\cdots )W^M+b^M)W^{M+1}+b^{M+1},$$

(2)

for selected activation functions ${\sigma }_{i+1}$, weights matrix $W^{i+1}$ between ith and $(i+1)$th layer and bias vector $b^{i+1}$ for $(i+1)$th layer neurons with $i=0,1,2,\dots ,M$ and $b_{M+1}\left(x\right)=x$.

The collection of all trainable parameters of the ANN, denoted as $\vartheta$, is consisted of weights $W^i={(w^{i,1},w^{i,2},\dots ,w^{i,s_i})}^T$ with $w^{i,j}=(w_i^{1,j},w_i{,}^{2,j},\cdots ,w_i^{s_{i-1},j})$ in which the components $w_i^{k,j}$ ($k=1,2,\cdots ,s_{i-1},j=1,2,\cdots ,s_i$) are the weights connected the neurons in $(i-1)$th layer to jth neuron in ith layer, and layer biases $b^i={(b_i^1,b_i^2,\dots ,b_i^{s_i})}^T(i=1, 2, \cdots , M + 1)$. Hence, $W^i$ is a $s_i\times s_{i-1}$ matrix. Let $\vartheta$ denote the set of all weights and biases parameters of the network. The unsupervised learning (training) method is used. That is, the optimal solution $N\left(x\right)$ at optimal parameters $\vartheta ={\vartheta }^{*}$ is searched such that the loss function to be given below is minimized.

Usually, the mean square error (MSE) is used to define the loss function for training the network as follows:

$$Loss={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}MS{E}_{f_i}+{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(N_i\left(0\right)-y_i^0)}^2,$$

(3)

where

$$MS{E}_{f_i}={\displaystyle \frac{1}{M}}\sum _{j=1}^M{(N_i^{\prime }\left(x_j\right)-f_i(x_j,N_1\left(x_j\right),N_2\left(x_j\right),\cdots ,N_n\left(x_j\right)))}^2,$$

and ${\left\{x_j\right\}}_{j=1}^M$ is a training data set sampled by a distribution sense from the solution interval I, and $N_i^{\prime }\left(x\right)$ is the derivative of the approximate solution $N_i\left(x\right)$ with respect to the independent variable x. In training the network, the automatic differentiation operation ∇ is used to realize $N_i^{\prime }\left(x\right)$. One of initialization methods [39] is performed on the neural network parameters when starting the training. A optimization technique [40] is used to adjust the network parameters in order to minimize the loss function (3) and carry out the backpropagation of errors. After completing the training of the network by minimizing the loss function ($Loss<ϵ$) or sufficient iterative parameter refinement (learning, training) ($Iter>maxit$), we obtain the network pattern (2) with a optimal parameter set ${\vartheta }^{*}$, which represents the approximate solution to problem (1). The basic structure of above mentioned ANN (e.g, PINN) is shown in Figure 1.

Particularly, what should be especially mentioned here is that the experiences of applying an ANN to solving the IVP of a (partial) differential equations show that it often yields high-accuracy approximate solutions merely near the initial value point rather than the entire interval $[0,T)$ [1]. That is, generally, there exists a constant $0<\delta \ll T$ such that the output $N\left(x\right)$, called a PINN solution, converges to the exact solution $y\left(x\right)$ over $[0,\delta )$ with high accuracy. We denote the approximation as

$$N\left(x\right)\approx y\left(x\right).$$

In other words, as the training point gradually moves away from the starting point $x=0$, the learning ability or generalization ability of the ANN gradually weakens. There are two main potential reasons for this phenomenon. First, the requirement of $Loss\to 0$ in network training is only a necessary condition for $N\left(x\right)$ to solve IVP (1). Second, with the expansion of the sampling data interval, the support of initial value information to the training ANN may gradually decline [7].

Therefore, in order to obtain the large interval solution to problem (1) by the ANN method, it is often necessary to construct a neural network with a large number of neurons and deep layers so that the influence of the initial value is deep and wide, which may make the network learn the solution as accuracy as possible [3]. However, the increase in network scale significantly increases the training difficulty of the neural network, and even makes it impossible to get normal training. In view of this problem, based on the advantage of PINN’s high-precision learning ability in a neighborhood of initial value points, we propose a PWNN method in the next section to obtain the large interval solution. In the method, the solution interval I is split into several intercell compartments, and a related small-scale-sized neural network is applied on each sub-interval to solve a sub-IVP corresponding to (1). Finally the neural network solutions of the sub-intervals are stitched together to obtain the approximate solution to problem (1) over the whole interval I.

3. A Piecewise ANN Method

In this section, based on the default assumptions (${\mathrm{H}}_1$)–(${\mathrm{H}}_2$) and good learning ability of a PINN, we present the PWNN method for finding the large interval approximate solution to IVP (1).

Method

The basic framework of the method is described as follows.

First, we divide the interval $[0,T]$ into p sub-intervals, namely insert $p-1$ points in interval $[0,T]$ making

$$0=a_0<a_1<\cdots <a_p=T,[0,T]=\bigcup _{i=1}^p{\Delta }_i,{\Delta }_i=[a_{i-1},a_i],$$

and suppose that sub-interval ${\Delta }_i$ is relatively small so that the good performance of the PINN is guaranteed. Then, we construct p ANNs (e.g., PINNs) with identical structures, as given in Section 2. Corresponding to the splitting of the interval $[0,T]$, denoted as ${\left\{m_i\right\}}_{i=1}^p$. Each neural network $m_i$ is used to solve a sub-IVP of the same differential equations in (1) on the ith sub-interval ${\Delta }_i$ with a well-defined initial value at $a_{i-1}$. The output of the network is denoted as $N^i\left(x\right)=(N_1^i\left(x\right),N_2^i\left(x\right),\cdots ,N_n^i\left(x\right))$, which has a composite expression like (2) and represents the approximation of the exact solution to the sub-IVP, denoted as ${\overline{y}}^i\left(x\right)=({\overline{y}}_1^i\left(x\right),{\overline{y}}_2^i\left(x\right),\cdots ,{\overline{y}}_n^i\left(x\right))$. Particularly, component $N_j^i\left(x\right)$ is the output of j-th neuron of network $m_i$ in the output layer for an input $x\in {\Delta }_i$, representing the approximation of ${\overline{y}}_j^i\left(x\right)$ for $j\in \{1,2,\cdots ,n\}$. The network is illustrated in Figure 2.

Specifically, first, a PINN $m_1$ is trained to solve the sub-IVP of the system in (1) on interval ${\Delta }_1$ with initial value ${\overline{y}}^1\left(0\right)=y^0$. The loss function used for the network is defined as

$$\begin{array}{cc}\hfill Los{s}_1& ={\displaystyle \frac{1}{M_{1}n}}\sum _{i=1}^n\sum _{j=1}^{M_1}{({\displaystyle \frac{\mathrm{d}{N}_i^1}{\mathrm{d}x}}\left(x_j\right)-f_i(x_j,{N_1}^1\left(x_j\right),{N_2}^1\left(x_j\right),\cdots ,N_n^1\left(x_j\right)))}^2\hfill \\ \hfill & +{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(N_i^1\left(0\right)-y_i^0)}^2.\hfill \end{array}$$

Here, the training data points ${\left\{x_j\right\}}_{j=1}^{M_1}$ are taken from the interval ${\Delta }_1$ by some distribution sense (the follow-up data samplings are similar). The training of the network $m_1$ is completed when the termination (convergence) conditions of the network training (generally the loss function value drops to near zero, or the iteration step size is sufficient) are satisfied. As a result, due to the approximation property of ANN, the ANN is an approximate solution to the sub-IVP, denoted by $N^1\left(x\right)=(N_1^1\left(x\right),N_2^1\left(x\right),\cdots ,N_n^1\left(x\right))$, i.e., it is the approximation of the exact solution ${\overline{y}}^1\left(x\right)=({\overline{y}}_1^1\left(x\right),{\overline{y}}_2^1\left(x\right),\cdots ,{\overline{y}}_n^1\left(x\right))$ to the sub-IVP over ${\Delta }_1$. Notably, compared with a neural network that solves IVP (1) over the entire interval I, network $m_1$ requires only a structure of relatively small scale and is trained well on ${\Delta }_1$ because it is a small neighborhood of the initial value point $x=0$. This gives full play to the strong local learning ability, faster convergence speed, and high approximation accuracy of PINN.

After obtaining the first network $m_1$ on ${\Delta }_1$, we train and use the next network $m_2$ to approximately solve the sub-IVP of the system in (1) with an initial value of ${\overline{y}}^2\left(a_1\right)=N^1\left(a_1\right)$ in terms of the previous trained network $m_1$ on the interval ${\Delta }_2$. The corresponding loss function for $m_2$ is defined as follows:

$$\begin{array}{cc}\hfill Los{s}_2& ={\displaystyle \frac{1}{M_{2}n}}\sum _{i=1}^n\sum _{j=1}^{M_2}{\left({\displaystyle \frac{\mathrm{d}{N}_i^2}{\mathrm{d}x}}\left(x_j\right)-f_i(x_j,{N_1}^2\left(x_j\right),{N_2}^2\left(x_j\right),\cdots ,N_n^2\left(x_j\right))\right)}^2\hfill \\ \hfill & +{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(N_i^2\left(a_1\right)-N_i^1\left(a_1\right)\right)}^2.\hfill \end{array}$$

(4)

Here, the training data points ${\left\{x_j\right\}}_{j=1}^{M_2}$ are taken from the interval ${\Delta }_2$. The second term on the right-hand side of the loss function ensures that network $m_2$ approximates $m_1$ as closely as possible at the initial point $a_1$. When the termination conditions are satisfied, network $m_2$ has been trained and yields approximate solution $N^2\left(x\right)=(N_1^2\left(x\right),N_2^2\left(x\right),\cdots ,N_n^2\left(x\right))$ of the exact solution ${\overline{y}}^2\left(x\right)=({\overline{y}}_1^2\left(x\right),{\overline{y}}_2^2\left(x\right),\cdots ,{\overline{y}}_n^2\left(x\right))$ to the sub-IVP.

Inductively, suppose we have already trained network $m_i$ and obtained approximate solution $N^i\left(x\right)=(N_1^i\left(x\right),N_2^i\left(x\right),\cdots ,N_n^i\left(x\right))$ of the exact solution ${\overline{y}}^i\left(x\right)=({\overline{y}}_1^i\left(x\right),{\overline{y}}_2^i\left(x\right),$$\cdots ,{\overline{y}}_n^i\left(x\right))$ to the sub-IVP of system in (1) with initial value ${\overline{y}}^i\left(a_i\right)=N^{i-1}\left(a_{i-1}\right)$ on the interval ${\Delta }_i$ for $i=1,2,\cdots ,k-1$. Now, network $m_k$ is trained and used to approximate the exact solution ${\overline{y}}^k\left(x\right)$ to the sub-IVP of the same system in (1) on the interval ${\Delta }_k$ with initial value ${\overline{y}}^k\left(a_{k-1}\right)=N^{k-1}\left(a_{k-1}\right)$ in terms of previous network $m_{k-1}$. The training data points ${\left\{x_j\right\}}_{j=1}^{M_k}$ are taken from the interval ${\Delta }_k$. The corresponding loss function for network $m_k$ is defined as

$$\begin{array}{cc}\hfill Los{s}_k& ={\displaystyle \frac{1}{M_{k}n}}\sum _{i=1}^n\sum _{j=1}^{M_k}{\left({\displaystyle \frac{\mathrm{d}{N}_i^k}{\mathrm{d}x}}\left(x_j\right)-f_i(x_j,{N_1}^k\left(x_j\right),{N_2}^k\left(x_j\right),\cdots ,N_n^k\left(x_j\right))\right)}^2\hfill \\ \hfill & +{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(N_i^k\left(a_{k-1}\right)-N_i^{k-1}\left(a_{k-1}\right)\right)}^2.\hfill \end{array}$$

(5)

When the termination conditions are satisfied, network $m_k$ has been trained and yields approximate solution $N^k\left(x\right)=(N_1^k\left(x\right),N_2^k\left(x\right),\cdots ,N_n^k\left(x\right))$ of the exact solution ${\overline{y}}^k\left(x\right)=({\overline{y}}_1^k\left(x\right),{\overline{y}}_2^k\left(x\right),\cdots ,{\overline{y}}_n^k\left(x\right))$ to the sub-IVP.

In this way, we train and use all ANNs $m_i$ or, equivalently, $N^i\left(x\right)$, subsequently to approximate the exact solution ${\overline{y}}^i\left(x\right)$ to the sub-IVP over sub-intervals ${\Delta }_i$ with initial value ${\overline{y}}^i\left(a_{i-1}\right)=N^{i-1}\left(a_{i-1}\right)$ for $i=1,2,\cdots ,p$.

Finally, we construct a piecewise function over $[0,T)$ with the PINN’s solutions, given by

$$\widehat{y}\left(x\right)=\left\{\begin{array}{cc}{N}^1\left(x\right),\hfill & x\in [0,a_1],\hfill \\ N^2\left(x\right),\hfill & x\in (a_1,a_2],\hfill \\ ⋮\hfill \\ N^p\left(x\right),\hfill & x\in (a_{p-1},T),\hfill \end{array}\right.$$

where

$$N^k\left(x\right)=(N_1^k\left(x\right),N_2^k\left(x\right),\cdots ,N_n^k\left(x\right)),k=1,2,\cdots ,p.$$

4. Theoretical Analysis and a Parameter Transfer Method

In the section, we theoretically demonstrate that the function $\widehat{y}\left(x\right)$ given in last section is an approximation of the large interval solution $y\left(x\right)$ of IVP (1) we are looking for. Meanwhile, we give a parameter transfer method and multiple rounds of pre-training approach to train PWNN.

4.1. Approximation of Large Interval Solution

In fact, the network $m_k$, or equivalently $N^k\left(x\right)$ with $N^0\left(x\right)=y^0$, presented in the last section, approximately solved the following sub-IVP of the system in (1):

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}{\overline{y}}^k}{\mathrm{d}x}}=f(x,{\overline{y}}^k),x\in {\Delta }_k,\hfill \\ {\overline{y}}^k\left(a_{k-1}\right)=N^{k-1}\left(a_{k-1}\right),\hfill \end{array}\right.$$

(6)

for dependent variable ${\overline{y}}^k\left(x\right)=({\overline{y}}_1^k\left(x\right),{\overline{y}}_2^k\left(x\right),\cdots ,{\overline{y}}_n^k\left(x\right))$ and for each $k\in \{1,2,\dots ,p\}$, inductively, we have

$$N^k\left(x\right)\approx {\overline{y}}^k\left(x\right)\mathrm{on}{\Delta }_k,k=1,2,\cdots ,p.$$

(7)

In addition, we also set a sub-IVP of (1) on each sub-interval ${\Delta }_k$ as

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}{y}^k}{\mathrm{d}x}}=f(x,y^k),x\in {\Delta }_k,\hfill \\ y^k\left(a_{k-1}\right)=y^{k-1}\left(a_{k-1}\right),\hfill \end{array}\right.$$

(8)

with $y^k\left(x\right)=(y_1^k\left(x\right),y_2^k\left(x\right),\cdots ,y_n^k\left(x\right))$ and $y^0\left(0\right)=y^0$ for $k\in \{1,2,\cdots ,p\}$. Due to the uniqueness of the solution to the IVP of an ODE under assumption (${\mathrm{H}}_1$) given in the last section, we know that solution $y^k\left(x\right)$ to sub-IVP (8) is the restriction of the exact solution $y\left(x\right)$ to IVP (1) on ${\Delta }_k$. That is,

$$y^k\left(x\right)=y\left(x\right)\mathrm{on}{\Delta }_k,k=1,2,\cdots ,p,$$

(9)

and particularly, $y\left(x\right)=y^1\left(x\right)={\overline{y}}^1\left(x\right)\mathrm{on}{\Delta }_1.$ Hence, noticing (7), approximation

$$N^1\left(x\right)\approx y\left(x\right),$$

(10)

holds on ${\Delta }_1$.

For $k=2$, we have

$$y^2\left(a_1\right)\stackrel{\left(8\right)}{=}{y}^1\left(a_1\right)={\overline{y}}^1\left(a_1\right)\stackrel{\left(7\right)}{\approx }{N}^1\left(a_1\right)\stackrel{\left(6\right)}{=}{\overline{y}}^2\left(a_1\right).$$

Hence, on sub-interval ${\Delta }_2$ the initial values of IVP (6) and IVP (8) are approximate. Consequently, by the qualitative property that the solution to the IVP of an ODE with assumption (${\mathrm{H}}_1$) continuously depends on the initial value, approximation

$$N^2\left(x\right)\approx y\left(x\right),$$

(11)

holds on ${\Delta }_2$.

Similarly, we can inductively prove

$$N^k\left(x\right)\approx y\left(x\right),$$

(12)

hold on intervals ${\Delta }_k$ for $k=1,2,\cdots ,p$. This shows that approximation $\widehat{y}\left(x\right)\approx y\left(x\right)$ holds on each ${\Delta }_k$.

Furthermore, from the construction of PINN $N\left(x\right)$ (see the multiple composite structure of the PINN’s output expression (2)), we see that function $N^k\left(x\right)$ is continuous and differential over each sub-interval ${\Delta }_k$. Therefore, except for the end points $a_k$ of ${\Delta }_k(k=1,2,\cdots ,p-1)$, function $\widehat{y}\left(x\right)$ is continuous and differential over large interval $[0,T)$. Although we do not confirm the continuity of $\widehat{y}\left(x\right)$ on those end points, we, from (6) and (7), know approximation

$$N^k\left(a_k\right)\approx N^{k+1}\left(a_k\right),$$

implying the approximation of the two side limitations

$$\widehat{y}\left(a_k^{-}\right)\approx \widehat{y}\left(a_k^{+}\right),$$

(13)

hold for $k=1,2,\cdots ,p-1$. This indicates that there may exist just a small ’jump’ in the value of function $\widehat{y}\left(x\right)$ at an end point. However, in numerical solutions, the errors can be controlled within the tolerable error range by improving the convergence accuracy of PINN solutions $N^k\left(x\right)$.

To sum up the above procedure, we proved the following conclusions.

Theorem 1.

On the solution interval $[0,T)$, the piecewise function $\widehat{y}\left(x\right)$ satisfies the approximation $\widehat{y}\left(x\right)\approx y\left(x\right)$ and is continuously differentiable except for the finite points $a_k(k=1,2,\cdots ,p-1)$, where the approximation (13) holds.

The Theorem is the theoretical basis of our PWNN method.

4.2. Transfer of Network Parameters

According to the research by Xavier et al., the parameter initialization of a neural network directly affects whether the network can be trained successfully [39]. In the proposed PWNN method, we need to train p interrelated PINNs. Therefore, the parameter initialization and transfer of the PWNNs in training them are key steps to successfully obtain large interval approximate solution $\widehat{y}\left(x\right)$. In order to assess and improve the stability and performance in certain cases, there are some additional network parameter transfer and multiple rounds of pre-training techniques that we employ beyond the basic setup.

1. A parameter transfer technique. In fact, we already used a parameter initialization technique in the training and theoretical analysis of PWNN. This is shown in IVP (6) and the procedure of constructing ANNs $m_k(k=1,2,\cdots ,p)$, which is explained in more detail in the following steps.

Start with $k=1$. By the standard steps of PINN, initializing network parameters ${\vartheta }_1$ with the Xavier technique or some other ones, we then train first PINN $m_1$ on ${\Delta }_1$ to solve IVP (6) for $k=1$ and determine $N^1\left(x\right)$.

For $k=2,3,\cdots ,p$, initializing parameters ${\vartheta }_k$ of $m_k$ using parameters ${\vartheta }_{k-1}$ obtained in $m_{k-1}$, we then train the kth PINN $m_k$ on ${\Delta }_k$ to solve IVP (6) and determine $N^k\left(x\right)$. That is, once network $m_{k-1}$ has completed training, we pass parameters ${\vartheta }_{k-1}$ to network $m_k$ as initialization of parameters ${\vartheta }_k$ of $m_k$, as shown in Figure 2.

It is worth noting here that since network $m_k$ inherits the parameters of $m_{k-1}$, the computational effort of loss function $Los{s}_k$ will be greatly reduced, thus speeding up the training of the network.

2. Multiple rounds of pre-training method. In the first time training PINN, there is not always satisfactory performance from the network being trained due to the random parameter initialization. In this case, it is common to train the network in several rounds (pre-training) based on the latest obtained parameters. That is, the obtained parameters after completing the training of an ANN are used as initialization of the network to train again, and so on, until the end of the training round. This multi-round pre-training gradually optimizes the parameter initialization, avoids the uncertainty caused by random initialization, and guides the improvement of training quality.

In our PWNN case, the above parameter transfer method (see above the case $k=1$) is used in the first round of training, and then the multi-round pre-training method is used after the second round. Specifically, each neural network $m_j,j=2,3,\cdots ,p$ receives the parameters of $m_{j-1}$ as parameter initialization in the first round of training. For subsequent rounds, in training a network, it receives the network parameters obtained from the latest training round of this network as its initialization. This multi-round pre-training will progressively improve the approximation of $\widehat{y}\left(x\right)$ to the exact solution to IVP (1). To further improve the approximation accuracy of $\widehat{y}\left(x\right)$ to exact solution $y\left(x\right)$, we even apply the two kinds of techniques interactively in training PWNNs.

4.3. Implementation of PWNN

Let ${\vartheta }_k^i$ denote the training parameter set of network $m_k$ in the ith round and $maxit$ denote the maximum number of iterations. $Loss$ represents the loss value of the neural network, which is determined by Equation (5), and $\epsilon >0$ represents a pre-specified error limitation.

Now, summarizing the above statement of the PWNN method, we have the following Algorithm 1 as an implementation of our proposed PWNN method.

Algorithm 1 PWNN

Initialization: Define a generic functional module Backpropagation=     {        While $it\le maxit\wedge loss\ge \epsilon$            Update ${\vartheta }_k^i$ via a optimizer [40]            Update the loss based on Equation (5) and backpropagation method     } Begin: for$i=1$ to n do     if $i=1$ then         for $k=1$ to p do            if $k=1$ then                ${\vartheta }_k^i$ is initialized using Xavier method or some others [39]            else                initializing ${\vartheta }_k^i={\vartheta }_{k-1}^i$            end if                Backpropagation         end for     else         initializing ${\vartheta }_k^i={\vartheta }_k^{i-1}$         Backpropagation     end if end for End

5. Experiment

In this section, we give several numerical experiments using the PWNN algorithm and compare the results with those of the PINN method and Runge–Kutta method to show the validity of our proposed method.

5.1. Example 1

Consider the following IVP of an ODE

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}{y}_1}{\mathrm{d}x}}=y_2,\hfill \\ {\displaystyle \frac{\mathrm{d}{y}_2}{\mathrm{d}x}}=-y_2-(2+sinx)y_1,\hfill \\ y_1\left(0\right)=0,y_2\left(0\right)=1,\hfill \end{array}\right.$$

(14)

with dependent variable $y\left(x\right)=(y_1\left(x\right),y_2\left(x\right))$ on interval $[0,10]$. We first use normal PINN to find the ANN solution to the problem. We construct a 3-layer neural network with 1 input neuron, 2 output neurons, and 2 hidden layers of 20 neurons each. We uniformly sampled 1000 points on the interval $[0,10]$ as the training data set. The learning rate is set to $0.01$, and the number of iterations is 10,000. The training results of the network are shown in Figure 3.

Then, we use PWNN to solve the problem with less training data and the same structure as PINN above on each sub-interval. Interval $[0,10]$ is divided into five equidistant sub-intervals ${\Delta }_i=[2(i-1),2i](i=1,2,\cdots ,5)$. The neural network employed in each sub-interval consists of two hidden layers, with each hidden layer containing 20 neurons. In order to compare it with PINN, we uniformly sample 200 data points from each sub-interval as the training data. The learning rate is set to 0.01, and each neural network performed 2000 iterations. The results of training PWNNs $m_i(i=1,2,\cdots ,5)$ are shown in Figure 4. The first five graphs represent the PWNNs on each interval, as well as loss function values, and the last graph is their combination $\widehat{y}\left(x\right)=({\widehat{y}}_1\left(x\right),{\widehat{y}}_2\left(x\right))$.

We also solve the problem using the fourth-order Runge–Kutta method to compare the results with those of using the above methods.

In Figure 5, the results obtained by using the PINN method, PWNN approach, and fourth-order Runge–Kutta method (RK4) are presented. It can be observed that the overall results of the three methods are similar. However, compared with PINN, the results of PWNN and RK4 are closer.

It shows that PWNN is more efficient than PINN in the sense of numerical solution.

5.2. Example 2

Consider the following IVP of a SIR model of epidemic dynamics

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}{y}_1}{\mathrm{d}x}}=-0.003y_1{y}_2,\hfill \\ {\displaystyle \frac{\mathrm{d}{y}_2}{\mathrm{d}x}}=0.003y_1{y}_2-0.1y_2,\hfill \\ {\displaystyle \frac{\mathrm{d}{y}_3}{\mathrm{d}x}}=0.1y_2,\hfill \\ y_1\left(0\right)=98,y_2\left(0\right)=2,y_3\left(0\right)=0.\hfill \end{array}\right.$$

(15)

We use PINN to find its solution over the interval $[0,50]$. The used PINN consists of 1 input neuron, 3 hidden layers of 20 neurons each, and 3 output neuron. The 200 training points are taken from $[0,20]$ equidistantly. The network adopts Xavier initialization, employs the Adam optimization algorithm with a learning rate $0.01$, and runs for 10,000 iterations. Figure 6 illustrates the solutions given by the PINN for IVP (15) over the interval $[0,20]$. Observably, the PINN successfully computes solutions merely within the intervals $[0,14]$ (left graph) instead of large interval $[0,20]$ (right graph). That is, as the solution interval further extends the learning results of PINN notably deviate from the real case.

We now solve the problem over a large interval $[0,50]$ using the PWNN method with a smaller size structure than the PINN above. Divide interval $[0,50]$ into 10 equally sized sub-intervals and Take 100 training points in each sub-interval by uniform distribution. Construct a neural network for each segment, where each network consists of 1 input neuron, 1 hidden layer with 20 neurons, and 3 output neurons. Utilize Xavier initialization and Adam optimization algorithms for each network with learning rate $0.01$ and 10,000 iterations to train $m_1$ on interval $[0,5]$.

Figure 7 illustrates the training results of PWNNs for all sub-intervals. The titles of the figures specify the names of the networks for each interval along with the respective values of the loss function after the completion of network training. Figure 8 displays the comparisons between the results of both PWNNs and the RK4 method. The solid lines represent the solutions $\widehat{y}\left(x\right)$ obtained from PWNNs, while the discrete points depict the solutions obtained using the RK4 method. Obviously, PWNN gives a large interval solution to problem (15) that is highly consistent with RK4.

Example 2 shows that the direct application of conventional PINN can only obtain the intercell solution (local solution) of the problem, while PWNN directly gives a large interval solution consistent with RK4, which demonstrates the effectiveness of the given PWNN algorithm.

5.3. Example 3

The Lotka–Volterra system describes the interaction between predator and prey, where the predator depends on the prey for food, and the prey is threatened by the predator. The model can be mathematically expressed as

$$\left\{\begin{array}{cc}{\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}\hfill & =\alpha x-\beta xy,\hfill \\ {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}\hfill & =-\gamma y+\delta xy.\hfill \end{array}\right.$$

(16)

We take $\alpha =\beta =0.5$, $\gamma =\delta =2$, considering initial condition $IV=[1.2,1.0]$, on interval [0, 20] to solve it. We first solve it with a [1, 20, 20, 2] PINN, where [1, 20, 20, 2] refers to the network having one input neuron, with two hidden layers in the middle, each with 20 neurons, and the output layer with two neurons. The network is trained for 10 rounds with 10,000 iterations in each round, the learning rate is $1\times {10}^{-4}$ and the optimizer is Adam. The training results are shown in the Figure 9, which shows that PINN has a large gap with the results predicted by RK4 after 10 rounds of training. Furthermore, we also attempted to separately increase the width and depth of the PINN and conduct training with the same training settings. The network’s prediction results are presented in the Figure 9, and the loss results are shown in the

Table 1. The loss value for problem (16) in PINN.

PINN	[1, 20, 20, 2]	[1, 20, 20, 20, 2]	[1, 50, 50, 2]
Round 1	$1.23\times {10}^{-2}$	$1.20\times {10}^{-2}$	$7.01\times {10}^{-4}$
Round 5	$1.20\times {10}^{-2}$	$1.20\times {10}^{-2}$	$2.85\times {10}^{-4}$
Round 10	$1.20\times {10}^{-2}$	$1.20\times {10}^{-2}$	$2.14\times {10}^{-4}$

. It can be found that, in the broader interval [0, 20], the increase in the depth of the PINN failed to enhance the performance of the model (the loss after 10 rounds of training was not lower than that of the shallow PINN). The PINN with increased depth, although the loss was decreasing, did not have a high accuracy after 10 rounds of training.

We now uniformly divide the interval $[0,20]$ into five segments, training each sub-interval with a $[1,20,20,2]$ PINN. The training configuration remains consistent with the previously described settings, with network prediction results presented in the accompanying Figure 9 and loss outcomes summarized in the

Table 2. The loss value for problem (16) in PWNN.

	${\mathit{m}}_1$	${\mathit{m}}_2$	${\mathit{m}}_3$	${\mathit{m}}_4$	${\mathit{m}}_5$
Round 1	$9.62\times {10}^{-6}$	$1.21\times {10}^{-3}$	$1.96\times {10}^{-3}$	$6.47\times {10}^{-4}$	$5.01\times {10}^{-4}$
Round 5	$4.99\times {10}^{-7}$	$2.17\times {10}^{-5}$	$2.72\times {10}^{-5}$	$4.66\times {10}^{-5}$	$4.24\times {10}^{-4}$
Round 10	$1.81\times {10}^{-7}$	$4.07\times {10}^{-6}$	$1.37\times {10}^{-5}$	$7.39\times {10}^{-6}$	$7.18\times {10}^{-5}$

. It is evident that from the first to the fifth iteration, as well as from the fifth to the tenth iteration, the loss for each sub-network shows a significant reduction, with the network predictions closely approximating those obtained using the high-accuracy RK4 method. Let us denote the parameter count of a $[1,20,20,2]$ PINN as $n_1$. The parameter count for a deeper $[1,20,20,20,2]$ PINN is approximately $20n_1$, while the parameter count for a wider $[1,50,50,2]$ PINN is about $6n_1$. In comparison, the parameter number of PWNN with the added number of networks is $5n_1$. This indicates that, without increasing the overall parameter count, enhancing the number of networks is more effective than augmenting their depth or width.

For a clearer comparison with the accepted traditional methods, we calculate the mean square error between each network and RK4. From the

Table 3. The mean square error between the predicted results of each network and the RK4 results.

Net	PINN			PWNN
	[1, 20, 20, 2]	[1, 20, 20, 20, 2]	[1, 50, 50, 2]
MSE (Net,RK4)	$8.18\times {10}^{-1}$	$8.17\times {10}^{-1}$	$2.53\times {10}^{-2}$	$2.95\times {10}^{-5}$

, it can be seen quantitatively that among the various networks, our proposed piecewise neural network method is the closest to the results of RK4.

5.4. Example 4

Consider the following IVP:

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathrm{d}{y}_1}{\mathrm{d}x}}=y_2{y}_3,\hfill \\ {\displaystyle \frac{\mathrm{d}{y}_2}{\mathrm{d}x}}=-y_1{y}_3,\hfill \\ {\displaystyle \frac{\mathrm{d}{y}_3}{\mathrm{d}x}}=-0.51y_1{y}_2,\hfill \\ y_1\left(0\right)=0,y_2\left(0\right)=y_3\left(0\right)=1.\hfill \end{array}\right.$$

(17)

As in previous examples, we first use PINN to find the numerical solutions of the problem in the interval $[0,20]$. We perform four rounds of training for PINN, with 10,000 iterations per round. The learning rate is set to 0.001 for the first round and 0.0001 for the subsequent rounds. In these experiments, the values of the loss function remains around $1.99\times {10}^{-3}$ for different rounds. This indicates that further increasing the number of training rounds and iterations does not significantly reduce the loss function value. The training results are shown in Figure 10.

When using PWNN, we divide the solution interval $[0,20]$ into five equal parts and construct corresponding PWNNs. Then, train each PWNN for four rounds, with each round consisting of 10,000 iterations on each sub-interval. The learning rate for the first round of training is set to $0.001$, while for the remaining rounds, it is set to $0.0001$. The training results of the PWNNs are shown in Figure 11.

The values of the loss function corresponding to each round of PWNN are shown in

Table 4. Comparisons of the results obtained by PWNN and RK4 for solving IVP (17).

Loss		PWNN	${\mathit{m}}_1$	${\mathit{m}}_2$	${\mathit{m}}_3$	${\mathit{m}}_4$	${\mathit{m}}_5$
Rounds
1			$1.41\times {10}^{-5}$	$2.17\times {10}^{-4}$	$2.38\times {10}^{-4}$	$3.74\times {10}^{-3}$	$1.30\times {10}^{-4}$
2			$1.53\times {10}^{-6}$	$1.67\times {10}^{-5}$	$1.50\times {10}^{-5}$	$5.28\times {10}^{-3}$	$1.73\times {10}^{-4}$
3			$6.98\times {10}^{-7}$	$4.97\times {10}^{-6}$	$5.68\times {10}^{-6}$	$6.44\times {10}^{-5}$	$1.71\times {10}^{-4}$
4			$4.43\times {10}^{-7}$	$2.20\times {10}^{-6}$	$4.09\times {10}^{-6}$	$1.94\times {10}^{-5}$	$3.61\times {10}^{-5}$

. It can be observed that as the number of training rounds increases, the loss function values of the PWNNs gradually decreases. In Figure 12, we show the comparisons of results of the PWNN in the fourth round with those of RK4. It can be seen that the results of the PWNN method are in good agreement with those of the RK4 method.

6. Discussion and Conclusions

The neural network-based method for solving differential equations provides solutions with excellent generalization properties. Models based on neural networks offer opportunities to theoretically and practically tackle differential equation problems across various sciences and engineering applications, while most other techniques offer discrete solutions, solutions with limited differentiability, or strongly depend on the discrete scheme of the variable domain. Effectively using the advantages of the nonlinear approximation of artificial neural networks and overcoming the shortcomings of traditional numerical methods is a hot research direction at present.

In this paper, based on the advantages of PINN local strong convergence, we propose a piecewise neural network method for solving initial value problems of ODEs over large intervals of the independent variable. This method not only provides an approach for the coordination of multiple networks to solve an IVP of ODEs but also introduces a parameter transfer and multiple rounds of pre-training technique to effectively enhance the accuracy of network solutions in training multi-correlation ANN models. On one hand, we offer a new method to address the solution extension problem of initial value problems of differential equations. On the other hand, our work aims to contribute to both the theoretical and practical aspects of applying ANNs, providing deep insights into the application of modern methods to traditional mathematical problems.

Through comparative experiments, we prove that under almost the same network training environment, the training time of piecewise neural networks is shorter, the convergence speed to the solution to the studied problem is faster, and it can also make up for the defects of the basic ANN algorithm. Furthermore, in Section 5.2, when PINN is used to solve the Equation (15) over a large solution domain, although the final loss value obtained by PINN training is very low, the approximate solution obtained does not match the actual solution. This may be attributed to two reasons. On the one hand, it shows that the optimization of the residual loss function is only a necessary condition for the artificial neural network output to be an approximate solution to the problem, but not a sufficient condition. On the other hand, this is due to small numerical instabilities during backpropagation because of the complexity of the loss hypersurface, where the ANN can settle on a local minimum with a small value for the loss function [2]. The piecewise neural network method proposed in this paper solves these problems effectively to some extent.

Certainly, there are some limitations to this study. We have not yet compared it with some newer methods, such as in [38]. We did not consider applying the proposed method to partial differential equation problems, which would involve partitioning high-dimensional domains of independent variables. The successful application of this method strongly depends on the effective training of sub-neural networks on each corresponding interval. In other words, if an ANN in a certain link cannot be effectively trained in the training of a PWNN, the whole algorithm may not complete the solving task. Furthermore, the continuity and differentiability of network solution $\widehat{y}\left(x\right)$ over the entire interval have yet to be guaranteed theoretically. These are topics for further study in the future.