Algorithms for Solving Ordinary Differential Equations Based on Orthogonal Polynomial Neural Networks

Abstract

This article proposes single-layer neural network algorithms for solving second-order ordinary differential equations, based on the principles of functional connection. According to this principle, the hidden layer of the neural network is replaced by a functional expansion unit to improve input patterns using orthogonal Chebyshev, Legendre, and Laguerre polynomials. The polynomial neural network algorithms were implemented in the Python programming language using the PyCharm environment. The performance of the polynomial neural network algorithms was tested by solving initial-boundary value problems for the nonlinear Lane–Emden equation. The solution results are compared with the exact solution of the problems under consideration, as well as with the solution obtained using a multilayer perceptron. It is shown that polynomial neural networks can perform more efficiently than multilayer neural networks. Furthermore, a neural network based on Laguerre polynomials can, in some cases, perform more accurately and faster than neural networks based on Legendre and Chebyshev polynomials. The issues of overtraining of polynomial neural networks and scenarios for overcoming it are also considered.

1. Introduction

In recent years, machine learning methods, particularly artificial neural networks (ANNs), have found widespread application beyond traditional data analysis tasks, entering the realm of scientific computing [1]. One promising direction is the use of ANNs for the numerical solution of differential equations, opening new possibilities for modeling complex physical and engineering systems [2]. Compared to classical numerical methods (such as the finite difference or finite element method), neural network approaches offer several potential advantages: they allow obtaining a solution in the form of a continuous and differentiable function over the entire domain, can work efficiently in high-dimensional spaces, and are well suited for problems with inverse parameters or incomplete data.

Various neural network architectures are used to solve ordinary differential equations (ODEs). The most common are multilayer networks, such as the classic multilayer perceptron (MLP) [3] or specialized Physics-Informed Neural Networks (PINNs), which explicitly incorporate governing laws into the loss function [4]. However, these architectures often require significant computational resources for training due to the large number of tunable parameters and can suffer from the vanishing gradient problem. An alternative is offered by single-layer neural networks based on the functional link principle (Functional Link Artificial Neural Network, FLANN) [5,6]. In the FLANN model, the hidden layer is replaced by a functional expansion block that increases the dimensionality of the input data through decomposition into orthogonal polynomial bases (e.g., Chebyshev, Legendre). This significantly reduces the number of network parameters and the number of iterations required for convergence, making FLANN a computationally efficient model with a high training speed.

FLANNs based on Chebyshev (ChNN) and Legendre (LeNN) polynomials have already been successfully applied to solve various ODEs [7,8,9], demonstrating competitive accuracy. However, the literature lacks a systematic investigation and direct comparison of the effectiveness of these two approaches. Moreover, the potential of neural networks based on other classical orthogonal polynomials, particularly Laguerre polynomials (LaNN), for solving initial and boundary value problems for ODEs remains virtually unexplored. Thus, there is a clear knowledge gap: there is no comparative analysis within the entire family of polynomial FLANNs, nor an assessment of their advantages compared to standard deep architectures.

The aim of this work is to fill this gap through a comprehensive study of the performance of the family of single-layer polynomial neural networks ChNN, LeNN, and LaNN. The test examples chosen are the Cauchy problem and the boundary value problem for a second-order nonlinear differential equation of the Lane–Emden type, for which an exact analytical solution is known. The Lane–Emden equation is frequently encountered in astrophysics, in theories of stellar structure, in theories of the thermal behavior of spherical gas clouds, and in theories of thermionic currents [8,10].

It should be noted that the system of Laguerre polynomials forms a complete orthogonal basis in $(0,\infty )$. When restricted to the finite interval $[0,1]$, they retain linear independence and can effectively approximate functions, especially those exhibiting exponential or singular behavior near zero—characteristic of many physical models, including the Lane–Emden equation. Therefore, one should expect superior performance of LANN compared to LeNN and ChNN.

This allows for an objective assessment of the accuracy of each method. The paper conducts a comparative analysis of the polynomial networks among themselves, as well as with the reference multilayer neural network MLP, based on two key criteria: the accuracy of the solution approximation and the algorithm execution time. It is demonstrated that properly configured polynomial networks can outperform MLP in computational efficiency while maintaining high accuracy, making them a promising tool for rapid numerical integration of differential equations.

The research plan in this paper contains the following sections. Section 1 provides an introduction to the research problem and its objectives. Section 2 provides information on orthogonal polynomials. Section 3 describes the architecture of a neural network built on orthogonal polynomials. Section 4 presents the formulation of the Cauchy problem and the edge problem for the Lane–Emden equation. Section 5 presents algorithms for solving the problems described above using polynomial neural networks, organized as pseudocode in the Python 3.13 programming language. Section 6 presents the results of studies of various polynomial neural networks and compares them with a multilayer neural network (MLP). Section 7 discusses issues of overfitting polynomial neural networks. Section 8 concludes the study.

2. Preliminaries

Orthogonal polynomials are widely used in approximation theory, in particular for constructing quadrature formulas for approximate calculation of integrals or function approximation [11].

Let $f\left(x\right)$ and $g\left(x\right)$ be real functions belonging to the class $L_w^2(a,b)$.

Definition 1.

Legendre polynomials of degree n for $x\in [-1,1]$ form an orthogonal system and can be computed by the recurrence formula:

$$P_0\left(x\right)=1,P_1\left(x\right)=x,P_{n+1}\left(x\right)={\displaystyle \frac{2n+1}{n+1}}x{P}_n\left(x\right)-{\displaystyle \frac{n}{n+1}}{P}_{n-1}\left(x\right).$$

(1)

Definition 2.

Chebyshev polynomials of degree n for $x\in [-1,1]$ form an orthogonal system and can be computed by the recurrence formula:

$$T_0\left(x\right)=1,T_1\left(x\right)=x,T_{m+1}\left(x\right)=2x{T}_m\left(x\right)-T_{m-1}\left(x\right).$$

(2)

Definition 3.

Laguerre polynomials of degree n for $x\in [0,\infty )$ form an orthogonal system and can be computed by the recurrence formula:

$$L_0\left(x\right)=1,L_1\left(x\right)=1-x,L_{m+1}\left(x\right)={\displaystyle \frac{(2m+1-x)L_m\left(x\right)-m{L}_{m-1}\left(x\right)}{m+1}}.$$

(3)

Recall that orthogonality of systems (1)–(3) means that the scalar product of any two functions of this system is equal to zero. More details about the properties of orthogonal polynomials can be found in [11].

For simplicity, we introduce the notation $O{P}_m\left(x\right)$, which defines the family of orthogonal polynomials (1)–(3).

3. Neural Network Model Based on Orthogonal Polynomials

Consider a single-layer neural network based on the functional-link principle (Figure 1).

Figure 1. Structure of a single-layer neural network based on orthogonal polynomials.

Figure 1 shows the structure of a single-layer neural network, which consists of one input node, one output layer, and a functional expansion block based on Legendre, Chebyshev, and Laguerre orthogonal polynomials. Here, by the notation OPNN, we represent one of the three neural networks LeNN, LaNN, or ChNN. The hidden layer is eliminated by transforming the input pattern into a higher-dimensional space using polynomials (1)–(3).

As input data, we consider a vector $x=\left\{x_1,x_2,\dots ,x_n\right\}$ of dimension n. The enhanced pattern is obtained using orthogonal polynomials (1)–(3):

$$\begin{array}{c}\hfill \left[1,O{P}_1\left(x_1\right),\dots ,O{P}_{m-1}\left(x_1\right);1,O{P}_1\left(x_2\right),\dots ,O{P}_{m-1}\left(x_2\right);\right.\\ \hfill \left. \dots ;1,O{P}_1\left(x_n\right),\dots ,O{P}_{m-1}\left(x_n\right)\right].\end{array}$$

(4)

According to (4), n-dimensional input x is extended into m-dimensional improved polynomials. Then, a weighted sum z of the expanded input data is formed, which is written as

$$z=\sum _{j=1}^m{w}_{j}O{P}_{j-1}\left(x\right).$$

(5)

To update the $w_j$ weights of the neural network, a backpropagation learning algorithm using the Adam (Adaptive Moment Estimation) optimizer is used. This algorithm provides faster convergence and is better at choosing an effective learning rate. Thus, the gradient of the error function with respect to the tunable parameters $w_j$ is determined.

The nonlinear hyperbolic tangent function $tanh\left(z\right)$ is considered as the activation function. For training, the gradient descent algorithm is used, and the weights are updated using the negative gradient at each iteration. The weights are initialized randomly and then updated:

$$w_j^{k+1}=w_j^k-\eta {\left[{\displaystyle \frac{\partial E\left(x,\mathbf{w}\right)}{\partial w_j}}\right]}^k,$$

(6)

where $\eta$ is the learning parameter taking values from 0 to 1, k is the iteration step used to update the weights, as usual in neural networks, and $E\left(x,\mathbf{w}\right)$ is the error function, $\mathbf{w}=\{w_1,\dots ,w_m\}$.

The output layer $N(x,\mathbf{w})$ is determined by the input vector x and the tunable parameters $w_j$ of the selected neural network by the formula

$$N(x,\mathbf{w})=tanh\left(z\right).$$

(7)

The activation function (7) can be chosen differently depending on the problem [12].

4. Problem Statement

Consider the following problem for a second-order ODE for $x\in [0,1]$:

$$y^{″}\left(x\right)+p\left(x\right)y^{\prime }\left(x\right)+q\left(x\right)y\left(x\right)=f\left(x\right),y\left(0\right)=y_0,y^{\prime }\left(0\right)=y_0^{\prime }.$$

(8)

Problem (8) is a Cauchy problem.

Definition 4.

The trial solution of the Cauchy problem (8) will be referred as the function

$$y_t\left(x\right)=y_0+y_0^{\prime }x+x^{2}N(x,\mathbf{w}).$$

(9)

Consider the following boundary value problem for a second-order ODE for $x\in [a,b]$:

$$y^{″}\left(x\right)+p\left(x\right)y^{\prime }\left(x\right)+q\left(x\right)y\left(x\right)=f\left(x\right),y\left(a\right)=y_a,y\left(b\right)=y_b.$$

(10)

Definition 5.

The trial solution of the boundary value problem (10) will be referred to as the function

$$y_t\left(x\right)=y_a+{\displaystyle \frac{x-a}{b-a}}(y_b-y_a)+(x-a)(b-x)N(x,\mathbf{w}).$$

(11)

The error function for recalculating the weights according to Formula (6), taking into account representations (9) or (11), is given as a residual of the following form:

$$E\left(x,\mathbf{w}\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left[y_t^{″}\left(x_i\right)+p\left(x_i\right)y_t^{\prime }\left(x_i\right)+q\left(x_i\right)y_t\left(x_i\right)-f\left(x_i\right)\right]}^2.$$

(12)

The calculation of derivatives (gradients) in Formula (12) is performed, taking into account Formulas (9) and (11). In the algorithms for implementing polynomial networks, gradient computation was performed automatically using the Python library [13].

5. Algorithms for Implementing Polynomial Neural Networks

The polynomial neural network algorithms were implemented in the Python programming language [13] in the PyCharm 3.13 environment [14] in the OPNN computer program. The OPNN program flowchart is shown in Figure 2.

Figure 2. (a) Stage 1: Data input and neural network initialization; (b) stage 2: training and output of simulation results.

It consists of two stages: The first stage defines the input parameters, i.e., Helper Functions: calculating orthogonal polynomials according to (1)–(3), automatic differentiation for calculating derivatives, functions for visualizing and estimating errors, and selects the appropriate polynomial neural network. The second stage contains the procedures for training it and inferring the obtained results. The main procedures in the algorithms are presented as pseudocode (Algorithms 1–7).

PolynomialBasis(x) is a procedure for calculating a polynomial basis for a given input x. Depending on the type of neural network, it invokes different procedures according to definitions (1)–(3).

Algorithm 1 Neural Network Training for ODE Solution

1: procedure MainTraining($network\_type$, $params$)   2:     Generate training data $x\_data\in [0.1,1]$   3:     Define exact solution: $exact\_solution\left(x\right)=-2ln(1+x^2)$   4:   5:     if $network\_type=1$ then   6:         $model\leftarrow LegendreNN(n\_polynomials)$   7:     else if $network\_type=2$ then   8:         $model\leftarrow ChebyshevNN(n\_polynomials)$   9:     else if $network\_type=3$ then 10:         $model\leftarrow LaguerreNN(n\_polynomials)$ 11:     else 12:          $model\leftarrow MLP(hidden\_size,num\_layers)$ 13:     end if 14:  15:     $start\_time\leftarrow \mathrm{current}\mathrm{time}\left(\right)$ 16:     $(trained\_model,loss\_history)\leftarrow$ TrainModel(model, x_data) 17:     $training\_time\leftarrow \mathrm{current}\mathrm{time}\left(\right)-start\_time$ 18:  19:     Visualize loss history and results 20:     Output comparative value table 21:     return $trained\_model,loss\_history,training\_time$ 22: end procedure

Algorithm 2 Model Training Procedure

1: procedure TrainModel($net$, $x\_data$, $epochs$, $\eta$)   2:     Initialize Adam optimizer with learning rate $\eta$   3:     Initialize ReduceLROnPlateau scheduler   4:     $loss\_history\leftarrow \left[\right]$   5:    6:     for $epoch\leftarrow 1$ to $epochs$ do   7:         $optimizer.zero\_grad\left(\right)$   8:         $loss\leftarrow$ ComputeLoss(net, x_data)   9:         $loss.backward\left(\right)$ 10:         $optimizer.step\left(\right)$ 11:          $scheduler.step(loss.item(\left)\right)$ 12:          $loss\_history.append(loss.item(\left)\right)$ 13:         if $epochmod100=0$ then 14:            Output current loss and learning rate 15:         end if 16:     end for 17:  18:     return $net,loss\_history$ 19: end procedure

Algorithm 3 Polynomial Neural Network Forward Pass

1: procedure Forward(x, $weights$)   2:     $polys\leftarrow$ PolynomialBasis(x)   3:     $output\leftarrow {\sum }_{i}weights\left[i\right]\times polys[:,i]$   4:     return $output$   5: end procedure

Algorithm 4 Loss Function Computation

1: procedure ComputeLoss($net$, x)   2:     Set $x.requires\_grad\leftarrow True$   3:     $(y,dy\_dx,d2y\_dx2)\leftarrow$ ComputeDerivatives(x, net)   4:    5:      Compute equation components:   6:     $term1\leftarrow d2y\_dx2$   7:     $term3\leftarrow 4\times (2\times exp(y)+exp(0.5\times y\left)\right)$   8:    9:     if $x>{10}^{-6}$ then 10:         $term2\leftarrow (2/x)\times dy\_dx$ 11:     else 12:          $term2\leftarrow 0$ 13:     end if 14:  15:     $eq\_residual\leftarrow term1+term2+term3$ 16:     $mask\leftarrow (x>{10}^{-6})$ 17:     $loss\leftarrow \mathrm{mean}(eq\_residual{\left[mask\right]}^2)$ 18:  19:     return $loss$ 20: end procedure

Algorithm 5 Trial Solution Derivative Computation

1: procedure ComputeDerivatives(x, $net$)   2:     $x\_tensor\leftarrow x$ (with $requires\_grad=True$)   3:     $y\leftarrow$ TrialSolution(x_tensor, net)   4:   5:    Compute first derivative:   6:     $dy\_dx\leftarrow {\displaystyle \frac{\partial y}{\partial x}}$ (using automatic differentiation)   7:    8:     Compute second derivative:   9:     $d2y\_dx2\leftarrow {\displaystyle \frac{\partial dy\_dx}{\partial x}}$ (using automatic differentiation) 10:  11:     return $y,dy\_dx,d2y\_dx2$ 12: end procedure

Algorithm 6 Trial Solution with Boundary Conditions (The Cauchy Problem)

1: procedure TrialSolution(x, $net$)   2:     $net\_output\leftarrow net\left(x\right)$   3:     $y\leftarrow x^2\times net\_output$   4:     return y   5: end procedure

A trial solution to the Cauchy problem according to Formula (9) is calculated using Algorithm 6.

Algorithm 7 Trial Solution with Boundary Conditions (Boundary Value Problem)

1: procedure TrialSolution(x, $net$)   2:     $net\_output\leftarrow net\left(x\right)$   3:     $y\leftarrow y_a(1-x)+y_{b}x+x(1-x)\times net\_output$   4:     return y   5: end procedure

A trial solution to the boundary value problem according to Formula (11) is calculated using Algorithm 7.

6. Research Results

Consider the operation of orthogonal polynomial networks on the example of solving the Cauchy problem for the singular nonlinear Lane–Emden differential equation

$$y^{″}\left(x\right)+{\displaystyle \frac{2}{x}}{y}^{\prime }\left(x\right)+4(2e^{y\left(x\right)}+e^{y\left(x\right)/2})=0,y\left(0\right)=0,y^{\prime }\left(0\right)=0,$$

(13)

as well as for the boundary value problem

$$y^{″}\left(x\right)+{\displaystyle \frac{2}{x}}{y}^{\prime }\left(x\right)+4(2e^{y\left(x\right)}+e^{y\left(x\right)/2})=0,y\left(0\right)=0,y\left(1\right)=-2ln2.$$

(14)

Remark 1.

It is known from refs. [8,10] that the Cauchy problem (13) has an exact solution of the form

$$y\left(x\right)=-2ln(1+x^2).$$

(15)

Based on Definition 4 by Formula (9), the trial solution for the Cauchy problem (13) is written as:

$$y_t\left(x\right)=x^{2}N(x,\mathbf{w}),$$

and for the boundary value problem (14), the exact solution (15) is also valid, taking into account representation (11) in the form

$$y_t\left(x\right)=-2ln2·x+x(1-x)N(x,\mathbf{w}).$$

The parameters of the orthogonal polynomial neural networks were set as follows: order of polynomials, $m=5$; number of training points from the segment $[0.1,1]$, 11; number of epochs 10.000; initial learning rate, $\eta =0.01$. The calculation results for solving the Cauchy problem (13) are given in Table 1.

Table 1. Solution of the Cauchy problem (13) using polynomial neural networks ($m=5$).

x	Exact $\mathit{y}\left(\mathit{x}\right)$	LeNN	Err	ChNN	Err	LaNN	Err
0.0	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
0.1	−0.019901	−0.020136	0.000235	−0.020098	0.000197	−0.020428	0.000528
0.2	−0.078441	−0.078844	0.000402	−0.078784	0.000342	−0.079432	0.000991
0.3	−0.172355	−0.172545	0.000190	−0.172517	0.000162	−0.172980	0.000624
0.4	−0.296840	−0.296616	0.000224	−0.296645	0.000195	−0.296431	0.000409
0.5	−0.446287	−0.445794	0.000493	−0.445859	0.000428	−0.444785	0.001502
0.6	−0.614969	−0.614560	0.000409	−0.614617	0.000353	−0.612906	0.002064
0.7	−0.797552	−0.797509	0.000044	−0.797525	0.000028	−0.795741	0.001811
0.8	−0.989392	−0.989695	0.000303	−0.989683	0.000290	−0.988527	0.000866
0.9	−1.186654	−1.186971	0.000317	−1.186994	0.000341	−1.186968	0.000315
1.0	−1.386294	−1.386297	0.000003	−1.386435	0.000140	−1.387423	0.001128

From Table 1, it can be seen that the polynomial neural networks give approximately the same result. Figure 3 shows the solution graph obtained by the LaNN, and Figure 4 shows the loss graph during training.

Figure 3. Example of LaNN operation.

Figure 4. Loss graph during training with LaNN.

The graph in Figure 3 demonstrates the efficiency of the Laguerre NN in solving the Cauchy problem (13). The loss function graph on a logarithmic scale is shown in Figure 4.

From Figure 4, it can be seen that the loss decreases with an increase in the number of epochs, which indicates successful model training. At the beginning of training (first 200 epochs), a rapid decrease in loss is observed, then the rate of decrease slows down, which is typical for the neural network training process. Note that the MLP method exhibits vanishing gradients. However, the single-layer polynomial approach inherently avoids this effect, contributing to the stability observed in Figure 4.

The loss graph allows for adjusting the hyperparameters of the neural network: learning rate, number of epochs, etc. The solution of the boundary value problem (14) is given in Table 2.

Table 2. Solution of the boundary value problem (14) using polynomial neural networks ($m=5$).

x	Exact $\mathit{y}\left(\mathit{x}\right)$	LeNN	Err	ChNN	Err	LaNN	Err
0.0	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
0.1	−0.019901	−0.018406	0.001495	−0.019366	0.000535	−0.014775	0.005125
0.2	−0.078441	−0.076801	0.001640	−0.077866	0.000575	−0.072799	0.005643
0.3	−0.172355	−0.170777	0.001579	−0.171657	0.000698	−0.167410	0.004945
0.4	−0.296840	−0.295251	0.001589	−0.295958	0.000882	−0.292378	0.004462
0.5	−0.446287	−0.444761	0.001526	−0.445407	0.000880	−0.441909	0.004378
0.6	−0.614969	−0.613748	0.001221	−0.614417	0.000553	−0.610649	0.004320
0.7	−0.797552	−0.796854	0.000699	−0.797530	0.000022	−0.793689	0.003863
0.8	−0.989392	−0.989202	0.000191	−0.989772	0.000380	−0.986570	0.002823
0.9	−1.186654	−1.186694	0.000040	−1.187007	0.000354	−1.185287	0.001367
1.0	−1.386294	−1.386294	0.000000	−1.386294	0.000000	−1.386294	0.000000

Here, it can be noted that the LaNN solved the problem more accurately than the ChNN, but was less accurate than the LeNN.

Table 3 shows the work of the multilayer neural network MLP with two hidden layers of 20 and 40 neurons respectively for solving the Cauchy problem (13).

Table 3. Solution of the Cauchy problem (13) using the multilayer MLP network.

x	Exact $\mathit{y}\left(\mathit{x}\right)$	MLP(20,2)	Err	MLP(40,2)	Err
0.0	0.000000	0.000000	0.000000	0.000000	0.000000
0.1	−0.019901	−0.019918	0.000017	−0.019996	0.000096
0.2	−0.078441	−0.078284	0.000158	−0.078554	0.000112
0.3	−0.172355	−0.171822	0.000534	−0.172433	0.000078
0.4	−0.296840	−0.295774	0.001066	−0.296928	0.000088
0.5	−0.446287	−0.444465	0.001823	−0.446377	0.000090
0.6	−0.614969	−0.612116	0.002853	−0.615030	0.000061
0.7	−0.797552	−0.793419	0.004134	−0.797606	0.000053
0.8	−0.989392	−0.983745	0.005647	−0.989446	0.000053
0.9	−1.186654	−1.179241	0.007413	−1.186684	0.000031
1.0	−1.386294	−1.376869	0.009425	−1.386322	0.000028

It can be seen from Table 3 that to achieve acceptable accuracy in MLP, it is necessary to increase the number of neurons in each layer. This leads, as we will show below, to an increase in the algorithm execution time.

Table 4 shows the solution of the boundary value problem (14) using the MLP network.

Table 4. Solution of the boundary value problem (14) using the multilayer MLP network.

x	Exact $\mathit{y}\left(\mathit{x}\right)$	MLP(20,2)	Err	MLP(40,2)	Err
0.0000	0.000000	0.000000	0.000000	0.000000	0.000000
0.1000	−0.019901	−0.019104	0.000796	−0.015659	0.004242
0.2000	−0.078441	−0.077559	0.000882	−0.072762	0.005679
0.3000	−0.172355	−0.171502	0.000854	−0.166195	0.006160
0.4000	−0.296840	−0.296061	0.000779	−0.291002	0.005838
0.5000	−0.446287	−0.445638	0.000649	−0.441194	0.005093
0.6000	−0.614969	−0.614448	0.000521	−0.610297	0.004672
0.7000	−0.797552	−0.797148	0.000405	−0.793336	0.004217
0.8000	−0.989392	−0.989135	0.000258	−0.986310	0.003083
0.9000	−1.186654	−1.186553	0.000100	−1.185103	0.001551
1.0000	−1.386294	−1.386294	0.000000	−1.386294	0.000000

From Table 4 it can be seen that increasing the number of neurons in each hidden layer leads to a deterioration in the accuracy of solving the boundary value problem (14). Let us show how the execution time of algorithms for solving problems (13) and (14) depends on the choice of neural network architecture. Neural network parameters: initial learning rate $\eta =0.01$, number of epochs 1000, order of polynomials $m=5$.

Table 5 shows the times spent on executing the algorithm for solving the Cauchy problem (13).

Table 5. Execution times of the algorithm for solving the Cauchy problem (13).

Network Architecture	Execution Time, s
LeNN	6.48
ChNN	4.81
LaNN	5.77
MLP(20,2)	4.77
MLP(40,2)	5.69

From Table 5 it can be seen that $\mathrm{ChNN}<\mathrm{MLP}(20,2)<\mathrm{LaNN}<\mathrm{LeNN}$. Here it should be noted that the neural network based on Laguerre orthogonal polynomials coped with the task faster than the neural network based on Legendre orthogonal polynomials. Moreover, the LaNN and ChNNs were faster in solving the Cauchy problem (13) than the multilayer MLP network.

The results of solving the boundary value problem (14) are given in Table 6.

Table 6. Execution times of the algorithm for solving the boundary value problem (14).

Network Architecture	Execution Time, s
LeNN	4.03
ChNN	3.59
LaNN	3.85
MLP(20,2)	3.71
MLP(20,3)	4.13

From Table 6, it follows that $\mathrm{ChNN}<\mathrm{MLP}(20,2)<\mathrm{LaNN}<\mathrm{LeNN}$, and therefore the trend in the execution times of algorithms for solving the boundary value problem (14) remains as in Table 5.

Here, a multilayer neural network MLP with three hidden layers of 20 neurons in each layer was used. From Table 6, it follows that single-layer orthogonal polynomial neural networks can work faster than multilayer neural networks of the MLP type, for example, the LeNN.

It should be noted that increasing the order of the polynomial n can also lead to a deterioration in the result. Let us take the parameters for the LeNN from the previous example (Table 3) and find solutions to problems (13) and (14) for polynomial orders $m=5$ and $m=7$.

The research results are given in Table 7 and Table 8.

Table 7. Solution of the Cauchy problem (13) using the LeNN with $m=5$ and $m=7$.

x	Exact $\mathit{y}\left(\mathit{x}\right)$	LeNN (m = 5)	Err	LeNN (m = 7)	Err
0.0000	0.000000	0.000000	0.000000	0.000000	0.000000
0.1000	−0.019901	−0.019803	0.000097	−0.015253	0.004647
0.2000	−0.078441	−0.078067	0.000374	−0.070342	0.008100
0.3000	−0.172355	−0.171714	0.000642	−0.170697	0.001658
0.4000	−0.296840	−0.296249	0.000591	−0.310144	0.013304
0.5000	−0.446287	−0.446258	0.000029	−0.474329	0.028041
0.6000	−0.614969	−0.615900	0.000931	−0.647436	0.032467
0.7000	−0.797552	−0.799405	0.001853	−0.819858	0.022305
0.8000	−0.989392	−0.991566	0.002174	−0.993316	0.003924
0.9000	−1.186654	−1.188236	0.001583	−1.178808	0.007846
1.0000	−1.386294	−1.386824	0.000529	−1.381549	0.004746

Table 8. Solution of the boundary value problem (14) using the single-layer LeNN network with $m=5$ and $m=7$.

x	Exact $\mathit{y}\left(\mathit{x}\right)$	LeNN (m = 5)	Err	LeNN (m = 7)	Err
0.0000	0.000000	0.000000	0.000000	0.000000	0.000000
0.1000	−0.019901	−0.020146	0.000246	−0.017843	0.002058
0.2000	−0.078441	−0.078711	0.000270	−0.076163	0.002278
0.3000	−0.172355	−0.172609	0.000254	−0.170318	0.002038
0.4000	−0.296840	−0.297066	0.000226	−0.294993	0.001847
0.5000	−0.446287	−0.446488	0.000201	−0.444588	0.001699
0.6000	−0.614969	−0.615148	0.000179	−0.613516	0.001454
0.7000	−0.797552	−0.797695	0.000142	−0.796478	0.001075
0.8000	−0.989392	−0.989474	0.000082	−0.988740	0.000652
0.9000	−1.186654	−1.186676	0.000022	−1.186362	0.000292
1.0000	−1.386294	−1.386294	0.000000	−1.386294	0.000000

From Table 7, we see that with an increase in the order of the polynomial, the calculation error increased, although the algorithm execution time decreased from 10.01 s for $m=5$ to 8.92 s for $m=7$.

From Table 8, we see that increasing the order of the Legendre polynomial leads to a deterioration in accuracy, and also to a decrease in the algorithm execution time for Table 8 at $m=5$–6.54 s, and at $m=7$–8.66 s.

Based on the above, we can conclude that the order of the polynomial requires careful tuning, as well as other hyperparameters of the neural network.

7. Overfitting Issues

It should be noted that the issue of overfitting remains relevant when using neural networks to solve differential equations. Overfitting here in the context of the problem of solving a differential equation is understood differently. Here, the training sample is the differential equation itself, which essentially defines an infinite set of constraints that the solution function must satisfy. The neural network here tries to approximate this function that satisfies these constraints everywhere in a given domain. It should also be said that there is no division of data into test and training samples here. We are looking for a single solution for the entire domain. The concept of “new data” from the same domain is erased—we want the network to work well on the entire domain at once. Thus, pure “memorization of noise in the training data” is not applicable here, since there are no discrete data initially. The concept of overfitting is understood mainly as overfitting at collocation points. We cannot check the fulfillment of the equation on the entire infinite domain. Therefore, we choose a finite set of points (collocation points) at which we calculate the residual of the equation and minimize it. The neural network between collocation points can behave incorrectly, for example, oscillate. The residual at the training collocation points will be very low (sometimes almost zero), but if we check the solution on another, denser grid of points, the residual will be huge. This is a sign of overfitting. Similarly, the concept of overfitting can arise at boundary (initial) points of the domain.

Here, overfitting can be combated by various methods similar to solving the issue of classical overfitting. For example, known methods are L1/L2 regularization, adaptive weights for the loss function, etc. However, in our opinion, the most effective and common method is dynamic selection of collocation points. Instead of using a fixed set of points, at each iteration (or after several epochs) collocations are selected anew (often randomly). Here, it is necessary to remember about the non-optimal choice of collocation points. If the points at which the equation residual is calculated are chosen unsuccessfully (for example, uniformly over the entire domain), the network may learn well in some areas and poorly in others (for example, in areas of high gradients) [15]. This approach does not allow the network to “memorize” their location and forces it to learn to satisfy the equation in the entire domain. We also note article [4], in which the authors study the issues of overfitting of the multilayer PINN caused by gradient imbalance.

In this article, the selection of collation points was static. However, dynamic selection algorithms are currently being developed using Chebyshev interpolation, random uniform distribution, and a number of other methods. The results of this research will be discussed in a future paper.

8. Conclusions

To our knowledge, this work is the first instance of an investigation on the approximate solution of the Cauchy problem and the boundary value problem for the Lane–Emden differential equation using three single-layer neural networks based on Legendre, Chebyshev, and Laguerre orthogonal polynomials. It is shown that the new neural network based on Laguerre polynomials gives acceptable results, and in some cases can be more accurate than the ChNN and faster than the LeNN. All three polynomial neural networks were also compared with the multilayer MLP neural network. Research has shown that single-layer polynomial neural networks can work better than the multilayer MLP neural network. Key parameters for polynomial neural networks, in addition to choosing the number of training epochs and learning rate, is the order of the polynomial. However, this order must be chosen optimally; its further increase or decrease can lead to a deterioration in results. Note that in multilayer neural networks it is necessary to adjust the number of hidden layers and the number of neurons in each of them. The latter means that single-layer polynomial neural networks have a simpler architecture than multilayer neural networks and, accordingly, they are easier to implement on a computer. The choice of neural network architecture depends on the specific task for an ordinary differential equation.

It should be noted that the effectiveness of LaNN on stiff systems or systems of integer-order partial differential equations has yet to be tested. This is a topic for further research of the LaNN method. Another direction of research is related to solving fractional differential equations and their systems [16,17,18,19,20], as well as for solving partial differential equations, similar to the work of [21].

From an applied research perspective, polynomial neural network methods can be applied to solving more global economic problems in various regions, for example, in modeling the energy sector, similar to work [22].