Improving PINN Convergence in Nonlinear Multiphase Flow Problems Through Weight Gradient Consistency Analysis

Abstract

The training of physics-informed neural networks (PINNs) for nonlinear multiphase flow in porous media is hampered by gradient conflicts between the individual components of the composite loss function. To address this problem, we propose a weighted gradient consistency metric that jointly accounts for the magnitudes and directions of the gradients of each loss term. Theoretical estimates of the convergence rate are derived, relating the proposed metric to the spectral properties of the preconditioner. The method is evaluated through a comparative study of optimizers—Adam, L-BFGS, and self-scaled Broyden—applied to three formulations of increasing complexity: a linear Buckley–Leverett model, a compressible two-phase model, and a fully nonlinear model with non-Newtonian rheology. The experiments demonstrate that self-scaled methods consistently achieve higher gradient alignment, faster loss reduction, and improved approximation accuracy compared to standard quasi-Newton and first-order baselines.

1. Introduction

The modeling of a two-phase flow process with oil displacement from a porous medium, by water or its solutions of various consistencies, is one of the key tasks of the theory and practice of oil field development [1]. Maximum possible accuracy in the description of such processes is a necessary condition for an adequate forecast of production, the optimization of well operation modes, and evaluation of the effectiveness of increasing oil recovery methods [2,3]. Modern mathematical models of two-phase (even isothermal) flow, as a rule, are a system of nonlinear partial differential equations describing the mass and momentum conservation laws of phases, as well as closing semiempirical relations specifying the compressibility of reservoir fluids and rocks, phase permeability, capillary forces, and rheological laws. Except for simplified problem statements, the integration of a system of equations is possible only numerically. The classical approach is based on the application of spatial discretization based, for example, on finite element or finite volume methods; time approximation with an implicit or explicit Euler scheme; and linearization procedures such as Newton’s method [4]. Eventually, a system of algebraic equations with poor conditionality is formed, which significantly complicates its solution and challenges the convergence and adequacy of the numerical solution. This circumstance has led to the development of iterative sequential methods, the efficiency, stability, and convergence of which are largely determined by the chosen strategy of discretization of differential operators [5,6].

In recent years, the PINN approach [7], proposed as an alternative to classical numerical methods for solving partial differential equations, has attracted considerable interest. The PINN approach is based on the insertion of physically valid constraints directly into the neural network learning process, which makes it possible to eliminate explicit grid discretization and potentially simplify the solution of multidimensional and parametrically complex problems. In the context of describing flow problems, the PINN algorithm is considered a promising tool for accelerated modeling, building surrogate models, and solving inverse problems [8], including identifying reservoir and fluid parameters.

However, the application of PINNs to two-phase flow problems faces several additional difficulties in comparison with the description of the flow in a porous medium of a single-phase fluid [8]. PINN does not cope well with tasks with discontinuity in their solutions—for example, “saturation waves” [5] in the so-called Buckley–Leverett formulation, when the relative phase permeabilities (RFPs) are characterized by strong nonlinearity—which leads to coefficient degeneracy and a tight connection between the equations. These problems are significantly aggravated when the compressibility of all phases is considered, which leads to a deterioration in convergence during the neural network learning process.

The correct accounting of “non-Newtonian” rheology, even for a single oil phase, is particularly difficult. Experimental and field data [9] show that heavy and highly viscous oils exhibit nonlinear rheological behavior, which requires the use of more complex rheological models such as viscoplastic or pseudoplastic fluids. In addition, processes of sequential loading and unloading are accompanied by thixotropy, which involves changes in the internal structure of the oil. Non-Newtonian flow can be observed not only for highly viscous fluids but also for less viscous ones when moving in porous media with low porosity and permeability, as well as during special chemical reagent (polymers, gels) flow, which is applied in oil recovery enhancement technologies. Ignoring these properties during subsurface flow modeling can lead to significant errors in the forecasting of flow rates, the oil saturation of the reservoir, and the waterlogging of the products. Despite these findings, most of studies devoted to the problems of reservoir oil displacement, where the PINN approach is used, are limited to Newtonian rheology. This ultimately leads to an erroneous forecast of the values of the oil recovery coefficient and to an incorrect estimation of the effectiveness of the applied development technologies.

In [10], an extended XPINN approach is considered, designed to solve flow problems in highly inhomogeneous porous media. The method is based on the computing domain separation into homogeneous subdomains; a separate neural network is trained in each of them, and the interaction between the subdomains is provided through the interface conditions of pressure and flow continuity. Numerical experiments demonstrate the high accuracy of the obtained solutions; however, this approach requires preliminary domain partitioning and the simultaneous training of several neural networks, which significantly complicates its implementation and increases computational costs.

An alternative approach was proposed in [11], where generative adversarial networks integrated with physical constraints were applied to solve the Buckley–Leverett equation describing a two-phase flow in a porous medium. It was shown that this method effectively approximates the saturation front and has increased resistance to noisy data compared to classical PINN approaches. At the same time, the formulation under consideration was limited to a simplified model without considering the compressibility of phases and was based on Darcy’s linear law, which narrows the scope of the applicability of the results obtained.

In [12], a hybrid approach to solving the Burgers equation for a viscous medium is proposed, combining classical boundary layer theory (BLT) and modern physics-informed neural networks (PINNs). Boundary layer theory is applied to obtain an analytical approximation in areas where viscosity dominates (near boundaries), and PINNs are further trained to accurately reproduce nonlinear dynamics, including the formation of shock waves, in the entire solution domain.

In [13], a systematic analysis of the application of a PINN to the problem of two-phase flow, considering capillary pressure, was carried out within the framework of the Masket–Leverett model. The influence of the neural network’s architecture was researched, as was how the loss function’s shape and the placement strategy of the calculated points influenced method accuracy and stability. The results of numerical experiments show that the application of adaptive schemes that generate calculation points and modification of the loss function can improve the quality of approximation; however, the implementation of standard PINN algorithms still does not exclude problems during modeling problems of liquids of different viscosities and mobility saturation fronts’ propagation in a porous medium, which underlines the limitations of such a “classical approach” when solving more general problems of two-phase flow.

In recent years, PINN-related papers have payed more attention not only to the architecture of the neural network and the form of the loss function but also to the choice of optimization algorithm [14]. Several studies show that the learning task for PINNs differs significantly from standard machine learning tasks; the loss function, as a rule, consists of several terms that require satisfying equations, boundary and initial conditions, etc., which can be characterized by significantly different scales and have different “geometry” in the parameter space. As a result, a highly heterogeneous and poorly conditioned landscape of the loss function is formed, which leads to slow convergence, deceleration of the process up to a stop at local minimum points, and high sensitivity to the choice of hyperparameters of the optimizer [15].

In this paper, we investigate how the choice of optimization algorithm affects the quality and stability of PINN training for progressively more complex formulations of nonlinear two-phase flow problems in porous media, ranging from the classical Buckley–Leverett model to compressible and non-Newtonian flow regimes.

We introduce a weighted gradient consistency metric that quantifies the agreement between gradients of different loss components by considering both their directions and magnitudes. Unlike conventional gradient alignment measures based solely on cosine similarity, the proposed metric emphasizes the relative contribution of competing loss terms and enables analysis of optimization conflicts in stiff nonlinear PINN problems.

In contrast to adaptive loss balancing and optimizer adaptation strategies commonly used in the PINN literature [16], the proposed metric is not employed to modify the training process directly. Instead, it serves as a diagnostic and analytical tool for studying the geometry of the optimization landscape and the interaction between physical constraints during training.

In addition, we consider a self-scaled modification of the BFGS optimizer [17] that dynamically adjusts the parameter update step based on the local structure of the gradient field. A comparative study of standard and modified optimization algorithms has been conducted to evaluate convergence stability, gradient consistency, and approximation accuracy in the presence of saturation fronts and strong model nonlinearity.

2. Mathematical Statement

To describe the process of oil being displaced by water in a porous medium, we consider a one-dimensional unsteady two-phase flow model that accounts for phase compressibility and the non-Newtonian rheology of the displaced fluid.

The equations of phase continuity are defined as follows:

$${\displaystyle \frac{\partial \left({\rho }_{\alpha }{S}_{\alpha }m\right)}{\partial t}}+{\displaystyle \frac{\partial }{\partial x}}\left({\rho }_{\alpha }{W}_{\alpha }\right)=0,$$

(1)

where $\alpha \in \{o,w\}$ corresponds to the water and oil phases accordingly; ${\rho }_{\alpha }$ is the density of the $\alpha$ phase; matrix porosity responds with m, the porosity coefficient; $S_{\alpha }$, the phase saturation; and $W_{\alpha }$, the flow rate of the corresponding phase $\alpha$.

Motion (flow) equations of each phase are set without considering capillary effects:

$$W_{\alpha }=-{\displaystyle \frac{k_{r\alpha }k}{{\mu }_{\alpha }}}{\displaystyle \frac{\partial p}{\partial x}},$$

(2)

where k is the absolute permeability of the porous medium, $k_{r\alpha }$ is the relative phase permeability, ${\mu }_{\alpha }$ is the dynamic viscosity of the phase, and p is the porous pressure.

The system is closed using the following relations, which account for the presence of two mobile phases in a deformable porous matrix, the dependence of oil viscosity on the pressure gradient, and the governing equations for weakly compressible phases:

$$\begin{array}{c}{S}_o+S_w=1,\hfill \end{array}$$

(3)

$$\begin{array}{c}\hfill m\left(p\right)=m_0+{\beta }_r(p-p_0),\end{array}$$

(4)

$$\begin{array}{c}\hfill {\mu }_o={\displaystyle \frac{{\mu }_{oh}-{\mu }_{ol}}{1+\exp \left[-B\left(\left|\frac{\partial p}{\partial x}\right|-G\right)\right]}}+{\mu }_{ol},\end{array}$$

(5)

$$\begin{array}{c}\hfill {\rho }_w\left(p\right)={\rho }_{w0}\left(1+{\beta }_w(p-p_0)\right),\end{array}$$

(6)

$$\begin{array}{c}\hfill {\rho }_o\left(p\right)={\rho }_{o0}\left(1+{\beta }_o(p-p_0)\right),\end{array}$$

(7)

where ${\beta }_r$, ${\beta }_w$, ${\beta }_o$ are the compressibility coefficients of the porous matrix, the water and oil phases, accordingly; ${\rho }_{w0}$, ${\rho }_{o0}$ are the phase densities at porous pressure $p_0$.

The nonlinearity of the flow process is due to the dependence of oil viscosity ${\mu }_o$ on the modulus of the pressure gradient $\left|\partial p/\partial x\right|$, which reflects the transformation of the internal structure of the oil with a change in the intensity of the flow process. The parameter ${\mu }_{oh}$ characterizes the viscosity of oil within an undisturbed structure, and ${\mu }_{ol}$—within a destroyed structure. Coefficient B determines the intensity of structural changes, and the parameter G sets the threshold value of the pressure gradient, above which the destruction of the structure begins.

The problem is defined over the following spatial domain and time interval:

$$x\in [0,L],t\in [0,T].$$

(8)

The initial conditions assume uniform pressure and phase saturation throughout the domain, with water saturation set to the critical bound water value, while the boundary conditions represent injection at constant water pressure at the left boundary of the domain:

$$\begin{array}{c}p(x,0)=p_0,S_w(x,0)=S_{w0},\hfill \end{array}$$

(9)

$$\begin{array}{c}p(0,t)=p_1,S_w(0,t)=S_{w1},\hfill \end{array}$$

(10)

$$\begin{array}{c}p(L,t)=p_0.\hfill \end{array}$$

(11)

3. The Optimization Problem

Within the PINN approach, the solution of the two-phase flow problem is approximated by a parameterized function depending on the space–time coordinates. A neural network is considered as follows:

$${\mathcal{N}}_{\mathit{\theta }}:(x,t)\in \mathrm{\Omega }\times [0,T]⟼\left(p(x,t),S_w(x,t)\right)\in {\mathbb{R}}^2,$$

where $\theta$ is a set of trainable network parameters (a vector in the network parameter space $R^d$, where d is the number of parameters to be trained), and the output components are the required pressure and water saturation fields.

This representation makes it possible to consider a neural network as an element of a finite-dimensional parametric subspace of the functional solution space. At the same time, the original problem of finding a continuous solution over the domain $\mathrm{\Omega }\times [0,T]$ is reduced to a numerical optimization problem defined on a finite set of spatiotemporal points, including collocation points within the domain where the governing equations are enforced, as well as points where the initial and boundary conditions are satisfied.

A set of spatiotemporal points is defined as

$$\mathcal{X}={\left\{(x_i,t_i)\right\}}_{i=1}^N\subset [0,L]\times [0,T].$$

Then the neural network approximation of the solution (${\widehat{p}}_i,{\widehat{S}}_i$) with a set of trainable network parameters $\theta$ is defined as follows:

$$\left({\widehat{p}}_i,{\widehat{S}}_i\right):={\mathcal{N}}_{\theta }(x_i,t_i)=\left(\widehat{p}(x_i,t_i;\theta ),{\widehat{S}}_w(x_i,t_i;\theta )\right),i=1,\dots ,N.$$

(12)

For other physical parameters of the problem, a similar representation will be applied.

3.1. Residual Pressure Equation

Substituting the neural network approximation expressions from (12) into the mass conservation Equation (1) and summing over both phases, the pressure discrepancy is determined as follows:

$${\mathcal{R}}_p={\displaystyle \frac{\partial }{\partial t}}\left({\rho }_w{\widehat{S}}_w\varphi +{\rho }_o(1-{\widehat{S}}_w)\varphi \right)+{\displaystyle \frac{\partial }{\partial x}}\left({\rho }_w{\widehat{W}}_w+{\rho }_o{\widehat{W}}_o\right).$$

(13)

Here phase flow rates are determined as follows:

$${\widehat{W}}_w=-{\lambda }_w\left({\widehat{S}}_w\right){\displaystyle \frac{\partial \widehat{p}}{\partial x}},{\widehat{W}}_o=-{\lambda }_o({\widehat{S}}_w,{\mu }_o){\displaystyle \frac{\partial \widehat{p}}{\partial x}},$$

(14)

and the phase mobilities are determined as follows:

$${\lambda }_w={\displaystyle \frac{k{k}_{rw}\left({\widehat{S}}_w\right)}{{\mu }_w}},{\lambda }_o={\displaystyle \frac{k{k}_{ro}\left({\widehat{S}}_w\right)}{{\mu }_o(\partial \widehat{p}/\partial x)}}.$$

3.2. Residual Equation for Saturation

The residual for water saturation is determined from Equation (1) for the aqueous phase (w) as follows:

$${\mathcal{R}}_S^{\mathrm{inv}}={\displaystyle \frac{\partial }{\partial t}}\left({\rho }_w{\widehat{S}}_w\varphi \right)+{\displaystyle \frac{\partial }{\partial x}}\left({\rho }_w{\widehat{W}}_w\right).$$

(15)

To increase the stability of PINN learning in the presence of saturation fronts (large saturation gradients) and the pronounced nonlinearity of the flow equations, we introduce an artificial viscosity with a constant coefficient. Adding an additional “diffusion” term makes it possible to smooth out numerical oscillations occurring in areas of large saturation gradients and provides a more stable approximation of the solution by a neural network. The coefficient of artificial viscosity is determined by a constant value and is assumed to be equal in all experiments: $\nu =35\times {10}^{-4}$. The selection of the artificial viscosity parameter $\nu$ is discussed in Appendix A, where a sensitivity analysis based on RMSE and residual metrics is presented in Figure A1.

The constant artificial viscosity leads to a modified saturation equation containing an additional diffusion term with a coefficient $\nu$.

$${\mathcal{R}}_S={\mathcal{R}}_S^{\mathrm{inv}}-{\displaystyle \frac{\partial }{\partial x}}\left(\nu {\displaystyle \frac{\partial {\widehat{S}}_w}{\partial x}}\right)=0,$$

(16)

which is used during formation of the loss function in the PINN algorithm. Numerical experiments show that the proposed modification significantly increases the stability of learning and makes it possible to reproduce saturation fronts correctly without the need for domain partitioning or the training of several neural networks.

3.3. Loss Function

The loss function $\mathcal{L}$ is formed as a weight function considering the weight $w_i$ which corresponds to the i -term of the loss function (loss component)

$$\mathcal{L}=w_{ps}·({\mathcal{L}}_p+{\mathcal{L}}_S)+w_{ic}·({\mathcal{L}}_{ic}^p+{\mathcal{L}}_{ic}^S)+w_{bc}·({\mathcal{L}}_{bc}^p+{\mathcal{L}}_{bc}^S)=w_{ps}·{\mathcal{L}}_{pde}+w_{ic}·{\mathcal{L}}_{ic}+w_{bc}·{\mathcal{L}}_{bc}$$

(17)

where the contributions of the equation residuals (the first two terms), the residuals from the initial conditions (the third and fourth terms), and the residuals from the boundary conditions (the last two terms) are taken into account:

$$\begin{array}{cc}\hfill {\mathcal{L}}_p& ={\displaystyle \frac{1}{N_f}}{\displaystyle \sum _{i=1}^{N_f}}{\left|{\mathcal{R}}_p(x_i,t_i)\right|}^2,\hfill \\ \hfill {\mathcal{L}}_S& ={\displaystyle \frac{1}{N_f}}{\displaystyle \sum _{i=1}^{N_f}}{\left|{\mathcal{R}}_S(x_i,t_i)\right|}^2,\hfill \\ \hfill {\mathcal{L}}_{ic}^p& ={\displaystyle \frac{1}{N_{ic}}}\sum _i{\left|\widehat{p}(x_i,0)-p_0\right|}^2,\hfill \\ \hfill {\mathcal{L}}_{ic}^S& ={\displaystyle \frac{1}{N_{ic}}}\sum _i{\left|{\widehat{S}}_w(x_i,0)-S_{w0}\right|}^2,\hfill \\ \hfill {\mathcal{L}}_{bc}^p& ={\displaystyle \frac{1}{N_{bc}}}\sum _i\left(|\widehat{p}(0,t_i)-p_1{|}^2+{|\widehat{p}(L,t_i)-p_0|}^2\right),\hfill \\ \hfill {\mathcal{L}}_{bc}^S& ={\displaystyle \frac{1}{N_{bc}}}\sum _i{\left|{\widehat{S}}_w(0,t_i)-1\right|}^2.\hfill \end{array}$$

4. Gradient Consistency Metrics

To quantify the consistency of gradients across different components of the loss function, we introduce a gradient consistency metric. It considers not only the directions of the corresponding loss gradients, but also their modules, which makes it possible to more accurately assess the contribution of each loss component to the overall optimization direction, especially in the presence of sharp edges and local conflicts between the physical terms of the loss function.

The network loss function is represented as the sum of n components:

$$\mathcal{L}\left(\theta \right)={\displaystyle \sum _{i=1}^n}{\mathcal{L}}_i\left(\theta \right),$$

(18)

where ${\mathcal{L}}_i$ corresponds to the equations residuals, boundary and initial conditions, or observational data. In the researched case $n=3$. At the k-step of optimization, individual gradients are calculated with the neural network parameter of each component:

$$g_k^i={\nabla }_{\theta }{\mathcal{L}}_i\left({\theta }_k\right),$$

(19)

and the full gradient:

$$g_k={\displaystyle \sum _{i=1}^n}{g}_k^i.$$

(20)

At a given iteration, the consistency metric is defined as the generalized cosine similarity between all pairs of gradients, taking into account their magnitudes.

$${\tilde{C}}_k={\displaystyle \frac{{\sum }_{i<j}\parallel g_k^i{\parallel }^{\beta }{\parallel g_k^j\parallel }^{\beta }\frac{g_k^i·g_k^j}{\parallel g_k^i\parallel \parallel g_k^j\parallel }}{{\sum }_{i<j}\parallel g_k^i{\parallel }^{\beta }{\parallel g_k^j\parallel }^{\beta }}},\beta >0.$$

(21)

Here and thereafter $\parallel ·\parallel$ denotes the Euclidean norm in the parameter space of the network $R^d$.

Parameter $\beta$ adjusts the sensitivity of the metric ${\tilde{C}}_k$ to gradient scale: at $\beta >1$ larger gradients get more weight, at $\beta <1$ smaller ones do. This makes it possible to adapt the metric to the specifics of the task. Within the framework of this paper, systematic study of the effect of $\beta$ was not conducted and $\beta =1$ is fixed in all experiments to ensure the simplicity and interpretability of the results obtained. This choice ensures that the weights are linearly dependent on the gradient norms.

${\tilde{C}}_k$ values lie in the range $[-1,1]$:

• ${\tilde{C}}_k\approx 1$—large gradients are aligned (co-directional), and learning is effective;
• ${\tilde{C}}_k\approx 0$—the gradients are orthogonal on average, and the optimizer’s step is partially compensated;
• ${\tilde{C}}_k\approx -1$—large gradients counteract each other, and there is a conflict between the loss components.

Two types of estimations are set to analyze learning dynamics:

• Component consistency (intrastep), which evaluates the consistency of different components gradients in one iteration

  $${\tilde{C}}_k^{\mathrm{intra}}={\tilde{C}}_k(g_k^1,g_k^2,\dots g_k^n);$$

  (22)
• Full consistency (interstep) comparing the direction of the full gradient between neighboring iterations

  $${\tilde{C}}_k^{\mathrm{inter}}={\displaystyle \frac{g_{k-1}·g_k}{\parallel g_{k-1}\parallel \parallel g_k\parallel }}.$$

  (23)

The use of such metrics allows us to:

• Identify local conflicts between loss components that lead to learning delays;
• Evaluate the contribution of large and small (modulo) gradients, which is especially important for tasks with sharp saturation front;
• Compare the effectiveness of different optimizers (Adam, L-BFGS, self-scaled modifications);
• Quantify the dynamics of network convergence through gradient consistency indicators.

5. Theoretical Analysis of the Influence of Gradient Consistency

In this section, we formalize the relationship between the proposed gradient consistency metric ${\tilde{C}}_k$ and the rate of decrease of the loss function under conditional gradient descent. The loss function is considered as follows: $\mathcal{L}\left(\theta \right)={\sum }_{i=1}^n{\mathcal{L}}_i\left(\theta \right)$ where $\theta \in {\mathbb{R}}^d$. $\mathcal{L}$ is considered to be L-smooth relative to the Euclidean norm:

$$\parallel \nabla \mathcal{L}\left(\theta \right)-\nabla \mathcal{L}\left({\theta }^{\prime }\right)\parallel \le L\parallel \theta -{\theta }^{\prime }\parallel \forall \theta ,{\theta }^{\prime }.$$

At each optimization step (iteration) k, preconditioned gradient descent with a preconditioning matrix $Q_k$ and a learning rate $\eta$ are applied:

$${\theta }_{k+1}={\theta }_k-\eta b_k,b_k=Q_k{g}_k,g_k={\displaystyle \sum _{i=1}^n}{g}_k^i,$$

where the $g_k^i=\nabla {\mathcal{L}}_i\left({\theta }_k\right)$. $Q_k$ matrix is considered to be symmetric and positively defined, and its spectrum is limited:

$$0<\xi \le {\lambda }_{\min }\left(Q_k\right)\le {\lambda }_{\max }\left(Q_k\right)\le E,\forall k.$$

Values characterizing the spread of gradient norms are set:

$$m_{\beta }=\underset{i<j}{\min }\parallel g_k^i{\parallel }^{1-\beta }\parallel g_k^j{\parallel }^{1-\beta },M_{\beta }=\underset{i<j}{\max }\parallel g_k^i{\parallel }^{1-\beta }{\parallel g_k^j\parallel }^{1-\beta }.$$

The actual decrease in the loss function at the k-iteration is ${\mathrm{\Delta }}_k=\mathcal{L}\left({\theta }_k\right)-\mathcal{L}\left({\theta }_{k+1}\right)$. Although the global loss landscape of a PINN is generally non-convex and may violate global L-smoothness, the optimization trajectories considered here originate from parameters obtained after a preliminary Adam phase. In this local region, the loss function typically exhibits a more regular, approximately quadratic behavior, making the L-smoothness assumption a reasonable local approximation. This is precisely the regime in which quasi-Newton methods (L-BFGS and the self-scaled Broyden method) operate, and thus the local analysis is directly relevant to the second stage of the training strategy. The assumption that the eigenvalues of $Q_k$ lie within $[\xi ,E]$ is consistent with standard BFGS-type approximations when combined with Wolfe line search and a positive definite initial Hessian. The self-scaled Broyden variant further reduces the ratio $E/\xi$, effectively improving the conditioning of the inverse Hessian approximation.

Assumption 1

(about the lower limit of loss reduction). All the above assumptions are considered fulfilled, and the learning rate (η) satisfies the condition $\eta \le 1/\left(LE\right)$. Then the estimate is valid for any k:

$${\mathrm{\Delta }}_k\ge A{\displaystyle \sum _{i=1}^n}{\parallel g_k^i\parallel }^2+B{W}_k{\tilde{C}}_k,$$

(24)

where

$$A=\eta (2\xi -E)-{\displaystyle \frac{L{\eta }^2}{2}}{E}^2,B=2\eta \xi m_{\beta }-L{\eta }^2{E}^2{M}_{\beta },$$

$$W_k=\sum _{i<j}\parallel g_k^i{\parallel }^{\beta }{\parallel g_k^j\parallel }^{\beta }.$$

Proof of Assumption 1.

Step 1

The standard inequality follows from the L-smoothness of the $\mathcal{L}$:

$$\mathcal{L}\left({\theta }_{k+1}\right)\le \mathcal{L}\left({\theta }_k\right)+\nabla \mathcal{L}{\left({\theta }_k\right)}^{\top }({\theta }_{k+1}-{\theta }_k)+{\displaystyle \frac{L}{2}}{\parallel {\theta }_{k+1}-{\theta }_k\parallel }^2.$$

(25)

Substituting ${\theta }_{k+1}-{\theta }_k=-\eta b_k$ and transferring $\mathcal{L}\left({\theta }_k\right)$ to the left, the inequality is obtained:

$${\mathrm{\Delta }}_k\ge \eta \nabla \mathcal{L}{\left({\theta }_k\right)}^{\top }{b}_k-{\displaystyle \frac{L{\eta }^2}{2}}{\parallel b_k\parallel }^2.$$

As $\nabla \mathcal{L}\left({\theta }_k\right)={\sum }_i{g}_k^i$ and $b_k=Q_k{\sum }_i{g}_k^i={\sum }_i{b}_k^i$, where $b_k^i=Q_k{g}_k^i$ then

$${\mathrm{\Delta }}_k\ge \eta \sum _i{g}_k^{i\top }{b}_k^i+\eta \sum _{i\ne j}{g}_k^{i\top }{b}_k^j-{\displaystyle \frac{L{\eta }^2}{2}}{∥\sum _i{b}_k^i∥}^2.$$

(26)

• Step 2: Evaluation of diagonal terms

Spectral decomposition $Q_k$ is applied, as $\xi I⪯Q_k$ then $x^{\top }{Q}_{k}x\ge \xi {\parallel x\parallel }^2$ is done for any vector x. Thus,

$$\sum _i{g}_k^{i\top }{b}_k^i=\sum _i{g}_k^{i\top }{Q}_k{g}_k^i\ge \xi \sum _i{\parallel g_k^i\parallel }^2.$$

(27)

• Step 3: Evaluation of the cross terms

$u_k^i=Q_k^{1/2}{g}_k^i$ is defined. Then $b_k^i=Q_k^{1/2}{u}_k^i$ and then

$$\sum _{i\ne j}{g}_k^{i\top }{b}_k^j=\sum _{i\ne j}{u}_k^{i\top }{u}_k^j={∥\sum _i{u}_k^i∥}^2-\sum _i{\parallel u_k^i\parallel }^2.$$

Applying spectral estimates to $Q_k^{1/2}$ results in $\parallel {\sum }_i{u}_k^i\parallel \ge \sqrt{\xi }\parallel {\sum }_i{g}_k^i\parallel$ and $\parallel u_k^i{\parallel }^2\le E{\parallel g_k^i\parallel }^2$. Therefore,

$$\sum _{i\ne j}{g}_k^{i\top }{b}_k^j\ge \xi {∥\sum _i{g}_k^i∥}^2-E\sum _i{\parallel g_k^i\parallel }^2.$$

(28)

Revealing the square of the norm of the original gradients sum,

$${∥\sum _i{g}_k^i∥}^2=\sum _i\parallel g_k^i{\parallel }^2+2\sum _{i<j}\parallel g_k^i\parallel \parallel g_k^j\parallel cos{\theta }_{ij}^k,$$

where $cos{\theta }_{ij}^k={\displaystyle \frac{g_k^{i\top }{g}_k^j}{\parallel g_k^i\parallel \parallel g_k^j\parallel }}$. Substituting the square of the norm in inequality (28), further obtains

$$\sum _{i\ne j}{g}_k^{i\top }{b}_k^j\ge (\xi -E)\sum _i\parallel g_k^i{\parallel }^2+2\xi \sum _{i<j}\parallel g_k^i\parallel \parallel g_k^j\parallel cos{\theta }_{ij}^k.$$

(29)

• Step 4: The relation of the sum of pairwise intersections with the metric ${\tilde{C}}_k$

It is noted that

$$\parallel g_k^i\parallel \parallel g_k^j\parallel cos{\theta }_{ij}^k=\left(\parallel g_k^i{\parallel }^{1-\beta }{\parallel g_k^j\parallel }^{1-\beta }\right)\left(\parallel g_k^i{\parallel }^{\beta }{\parallel g_k^j\parallel }^{\beta }cos{\theta }_{ij}^k\right).$$

$a_{ij}=\parallel g_k^i{\parallel }^{1-\beta }{\parallel g_k^j\parallel }^{1-\beta }$, $b_{ij}=\parallel g_k^i{\parallel }^{\beta }{\parallel g_k^j\parallel }^{\beta }cos{\theta }_{ij}^k$ is defined. It was also defined that $m_{\beta }\le a_{ij}\le M_{\beta }$ for each $i<j$. Then

$$m_{\beta }\sum _{i<j}{b}_{ij}\le \sum _{i<j}{a}_{ij}{b}_{ij}\le M_{\beta }\sum _{i<j}{b}_{ij}.$$

But ${\sum }_{i<j}{b}_{ij}=W_k{\tilde{C}}_k$ as defined by metric. Therefore,

$$m_{\beta }{W}_k{\tilde{C}}_k\le \sum _{i<j}\parallel g_k^i\parallel \parallel g_k^j\parallel cos{\theta }_{ij}^k\le M_{\beta }{W}_k{\tilde{C}}_k.$$

(30)

• Step 5: Evaluation of the last term in (26)

Similarly, with application of the inequality $\parallel {\sum }_i{b}_k^i\parallel \le E\parallel {\sum }_i{g}_k^i\parallel$, it is obtained that

$${∥\sum _i{b}_k^i∥}^2\le E^2{∥\sum _i{g}_k^i∥}^2=E^2\left(\sum _i\parallel g_k^i{\parallel }^2+2\sum _{i<j}\parallel g_k^i\parallel \parallel g_k^j\parallel cos{\theta }_{ij}^k\right).$$

Applying the upper boundary from (30) for the sum of pairwise intersections, it is obtained that

$${∥\sum _i{b}_k^i∥}^2\le E^2\left(\sum _i{\parallel g_k^i\parallel }^2+2M_{\beta }{W}_k{\tilde{C}}_k\right).$$

(31)

• Final estimation

Terms (27), (29) (using the lower boundary from (30) for the cross terms) and (31) are substituted into (26):

$$\begin{array}{cc}\hfill {\mathrm{\Delta }}_k& \ge \eta \xi \sum _i{\parallel g_k^i\parallel }^2+\eta \left[(\xi -E)\sum _i{\parallel g_k^i\parallel }^2+2\xi m_{\beta }{W}_k{\tilde{C}}_k\right]\hfill \\ & -{\displaystyle \frac{L{\eta }^2}{2}}{E}^2\left(\sum _i{\parallel g_k^i\parallel }^2+2M_{\beta }{W}_k{\tilde{C}}_k\right)\hfill \\ & =\left[\eta (2\xi -E)-{\displaystyle \frac{L{\eta }^2}{2}}{E}^2\right]\sum _i{\parallel g_k^i\parallel }^2+\left[2\eta \xi m_{\beta }-L{\eta }^2{E}^2{M}_{\beta }\right]W_k{\tilde{C}}_k.\hfill \end{array}$$

  □

5.1. Analysis of the Estimation Obtained

Let us consider the main factors influencing the value of the lower boundary.

• The coefficient A in the sum of the squares of the norms

This coefficient in inequality (24) is responsible for the contribution that does not depend on the consistency of the directions. It is quadratic relating to $\eta$ and reaches a maximum at

$${\eta }^{\ast }={\displaystyle \frac{2\xi -E}{L{E}^2}},A_{max}={\displaystyle \frac{{(2\xi -E)}^2}{2L{E}^2}}.$$

If $2\xi -E\le 0$ (that is, the condition number $\kappa =E/\xi \ge 2$) then $A\le 0$ at any $\eta$. In this case, the first term can slow down convergence, and a positive contribution can be obtained only at the cost of the second term at ${\tilde{C}}_k>0$. A well-conditioned preconditioner $\xi \approx E$ ensures the positivity of A at small $\eta$.

• Coefficient B for a weight metric

Coefficient B in inequality (24) determines how strongly the consistency of the gradients affects the loss reductions. The condition of B-positivity is provided by the condition

$$\eta <{\displaystyle \frac{2\xi m_{\beta }}{L{E}^2{M}_{\beta }}}.$$

The multipliers $m_{\beta }$ and $M_{\beta }$ reflect the spread of gradient norms. With $\beta =1$ $m_1=M_1=1$ and the condition simplifies to $\eta <{\displaystyle \frac{2\xi }{L{E}^2}}$. With $\beta \ne 1$ the $m_{\beta }/M_{\beta }$ ratio can be less than one, which tightens the restriction on $\eta$. Thus, a strong spread of norms (${\displaystyle \frac{M_{\beta }}{m_{\beta }}}\gg 1$) requires a more careful choice of steps to preserve the positive contribution of consistency.

Thus, the metric ${\tilde{C}}_k$ enables quantification of the relationship between gradient consistency and the efficiency of the optimization step, taking into account both the direction and magnitude of the gradients of each loss component. Under the additional assumption of the scalar of the preconditioner ($Q_k=cI$) and balanced norms $\parallel b_k^i\parallel =\lambda$ the estimation (24) agreed with results in [18].

5.2. Two-Step Evaluation and Interstep Consistency

To complete the analysis, let us consider the effect of the step-by-step consistency of gradients on the total decrease in the loss function over two consecutive iterations. Let us define the interstep metric as the cosine of the angle between complete gradients at the nearest steps

$${\tilde{C}}_k^{\mathrm{inter}}={\displaystyle \frac{g_{k-1}·g_k}{\parallel g_{k-1}\parallel \parallel g_k\parallel }}.$$

(32)

Assumption 2

(On the two-step lower boundary). Let the assumptions introduced in the one-step evaluation be fulfilled. Then the estimate is valid for any two consecutive iterations

$$\begin{array}{cc}\hfill {\mathrm{\Delta }}_{k-1}+{\mathrm{\Delta }}_k& \ge \left(\eta \xi -{\displaystyle \frac{L{\eta }^2}{2}}{E}^2\right)\parallel g_{k-1}{\parallel }^2-{\displaystyle \frac{L{\eta }^2}{2}}{E}^2{\parallel g_k\parallel }^2\hfill \\ \hfill & +\eta \xi \parallel g_{k-1}\parallel \parallel g_k\parallel {\tilde{C}}_k^{\mathrm{inter}}-\left(\eta (E-\xi )+L{\eta }^2{E}^2\right)\parallel g_{k-1}\parallel \parallel g_k\parallel .\hfill \end{array}$$

(33)

Proof of Assumption 2.

Step 1

By the definition of L-smoothness of the function $\mathcal{L}$, for any pair of vectors—in particular, for ${\theta }_{k-1}$ and ${\theta }_{k+1}$—it holds that

$$\mathcal{L}\left({\theta }_{k+1}\right)\le \mathcal{L}\left({\theta }_{k-1}\right)+\nabla \mathcal{L}{\left({\theta }_{k-1}\right)}^{\top }({\theta }_{k+1}-{\theta }_{k-1})+{\displaystyle \frac{L}{2}}{\parallel {\theta }_{k+1}-{\theta }_{k-1}\parallel }^2.$$

Considering ${\theta }_{k+1}-{\theta }_{k-1}=-\eta (b_{k-1}+b_k)$, and switching $\mathcal{L}\left({\theta }_{k-1}\right)$ to the left, the following is obtained:

$${\mathrm{\Delta }}_{k-1}+{\mathrm{\Delta }}_k\ge \eta \nabla \mathcal{L}{\left({\theta }_{k-1}\right)}^{\top }(b_{k-1}+b_k)-{\displaystyle \frac{L{\eta }^2}{2}}{\parallel b_{k-1}+b_k\parallel }^2.$$

(34)

Let us evaluate the first linear term on the right side of the inequality (34). As $\nabla \mathcal{L}\left({\theta }_{k-1}\right)=g_{k-1}$, the following is obtained:

$$g_{k-1}^{\top }{b}_{k-1}=g_{k-1}^{\top }{Q}_{k-1}{g}_{k-1}\ge \xi {\parallel g_{k-1}\parallel }^2,$$

where the following condition is applied: $Q_{k-1}⪰\xi I$.

• Step 2

To evaluate the second term on the right side consider $Q_k$ as $Q_k=\xi I+(Q_k-\xi I)$, then

$$g_{k-1}^{\top }{b}_k=\xi g_{k-1}^{\top }{g}_k+g_{k-1}^{\top }(Q_k-\xi I)g_k.$$

As $\parallel Q_k-\xi I\parallel \le E-\xi$ (the spectral norm of the difference does not exceed the difference between the maximum and minimum eigenvalues), the following is obtained:

$$|g_{k-1}^{\top }(Q_k-\xi I)g_k|\le (E-\xi )\parallel g_{k-1}\parallel \parallel g_k\parallel ,$$

Therefore, low estimation follows:

$$g_{k-1}^{\top }(Q_k-\xi I)g_k\ge -(E-\xi )\parallel g_{k-1}\parallel \parallel g_k\parallel .$$

Thus,

$$g_{k-1}^{\top }{b}_k\ge \xi \parallel g_{k-1}\parallel \parallel g_k\parallel {\tilde{C}}_k^{\mathrm{inter}}-(E-\xi )\parallel g_{k-1}\parallel \parallel g_k\parallel .$$

(35)

Combining (35) with an estimate for $g_{k-1}^{\top }{b}_{k-1}$, the following is obtained:

$$\eta g_{k-1}^{\top }(b_{k-1}+b_k)\ge \eta \xi \parallel g_{k-1}{\parallel }^2+\eta \xi \parallel g_{k-1}\parallel \parallel g_k\parallel {\tilde{C}}_k^{\mathrm{inter}}-\eta (E-\xi )\parallel g_{k-1}\parallel \parallel g_k\parallel .$$

(36)

• Step 3

The estimation of the quadratic term in (34). With application of spectral boundaries:

$$\parallel b_{k-1}\parallel \le E\parallel g_{k-1}\parallel ,\parallel b_k\parallel \le E\parallel g_k\parallel ,$$

and the triangle inequality, the following is obtained:

$$\begin{array}{cc}\hfill \parallel b_{k-1}+b_k{\parallel }^2& \le (\parallel b_{k-1}\parallel +\parallel b_k{\parallel )}^2\hfill \\ \hfill & \le E^2(\parallel g_{k-1}\parallel +\parallel g_k{\parallel )}^2\hfill \\ \hfill & =E^2\left(\parallel g_{k-1}{\parallel }^2+\parallel g_k{\parallel }^2+2\parallel g_{k-1}\parallel \parallel g_k\parallel \right)\hfill \end{array}$$

(37)

Substituting (36) and (37) into (34), the following inequality is obtained:

$$\begin{array}{cc}\hfill {\mathrm{\Delta }}_{k-1}+{\mathrm{\Delta }}_k& \ge \eta \xi \parallel g_{k-1}{\parallel }^2+\eta \xi \parallel g_{k-1}\parallel \parallel g_k\parallel {\tilde{C}}_k^{\mathrm{inter}}-\eta (E-\xi )\parallel g_{k-1}\parallel \parallel g_k\parallel \hfill \\ & -{\displaystyle \frac{L{\eta }^2}{2}}{E}^2\left(\parallel g_{k-1}{\parallel }^2+\parallel g_k{\parallel }^2+2\parallel g_{k-1}\parallel \parallel g_k\parallel \right).\hfill \end{array}$$

Grouping the terms, the required estimate is obtained (33).  □

• Interpretation of the obtained estimate

Inequality (33) reveals the structure of the influence of interstep consistency on the dynamics of optimization. Let us consider separately the corresponding terms of inequality (33):

• Terms that depend only on the gradient norms:

  $$\left(\eta \xi -{\displaystyle \frac{L{\eta }^2}{2}}{E}^2\right)\parallel g_{k-1}{\parallel }^2-{\displaystyle \frac{L{\eta }^2}{2}}{E}^2{\parallel g_k\parallel }^2.$$
  The first term reflects the potential contribution of the previous step, which can be positive for a sufficiently small $\eta$; the second term is always negative and is determined by the upper boundary of the norm of the preconditioned direction at the current step.
• Contribution of interstep consistency:

  $$+\eta \xi \parallel g_{k-1}\parallel \parallel g_k\parallel {\tilde{C}}_k^{\mathrm{inter}}.$$
  This term is positive with ${\tilde{C}}_k^{\mathrm{inter}}>0$ and it becomes bigger as it approaches the gradient directions in neighboring iterations. Multiplier $\xi$ emphasizes that the effectiveness of consistency directly depends on the minimum eigenvalue of the preconditioner; a well-conditioned preconditioner enhances the positive effect.
• Negative cross term:

  $$-\left(\eta (E-\xi )+L{\eta }^2{E}^2\right)\parallel g_{k-1}\parallel \parallel g_k\parallel .$$
  This term is due to the spread of the preconditioner’s eigenvalues (the first term) and the quadratic approximation (the second term). The more poorly $Q_k$ is conditioned (the more $E-\xi$), the higher the negative contribution of this term.

Thus, the total loss reduction in two steps is determined by the balance between the gains from gradient consistency and losses due to poor conditioning of the preconditioner. Well-designed optimizers (for example, self-scaled methods) tend to reduce $E-\xi$, getting $Q_k$ closer to the scalar matrix; this enhances the role of consistency and weakens negative terms. In the special case of a scalar preconditioner, the estimate (33) agrees with the results of [18].

5.3. Gradient and Quasi-Newton Methods

Consider the problem of minimizing a smooth loss function

$$L\left(\theta \right)={\displaystyle \sum _{i=1}^n}{L}_i\left(\theta \right),g\left(\theta \right)=\nabla L\left(\theta \right).$$

(38)

First-order methods define an iterative process as follows:

$${\theta }_{k+1}={\theta }_k-{\eta }_{k}g\left({\theta }_k\right),$$

(39)

where ${\eta }_k>0$ is the learning rate, which, in the general case, can change from iteration to iteration or adapt during the learning process (for example, in the Adam and RMSProp methods). These methods use only information about the gradient and do not consider the local curvature of the loss function. If the problem is poorly conditioned (a wide range of Hessian eigenvalues ${\nabla }^{2}L\left(\theta \right)$), this leads to slower convergence and the appearance of oscillations along directions with high curvature. The second-order methods form an iterative process set as follows:

$${\theta }_{k+1}={\theta }_k-{\eta }_{k}H{\left({\theta }_k\right)}^{-1}g\left({\theta }_k\right),$$

(40)

where $H\left({\theta }_k\right)={\nabla }^{2}L\left({\theta }_k\right)$ is the Hessian of the loss function. For a quadratic local approximation of the loss function in one iteration (at ${\eta }_k=1$) the step is achieved by the exact minimum of this approximation. However, explicit calculation and inversion of the Hessian are computationally expensive in high-dimensional problems. Quasi-Newton methods accelerate gradient optimization by approximating the curvature of the loss function without explicitly calculating the Hessian. Instead of an exact matrix of second derivatives ${\nabla }^2\mathcal{L}\left(\theta \right)$ a $B_k$ sequence of matrices is constructed, approximating $H\left({\theta }_k\right)$ (Hessian itself), or $H{\left({\theta }_k\right)}^{-1}$—approximations of the inverse Hessian (as, for example, in the BFGS method, where the $B_k\approx H{\left({\theta }_k\right)}^{-1}$). The matrix $B_k$ is updated recursively at each iteration, using only information about previous gradients and parameter changes, which makes it possible to consider the curvature of the landscape without explicitly reversing the Hessian and calculating the second derivatives. The Broyden set of methods is a general class of quasi-Newton algorithms in which a new approximation $H{\left({\theta }_{k+1}\right)}^{-1}$ is obtained as a minimal change from the previous one $H{\left({\theta }_k\right)}^{-1}$, satisfying the secant condition from [19]. The obtained inverse Hessian approximation is applied as a precondition for the gradient, forming the direction of the step in the iterative process (40).

In the self-scaled modifications of the Broyden methods [17], an additional positive scaling of the inverse Hessian approximation is introduced. This scaling automatically corrects the overall level of curvature, reducing the impact of poor task conditioning. From a practical point of view, the self-scaled correction reconciles the accumulated quasi-Newton information with the current gradient changes, improving the stability of the search direction and the rate of convergence in problems with a significantly anisotropic landscape of the loss function.

In the context of the estimation, this leads to the following factors:

• Growth of the A-coefficient;
• An increase in the permissible range of $\eta$, at which $B>0$.

These theoretical conclusions are consistent with experimental observations, where self-scaled methods demonstrate faster and more stable convergence on stiff problems.

6. Examples of Computational Experiments

The solution of the same two-phase flow problem is now considered with a progressively more complex mathematical formulation. This approach makes it possible to trace how changes in the nonlinearity of the flow law and the stiffness of the system (phase compressibility) affect the dynamics of the optimization process and the evolution of the gradient consistency metric. Three levels of complexity of the two-phase flow model are considered:

• The first model is a linear formulation without considering the compressibility of the phases and with a constant viscosity for all phases, including oil (phase $\alpha =o$), as in the classical Buckley–Leverett model (BL model);
• The second model, which considers the compressibility of the phases;
• The third model is a completely nonlinear model that includes both the equations of compressibility of the phases and the dependence of the viscosity of the displaced phase $\alpha =o$ on the modulus of the pressure gradient.

This sequentially more complex formulation makes it possible to trace the influence of the system’s nonlinearity and stiffness on the optimization dynamics and the gradient consistency.

All numerical experiments assume the application of a fully connected neural network with eight hidden layers of 20 neurons each and a tanh activation function. The network approximates pressure fields $\widehat{p}(x,t)$ and water saturation ${\widehat{S}}_w(x,t)$. Table A1 presents the parameters and their values used in the experiments.

The calculation options in the three formulations described above were compared between two neural network training strategies: (i) 1000 iterations of Adam followed by 200 iterations of L-BFGS; (ii) 400 iterations of Adam followed by 200 iterations of self-scaled Broyden.

This scheme reflects the common practice of first performing a first-order pre-training step, followed by refinement of the solution using quasi-Newton methods. For each case, we analyzed the dynamics of the loss function, intrastep gradient consistency ${\tilde{C}}_k^{\mathrm{intra}}$, interstep consistency $C_k^{\mathrm{inter}}$, the RMSE of the system of equations solution, and the stability of results across multiple runs.

The proposed analysis is complementary to existing PINN stabilization strategies for handling sharp fronts and discontinuities. A recent review [20] classifies such approaches into three main categories: (i) physics-based modifications, including artificial viscosity and entropy constraints; (ii) loss and training adaptations, such as adaptive weighting, causal loss formulations, and residual-based sampling; and (iii) architectural extensions, including domain decomposition methods (e.g., XPINNs), feature-enhanced networks, and kernel-based architectures such as KAN.

However, in our numerical experiments, the application of Fourier feature embeddings, causal loss formulations, adaptive artificial viscosity, residual-based sampling, and KAN-type architectures did not lead to a significant improvement in the accuracy of saturation front capturing. This observation is consistent with findings reported in [20], where many stabilization techniques demonstrate good performance for simplified formulations (e.g., the Buckley–Leverett equation), but their effectiveness deteriorates in fully coupled nonlinear systems involving multiple governing equations, where the approximation error increases significantly.

At the same time, we note that the combination of self-scaled quasi-Newton methods with the aforementioned stabilization techniques has not been explored in this study and remains an important direction for future research. In this work, we focus on analyzing the influence of optimization algorithms on training quality and stability.

6.1. First Formulation (Without Considering Compressibility and with Constant Phase Viscosity)

With application of the mathematical BL model, which assumes the constancy of densities and viscosities of the phases, the numerical solution is characterized by the dynamics of the loss function shown in the graphs in Figure 1 on the left (a) and demonstrating the degree of interstep and intrastep consistency ((b) and (c) of Figure 1, accordingly) when implementing the problem of minimizing the loss function using first-order method (39) and quasi-Newton methods (40). The dotted line on the (a) chart shows the change in the loss function during Adam + self-scaled Broyden optimization, and the solid lines show the change during Adam + LBFGS optimization.

Figure 1. The dynamics of the loss function (a), intrastep consistency ${\tilde{C}}_k^{\mathrm{intra}}$ (b), and interstep consistency $C_k^{\mathrm{inter}}$ (c) for the formulation without considering compressibility and with constant viscosities of the phases.

The (a) part of Figure 2 shows a comparison of saturation plots using the self-scaled Broyden and LBFGS optimization methods in comparison with the BL reference solution. The (b) side of Figure 2 shows the results of the RMSE depending on the grid resolution for linear formulation.

Figure 2. (a) Comparison with the reference solution (dotted black line) of saturation plots with optimization with self-scaled Broyden (solid blue line) and LBFGS (dotted red line). Dependence of the approximation error (RMSE) on the grid resolution for linear formulation (b).

For the BL formulation, the dynamics of the loss function and consistency metrics (Figure 1) demonstrate a characteristic difference between the first-order and quasi-Newton methods. At the Adam learning stage, there is a slow decrease in the loss function and pronounced fluctuations in the interstep consistency of $C_k^{\mathrm{inter}}$, which indicates an unstable optimization trajectory and alternating dominance of various loss components. The intrastep consistency ${\tilde{C}}_k^{\mathrm{intra}}$ quickly assumes values close to −1, which indicates a strong conflict between the gradient directions of individual components.

After switching to quasi-Newton methods, the dynamics of the optimization process stabilizes; the interstep consistency increases and remains close to 1, which reflects the alignment of the parameter update trajectory. The intrastep consistency remains negative, but its modulus decreases (values of the order of −0.8), which indicates a decrease in the degree of “conflict” between the various components (gradients) of the loss function.

The self-scaled modification demonstrates the smallest variation in $C_k^{\mathrm{inter}}$ and provides faster and deeper reduction of the loss function (by almost an order of magnitude compared to the classic L-BFGS approach), which confirms the effectiveness of automatic scaling in the presence of competing loss components.

To clarify, in order to analyze the convergence over the space–time grid, the problem solution was restarted at least five times for each resolution, after which RMSE boxplot graphs were constructed (Figure 1). The solution obtained by the IMPES method with a spatial resolution of 100 cells and 3000 time steps was used as a reference. It is shown that the self-scaled modification demonstrates a faster decrease in error with increasing resolution and a significantly smaller spread of results, which indicates increased learning stability. The spatial distribution of the approximation error and PDE residuals are presented in Figure A2. The RMSE is largest near the saturation front while the residuals remain close to zero elsewhere, confirming that the PINN solution satisfies the governing equations with good fidelity.

6.2. Second Formulation (Considering Compressibility and with Constant Phase Viscosity)

Figure 3 shows the dynamics of changes in the loss function considering the compressibility of phases (according to the model of a weakly compressible medium). The dynamics of consistency metrics differ from the linear formulation. The dotted line on the (a) chart shows the change in the loss function during Adam + self-scaled Broyden optimization, while the solid lines show the change during Adam + LBFGS optimization. As seen in Figure 4, which presents pressure plots (panel (a)) and graphs showing the progression of the saturation front (panel (b)), the flow hydrodynamics change significantly when accounting for the pressure dependence of phase densities. The RMSE representation (panel (c)) also varies depending on the grid resolution in the linear formulation.

Figure 3. The dynamics of loss function (a), intrastep consistency ${\tilde{C}}_k^{\mathrm{intra}}$ (b) and interstep consistency $C_k^{\mathrm{inter}}$ (c) for the formulation considering compressibility and with constant viscosities of the phases.

Figure 4. Comparison with the reference solution (dotted black line) of pressure diagrams (a) and saturation (b) when optimized with self-scaled Broyden (solid blue line) and LBFGS (dotted red line). Graph (c) shows the dependence of the approximation error (RMSE) on the grid resolution for linear formulation, considering the compressibility of the phases.

The solution of the optimization problem becomes more structurally consistent: when training with the Adam method, the intrastep consistency ${\tilde{C}}_k^{\mathrm{intra}}$ remains closer to −1, however, the spread of values almost disappears, indicating that the “conflict” has been stabilized between the loss components. At the same time, the interstep consistency of $C_k^{\mathrm{inter}}$ remains close to 1 throughout the training, reflecting a steady trajectory of parameter updates.

The transition to quasi-Newton methods leads to a moderate improvement in the intrastep consistency of ${\tilde{C}}_k^{\mathrm{intra}}$, but is accompanied by a slight decrease in the interstep consistency of $C_k^{\mathrm{inter}}$ compared to Adam. The self-scaled modification demonstrates a more pronounced improvement in intrastep consistency and less degradation of interstep consistency (values stabilize around 0.8), providing the most balanced dynamics.

The behavior of the loss function qualitatively repeats the BL formulation; the self-scaled strategy provides the deepest reduction in loss with stable convergence.

6.3. The Third Formulation (Considering the Compressibility of the Phases and the Nonlinear Change in the Viscosity of the Displaced Phase)

In the fully nonlinear formulation, where both the phase densities and the viscosity of the oil phase ($\alpha =o$) vary with pressure, the characteristics of the numerical solution are illustrated in Figure 5 and Figure 6. These figures clearly show an increase in effects associated with the stiffness of the system.

Figure 5. The dynamics of the loss function (a), intrastep consistency ${\tilde{C}}_k^{\mathrm{intra}}$ (b) and interstep consistency $C_k^{\mathrm{inter}}$ (c) for the formulation considering compressibility and with a nonlinear law of viscosity change. The dotted line on the left graph is the change in the loss function during optimization with Adam + self-scaled Broyden. The solid lines are for Adam + LBFGS.

Figure 6. Comparison with the reference solution (dotted black line) of pressure (a) and saturation (b) plots when optimized with self-scaled Broyden (solid blue line) and LBFGS (dotted red line). The dependence of the approximation error (RMSE) on the resolution of the grid for formulation, taking into account the compressibility of phases and the nonlinear law of viscosity variation (c).

Intrastep consistency in Adam training is again characterized by values close to the value of $-1$, which indicates an increase in the multidirectional influence of the components of the loss function. In contrast to the previous case (considering only compressibility), there remains a noticeable variability in intrastep consistency function ${\tilde{C}}_k^{\mathrm{intra}}$, reflecting the increased sensitivity of the iterative process to local changes in pressure and saturation. At the same time, the interstep consistency of $C_k^{\mathrm{inter}}$ with Adam shows more pronounced fluctuations compared to previous formulations, which indicates the instability of the optimization trajectory in conditions of strong nonlinearity.

Quasi-Newton methods partially compensate for this effect; the ${\tilde{C}}_k^{\mathrm{intra}}$ function is characterized in terms of negative values somewhat closer to the $-1$ level, and $C_k^{\mathrm{inter}}$ on the contrary, stabilizes at a higher level, although periodic drawdowns associated with local rearrangements of the approximation process are occurred.

The self-scaled modification ensures the most stable behavior of control functions: intra–step consistency improves more significantly, and interstep consistency remains at the level of 0.8–0.9 with a smaller spread compared to the standard L-BFGS variant.

The dynamics of the loss function retains the qualitative structure observed in previous experiments; the self-scaled strategy provides the deepest reduction in loss. At the same time, the absolute approximation error increases in comparison with linear cases, which is naturally caused by the increased nonlinearity of the model. Thus, a stable pattern is observed in all three formulations, considering the curvature of the loss function increases the intrastep consistency of gradients and stabilizes the interstep dynamics, which directly affects the speed and stability of the loss function reduction. The self-scaled modification further improves the balance between loss components, providing more stable and predictable optimization, especially in nonlinear tasks.

6.4. Computational Cost Considerations

A preliminary assessment of the computational cost is provided below, while a systematic comparison with classical solvers across a wider range of problem sizes is deferred to future work. In the present experiments, the complete training schedule (1000 Adam iterations followed by 200 BFGS iterations) requires approximately 10 min on a single GPU, while the reduced schedule (400 Adam + 200 Broyden iterations) completes in about 6 min. We acknowledge that this is longer than a single run of the IMPES reference solver on the same spatiotemporal grid.

The practical advantage of the PINN approach, however, becomes apparent when pre-trained models are reused. As demonstrated in recent work [21], a pre-trained PINN can serve as a computational cache for related problem settings. When training starts from a pre-trained state and an early-stopping criterion (e.g., five consecutive iterations without loss improvement) is applied, the fine-tuning time drops to approximately 30 s, which is comparable to or even faster than a conventional IMPES solve. Moreover, unlike IMPES, the PINN solution is less sensitive to grid resolution.

7. Conclusions

The paper examines the role of gradient consistency in PINN training for two-phase flow problems in porous media under various task formulations. The proposed interpretation of intrastep and interstep consistency metrics made it possible to link the geometry of the gradient structure of the loss function with the dynamics of optimization and learning stability.

Theoretical analysis has shown that considering the curvature of the loss function through conditional steps reduces the “conflict” between the loss components and stabilizes the trajectory of parameter updates. Numerical experiments for linear, compressible, and fully nonlinear formulations have confirmed this hypothesis; quasi-Newton methods demonstrate higher intrastep consistency, more stable interstep dynamics, and a deeper reduction in the loss function compared to Adam. Self-scaled modifications provide additional improvements by automatically scaling the gradient components.

It is shown that, as the nonlinearity of the model increases, the benefits of methods that account for curvature become more pronounced. This is particularly important for applied oil and gas modeling problems, where stiffness and physical realism—including the nonlinearity of phase properties—are typical features of the mathematical formulation. These factors significantly affect the accuracy of numerical solutions in representing the real characteristics of the development process.

It is worth noting that using the PIKAN architecture in the tasks considered did not lead to a noticeable improvement in approximation accuracy. This suggests that under conditions of strong “competition” (multidirectionality) among the components of the loss function, the key factor for solution quality is not the network architecture itself, but rather the choice and configuration of the optimization algorithm.

As a direction for further research, it seems promising to study methods based on the Newton–Krylov approaches, which make it possible to more fully consider the Hessian structure when solving highly nonlinear problems. An analysis of their impact on consistency metrics and the stability of PINN learning can provide additional insight into the role of high-precision quasi-Newtonian and Newtonian schemes in flow problems with expressed nonlinearity.