A Neural Network Constitutive Model and Automatic Stiffness Evaluation for Multiscale Finite Elements

Abstract

A neural network model for a constitutive law in nonlinear structures is proposed in this paper. The artificial neural network (ANN) model is constructed based on a data set of responses from representative volume elements, which was calculated by finite elements and using an open scientific software machine learning platform. The tangential stiffness matrix, which can be used within a multiscale finite element analysis, is calculated via the method of automatic differentiation. Two types of constitutive neural networks are proposed. The first approach involves training a residual model with three respective surrogate models for the stress components, ensuring less computing cost. The second approach considers a separate ANN for each stress component, ensuring a high rate of convergence. The numerical results are compared with the given data set as well as with the results obtained after applying a polynomial regression. The loss function, without and including the Sobolev metrics, is considered. In addition, a physics-informed constitutive neural model, which enforces hyperelasticity principles, is also analyzed. The choice of the hyperparameters is discussed.

1. Introduction

Multiscale modeling is a simulation technique that describes a behavior at a given length scale based on the physics at a finer scale, which in turn is considered to be better known or easy to model. Structures made of composite materials or materials with microstructure constitute typical candidates for multiscale modeling. In fact, it is impossible to model a whole structure by taking into consideration all constituents and their possible nonlinear interactions within the model. Therefore, a compromise must be considered wherein a reduction in precision and an increase in uncertainty related to multiscale modeling should be accepted in order to avoid the complexity of a full-scale model. In engineering, homogenization is a representative example of applied multiscale modeling [1]. In linear problems, homogenization is usually based on analytical expressions that give the homogenized properties for a given microstructure or composite material [2,3]. This approach is powerful, despite certain theoretical difficulties for the calculation of homogenized properties. Numerical homogenization allows for the extension of classical analytical homogenization in order to describe more complicated problems. Homogenized properties are load dependent or even path dependent; therefore, analytical solutions are either difficult to calculate or do not exist. A path-independent problem is considered here as a first example.

ANNs have been applied in different areas of science and engineering, where data sets are available for training and testing. In particular, ANNs are employed in elastoplastic and contact problems in mechanics by using a minimization of energy. The Hopfield and Tank neural networks have been proposed by Kortesis and Panagiotopoulos [4] and Avdelas et al. [5]. The feedforward NNs trained by the backpropagation algorithm have been used for an approximation of several problems in mechanics based on examples (supervised learning). The buckling loads in nonlinear problems for elastic plates have been calculated by neural networks in Muradova and Stavroulakis [6]. Inverse and parameter identification problems in mechanics have been solved by using backpropagation neural networks in Stavroulakis [7]. A recent review of the classical usage of neural networks within computational mechanics has been published by Yagawa and Oishi [8].

Several neural network-based constitutive models have been proposed in recent works. The main directions are described in the review article by Dornheim et al. [9].

The work described here belongs to the more general topic of CANN (constitutive ANN). In fact, from input–output relations, an ANN is able to learn and replace a mathematically consistent constitutive material model. Adding additional information, like thermomechanical restrictions, enhances the effectiveness and allows for training with less data. Further information can be found in, among others, references [9,10,11].

Here, an efficient way to store the constitutive relation based on a database of responses from a representative volume element (RVE) is proposed through the construction of a neural network constitutive model. Subsequently, the usage of this model within a nonlinear, upper-level finite element model is straightforward.

The Sobolev metrics (e.g., [12,13]) is also tested in the minimization problem. The proposed neural model allows a prediction of the material response from the strain effects based on data set $\{{\mathrm{\epsilon }}_i,{\mathrm{\sigma }}_i,{(\partial \sigma /\partial \epsilon )}_i,i=1,2,\dots ,N\}$.

Two types of constitutive neural networks are proposed. The first approach involves the training of a residual model with three respective surrogate models for stress components. The second approach considers a separate ANN for each stress component. The first neural model requires less computational cost, since the training of only one residual model is required. However, the second approach is preferred for the fitting of the data set of the tangent stiffness tensor (partial derivatives of the stress tensor) in the loss function. In this case, the desired accuracy can be reached with a smaller number of epochs. The applicability of the proposed method has a large range, e.g., in nonlinear structures and in hyperelastic materials.

For the fitting of the backpropagation model, data from the representative volume element calculated using finite elements have been taken. Thus, the target and test data have been collected from the numerical experiments. Two splits of the data set from the responses of representative volume element are chosen. First, the data are split into training ($50\%$) and validation/test data ($50\%$) in Examples 1, 2, and 4, and then, the data samples are split into $80\%$ training and $20\%$ ($10\%$ for validation and $10\%$ for testing/prediction) for the data in Example 3. The computed loss, validation, and prediction errors are presented in tables. As numerical experiments have shown, there is no significant difference between these two splits.

The results are also compared with a linear polynomial regression method of the third degree. The results of the comparison are presented in figures. Although the computational cost of implementation of the linear regression model is less than that required for the training of the neural network, there are advantages to using ANNs. Namely, the material stiffness matrix, i.e., the partial derivatives of the stress function with respect to the components of the strain tensor, are more accurately approximated because of deep learning performance based on the tanh activation (basis) function of harmonic analysis. Fitting the stress data and their derivative data involved in the loss function provides stability and good accuracy.

The data should be carefully arranged in order to prevent overfitting. The developed programming code provides conditions to overcome this phenomenon by using Python’s tools. The data are not shuffled before the training process in order to keep them smoothly increasing or decreasing. Numerous numerical experiments have shown that this guarantees the stability of the training process, allows for an optimal updating of weights and biases in the backpropagation, and increases the rate of convergence to the original (target) data set. Moreover, it properly captures the nonlinear behavior of the considered materials.

The data set, derived from the nonlinear polynomial constitutive model for hyperelastic materials, has also been trained in the constructed neural model. Here, the loss function includes constraints on the second derivative of the strain energy function $(\partial \sigma /\partial \epsilon ={\partial }^2\mathrm{\Psi }/\partial {\epsilon }^2\ge 0)$, which is namely a convexity that enforces realistic physical laws and provides material stability conditions during deformation. The numerical results obtained from the constructed neural network are compared with the exact values at the predicted points.

An ANN together with the method of automatic differential of Tensorflow-scientific library 2.14.0 in Python 3.11 programming [14] is used here. The neural model includes automatic differentiation [15] to calculate the partial derivatives of the ANN output. Automatic differentiation is performed through the chain rule applied to the neural stress models. In the residual model, except for fitting the stress tensor data set, fitting the given tangent stiffness matrix, obtained from the representative volume element, is also provided.

2. Background of the Constitutive Metamodel Based on Responses of Representative Volume Elements

Numerical homogenization uses an RVE with several materials (composite) or microstructure and calculates homogenized properties, which in turn are used for the study of a whole structure with a continuous model [16,17]. In nonlinear behavior, this step must be repeated for each level of stress response. This approach has been proposed to facilitate multiscale modeling and avoid the more accurate but expensive FEM2 [18]. FEM2 requires the solution of the detailed model of the RVE for every different combination of loadings and points in the structure that appear during the incremental iterative solution of the homogenized finite element model.

Classical polynomial surrogates as well as a neural network and other ones have been proposed and tested in [1,19,20,21,22,23,24,25].

A feedforward neural network has the ability to correlate vector inputs with vector outputs provided that the parameters involved have been suitably calculated. This step is accomplished with the help of data (examples) within a step called training. The topology of the neural network (number of layers and neurons) as well as the activation functions (basis functions) at nodes in correlation with the number of examples define the ability of the network to approximate well an unknown target function. Trial and error or optimal design principles can be used toward a satisfactory result [26].

A modern approach for introducing physics into the neural network metamodel uses differential equations of mechanics in combination with the automatic differentiation of output of the neural network, in order to train the network, without using specific input–output sample examples. This is the so-called physics-informed neural networks approach ([14,25,27,28,29,30], etc.). Within the homogenization metamodel, PINN could provide thermomechanically consistent approximations that are more stable than those of the blind neural network.

Using ANNs is able to simulate classical mechanical problems if sufficient samples of input–output data are provided for training. The concept is shown schematically in Figure 1 and Figure 2 ([1]) for the stress–strain relation of the RVE. The RVE for every possible complexity is solved for different combinations of stresses, and the corresponding strains are calculated and used for training. More refined approaches introduce additional physical restrictions on the corresponding constitutive metamodel so the thermomechanical principles are not violated [25]. Furthermore, loading sequences can be used in a path-dependent problem to train a corresponding neural network that will provide the response based on previous loading sequences at a given point.

The trained neural network can be integrated into a finite element program and replaces the required constitutive response, as shown in Figure 1 (see, among others, [1,2,31,32]). The material constitutive law is based on experimental or RVE data. It is based either on an established mechanical material law or a metamodel (interpolation). This approach is close to the classical usage of constitutive relations, and it can be integrated with existing commercial finite element packages [33,34,35,36].

The results used here correspond to a masonry structure with different stones and mortar materials subjected to various stress–strain levels. The Representative Volume Element has been discretized by the finite element method. The results have been collected in a database that are used for the training of the neural network. A MATLAB v.2021a implementation for the multiscale analysis of two-dimensional problems as well as the database is available as a Supplementary Material to reference [1]. The masonry RVE has been used for the creation of the database.

The plastic deformations calculated under a given strain loading are shown in Figure 3 ([1]). The method has been described in reference [37], where the reader can find more details.

3. Feedforward and Backpropagation in the Neural Model with a Computation of Tangential Stiffness Matrix

The computational approach initially proposed and presented in [1] uses two neural network metamodels, one for the approximation of the constitutive relation and the second for the approximation of its first derivative, that is, the calculation of the tangential stiffness matrix. Here, following ideas from PINN’s community, we construct a neural network based on scientific software Tensorflow v.2.14.0, which provides automatic differentiation for the calculation of the partial derivatives and also enforces material consistency.

Two types of neural networks are introduced and trained. The first neural network consists of one residual model and three surrogate models, which are intended to compute the components of the stress tensor ${\sigma }_{xx},$${\sigma }_{yy}$ and ${\sigma }_{xy}({\sigma }_{yx}={\sigma }_{xy}).$ Namely, there are three NNs with three inputs, strain components ${\epsilon }_{xx},$${\epsilon }_{yy}$ and ${\epsilon }_{xy}({\epsilon }_{yx}={\epsilon }_{xy})$ and one output, which is a component of the stress tensor; that is, the first NN has output ${\sigma }_{xx}^{NN}$, the second one has output ${\sigma }_{yy}^{NN}$ and the third has output ${\sigma }_{xy}^{NN}$. The residuals of the outputs are treated simultaneously. The architecture of the proposed bundle of NNs is presented in Figure 4. In the second approach, there exist three separate neural networks for each component of the stress tensor (Figure 5). The loss error function for each output is minimized separately. This can be accomplished because the components of the stress tensor can be computed independently.

In the feedforward of the neural network, the output vector for each layer is calculated as follows

$$\begin{array}{ccc}\hfill z_0& =& \hfill {({\mathrm{\epsilon }}_{11},{\mathrm{\epsilon }}_{22},{\mathrm{\epsilon }}_{12})}^T,\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill z_k& =& f(W_k^T{z}_{k-1}+b_k),k=1,2,\dots ,N_L-1,\hfill \\ \hfill {\sigma }_{ij}^{NN}=z_{N_L}& =& W_{N_L}^T{z}_{N_L-1}+b_{N_L},i,j=1,2,\hfill \end{array}$$

(2)

where ${\mathrm{\epsilon }}_{11}={\mathrm{\epsilon }}_{xx},{\mathrm{\epsilon }}_{12}={\mathrm{\epsilon }}_{21}={\mathrm{\epsilon }}_{xy}={\mathrm{\epsilon }}_{yx},{\mathrm{\epsilon }}_{22}={\mathrm{\epsilon }}_{yy},$ ${\mathrm{\sigma }}_{11}={\mathrm{\sigma }}_{xx},{\mathrm{\sigma }}_{12}={\mathrm{\sigma }}_{21}={\mathrm{\sigma }}_{xy}={\mathrm{\sigma }}_{yx},{\mathrm{\sigma }}_{22}={\mathrm{\sigma }}_{yy},$ and $W_k^T$ is the matrix of the dimension $(n_{k-1}\times n_k)$, containing the weights between $k-1$ and k hidden layers with $n_{k-1}$ and $n_k$ neurons, respectively. $W_{N_L}^T$ is the matrix that contains the weights between the last hidden layer and the output of the neural network. The $b_k$ denote biases. There is no activation function f for the output layer in the proposed Keras module, but the tf.keras.layers.Activation function can be used to apply a custom activation function to the output if needed, for example, tf.keras.layers.Activation(tf.nn.relu).

The residual parts are computed as

$$MSE=MS{E}_{\sigma }+MS{E}_S,$$

(3)

where $MS{E}_{\sigma }$ is the loss error function while training the data for the strain and stress tensor and $MS{E}_S$ is the Sobolev loss error function [12]. The residual $MS{E}_{\sigma }$ is written as

$$MS{E}_{\sigma }=|{\mathrm{\sigma }}_{11}^{NN}-{\mathrm{\sigma }}_{11}^{\mathrm{data}}|+|{\mathrm{\sigma }}_{22}^{NN}-{\mathrm{\sigma }}_{22}^{\mathrm{data}}|+|{\mathrm{\sigma }}_{12}^{NN}-{\mathrm{\sigma }}_{12}^{\mathrm{data}}|$$

(4)

where ${\mathrm{\sigma }}_{ij}^{NN}={\mathrm{\sigma }}_{ij}^{NN}\left({\mathrm{\epsilon }}_i\right),i,=1,2,$$({\mathrm{\sigma }}_{12}={\mathrm{\sigma }}_{21})$ are the output of the neural network, namely, the values of the stress tensor which are computed by the composed constitutive neural network of the variable data set $\mathrm{\epsilon }=({\mathrm{\epsilon }}_{11}, {\mathrm{\epsilon }}_{22}, {\mathrm{\epsilon }}_{12})$, and N is the number of the values used for fitting the given data set for the stress tensor (samples). If the residuals for the surrogate models, ${\mathrm{\sigma }}_{xx}^{NN}\left(\mathrm{\epsilon }\right),$${\mathrm{\sigma }}_{yy}^{NN}\left(\mathrm{\epsilon }\right),$${\mathrm{\sigma }}_{xy}^{NN}\left(\mathrm{\epsilon }\right)$ are treated separately, i.e., there are three neural networks (Figure 5), then instead of (4), the following formulas yield

$$\begin{array}{c}\hfill MS{E}_{{\sigma }_{ij}}=|{\mathrm{\sigma }}_{ij}^{NN}-{\mathrm{\sigma }}_{ij}^{\mathrm{data}}|,\end{array}$$

(5)

and in Figure 5, we have three $MSE{\sigma }_{11}$, $MSE{\sigma }_{22}$ and $MSE{\sigma }_{12}$ for each surrogate neural network, respectively.

In the case of the Sobolev metrics in the neural network [12], the derivative information can be easily incorporated into the neural network model training process for $\mathrm{\sigma }$ by making derivatives of the neural network match the ones given by $\mathrm{\sigma }$.

If we have access to the partial derivatives of the stress tensor $\mathrm{\sigma }$ with respect to the input $\mathrm{\epsilon }$ to K order

$$\{\mathrm{\epsilon },\mathrm{\sigma }\left(\mathrm{\epsilon }\right),D_{\mathrm{\epsilon }}^1\left(\mathrm{\sigma }\left(\mathrm{\epsilon }\right)\right),\dots ,D_{\mathrm{\epsilon }}^K\left(\mathrm{\sigma }\left(\mathrm{\epsilon }\right)\right)\},$$

where $D_{\mathrm{\epsilon }}^k\left(\mathrm{\sigma }\left(\mathrm{\epsilon }\right)\right)=\left({\partial }^k\mathrm{\sigma }/\partial {\mathrm{\epsilon }}^k\right),$$k=1,2,\dots ,K.$ Then, the Sobolev error loss function $MS{E}_S$ can be included as well in (4), i.e.,

$$MSE=\sum _{ij}\left[|{\mathrm{\sigma }}_{ij}^{NN}-{\mathrm{\sigma }}_{ij}^{\mathrm{data}}|++\sum _{k=1}^K|D_{\mathrm{\epsilon }}^k\left({\mathrm{\sigma }}_{ij}^{NN}\right)-D_{\mathrm{\epsilon }}^k\left({\mathrm{\sigma }}_{ij}^{\mathrm{data}}\right)|\right],$$

(6)

and for the neural architecture Figure 5 in virtue of (5),

$$\begin{array}{c}\hfill MS{E}_{{\sigma }_{ij}}=|{\mathrm{\sigma }}_{ij}^{NN}-{\mathrm{\sigma }}_{ij}^{\mathrm{data}}|+\sum _{k=1}^K|D_{\mathrm{\epsilon }}^k\left({\mathrm{\sigma }}_{ij}^{NN}\left(\mathrm{\epsilon }\right)\right)-D_{\mathrm{\epsilon }}^k\left({\mathrm{\sigma }}_{ij}^{\mathrm{data}}\right)|.\end{array}$$

(7)

The numerical experiments have shown that the results are improved after using the Sobolev function. Since the process of numerical differentiation is not stable, to obtain good accuracy for computations of derivatives, a lot of training iterations are needed. With small perturbations (errors) in the output of the neural network (components of the stress tensor), one can obtain quite a large error for the derivatives of the components. The Sobolev training can overcome these disadvantages by matching the derivatives of the NN output. Then, the rate of convergence is higher, and a good accuracy can be reached quickly.

In backpropagation, the chain rule from calculus is used to compute the gradients. The output of an NN is considered as a composite function. Therefore, its derivative is equal to the product of the derivatives of its components. In the context of backpropagation, this means that the gradients of the loss with respect to the weights of a given layer depend on the gradients of the loss with respect to the weights of the subsequent layer. Once the gradients have been computed, they are used to update the weights of the network using a suitable optimization algorithm (e.g., gradient descent). The optimization method adjusts the weights in the direction that minimizes the loss, allowing the network to improve its performance.

The target solution is presented through expansion from basis (sigmoid, hyperhyperbolic tangent, etc.) functions. The network is trained iteratively on a set of input–output ($\mathrm{\epsilon }$, $\mathrm{\sigma }$) data, the weights are updated, and the process is repeated until the network has learned to produce accurate outputs for a wide range of inputs. One of the advantages of backpropagation is that it allows neural networks to learn from their mistakes and improve their performance over time. This is particularly useful in applications where the relationship between the inputs and outputs is nonlinear (e.g., in macrostructures). However, the procedure of backpropagation can be computationally expensive, particularly for large networks and when the network becomes too complex and starts to fit the training data too closely, leading to poor performance on new, unseen data.

The automatic differentiation (AD) method, a built-in Tensorflow 2.14.0 library, computes exact gradients using the chain rule, resulting in more accurate derivatives in comparison with finite differences. In backpropagation, the gradients are computed with respect to the weights and the biases. The derivatives of the output function are computed using AD with the help of the chain rule with respect to the input variables. However, since the output of the neural network is an approximation of the exact solution, i.e., the output data have some perturbation with the AD, this perturbation (error) will not decrease and more likely increases. The results of computing the derivatives by the AD can be compared with numerical differentiation, e.g., FD approximation (forward, backward, and central) or with some other well-known numerical methods for numerical differentiation. For the error of approximation, for example, of the forward differences for the partial derivatives for the $\mathrm{\sigma }$, we have

$$D_{{\epsilon }_{kl}}^1{\sigma }_{ij}={\displaystyle \frac{\Delta {\sigma }_{ij}}{\Delta {\epsilon }_{kl}}}+O(\Delta {\epsilon }_{kl}),i,j=1,2,k,l=1,2.$$

(8)

If the output of the neural network has an error ${\mathbf{\delta }}_{ij}^{NN}={\mathrm{\sigma }}_{ij}^{NN}-{\mathrm{\sigma }}_{ij}^{\mathrm{data}}$, then

$$D_{{\epsilon }_{kl}}^1{\sigma }_{ij}^{NN}-D_{{\epsilon }_{kl}}^1{\sigma }_{ij}={\displaystyle \frac{\Delta {\delta }_{ij}^{NN}}{\Delta {\epsilon }_{kl}}}+O(\Delta {\epsilon }_{kl}),i,j=1,2.$$

(9)

Thus,

$$\left|\right|D_{\mathrm{\epsilon }}^1{\mathrm{\sigma }}^{NN}-D_{\mathrm{\epsilon }}^1\mathrm{\sigma }\left|\right|=O\left({\displaystyle \frac{\left|\right|{\mathit{\delta }}^{NN}{\left|\right|}_{\infty }}{\mathit{ϵ}}}\right)+{O\left(\right||\Delta \mathrm{\epsilon }||}_{\infty }),$$

(10)

where $\left|\right|D_{\mathrm{\epsilon }}^1{\mathrm{\sigma }}^{NN}-D_{\mathrm{\epsilon }}^1\mathrm{\sigma }\left|\right|$ is the Euclidean norm, $\left|\right|{\mathit{\delta }}^{NN}{\left|\right|}_{\infty }={max}_{ij}\left|{\delta }_{ij}^{NN}\right|,$$\mathit{ϵ}={min}_{ij}|\Delta {\mathrm{\epsilon }}_{ij}|,$ ${\left|\right|\Delta \mathbf{\epsilon }\left|\right|}_{\infty }={max}_{ij}|\Delta {\mathrm{\epsilon }}_{ij}|.$ For the backward, the same estimate is true. Analogously, for the central differences, we obtain

$$\left|\right|D_{\mathrm{\epsilon }}^1{\mathrm{\sigma }}^{NN}-D_{\mathrm{\epsilon }}^1\mathrm{\sigma }\left|\right|=O\left({\displaystyle \frac{\left|\right|{\mathit{\delta }}^{NN}{\left|\right|}_{\infty }}{\mathit{ϵ}}}\right)+{O\left(\right||\Delta \mathrm{\epsilon }||}_{\infty }^2).$$

(11)

Hence, one concludes that the error of numerical differentiation increases for a large perturbation $\left|\right|{\mathit{\delta }}^{NN}{\left|\right|}_{\infty }$. One can compare the values of the derivatives computed from the neural model with the use of AD with the approximated derivatives computed by the FD.

According to the results, presented in [38] for a common class of ANNs with one hidden layer, the mean integrated squared error between the estimated network and a target function $\sigma$ is bounded by

$$O\left({\displaystyle \frac{C_{\sigma }^2}{n}}\right)+O\left({\displaystyle \frac{nd}{N}}logN\right),$$

(12)

where n is the number of neurons in the hidden layer, d is the input dimension, N is the number of training observations (samples), and $C_{\sigma }$ is the first absolute moment of the Fourier magnitude distribution of the target function $\sigma$; i.e., $C_{\sigma }$ quantifies the regularity of the function via an integral involving the Fourier transform (see Formula (2) [38]). There are two contributions to this total risk: the approximation error and the estimation error. The approximation error is the distance between the target function and the closest neural network function of the given architecture, and an estimation error refers to the distance between this ideal network function and an estimated network function. The constant $C_{\sigma }$ can be exponentially large in d for sequences of functions $\sigma$ of increasing dimensionality. Networks with many hidden layers can improve accuracy in some cases.

From Formulas (10), (11) and the estimate (12), one concludes that the error of the neural network influences the total estimate of the error of computation of the derivatives of the output function. However, with AD, the computations are more accurate because the derivatives are computed exactly by the chain rule.

Since the tangent stiffness matrix is symmetric, only the components, located in the lower or upper triangular parts of this matrix, are used, which thereby decreases the time of computations. Here in feedforward, the nonlinear tahn activation function is used. It provides better approximation in comparison with the other activation functions. In backpropagation, the weights and biases are usually updated using a stochastic optimizer, such as Stochastic Gradient Descent (SGD) and Adam’s method [39]. The residual network updates the surrogate network weights and biases using the residual obtained from the residual network. Here, we use the Adam optimization algorithm.

4. Computational Procedure

In this section, a computational algorithm for the construction of a neural network constitutive metamodel is presented. The procedure is implemented with the use of automatic differentiation of Tensorflow 2.14.0 and Keras library of Python [40]. To optimize weights and biases, Adam’s optimization algorithm is applied, and the training set is divided into batches. The flowchart summarizing the process of constructing a neural network is presented in Figure 6.

The steps of implementation of the procedure with some methods/functions of the Python programming code are described below. (The programming code is available from the Supplementary Material of the article).

Computational Algorithm

• Import all necessary Python’s libraries:
  import tensorflow as tf, import keras, import matplotlib.pyplot as plt, and import csv.
• Set up the input for neural surrogate networks from the data set of the strain tensor. There are three inputs, $\mathrm{\epsilon }=[{\epsilon }_{xx},{\epsilon }_{yy},{\epsilon }_{xy}]$, for each NN, and each input is a vector of values.
• Set up the output, training and test samples for the neural networks as well as the data set of the stress tensor $\mathrm{\sigma }=[{\sigma }_{xx},{\sigma }_{yy},{\sigma }_{xy}]$ for fitting while minimizing the error loss function.
• Set up a number of training iterations, epochs and batches for the neural networks.
• Read the data set, $\mathrm{\epsilon },$$\mathrm{\sigma }$ and $\partial \mathrm{\sigma }/\partial \mathrm{\epsilon }$ if they are available, from txt/csv files with np.genfromtxt function. A part of the data set is used for training, and the remaining data are used for the test.
• Normalize the $\mathrm{\epsilon }$ and $\mathrm{\sigma }$ if it is necessary with

  $$\widehat{\mathrm{\epsilon }}=a+(b-a){\displaystyle \frac{\mathrm{\epsilon }-\underline{\mathrm{\epsilon }}}{\overline{\mathrm{\epsilon }}-\underline{\mathrm{\epsilon }}}},\widehat{\mathrm{\sigma }}=a+(b-a){\displaystyle \frac{\mathrm{\sigma }-\underline{\mathrm{\sigma }}}{\overline{\mathrm{\sigma }}-\underline{\mathrm{\sigma }}}},$$

  where $[a,b]$ is the new segment, $\underline{\mathrm{\epsilon }}={min}_{ij}{\epsilon }_{ij},$$\overline{\mathrm{\epsilon }}={max}_{ij}{\epsilon }_{ij},$$\underline{\mathrm{\sigma }}={min}_{ij}{\sigma }_{ij}$ and $\overline{\mathrm{\sigma }}={max}_{ij}{\sigma }_{ij},$ and use the chain rule for normalizing and computing derivatives, i.e.,

  $${\displaystyle \frac{\partial \widehat{\mathrm{\sigma }}}{\partial \widehat{\mathrm{\epsilon }}}}={\displaystyle \frac{\partial \widehat{\mathrm{\sigma }}}{\partial \mathrm{\sigma }}}{\displaystyle \frac{\partial \mathrm{\sigma }}{\partial \mathrm{\epsilon }}}{\displaystyle \frac{\partial \mathrm{\epsilon }}{\partial \widehat{\mathrm{\epsilon }}}}={\displaystyle \frac{\overline{\mathrm{\epsilon }}-\underline{\mathrm{\epsilon }}}{\overline{\mathrm{\sigma }}-\underline{\mathrm{\sigma }}}}{\displaystyle \frac{\partial \mathrm{\sigma }}{\partial \mathrm{\epsilon }}}.$$
• Create a class/function object in Python allowing automatic differentiation using tensorflow tf.GradientTape module.
• Set up a number of neurons and layers for the NNs.
• Group layers (input, hidden and output) and neurons into an object with training/inference features for the surrogate net metamodels with dimensions, with three inputs and one output, based on the Keras modules, tf.keras.Input, tf.keras.layers.Dense, tf. keras.models.Model, tf.keras.layers.Input.
• Call the class/function for Automatic Differentiation, which is defined in Step 7.
• Define the input and the output for the NNs using module tf. keras.models.Model for inputs and training items in the list of outputs.
• Create training and test data, including the training variables for the inputs $[{\epsilon }_{xx},{\epsilon }_{yy},{\epsilon }_{xy}]$ and for the output ${\sigma }_{xx},$${\sigma }_{yy},$ and ${\sigma }_{xy}.$ In the case of the Sobolev function, $D_{\mathrm{\epsilon }}^1\left(\mathrm{\sigma }\left(\mathrm{\epsilon }\right)\right),\dots ,D_{\mathrm{\epsilon }}^K\left(\mathrm{\sigma }\left(\mathrm{\epsilon }\right)\right)$ is also included if the corresponding data are available.
• Give a formulation for the residuals (4), (5), (6) or (7) for fitting the NNs to the data set for the output for $[{\mathrm{\sigma }}_{xx}^{NN},$${\mathrm{\sigma }}_{yy}^{NN}$, ${\mathrm{\sigma }}_{xy}^{NN}{]}_i$ and the partial derivatives for them if available.
• Choose an activation function. Here, the tanh function is used.
• Compile the residual neural models using the Keras module keras.models.Model.compile with the help of the built-in Adam optimizer and Mean Square Error modules.
• Train the neural metamodels with using keras.models.Model.fit as the training input and output data (backpropagation). If hyperelastic principles are enforced, include the inequality for the strain energy function,

  $${\displaystyle \frac{\partial \sigma }{\partial \epsilon }}={\displaystyle \frac{{\partial }^2\mathrm{\Psi }}{\partial {\epsilon }^2}}\ge 0$$

  in the loss function and use the module tf.nn.relu(-$d\sigma /de$) in tf. keras.models.Model.
• Stop simulation if overfitting happens (the validation loss starts to increase). In this case, add a penalty to the loss function for large weights. Simplify the NN by reducing the number of layers or neurons. Check the data set, increase data or correct them if necessary. Go back to the previous step.
• Go back to the true values from the normalized output results $\mathrm{\sigma }$ and the derivatives.
• Plot graphs for the model accuracy, model loss and prediction results of the outputs of the NNs with the use of plot and history() functions, which are readily available for use inside Python, and save the obtained data with, e.g., np.savetxt() and the figures with save plt.safefig().

5. Numerical Results

5.1. Example 1

The proposed neural constitutive network with the architecture presented in Figure 5 has been trained with different numbers of epochs, layers and neurons. The set of data for training and testing from the representative volume elements has in total 9261 couples of input–output values, i.e., for $\mathrm{\epsilon }$ and $\mathrm{\sigma }$. For the $\mathrm{\epsilon }$, we have

$$\begin{array}{ccc}\hfill {\epsilon }_{ij}^m& =& {\epsilon }_{ij}^{m-1}+h,h=0.001,\hfill \\ {\epsilon }_{ij}^1\hfill & =& \hfill -0.01,m=2,3,\dots M,M=21.\hfill \end{array}$$

(13)

In this example, neural networks, which consist of four hidden layers with [15, 20, 15, 20] neurons, have been considered. The batch size is 64, and from the data set, $M=168$ values of the stress tensor (84 for the training and 84 for the test validation) are used to construct the neural constitutive models with computations of the partial derivatives for the stress tensor. The batch size, which is an important key in machine learning and deep learning, refers to the number of training samples utilized in one iteration of model training. It can influence the time of training, the efficiency and the model performance. The batch size can be chosen while training the model, e.g., in the Keras library by the module "keras.models.Model.fit(xtrain, ytrain, validationdata= (xtest,ytest), epochs=nepochs, batchsize=64, verbose=2,callbacks=callbacks)".

Here, Formula (5) is used for the loss function; i.e., each neural model is trained separately. The results with 4000 training iterations are shown in Figure 7.

Figure 7 shows a match between the data set and the NN stress components. In

Table 1. The model loss error for different values of the training parameters.

Number of Layers	Neurons	Epochs	Training Samples	MSE$\mathit{\sigma }$	Time (min.)
2	[40,40]	2000	42	$1.4\times {10}^{-5}$	3
2	[40,40]	4000	42	$1.2\times {10}^{-6}$	6
3	[15,30,40]	2000	42	$1.2\times {10}^{-5}$	5
4	[15,20,15,20]	4000	84	$4.7\times {10}^{-6}$	8
4	[15,20,15,20]	8000	84	$1.6\times {10}^{-6}$	15.5

, the error loss function

$$\mathrm{MSE}\sigma =max\{\mathrm{MSE}{\sigma }_{xx},\mathrm{MSE}{\sigma }_{yy},\mathrm{MSE}{\sigma }_{xy}\}$$

is given for different numbers of layers, neurons and epochs.

The time of computations can be decreased by using the three surrogate neural networks with the residual neural model, which includes the residual (4). The loss error function includes all three components of the stress function (Figure 4). This approach is less expensive, but it is effective only for computations of the stress tensor. Figure 8 shows the loss function (4) while training the neural network to calculate the components of the stress tensor. The number of epochs is 4000. The neural network as before has four hidden layers with [15, 20, 15, 20] neurons, respectively. The batch size is 64, and from the data set, we have taken the $M=82$ (42 for the training and 42 for the test validation) values of the stress tensor.

5.2. Example 2

In this example, the data set ($M=9261$) has been trained for obtaining a neural constitutive model with the partial derivatives of the components of the stress tensor obtained via automatic differentiation. In Figure 9, the components of the stress tensor are presented, and in Figure 10, Figure 11 and Figure 12, the partial derivatives obtained from the components are sketched, respectively. The Sobolev error function (6) ($K=1$) is also used to train the neural network with the use of AD for the computation of the partial derivatives. Figure 10, Figure 11 and Figure 12 show that the accuracy is better in the case of using the Sobolev metrics with the same number of training iterations. The neural network, which consists of four hidden layers with [15, 20, 15, 20] neurons with 5000 epochs, has been trained. The batch size is 64 and the data set has 9261 values for the components of the stress tensor (from which 4631 are used for the training and 4630 are used for the test validation). The test data are used for the comparison and validation of the results. In

Table 2. The model loss error for different values of the training parameters.

Number of Layers	Neurons	Epochs	Batch Size	MSE$\mathit{\sigma }$	Time (min.)
2	[15,15]	1000	64	$2.0\times {10}^{-4}$	13.64
2	[15,15]	2000	64	$1.8\times {10}^{-4}$	27.62
2	[40,40]	2000	64	$6.3\times {10}^{-5}$	27.30
2	[40,40]	2000	84	$7.7\times {10}^{-5}$	26.43
3	[15,30,15]	4000	84	$4.7\times {10}^{-5}$	52.48
4	[15,20,15,20]	2000	84	$5.8\times {10}^{-5}$	26.31
4	[30,40,30,40]	2000	84	$3.8\times {10}^{-5}$	26.79
5	[15,20,15,20,15]	2000	84	$5.7\times {10}^{-5}$	26.27
5	[15,20,15,20,15]	4000	84	$3.6\times {10}^{-5}$	52.57

, the results of numerical experiments with different numbers of layers, neurons, and epochs are presented. It is noted that the accuracy of computations increases not only with an increase in the number of epochs but also with an increase in the number of layers and neurons.

The numerical experiments show that with a larger number of epochs, hidden layers, and neurons, the results are more accurate. In the case when derivative data are not available or Sobolev’s error is not included in the process, more training iterations are required in order to obtain good accuracy for the derivatives.

From Figure 10, Figure 11 and Figure 12, we can see the decreasing error after using the Sobolev metrics in the loss function of the training process. Therefore, if the data for a stiffness matrix are available, the calculations of the components of the stress tensor and its derivatives are more accurate.

5.3. Example 3

Here, the neural network is trained for each stress tensor component separately (Figure 5). The data set $M=9261$ is split into $80\%$ for training and $20\%$ ($10\%$ for validation and $10\%$ for test/prediction). The computed maximum of training and validation loss error, mean square error (MSE) and mean absolute error (MAE) with 84 batch size for the cases, without and with the Sobolev’s metrics, are given in

Table 3. The computed loss error, mean square error (MSE) and mean absolute error (MAE) of training of the NN for different hyperparameters with a batch size of 84.

Neurons in Layers	Epochs	Train. Loss	Valid. Loss	MSE$\mathit{\sigma }$	MAE$\mathit{\sigma }$	MSEstiff	MAEstiff	Time (min.)
[50,50]	2000	$1.7\times {10}^{-6}$	$6.8\times {10}^{-7}$	$6.8\times {10}^{-7}$	0.000661	0.0001885	0.006957	18.51
[50,50]	5000	$1.1\times {10}^{-6}$	$3.3\times {10}^{-6}$	$3.4\times {10}^{-6}$	0.001550	0.000164	0.006603	44.89
[30, 40, 30, 40]	2000	$1.1\times {10}^{-6}$	$9.45\times {10}^{-7}$	$9.1\times {10}^{-7}$	0.000600	0.000740	0.013892	21.82
[30, 40, 30, 40]	5000	$2.1\times {10}^{-6}$	$1.5\times {10}^{-6}$	$1.5\times {10}^{-6}$	0.000941	0.089927	0.135217	49.69
[15, 30, 15, 30, 15]	2000	$1.8\times {10}^{-6}$	$1.8\times {10}^{-6}$	$1.8\times {10}^{-6}$	0.001138	0.000648	0.013876	24.63
[15, 30, 15, 30, 15]	5000	$1.4\times {10}^{-6}$	$8.0\times {10}^{-7}$	$7.8\times {10}^{-7}$	0.000674	0.002472	0.026296	60.52

and

Table 4. The computed loss error, mean square error (MSE) and mean absolute error (MAE) of training of the NN with the Sobolev metrics for different hyperparameters with a batch size of 84.

Neurons in Layers	Epochs	Train. Loss	Valid. Loss	MSE$\mathit{\sigma }$	MAE$\mathit{\sigma }$	MSEstiff	MAEstiff	Time (min.)
[50,50]	2000	$4.2\times {10}^{-5}$	$4.2\times {10}^{-5}$	$8.4\times {10}^{-6}$	0.002398	$3.0\times {10}^{-5}$	0.004027	27.04
[50,50]	5000	$4.5\times {10}^{-5}$	$6.3\times {10}^{-6}$	$5.1\times {10}^{-6}$	0.002168	$5.1\times {10}^{-5}$	0.004628	66.73
[30, 40, 30, 40]	2000	$5.2\times {10}^{-5}$	$4.6\times {10}^{-5}$	$4.5\times {10}^{-6}$	0.002791	$2.7\times {10}^{-5}$	0.003114	26.94
[30, 40, 30, 40]	5000	$3.1\times {10}^{-5}$	$5.0\times {10}^{-5}$	$6.4\times {10}^{-6}$	0.001990	$5.8\times {10}^{-6}$	0.004675	66.50
[15, 30, 15, 30, 15]	2000	$4.1\times {10}^{-5}$	$5.1\times {10}^{-5}$	$1.3\times {10}^{-5}$	0.002822	$2.6\times {10}^{-5}$	0.003749	26.53
[15, 30, 15, 30, 15]	5000	$2.3\times {10}^{-5}$	$3.2\times {10}^{-5}$	$2.9\times {10}^{-6}$	0.001506	$2.8\times {10}^{-5}$	0.003678	66.68

, respectively. The above error results are presented for the stress tensor and the stiffness matrix.

As numerical experiments have shown, the hyperparameters have a significant impact on the results. More shallower networks sometimes can perform better for the computation of partial derivatives. From the values of the training loss, validation loss, MSE, and MAE, we can see that in the neural model with the Sobolev metrics, more epochs are needed to reach the desired accuracy for the stress components. The results are also compared with the third-order linear polynomial regression. In Figure 13, Figure 14, Figure 15 and Figure 16, the line fitting for the computed stress components, tangent stiffness and polynomial regression are sketched. Although the polynomial regression requires less computational effort, the ANN provides more accurate approximations. In Figure 13, the residual graphs for the stress components with [50,50] neurons and 5000 epochs are sketched, and in Figure 14, Figure 15 and Figure 16, the residual graphs for the derivatives with [30,40,30,40] neurons and 5000 epochs are illustrated.

5.4. Example 4

In this example, we consider uniaxial tension, which is stretching along a single direction (or axis). It is a common and useful method for the mechanical testing of materials. The experiments have been conducted on hyperelastic materials, using a polynomial model for the nonlinear elastic deformations, like the one used in polymers and rubbers (e.g., [41,42]). The stress–strain relation in tension can be written in a polynomial form for a nonlinear elastic material,

$$\sigma \left(\epsilon \right)=\sum _{i=1}^n=E_i{\epsilon }^i,$$

(14)

where $E_i$ is the Young’s modulus. The case $E_2=E_3=E_4=0$ corresponds to Hook’s law from the linear elasticity theory. Tension has been tested for $E1=2.14663$ MPa, $E2=-0.646588$ MPa, $E3=-0.0697791$ MPa, $E4=0.566989$ MPa (Dastjerdi [41]).

In (13), we have taken ${\epsilon }_{ij}^1=-0.6,$$h=0.05$ and $N=50.$ The neural network has been trained with a data set, $M=90$ (45 for training and 45 for validation and test ) with four hidden layers and with [15,20,15,20] neurons, respectively. The number of epochs is 8000 and the batch size is 64. The stress function with respect to the strain and its derivative have been predicted from the data set $\{{\epsilon }_i,i=1,2,\dots ,P\}$. In the NN, the loss function incorporates the physical constraints

$${\displaystyle \frac{\partial \sigma }{\partial \epsilon }}={\displaystyle \frac{{\partial }^2\mathrm{\Psi }}{\partial {\epsilon }^2}}\ge 0$$

and the Sobolev function. The physics-informed constitutive model enforces hyperelasticity principles, which ensures material stability and consistency.

The stress–strain curve and the derivative of the stress at the predicted points $P=40$ with the exact values of the stress from Equation (14) are shown in Figure 17.

The loss error $MSE=5.7\times {10}^{-7}$ on the last epoch and the relative errors of the computed stress values and their derivatives with respect to the strain at the predicted points are

$${\delta }_1={∥{\displaystyle \frac{{\sigma }^{\mathit{NN}}-\sigma }{\sigma }}∥}_{\infty }=0.000424,{\delta }_2={∥{\displaystyle \frac{{\sigma }_{\epsilon }^{\mathit{NN}}-{\sigma }_{\epsilon }}{{\sigma }_{\epsilon }}}∥}_{\infty }=0.008323.$$

Since the chain rule is used in automatic differentiation, the approximation of the derivatives can be completed with high accuracy. As the numerical results have shown, with even a small number of epochs, we can obtain a good rate of convergences of the neural network to the data values if the material tangent stiffness tensor is included in the training of the neural network.

6. Conclusions

A constitutive nonlinear neural network metamodel has been proposed for approximation of the constitutive relation in macrostructures and hyperelastic materials. The components of the stress and strain tensor and the partial derivatives of the stress tensor components have been used for neural network training. The data set has been obtained from the representative volume element using finite elements. The tangential stiffness matrix, i.e., partial derivatives of the stress tensor components within a multiscale model, have been calculated via the method of automatic differentiation.

A neural constitutive network model with a common residual MSE and three surrogate models for stress components has been trained to predict the stress components and tangent stiffness tensor with respect to the strain components, respectively. In addition, a second type of neural network with a separate ANN for each stress component has been trained as well. The loss function, without and including the Sobolev metrics, is investigated. The numerical results have been compared with the given data set of the stress–strain and partial derivatives as well as with the results, obtained after applying a polynomial regression, at prediction points. In addition, a physics-informed constitutive neural model, which enforces hyperelasticity principles, has also been discussed. A choice of the hyperparameters has been analyzed. The results can be integrated into a multiscale finite element analysis and provide a viable solution to nonlinear homogenization.

The NN has been developed based on the open scientific software machine learning platform Tensorflow 2.14.0 and its Keras library, and it was written in Python.

The residual model has been constructed with different loss functions, including mean square error and the Sobolev function with the derivatives of the components of the stress tensor. The latter gives better results after training the neural network. The proposed model allows us to predict the stress tensor in macrostructures and in hyperelastic materials, where the relations between stress and strain tensor are highly nonlinear. The technique has been tested on hyperelastic materials with incorporating constrains on the second derivative of the strain energy function in order to keep material stability and consistency.

The proposed techniques can be applied to nonlinear structures and hyperelastic materials with large deflections. In these cases, the constitutive behavior can be simulated by the proposed trained neural model based on experimental or statistical data sets for various applications, like biological tissue (e.g aorta, blood vessels), elastomers, rubbers, polymers, etc.

Extensions of this work in order to take into account the path-dependent constitutive relation of the RVE and dynamic effects are possible on more complicated problems and constitute the subject of our current research investigation.