Mathematical Tools for Simulation of 3D Bioprinting Processes on High-Performance Computing Resources: The State of the Art

Abstract

Three-dimensional (3D) bioprinting belongs to the wide family of additive manufacturing techniques and employs cell-laden biomaterials. In particular, these materials, named “bioink”, are based on cytocompatible hydrogel compositions. To be printable, a bioink must have certain characteristics before, during, and after the printing process. These characteristics include achievable structural resolution, shape fidelity, and cell survival. In previous centuries, scientists have created mathematical models to understand how physical systems function. Only recently, with the quick progress of computational capabilities, high-fidelity and high-efficiency “computational simulation” tools have been developed based on such models and used as a proxy for real-world learning. Computational science, or “in silico” experimentation, is the term for this novel strategy that supplements pure theory and experiment. Moreover, a certain level of complexity characterizes the architecture of contemporary powerful computational resources, known as high-performance computing (HPC) resources, also due to the great heterogeneity of its structure. Lately, scientists and engineers have begun to develop and use computational models more extensively to also better understand the bioprinting process, rather than solely relying on experimental research, due to the large number of possible combinations of geometrical parameters and material properties, as well as the abundance of available bioprinting methods. This requires a new effort in designing and implementing computational tools capable of efficiently and effectively exploiting the potential of new HPC computing systems available in the Exascale Era. The final goal of this work is to offer an overview of the models, methods, and techniques that can be used for “in silico” experimentation of the physicochemical processes underlying the process of 3D bioprinting of cell-laden materials thanks to the use of up-to-date HPC resources.

1. Introduction

In recent years, great interest has been devoted to three-dimensional (3D) bioprinting, which employs a numerically controlled dispensing system to deposit cells and biomaterial inks, named bioink, in a well-defined and controlled way. A wide number of papers have shown how beneficial it could be in the creation of smart patches for wound healing or thick tissue constructs, named scaffolds, for bone and cartilage tissue engineering [1,2,3,4]. Despite this, there are numerous difficulties with bioprinting, including the inability of cell-laden biomaterials to be printed, the severe circumstances that cells must endure while being printed, and the development of the bioprinted scaffold [5]. What you print in bioprinting is not necessarily what you obtain; natural cell rearrangements may alter the system, frequently for the better. Indeed, bioprinting is complicated at every stage of the process, from pre-processing to processing to post-processing.

Numerous geometrical characteristics and tunable material properties have been shown experimentally to have a clear and significant impact on cell viability and the appropriate creation of the intended tissue structure [6]. Considering the great number of possible combinations of geometrical constraints and material properties, as well as the great number of bioprinting techniques that are currently available, it can be crucial to create and apply computer models to gain a deeper knowledge of the bioprinting process, rather than exclusively depending on experimental research [7]. Indeed, in silico experiments like computational research and digital fabrication are thought to be important sources of innovation and to have a significant impact on the emerging field of 3D bioprinting and tissue engineering. While producing more accurate and realistic virtual experiments is still a difficulty, this can be overcome with a variety of strategies, including statistical tools and methodologies coupled with first-principles-based simulations (The contribution of Bradley et al. [8], with the terms “first principles models”, intend to represent models derived from fundamental laws of physics, chemistry, thermodynamics, kinetics, and transport phenomena, such as mass balance and energy balances. These models can take many different forms, such as a system of ordinary or partial differential equations (ODE or PDE), a system of fundamental algebraic equations, or a mix of the two to create a general nonlinear algebraic partial differential system of equations. Although the term “first principles models” was primarily used to represent methods for systems with continuous dynamics, systems with discrete or non-continuous relationships, as well as stochastic systems, can be represented by them as well [8]. On the contrary, purely data-driven models are distinct from first principles models in that their parameters are “fitted” according to the available data. Empirical models are another name for data-driven models. The data may originate from historical records, physical experiments, samples from simulations or first-principles models, or any combination of the aforementioned sources [8]). It is relevant to mention that computational science (which is the basis for any “in silico” experimentation) is a highly interdisciplinary discipline that requires the contribution of experts from the field of interest, mathematicians, and computer scientists [9] who jointly define the descriptive mathematical model of the problem, and implement, using appropriate tools, algorithms, and software, its numerical model (see Section 2 for a more complete description of the computational science (CS) process).

Numerous research papers have examined the effects of various hydrogel-based materials on cellular survival, “printability” (see [10] for a definition of the term), and shape fidelity [11], due to their significant role in the 3D bioprinting process. In fact, printability and shape accuracy are the most critical factors in bioprinting [12] since the printed construct should mimic the exact structure and architecture of the natural tissue.

In ref. [13], in the context of bone tissue engineering, a proof-of-concept study that integrates computationally guided design with customized bioprinting of cells and biomaterial carriers is reported. This work intends to be the necessary premise to the definition of a protocol for the “automatic” determination of the characteristics of “biological” inks fundamental to the success of scaffolds created through 3D printing: the characteristics of these bioinks must guarantee the mechanical features necessary to the printing process and at the same time the survival of the cells, suitably embedded into the ink, at every stage of the scaffold’s life.

Since each of the aforementioned stages is characterized by particular needs of the bioink [14], it was preferred to classify the above methods according to their applicability in each of the following phases:

• Pre-printing
  In the print cartridge, a cell-laden bioink can be considered a composite material. Therefore, even in the absence of different compounds such as rheological enhancers or additional biomaterials, high-cell-density suspensions behave as colloidal systems that exhibit printability. The impact of the cells embedding on the viscoelastic properties of the bioink is further complicated by the possibility that the cells are surrounded by a pericellular matrix, which could alter their mechanical characteristics, the hydrodynamic radius, and boundary conditions at the fluid interface [15].
  It is worth noting that cells can interact with each other, during the pre-printing phase and during the whole process, adding a further level of complexity to the overall system. Furthermore, different cell sources can be also considered when dealing with interface tissues such osteochondral one, which involves bone and cartilage tissues. Nonetheless, a bioink must prevent cell sedimentation to preserve a uniform cell suspension [16]. The cells in the ink are no longer evenly distributed when they settle. This may result in clogged nozzles as well as an uneven cell distribution in the finished printed structure (i.e., more cells in the early printed layers than in the latter ones). Creating bigger and more complex scaffolds may exacerbate the concerns related to cell sedimentation because these structures usually demand longer printing times (i.e., printing full-scale tissues or organs can take hours or even days). In conclusion, since cells can flow when a force is applied, a bioink would avoid cell sedimentation while yet remaining printable [14].
  Additionally, the volume occupied by the cells within a bioink varies according to their size and density. The hydrogel is excluded from the volume occupied by cells, which could affect the physicochemical and viscoelastic properties [15]. For example, cells may actually hinder the cross-linking process by limiting contact between reacting groups or acting as a physical barrier between various ink layers.
• During Printing
  This indicates that as a force is applied, their viscosity alters. Non-Newtonian fluids are primarily categorized into shear thickening (viscosity increases when shear rate increases) and shear thinning (viscosity reduces when shear rate increases) in response to this viscous tendency. Generally speaking, printability increases with material viscosity—at least to the point at which the internal pressures created can harm cells. Furthermore, some studies examine the impact of temperature on the viscosity of bioink. Nonetheless, they must be bioprinted at physiological conditions at 37 °C to preserve the cells [11]. Additionally, as a result of cell migration and proliferation, the distribution of cells during printing may change, which may have an impact on the bioink’s rheological characteristics. In fact, as cells interact with the matrix around them and with one other, traction forces are created that operate on the hydrogel macromolecules in a printed structure.
• Post-Printing
  Bioink needs to produce a 3D “milieu” that promotes cellular survival and function after printing. Because they offer an aqueous, cell-compatible environment that can mimic many of the mechanical and biochemical characteristics of the original tissue, hydrogels are frequently utilized in 3D cell culture. The simple passage of nutrients and waste products to and from encapsulated cells is made possible by their high water content and permeability. They can also provide a variety of signals to control cell phenotypic, differentiation, growth, and migration.

The goal of this scoping review is to provide an overview of the models, methods and techniques that can be used for “in silico” experimentation (or computational simulation) of the physico/chemical processes underlying the process of 3D printing of cell-laden materials [14,17] on high-performance computing (HPC) systems.

Although bioprinting for tissue engineering has made significant progress, additional research is required to determine the best bioink compositions and process parameters to create repeatable procedures that will result in the production of more functional tissue-engineered scaffolds. More knowledge is needed about the interplay between fluid flow, nutrient or oxygen distribution, cell–cell interaction, and flow shear stress in in vitro tissue morphogenesis. In this scenario, computational and mathematical modeling can be very helpful in explaining the intricate underlying mechanisms and facilitating the rapid and effective comparison of different conditions. Moreover, advances in “in silico” experimentation largely rely on concurrent advances in algorithms, software, and hardware needed to obtain high-performance levels for computational models. So many challenges related to the effective use of advanced computing platforms of the Exascale Era should be faced.

To this aim, this review paper proposes itself as a guide for future research to capitalize on the potential of mathematical models to find improved answers to the unmet clinical needs of modern society. The review work intends to answer the following research question:

“What are the relevant mathematical tools for the simulation of the different phases of 3D bioprinting with a particular focus on the first principles-based (FP) models in macro and mesa-scales and how such tools should be modified/integrated to be compliant with the emerging computational resource in the Exascale Era?”.

Literature searches were conducted in three electronic databases (PubMed, Google Scholar, and Web of Science). The following search keys were used for the literature search:

(Sub-Search-1) AND (Sub-Search-2) AND (Sub-Search-3) AND (Sub-Search-4) AND (Sub-Search-5)

where

Sub-Search-1:	(bioprinting OR (cell-laden hydrogels) OR bioinks)
Sub-Search-2:	(simulation OR modeling OR (computational science))
Sub-Search-3:	(macro or mesa scale FP models)
Sub-Search-4:	(numerical tools for computational solution of FP-model problems)
Sub-Search-5:	(Exascale models and algorithms).

The review is therefore organized as follows: in Section 2 a description of how computational science (CS) allows us to perform “in silico” experiments will be provided; Section 3 will give an inventory of the mathematical tools used in the literature for the computational simulation of the processes active in each of the above-mentioned phases (a description of the literature that is the result of the search [(Sub-Search-1) AND (Sub-Search-2)]); Section 4 will report a description of the “state of the art” of HPC systems in the Exascale Era; Section 5 and Section 6 will give a deeper insight into both models and numerical tools used in the literature (a description of the literature that is the result of the searches (Sub-Search-3) and (Sub-Search-4) respectively); Section 7 will describe the most advanced results in developing and implementing such mathematical tools on HPC systems in the Exascale Era (a description of the literature that is the result of the search (Sub-Search-5)); the final part of this paper (Section 8) will summarize the aims, results, potential usefulness of the work, and the open challenges and will highlight the future perspectives of the field in the emergent realm of biofabrication and process optimization thanks to approaches based on models that “learn from data”.

For quick access to the transversal contents and to the description of the mathematical tools that are the aim of the research question related to this work, we present

Table 1. An overview of the mathematical tools suggested in this review work that we included as relevant in consideration of our literature search.

Transversal Contents
Introduction on Computational Science (Section 2)
“State of the Art” of HPC Systems (Section 4)
Inventory of Mathematiacl Tools for Computational Simulation of Bioprinting Processes (Section 3)
Models	Numerical Tool		Exascale Numerical Tool
	Discretization	Algorithms
Fluid Flows Equations or coupled continuos equations (Two-Phase flow problem, transport and response of species, Section 5.1)     Cellular Particle Dynamics (Section 5.2.2)	FEM, FDM, FVM (Section 6.1.1)     Cellular Particle Dynamics Algorithm (Section 6.2.2)	Time Steppers (point 1 of Section 6.1.2) Non-linear problem solvers (point 2 of Section 6.1.2) Linear problem solvers (point 3 of Section 6.1.2) The Saddle Point Problem solvers (point 4 of Section 6.1.2)	General consideration on models suitable for Exascale (Section 7.1) Parallel Time Steppers (Section 7.2.1) Composite Non Linear Solver (Section 7.2.2) Parallel Iterative and Direct Linear Solver (Section 7.2.3) Fast Multipole Methods and “Hierarchical” matrices (Section 7.2.4)
Monte Carlo approach (Section 5.2.1)		The Markov Chain Monte Carlo Algorithms (Section 6.2.1)	Parallel Monte Carlo based methods (Section 7.2.5)
The Cellular Automata Model (Section 5.2.3)		The Cellular Automata Model Algorithms (Section 6.2.3)	General considerations on stochastic-based algorithms (Section 7.2)

.

2. The Role of Computational Science (CS)

Mathematical models have traditionally been created by theorists to better understand how physical systems function. But in the last several decades, there has been a tremendous advancement in processing power that has allowed for the creation of high-fidelity simulation software based on such models, used as a proxy for learning about the real world. We call this new method CS, which is a supplement to pure theory and experiment. Deep knowledge of mathematical modeling, numerical analysis, software engineering, high-performance computing, statistics, and a thorough comprehension of the technological application domain being investigated are essential for successful CS research. This is therefore a highly interdisciplinary work that requires the cooperation of application scientists, mathematicians, statisticians, and computer scientists [18], all acting in an integrated and interdependent ecosystem.

Figure 1 represents the process on which CS is based:

• Field experts and mathematicians jointly define the descriptive mathematical model of the problem.
• Mathematicians and computer scientists jointly define and implement the descriptive numerical model of the problem. The numerical model is described by an algorithm that can have a high computational cost due to the great number of operations. Therefore, it may be desirable to exploit the parallelism made available by resources for HPC [19,20,21,22,23,24,25].
• Field experts, mathematicians, and computer scientists jointly validate the correctness and accuracy of the numerical model and algorithm by executing its implementing software. Parallel algorithms and software are evaluated on the basis of their performance in terms of the number of performed operations per time unit, strong and weak scalability, and other useful metrics [26].
• If necessary, on the basis of the observations collected in the previous steps, a new formulation of the mathematical model is constructed.

In the CS process, each step has an impact on all the other steps in a loop where single-field experts can contribute just a small part (see the right part of Figure 1). Research endeavors may yield the dissemination of precisely designed software, empowering members of the wider scientific community to conduct their research based on the CS process [27,28].

Figure 1. The process of Computational Science. Image realized on the basis of materials from The German Priority Programme “Software for Exascale Computing” (SPPEXA) [29].

Figure 1. The process of Computational Science. Image realized on the basis of materials from The German Priority Programme “Software for Exascale Computing” (SPPEXA) [29].

3. An Inventory of the “In Silico” Experiments in 3D-Bioprinting of Cell-Laden Hydrogels

In past centuries, scientists have created mathematical models to understand how physical systems function. With the quick progress of computational capabilities, high-fidelity and high-efficiency “computational simulation” tools have been developed based on such models and used as a proxy for real-world learning.

Scientists and engineers in the context of material science have begun to develop and use computational models more extensively to better understand also the bioprinting process, rather than solely relying on experimental research, due to the large number of possible combinations of geometrical parameters and material properties, as well as the abundance of available bioprinting methods.

Figure 2 shows the logical structure graph of the relationships between 3D bioprinting computational simulation and concerned technologies and tools. In particular, bioprinting techniques could help find answers to some clinical needs of modern society, such as the regeneration of bone tissue using scaffolds or the “in vitro” study of the proliferation of tumor cells. Due to the enormous complexity of these techniques, experimental knowledge of the physical/chemical phenomena associated with them alone is not sufficient to fully understand their dynamics. Computational simulation is now widely recognized as a very powerful tool for broadening this understanding. Computational simulation obviously cannot exist without an effective investment in research on mathematical models and algorithms capable of guaranteeing effectiveness and efficiency in the solution of the involved numerical problems.

This section gives an inventory of the numerical approaches used in the literature for computational simulation of the processes active in each phase of the bioprinting process.

Table 2. Purview of the papers highlighting the most representative mathematical models based on the “first principles” approach used to describe the different phases of the bioprinting process.

	Model	Aim	Refs.
Pre-Printing	Fluid Flows Equations	Evaluate the influence of a seeding loading regime on suction pressure and infiltration velocity of the cell suspension.	[30]
		Introduce a new methodology to simulate the cell-seeding process under perfusion conditions. The cells are treated as spherical particles dragged by the fluid media.	[31]
		Design a model describing the fusion of growing cell clusters, such as embryo cell aggregates or tumor cell spheroids.	[32]
During Printing	Fluid Flows Equations	Used to study the influence of needle geometries on cell clogging during printing.	[33]
		Determine how printing process parameters affect the flow velocity of cell-laden printing material and the survivability of cells within printed structures.	[34]
		Aid the design of an extruder system for the biofabrication of vascular networks with the goal of maintaining cell viability throughout the printing process.	[35]
	Monte Carlo approach	Used to study cellular adhesiveness and motility, interactions with concentration fields, and concentration-dependent, stochastic cell dynamics, driven by metabolite-dependent cell death.	[36]
	Immersed Boundary Method	Used to study cell deformation during the extrusion/ injection-based bioprinting processes.	[37]
	Two-Phase Level Set Methods	Used to study droplet formation.	[38,39]
		Computational simulations of bioink printing process in suspension baths.	[40]
		Used to assess the influence of bioink temperature on shear stress, pressure, and velocity.	[11]
	Models for transport and response of biological and chemical species	Optimize printing process parameters in order to achieve the required cell dispersion and maintain cell viability.	[41]
		Investigate the distribution of oxygen and glucose in bioprinted structures, as well as examine the reality of the diffusion and consumption phenomena and their impact on the growth and demise of cells.	[42]
	Surface Tension Model	Study the filament deposition and the final bioprinted shape during the extrusion/ injection-based bioprinting processes.	[43]
Post-Printing	Monte Carlo approach	Study cell self-assembly and cellular aggregate fusion of multicellular aggregate systems.	[44]
		Study early morphogenesis of 3D tissue structures formed through the post-printing fusion of the bioink particles,	[45]
	Fluid Flows Equations	Study post-printing cell rearrangement to forecast the geometry and stability of printed structures	[46]
	Cellular Particle Dynamics	Study the discrete bioink units’ ability to self-assemble into the intended form after deposition and their deposition rate.	[47,48,49]
	Cellular Automata	Simulate post-printing cell behavior within the 3D bioprinted complex under a variety of bioprinting conditions.	[50]

reports a purview of the papers described in the following that used the most representative mathematical models based on the “first principles” approach to predict the different phases of the bioprinting process.

3.1. Pre-Printing

A crucial stage in biofabrication process is cell seeding. Using computational methodologies in conjunction with experimental activities, efficient cell seeding strategies are designed. For example, the relative impact of permeability and thickness, and coating on cell seeding efficiency and homogeneity was evaluated in [30] using a fully factorial experimental design, in conjunction with the Finite Elements Method (FEM) discretization of fluid flows equations, to predict the scaffold permeability effects on cell seeding effectiveness. For another example, the number of cells and the seeding time were optimized using the simulation of cellular interactions with biomaterials, cells, and dynamic medium in a situation where cell seeding takes place under perfusion conditions [31].

Furthermore, the theoretical framework, based on tissue fluidity, considers that cells tend to aggregate like droplets of an exceedingly viscous and incompressible fluid. In the mathematical model, cell clusters are assumed to have a size lower than 0.5 mm. Otherwise, gas and nutrients exchange are prevented, enabling the formation of necrotic areas [5]. A good agreement was found between continuum models of the coalescence of highly viscous liquid droplets and tissue cell cluster fusion studies. But there is a crucial difference between very viscous inanimate fluids and cell populations: they do not follow the principle of volume conservation. Dechristé et al. designed a compressible viscoelastic fluid model describing the fusion of growing cell clusters, such as embryo cell aggregates or tumor cell spheroids [32]. The authors initially gave the whole partial differential system, which describes the fusing of expanding spheroid, based on the Stokes equation. The combined model included a viscous model representing the velocity field and a tumor growth model. It is interesting to note that both models produced similar results since the velocity field’s divergence equals the rate at which volume changes. The authors reduced the complexity of the 3D model to a 0D model, which included an ordinary differential system, under the symmetry conditions. To show the consistency, the experiments were fitted to a numerical model. Similarly, Kuan et al. built a 2D continuum theory to model the aggregate formation in a system characterized by uniformly distributed cells, as well as the coalescence of aggregate doublets [51]. In particular, the authors explained the production of aggregates as an active-phase separation phenomenon, and the merging of aggregates as the coalescence of viscoelastic droplets, with the important timeframes associated with the active force’s turnover. The theory offered a broad framework for investigating the nonequilibrium dynamics and rheology of dense cellular aggregates [52].

3.2. During Printing

By clarifying the intricate interactions between the many printing parameters and forecasting their ideal configurations for certain goals, mathematical model techniques have shown to be effective in streamlining the bioprinting process [53,54]. There may be several goals for the computational optimization of bioprinting procedures [55].

Maximizing printing fidelity—the degree of resemblance between the product models and the real bioprinted product—is the goal of some models. Alternative strategies aim to optimize biomimetic fidelity, which refers to the construct’s biological, mechanical, and rheological resemblance to its in vivo equivalent. Alternative methods address the problem of simultaneously optimizing printing and the biomimetic fidelity in bioprinting. For instance, to assess the significance of different process parameters, such as ultraviolet (UV) intensity and UV exposure time, on cell viability in stereolithography-based 3D bioprinting, Xu et al. [56] successfully developed a predictive model using computational methods based on data to anticipate cell viability with good sensitivity. Several investigations have also attempted to create various methods to enhance the printability of bioinks. Using multiple regression analysis, Lee J. et al. showed how printability and the ink’s mechanical characteristics relate to one another. Nevertheless, depending on the goal of the study and the data-collection procedure, this approach highlighted certain limitations even with the significant advancements made in the application of the models in the biological sciences [57]. The biofabrication of high-resolution, fragile cell transfer of printed human-induced pluripotent stem cells (iPSC) using a low-cost 3D printer for exact cell placement and stem cell development is based on computational analysis, which uses fluid flow equations and it allows to tune flow parameters to improve needle geometry [33]. Simulations for studying cell deformation during the extrusion/injection-based bioprinting processes are described in [37]. Those simulations are based on the Immersed Boundary Method (IBM) [58]. Some authors [38,39] used the Two-Phase Level Set Methods (TPLSMs) [59,60] to study droplet formation.

Sego et al. used computational methods to predict the spheroids’ viability under different process parameters, during biofabrication prediction. The authors developed a discrete–continuous heuristic model that predicts metabolic effects over cells during spheroid-dependent construction combining field equations with a cellular Potts-type method, which is based on a Monte Carlo approach [36].

To determine how process parameters affect the flow velocity of cell-laden printing material and the survivability of cells within printed structures, researchers have explored the experimental effects of needle geometry and air pressure, and appropriately, models, based on equations for viscoelastic fluids, have been created to mimic these findings [34].

The models can be used to optimize process parameters in order to achieve the required cell dispersion and maintain cell viability. It has also been shown that computational modeling, based on models for the transport and response of biological and chemical species, can be used to produce well-designed tissue engineering structures that enhance bone and tissue regeneration by employing a particular spatial layout of cells in a matrix [41]. In particular, compared to uniform structures, a cell-gradient pattern of cell-laden hydrogels with different cell densities produced greater cell survivability after printing.

Using TPLSM and Carreau constitutive viscosity models, Prendergast et al. reported computational simulations of bioink printing in suspension baths in [40]. With IBM and the Surface Tension Model [61], a further publication [43] examined the filament deposition and the final bioprinted shape. The linear “Phan-Thien and Tanner (PTT)” model provided a clear explanation of the viscoelastic bioink’s rheology [62]. Employing a TPLSM-based simulation, Gomez-Blanco et al. assessed the influence of bioink temperature on shear stress, pressure, and velocity [11].

Meanwhile, the design of an extruder system for the biofabrication of vascular networks was aided by multiphysics computational models, based on fluid dynamics equations, with the goal of maintaining cell viability throughout the printing process [35].

Recently, Gironi et al. developed a computational model, based on models for the transport and response of biological and chemical species, to investigate the distribution of oxygen and glucose in bioprinted structures, as well as to examine the reality of the diffusion and consumption phenomena and their impact on the growth and demise of cells. The model’s initial validation was carried out on straightforward droplet-shaped structures. The investigation of constructions with and without channels, varying in size and geometry, was conducted using the model. In order to create vascularized bioprinted tissues, more intricate bioprinted construct configurations with bigger sizes and hierarchical vascular networks will be studied computationally and physically [42].

However, a summary of the different numerical models used to investigate cell viability and shape fidelity during the printing process can be found in [7]. The goal was to use these data to develop a mathematical model that could forecast the state of the cell resulting from the generated shear stresses. Additionally, this could aid in the optimization of the printing procedure and explore the effects of process-induced mechanical effects on cell vitality (e.g., refer to Nair et al. [63]).

3.3. Post-Printing

Computational methods may allow also one to target the overall design of the biofabrication process by forecasting the characteristics and quality of the end product based on the stimuli that are delivered and how they are arranged in space and time [64]. Furthermore, they may allow the evaluation of the cell maturation after the printing phase.

In particular, over time, cells embedded into the bioink proliferate and grow, tending to form a tissue. This represents an important aspect of the bioprinting process. Computational modeling can be used to simulate the fusing of cellular aggregates in biofabrication [7]. For instance, cell self-assembly and the cellular aggregate fusion of multicellular aggregate systems were studied in [44] utilizing kinetic Monte Carlo (KMC) methods [65]. Specifically, KMC was used to anticipate how post-printed scaffolds’ morphological characteristics would alter as tissue morphogenesis progressed. Predicted equilibrium tissue configurations were extrapolated from the interfacial tensions among the many cell types that comprise the bioprinted tissue. This was accomplished by applying a discrete multicellular lattice model that explained how cells interacted using the differential adhesion theory [44].

Computational techniques can be utilized not just to mimic the bioprinting process but also to investigate the mechanisms behind the fusing of multicellular structures. For example, post-printing cell rearrangement was predicted using the Lattice Boltzmann approach, which is based on fluid dynamics equations, to forecast the geometry and stability of printed structures [46]. A number of shapes were tested: a faulty hexagon, a square lattice that fuses into a structure resembling tissue, a hexagonal configuration of cylinders that fuses into a cylindrical tube, and a sequence of cylinders that fuses into a planar construct. New working hypotheses can be evaluated using computer simulations more quickly and cheaply than in a lab [46]. Jakab et al. mimicked, by using a Monte Carlo approach, the early morphogenesis of 3D tissue structures formed through the post-printing fusion of the bioink particles, in analogy with early structure-forming processes in the embryo that utilize the apparent liquid-like behavior of tissues composed of motile and adhesive cells [45].

In refs. [47,48,49], the predictive power of the computational method, named Cellular Particle Dynamics (CPDs) [65,66], was demonstrated in cases of simple printed constructs prepared with spherical multicellular bioink units. The discrete bioink units’ ability to self-assemble into the intended form after deposition and their deposition rate both affect the success of a bioprinting procedure. In particular, the creation of a tissue after printing is mainly guided by basic principles of biological organization. The development of biological structures in bioprinting, where mini-tissues are used as bioink, is characterized by their discrete fusion into multicellular spheroids or cylinders, which resembles the merging of liquid droplets. More specifically, the fusion of discrete bioink particles can be described by equations applicable to very viscous liquids.

Phase field theories were used to simulate the fusing of cellular aggregates in biofabrication in a different mathematical model, in which the surrounding hydrogel and cells were considered as a binary fluid combination consisting of two immiscible fluids [67]. In particular, the surrounding hydrogel ink and the cellular aggregates were modeled by considering a viscous fluid and a spheroid of complex fluids, respectively. A mean-field potential was then created, using a higher-order spectra model, by combining the long-range, attractive interactions between cells with the short-range, repulsive interactions resulting from the immiscibility. A two-phase model that can be used to study how active matter interacts with the surrounding viscoelastic medium was described in [68]. Moreover, an Oldroyd-B model was used to characterize the fluid rheology [62].

The development of a cellular automata model (CA) [69] to simulate post-printing cell behavior within the 3D bioprinted complex was first described in [50]. Preliminary “in vitro” studies were used to establish the criteria for cell proliferation, viability, and cluster formation in the CA model. In addition to quantitatively capturing the post-printing “in vitro” behavior of cells in the 3D scaffold, the CA model was able to predict and elucidate the behavior of the cells under a variety of bioprinting conditions. It was able to replicate, for example, the way that cells move and proliferate inside the pores. Furthermore, the in-silico data demonstrated how cell proliferation is directly influenced by the initial cell density in the bioink as well as the number of starting cells.

4. A “State of the Art” of HPC Systems in the Exascale Era

Progress in the deployment of computing platforms has constituted both a pull and a push factor for the advancement of scientific knowledge and engineering design. Historically, the HPC systems era started in the 1980s, when vector supercomputing dominated high-performance computing, as embodied in systems designed by Cray Research Inc. [70,71]. The 1990s saw the rise of massively parallel processing (MPP) in distributed memory systems and shared memory multiprocessors (SMPs) [72]. In turn, clusters of commodity (Intel/AMD x86) [73] and purpose-built processors (such as IBM’s BlueGene) [74] dominated the beginning of this Millenium.

Today, these clusters (which are made of hundreds of nodes and millions of processors/cores) are enriched by computational accelerators in the form of coprocessors, such as General Purpose Graphical Processing Units (GP-GPUs); they are based on high-speed, low-latency interconnects (such as Infiniband) (see Figure 3 for a representation of modern HPC systems) [71].

Parallel computers (in all their different architectural evolutions) made it possible to lay the foundations for overcoming the limits that, over the years, have slowed down the possibility of enhancing processor computational power [75]. Just a few years ago, teraflops (${10}^{12}$ floating point operations/second) was the state-of-the-art of HPC technology. Today, those same values can be obtained by a simple personal computer with just one accelerator, so advanced computing is now defined by multiple petaflop ((${10}^{15}$) floating operations/second) supercomputing systems. The exponential increase in advanced computing capability is outlined by the results of the well-known high-performance LINPACK (HPL) benchmark [76] which is used to compile the Top 500 list of the world’s fastest computers [77]. Such a list shows that the Exascale (${10}^{18}$ operations per second) Era is by now the current affairs in the long journey of performance increases that lasts for more than half a century. Indeed, in the second part of the 2022 year, the “Frontier” supercomputer installed at the US Oak Ridge National Laboratory broke the glass ceiling of the Exascale limit [78].

Historically, HPC advances largely relied on concurrent advances (in a sort of co-design approach) in algorithms, software, architecture, and hardware that led to higher performance levels for computational models. However, there are many challenges related to the effective use of advanced computing platforms of the Exascale Era, including, but not limited to, massive parallelism and high complexity in programming such heterogeneous computing facilities.

A series of studies identified research challenges in developing the generation of advanced computing systems for the Exascale objective [71,79,80]. From a computational scientist’s point of view, particular relevance has challenges related to

• Application programming support, which, in response to the complexity and scale of advanced computing hardware, makes available programming languages, numerical libraries, and programming models, eventually combined in hybrid and hierarchical approaches, exploit the multi-node, multi-core, and accelerators-based architecture of up-to-date HPC resources.
• New mathematical models and algorithms that can give solutions to the need for increasing amounts of data locality and the need to obtain much higher levels of concurrency and that can guarantee a high level of “scalability” [81] and “granularity” [82]. Then, it is now the time to develop new algorithms that are more energy-aware, more resilient, and characterized by reduced synchronization and communication needs.

Therefore, such scientists, supported by an appropriate programming environment, should spend effort developing models, re-designing algorithms, and re-implementing applications to fully exploit the power of Exascale architectures.

In addition to the technical ones, opportunities offered by the Exascale Era also brought challenges related to organizational, cultural, and economic issues. Among them, the issues related to the specific need to educate new students and workers to ensure a sufficiently large and capable workforce to manage “the Exascale transition” are worth noting. Despite these challenges, all the scientific communities cannot avoid facing them since every advance in computing technology has driven industry innovation and economic growth, spurring research in every domain of science. Managing the innumerable technical, political, and economic challenges will not be easy. Still, these challenges could be more addressable if they were faced in coordinated planning across different academic, industry, and social contexts. All the above considerations are the basis of the multiple efforts spent by different communities all around the world to reach and consolidate the objectives related to the effective and efficient use of up-to-date advanced computing platforms during the Exascale Era (i.e., see the various projects funded all around the world [80,83,84,85] or the initiatives from the civil society [86]).

Such efforts reveal the strategic role of HPC and SC for the present and the future. Indeed, the HPC community is already starting to question itself on the future of post-Exascale computing. For example, the organizers of both the International Conference on High-Performance Computing 2024 and the International Conference on Parallel Processing 2023 asked Kathy Yelick, Vice Chancellor for Research at the University of California Berkeley, to deliver the opening keynote talk, titled “Beyond Exascale Computing” [87,88]. The Yelick talk summarizes the results from two reports [89,90] both on Scientific Computing in the Post-Exascale Era. About her statements about the future of SC, one cannot help but agree that we have “... to continue to rethink applications...” and thus the mathematical models and the algorithms.

5. Mathematical Models

CS may be used to describe the dynamic behavior of physicochemical systems using simulations and models. Mathematical models can be created at the three conventional scales: microscopic, macroscopic, and kinetic [91].

On the microscopic scale, the state of the system is made up of all of its component states and is solely a function of time. The dynamics are modeled using a large system of ordinary differential equations, each representing a variable at the microscopic level. At the macroscopic level, continuum models for systems whose state variable can be a vector, are created. This kind of variable, which depends on both space and time, is provided by locally averaged data. Partial differential equations derived from equilibrium or conservation rules characterize the dynamics within the elementary volume of the space of microscopic states. At the mesoscopic scale, kinetic theory models can be also defined for systems composed of numerous interacting units, each of whose unique states is referred to as a microscopic state. The dynamics are described by a differential system created by conservation relations within the elementary volume of the space of microscopic states [91]. Furthermore, mathematical models at different scales can also be combined to form a hybrid technique called a multiscale approach [92].

Although the idea of multiscale modeling impacts many domains, material modeling has only benefited greatly from it in recent decades. Since the physics at a finer scale is better understood than at a coarser size, multiscale modeling refers to formulating equations, parameters, or simulation techniques that characterize behavior at a particular length scale starting from information obtained at the finer scale [92]. Many of these techniques have been created, employing different strategies to bridge various lengths and time spans. For a summary of multiscale computational methods and representative examples, the reader may refer to Table 1 of the work by Fish et al. [92]. As examples, we provide below two indicative samples taken from that table:

Upscaling methods based on the same  mathematical models: On different scales, the same (or similar) physics models are applied. The interscale coupling could be based on different techniques such as asymptotic theory, wavelets, or macrohomogeneity conditions [92].

Resolved-scale methods based on a Multigrid approach: Separate physics models and discretization schemes are used in different spatial domains that can be simulated concurrently. Coupling strategies are based on interscale operators (restriction and prolongation). The solution algorithms are similar to iterative Multigrid matrix solvers [92].

The extensive treatment of the “multiscale approaches” goes beyond the scope of this work but could be the objective of future developments. In any case, for those wishing to delve deeper into the topic, they can refer to the interesting review by Fish et al. [92].

According to what has already been tested and described in the literature cited in Section 3, we will only focus on models used at both macro and mesa scales, neglecting models that work at micro-scales.

5.1. Models at the Macro-Scale

The proposed models are all based on Fluid Flows Equations that are described in Section 5.1.1. Such equations could be coupled with other ones to describe more complex problems such as “The two-phase flow problem” (see Section 5.1.2) or the simulation of Transport and response of biological and chemical species (see Section 5.1.3).

In

Table 3. Summary of “Pros and Cons” of the models at the macro-scale.

Model	“Pros’’	“Cons”
The Navier–Stokes equations for Viscoelastic Fluids (Section 5.1.1)	The most comprehensive way to describe complex fluid dynamics.	The definition of model parameters could be very hard.
The Level Set Method (LSM) for the two-phase flow problem (Section 5.1.2)	LSM is based on a front-capturing approach that uses a scalar function on a stationary grid to capture the interface evolution. The ability to compute phenomena like bubble break-up and coalescence is made possible by the implicit interface capture.	The main difficulty in using this approach is keeping a clear boundary between the different fluids.
The Immersed Boundary Method (IBM) for the two-phase flow problem (Section 5.1.2)	IBM is based on a combination of front-capturing and front-tracking approaches. It can be useful to describe fluid–structure interaction when the structure is intended to be an immersed body.	IBM could inherit drawbacks of front-tracking approaches in describing interfaces with complex geometries. Moreover, the correct definition of the distribution function d could be crucial.
The continuum surface force (CSF) model for the two-phase flow problem (Section 5.1.2)	The probably most famous approach used to compute the surface tension force needed by the application of front capturing methods. The model considers surface tension force as a continuous, 3D effect across an interface, rather than as a boundary value condition on the interface.	The definition of the transition region and of the “color” function $\tilde{c}$.
Transport and response of biological and chemical species (Section 5.1.3)	Models used to study how the tissue reacts to the chemical and mechanical environment in the culture medium. They can be considered a strict framework of conservation laws for mixes of interacting continua that describe the transfer of mass and momentum between different component phases.	The definition of model parameters could be very hard.

we provide a summary of “Pros and Cons” of the described models.

5.1.1. The Navier–Stokes Equations for Viscoelastic Fluids

A fluid is said to be Newtonian if the local rate of change in its deformation over time is linearly proportional to the viscous stresses arising from its flow. On the other hand, non-Newtonian fluids defy this principle, and, for the most part, their viscosity depends on the present or past shear rate. Non-Newtonian fluids include blood, proteins, polymers, suspensions, emulsions, chemical reactants, and most other fluids found in both nature and industry. These fluids typically exhibit a wide range of remarkable properties, such as viscoelasticity, shear-thinning, and shear-thickening properties [93,94]. Shear-thinning (also called pseudoplastic) fluids are characterized by an apparent viscosity which decreases by increasing the shear rate. Conversely, shear-thickening fluids are those whose apparent viscosity rises as the shear rate decreases [95]. Finally, it should be noted that viscoelastic fluids exhibit time-dependent strain, a characteristic in which the fluids’ strain gradually approaches the equilibrium value when stress is applied.

Navier–Stokes equations are used to describe the conservation of mass and momentum of incompressible fluid, varying in temporal domain $\Theta$, in a spatial domain $\Omega \subseteq {\Re }^n$, $n=1,2,3$ with boundary $\delta \Omega$ as the following problem:

Problem 1. 

The Navier–Stokes equations defined in $\Omega \times \Theta$

$$\begin{array}{cccc}\hfill \frac{\partial }{\partial t}\left(\rho \mathit{v}\right)+\left(\rho \mathit{v}\right)\odot \nabla \mathit{v}& =\hfill & \hfill -\nabla p+div\mathit{T}+\rho \mathit{g}+\mathit{f}& (Momentum equation)\hfill \end{array}$$

(1)

$$\begin{array}{cccc}\hfill div\mathit{v}& =\hfill & \hfill 0& (Continuity equation)\hfill \end{array}$$

(2)

$$\begin{array}{cccc}\hfill {\mathit{v}}_{\left|t=0\right.}& =\hfill & \hfill {\mathit{v}}_0& \end{array}$$

(3)

$$\begin{array}{ccc}\hfill p_{\left|t=0\right.}& =\hfill & \hfill p_0\end{array}$$

(4)

where $\mathit{v}$ is the velocity field, ρ is the density of the fluid, p is the pressure field, $\mathit{g}$ is the gravitational acceleration, $\mathit{f}$ is an additional force field, and $\mathit{T}$ is the stress response to the deformation of the fluid.

An example for $\mathit{f}$ is the “Surface Tension” force [96] in a two-phase problem [59] (see Section 5.1.2).

The notations $\nabla \mathit{u}$, $div\mathit{u}$, $\nabla f$ and $\mathit{u}\odot \nabla \mathit{u}$ respectively are used to representing:

$$\begin{array}{ccc}\hfill \nabla \mathit{u}& =& \left[\begin{array}{ccc}\frac{\partial u_1}{\partial x_1}& \cdots & \frac{\partial u_n}{\partial x_1}\\ ⋮& \ddots & ⋮\\ \frac{\partial u_1}{\partial x_n}& \cdots & \frac{\partial u_n}{\partial x_n}\end{array}\right]\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill \mathrm{div}\mathit{u}& =& \sum _{i=1,\dots ,n}\frac{\partial u_i}{\partial x_i}\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill \mathrm{div}\mathit{T}& =& \left[\begin{array}{c}\sum _{j=1,\dots ,n}\frac{\partial t_{1,j}}{\partial x_j}\\ ⋮\\ \sum _{j=1,\dots ,n}\frac{\partial t_{n,j}}{\partial x_j}\end{array}\right]\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \nabla f& =& \left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill \mathit{u}\odot \nabla \mathit{u}& =& \left[\begin{array}{c}\sum _{j=1,\dots ,n}{u}_j\frac{\partial u_1}{\partial x_j}\\ ⋮\\ \sum _{j=1,\dots ,n}{u}_j\frac{\partial u_n}{\partial x_j}\end{array}\right]\hfill \end{array}$$

(9)

where $f\left(x,t\right)\in \Re$, $\mathit{u}\left(x,t\right)={\left[u_i\left(x,t\right)\right]}_{i=1,\dots ,n}^T\in {\Re }^n$, $\mathit{T}\left(x,t\right)={\left[t_{i,j}\left(x,t\right)\right]}_{i,j=1,\dots ,n}\in {\Re }^{n\times n}$ and $x=\left(x_1,\dots ,x_n\right)\in {\Re }^n$.

The solvent-polymer stress splitting technique is used in this formulation to decompose the stress response into two terms [93]: ${\mathit{T}}_s$, which represents the solvent’s instantaneous response, and ${\mathit{T}}_p$, which represents the polymeric contribution to the stress response which accounts for the polymers’ memory effects [97]:

$$\mathit{T}={\mathit{T}}_s+{\mathit{T}}_p$$

(10)

where the solvent part, ${\mathit{T}}_s$, is defined as below:

$${\mathit{T}}_s=2{\mu }_s\left(\dot{\gamma }\right)\mathit{D}$$

(11)

where

$$\begin{array}{ccc}\hfill \mathit{D}& =& \frac{1}{2}\left[\nabla \mathit{v}+{\left(\nabla \mathit{v}\right)}^T\right],\hfill \end{array}$$

(12)

where $\dot{\gamma }$ is the shear rate defined by [98]

$$\begin{array}{ccc}\hfill \dot{\gamma }& =& \sqrt{2}∥\mathit{D}∥,\hfill \end{array}$$

(13)

and where $∥·∥$ denotes the Hilbert–Schmidt norm of a matrix $\mathit{A}$ defined as

$$∥\mathit{A}∥=\sqrt{\sum _{i,j=1,\dots ,n}{a}_{ij}^2}$$

The following viscosity models ${\mu }_s$ for generalized Newtonian fluids could be considered [93,99]:

• Newtonian viscosity model

  $${\mu }_s^{\left(0\right)}={\mu }_{\infty }^{\left(0\right)}={\mu }_s,$$
• Power-Law model

  $${\mu }_s^{\left(1\right)}={\mu }_{\infty }^{\left(1\right)}{\dot{\gamma }}^{n-1},$$
• Cross model

  $${\mu }_s^{\left(2\right)}={\mu }_{\infty }^{\left(2\right)}+\left({\mu }_0^{\left(2\right)}-{\mu }_{\infty }^{\left(2\right)}\right){\left[1+{\left(\lambda \dot{\gamma }\right)}^b\right]}^{-1},$$
• Modified Cross model

  $${\mu }_s^{\left(3\right)}={\mu }_{\infty }^{\left(3\right)}+\left({\mu }_0^{\left(3\right)}-{\mu }_{\infty }^{\left(3\right)}\right){\left[1+{\left(\lambda \dot{\gamma }\right)}^b\right]}^{-a},$$
• Carreau model

  $${\mu }_s^{\left(4\right)}={\mu }_{\infty }^{\left(4\right)}+\left({\mu }_0^{\left(4\right)}-{\mu }_{\infty }^{\left(4\right)}\right){\left[1+{\left(\lambda \dot{\gamma }\right)}^2\right]}^{\frac{n-1}{2}},$$
• Carreau–Yasuda model

  $${\mu }_s^{\left(5\right)}={\mu }_{\infty }^{\left(5\right)}+\left({\mu }_0^{\left(5\right)}-{\mu }_{\infty }^{\left(5\right)}\right){\left[1+{\left(\lambda \dot{\gamma }\right)}^m\right]}^{\frac{n-1}{m}},$$

where ${\mu }_0^{\left(\theta \right)}$ and ${\mu }_{\infty }^{\left(\theta \right)}$ are the asymptotic viscosity values at zero and infinite shear rates (where $\theta =0, 1, 2, 4, 5$). The parameters of the non-Newtonian viscosity models are a, b, m, and n, and the relaxation time is denoted by the symbol $\lambda$. The Power-Law model [93] may explain shear-thinning behavior for $n<1$, shear-thickening behavior for $n>1$, and Newtonian behavior for $m=1$.

Several constitutive rheological models are available in the literature with polymeric stresses ${\mathit{T}}_p$ [93,97], which are typically functions A number of constitutive rheological models with polymeric stresses ${\mathit{T}}_p$ are available in the literature [93,97]. These stresses are usually functions ${\mathit{f}}_S\left(·\right)$ of the conformation tensor $\mathit{C}$,

$${\mathit{T}}_p=\frac{{\mu }_p{\mathit{f}}_S\left(\mathit{C}\right)}{\lambda }$$

(14)

where ${\mu }_p$ the polymeric viscosity.

According to [97], the conformation tensor $\mathit{C}\in {\Re }^{n\times n}$ can be viewed as an internal state variable that describes the molecular deformation of the polymer chains. Assuming symmetry and positive definiteness, the conformation tensor C behaves in accordance with the following equation:

$$\stackrel{▽}{\mathit{C}}=-\frac{{\mathit{f}}_R\left(\mathit{C}\right)}{\lambda }$$

(15)

where the relaxation function, which varies for each specific constitutive model, is ${\mathit{f}}_R\left(·\right)$. The operator upper-convected derivative, described in [97], is indicated by ▽. With the symbol •, we intend to represent the matrix–matrix operation.

$$\stackrel{▽}{\mathit{C}}=\frac{\partial \mathit{C}}{\partial t}+\nabla •\left(\mathit{v}\mathit{C}\right)-\mathit{C}•\nabla \mathit{v}-\nabla {\mathit{v}}^T•\mathit{C}$$

(16)

where

$$\begin{array}{ccc}\hfill \nabla •\left(\mathit{v}\mathit{C}\right)& =& \left[\begin{array}{ccc}\sum _{k=1,\dots ,n}{v}_k\frac{\partial c_{1,1}}{\partial x_k}& \cdots & \sum _{k=1,\dots ,n}{v}_k\frac{\partial c_{1,n}}{\partial x_k}\\ ⋮& \ddots & ⋮\\ \sum _{k=1,\dots ,n}{v}_k\frac{\partial c_{n,1}}{\partial x_k}& \cdots & \sum _{k=1,\dots ,n}{v}_k\frac{\partial c_{n,n}}{\partial x_k}\end{array}\right]\hfill \end{array}$$

(17)

Table 4. The relaxation and strain functions for some constitutive models are denoted by ${\mathit{f}}_S\left(\mathit{C}\right)$ and ${\mathit{f}}_R\left(\mathit{C}\right)$. L is the ratio of the length of a fully expanded polymer dumbbell to its equilibrium length, and $\mathrm{tr}\left(\mathit{C}\right)$ denotes the trace of the tensor $\mathit{C}$.

	Oldroyd B	FENE-P	FENE-CR	Linear PTT
${\mathit{f}}_R\left(\mathit{C}\right)$	$\mathit{C}-\mathit{I}$	$\frac{\mathit{C}}{1-\mathrm{tr}\left(\mathit{C}\right)/L^2}-\mathit{I}$	$\frac{\mathit{C}-I}{1-\mathrm{tr}\left(\mathit{C}\right)/L^2}$	$\left[1+ϵ\mathrm{tr}\left(\mathit{C}-\mathit{I}\right)\right]\left(\mathit{C}-\mathit{I}\right)$
${\mathit{f}}_S\left(\mathit{C}\right)$	$\mathit{C}-\mathit{I}$	$\frac{\mathit{C}}{1-\mathrm{tr}\left(\mathit{C}\right)/L^2}-\mathit{I}$	$\frac{\mathit{C}-I}{1-\mathrm{tr}\left(\mathit{C}\right)/L^2}$	$\mathit{C}-\mathit{I}$

collects the expressions of the strain ${\mathit{f}}_S\left(\mathit{C}\right)$ and relaxation functions ${\mathit{f}}_R\left(\mathit{C}\right)$ for some constitutive models [97].

Classically, in the case of the Oldroyd-B model, it is usual to skip the use of C by combining Equations (1) and (15). Then, the constitutive equations in terms of the viscoelastic stress tensor, $T_p$ is written as

$$\lambda T_p+\stackrel{▽}{T_p}=2{\mu }_p\mathit{D}.$$

(18)

The summation of ${\mu }_s$ and ${\mu }_p$ of the fluid is defined as total viscosity and denoted as ${\mu }_0={\mu }_s+{\mu }_p$. The ratio of ${\mu }_s$ to ${\mu }_p$ is defined as viscosity ratio of viscoelastic fluid also known as the retardation ratio and denoted as $\beta =\frac{{\mu }_s}{{\mu }_p}\le 1$.

Different boundary conditions can occur on the boundary $\partial \Omega$ of $\Omega$ depending on the different physical conditions. However, it is important to consider that while the Neumann condition specifies the gradient of the velocity perpendicular to the boundary, the Dirichlet boundary condition expresses the value of the velocity on the boundary. The decomposition of the velocity $\mathit{v}$ in its normal component $v_N:=\mathit{v}·\mathit{n}$ and its tangential part $v_T:=\mathit{v}-v_N\mathit{n}$ (where $\mathit{n}$ is the vector normal to boundary $\partial \Omega$) allows the description of five different types of boundary conditions [98] as reported in the

Table 5. Boundary conditions that are possible on the boundary $\partial \Omega$ of $\Omega$ depending on the different physical conditions.

No-slip	The boundary $\partial \Omega$ is considered as a rigid wall where no fluid comes out of the wall. On the boundary, the following equation is valid [98]: $$\mathit{v}=0$$
Free-slip	No fluid comes out of the wall, but in contrast to the no-slip condition, no friction acts along the wall. Therefore, on the boundary, the following equation is valid [98] $$v_N=0\mathrm{and}\mathbf{\tau }·\mathit{T}·\mathit{n}=0\mathrm{for}\mathrm{each}\mathbf{\tau }\perp \mathit{n}$$ With the symbol ·, we intend to represent the matrix-vector operation and the vector dot product operation depending on the type of the operands.
Inflow	In this case, all velocity components are stated [98], i.e., $$\mathit{v}={\mathit{v}}_{in}\mathrm{where}{\mathit{v}}_{in}\mathrm{is}\mathrm{given}$$
Outflow	In order to prescribe boundary conditions at the outlet, many technics exist. Practicable importance can be attributed to the so-called “do-nothing” method, which is described by the following equation $$\nabla \mathit{v}·\mathit{n}-p\mathit{n}=0$$ with a prescribed normalized pressure p [98].
Periodic	In the case of a one-directional periodical problem with periodic length $\pi$, the velocities and the pressure have to be equal on both sides of the periodical direction. If we assume that the problem is periodic in the $x_i$ direction, it follows that [98] $$\begin{array}{ccc}\hfill \mathit{v}\left(x_1,\dots ,x_i,\dots ,x_n,t\right)& =& \mathit{v}\left(x_1,\dots ,x_i+\pi ,\dots ,x_n,t\right)\hfill \\ \hfill {\partial }_{x_1}\mathit{v}\left(x_1,\dots ,x_i,\dots ,x_n,t\right)& =& {\partial }_{x_1}\mathit{v}\left(x_1,\dots ,x_i+\pi ,\dots ,x_n,t\right)\hfill \\ \hfill p\left(x_1,\dots ,x_i,\dots ,x_n,t\right)& =& p\left(x_1,\dots ,x_i+\pi ,\dots ,x_n,t\right)\hfill \end{array}$$

.

If the velocity is stated on the whole boundary of the domain (i.e., when pure Dirichlet boundary conditions are considered), the additional condition

$${\int }_{\partial \Omega }\mathit{v}·\mathit{n}dA=0$$

must be satisfied because the velocity field is divergence-free [98].

5.1.2. The Two-Phase Flow Problem

Two-phase flow describes the combined behavior of two different fluids (or phases) that are immiscible at a molecular level and both occupy domain $\Omega \in {\Re }^n$. For example, this method was used to describe a water droplet in the air or the description of cellular aggregates in the surrounding hydrogel. Within each phase, all physical quantities are taken to be continuous; however, at the interface, at least one physical quantity is discontinuous. Modeling and simulating multiphase fluid flows have been a big challenge over the years; we will describe some of the related models.

We denote the domain occupied by the two liquids with ${\Omega }_1$ and ${\Omega }_2$, respectively, such that $\Omega ={\Omega }_1\cup {\Omega }_2$ and such that they are separated by an interface $\Gamma ={\Omega }_1\cap {\Omega }_2$. In general, the phase index $i=1,2$ is used for all quantities of the liquid phase i. If the fluids are in motion, the domains ${\Omega }_i\left(t\right),i=1,2$, as well as the interface $\Gamma \left(t\right)$, are unknown at the time instant t getting a so-called free boundary problem.

When it comes to the numerical treatment of free boundary problems, we can often identify two different kinds of issues [98].

• The representation and evolution in time of the interface.
• The way the imposed surface boundary conditions are treated.

The first issue above will be treated at the following point 1, whereas point 2 deals with the second issue above.

• Representation and time evolution of the interface.
  To represent the two-phase flows with shifting inter-phase boundaries, various methodologies have been developed in response to the problem’s complexity, both numerically and physically. In principle, the methods can be divided into two classes: the so-called front-tracking (FT) methods and front-capturing (FC) methods.
  In these methods, two-phase flow is treated as a single flow with the density and viscosity smoothly varying across the moving interface which is captured in a Eulerian framework (FC) or in a Lagrangian, framework (FT), with the terms Eulerian and Lagrangian frameworks, meaning, meaning the following [100]:
  • Lagrangian framework: In classical field theories, the Lagrangian specification of the flow field is a way of looking at fluid motion where the observer follows an individual fluid parcel as it moves through space and time.
  • Eulerian framework: The Eulerian specification of the flow field is a way of looking at fluid motion that focuses on specific locations in the space through which the fluid flows as time passes.
  The front-tracking method is based on a formulation in which a separate unstructured grid (surface grid) with nodes converging at a local velocity is used to represent the interface. One benefit of the technique is that interfacial conditions can be easily included because the interface is precisely established by the surface grid’s position. Moreover, a good approximation of the curvature allows for the consideration of surface tension force. The method’s drawback is that it requires the use of extra algorithms to compute flows with substantial interface deformations such bubble break-up and coalescence since the surface grid will be severely distorted.
  In contrast, the front-capturing approach uses a scalar function on a stationary grid to capture the interface as part of its implementation of the interface evolution. The ability to compute phenomena like bubble break-up and coalescence is made possible by the implicit interface capture provided by front-capturing techniques. The two main difficulties in using this approach are keeping a clear boundary between the different fluids and accurately calculating the surface tension forces [98].
  The Level Set approach was developed as a result of the front-capturing approach [59,60]. A different method, known as the Immersed Boundary (IB) method [58,101,102,103,104], was developed for flows across challenging geometries. It can be thought of as a combination of front-capturing and front-tracking procedures.
  • Level Set Method The idea on which the level set method is based is quite simple. Given an interface $\Gamma$ in ${\Re }^n$ of codimension one, bounding an open region $\Omega$, the LS method intends to analyze and compute its subsequent motion under a velocity field $\mathit{v}$.
    This velocity depends on the position, time, the geometry of the interface (e.g., its normal or its mean curvature) and the external physics.
    The idea is to define a smooth function $\varphi \left(x,t\right)$ representing the interface as the set where $\varphi \left(x,t\right)=0$.
    The level set function $\varphi \left(x,t\right)$ has the following properties:

    $$\begin{array}{ccc}\hfill \varphi \left(x,t\right)>0& & \mathrm{for}x\in {\Omega }_1\hfill \\ \hfill \varphi \left(x,t\right)<0& & \mathrm{for}x\in {\Omega }_2\hfill \\ \hfill \varphi \left(x,t\right)=0& & forx\in \Gamma \left(t\right)\hfill \end{array}$$
    Thus, the interface is to be captured for all times t, by defining the set of points $\Gamma \left(t\right)$ for which the $\varphi$ values (levels) obey the following equation

    $$\frac{\partial \varphi }{\partial t}+\mathit{v}·\nabla \varphi =0.$$

    (19)
    Coupling the level set method with problems related to two-phase Navier–Stokes incompressible flow needs the reformulation of Problem 1 as follows

    Problem 2. 

    Equations of two-phase Navier–Stokes incompressible flow defined in $\Omega \times \Theta$ using the level set method

    $$\begin{array}{ccc}\hfill \frac{\partial }{\partial t}\left(\rho \mathit{v}\right)+\left(\rho \mathit{v}\right)\odot \nabla \mathit{v}& =& -\nabla p+div\mathit{T}+\rho \mathit{g}+\delta \left(\varphi \right)\sigma \kappa \mathbf{N}\hfill \end{array}$$

    (20)

    $$\begin{array}{ccc}\hfill div\mathit{v}& =& 0\hfill \end{array}$$

    (21)

    $$\begin{array}{ccc}\hfill \frac{\partial \varphi }{\partial t}+\mathit{v}·\nabla \varphi & =& 0\hfill \end{array}$$

    (22)

    $$\begin{array}{ccc}\hfill {\mathit{v}}_{\left|t=0\right.}& =& {\mathit{v}}_0\hfill \end{array}$$

    (23)

    $$\begin{array}{ccc}\hfill p_{\left|t=0\right.}& =& p_0\hfill \end{array}$$

    (24)

    $$\begin{array}{ccc}\hfill {\varphi }_{\left|t=0\right.}& =& {\varphi }_0\hfill \end{array}$$

    (25)

    where $\mathit{v}$ is the velocity field, p is the pressure field, $\mathit{g}$ is the gravitational acceleration, $\rho \left(\varphi \right)$ and $\mathit{T}\left(\varphi \right)$ are the piecewise constant fluid densities and stress tensors described by the Heaviside function, σ is the surface tension coefficient, κ is the curvature of the interface, $\mathbf{N}$ is the unit normal vector outward to $\Gamma \left(t\right)$, and $\delta \left(\varphi \right)$ is a delta function.

    The following equations hold [59]: With the symbol $\left|u\right|$, we denote the modulus of the vector $u\in {\Re }^n$, that is,

    $$\left|u\right|=\sum _{i=1,\dots ,n}{u}_i^2$$

    $$\begin{array}{ccc}\hfill \mathit{T}\left(\varphi \left(x,t\right)\right)& =& {\mathit{T}}_1{H}_1\left(x,t\right)+{\mathit{T}}_2\left(1-H_1\left(x,t\right)\right)\hfill \end{array}$$

    (26)

    $$\begin{array}{ccc}\hfill \rho \left(\varphi \left(x,t\right)\right)& =& {\rho }_1{H}_1\left(x,t\right)+{\rho }_2\left(1-H_1\left(x,t\right)\right)\hfill \end{array}$$

    (27)

    $$\begin{array}{ccc}\hfill H_1\left(x,t\right)& =& \left\{\begin{array}{cc}1\hfill & forx\in {\Omega }_1\left(t\right)\hfill \\ 0\hfill & forx\notin {\Omega }_1\left(t\right)\hfill \end{array}\right.\hfill \end{array}$$

    (28)

    $$\begin{array}{ccc}\hfill \delta \left(\varphi \left(x,t\right)\right)& =& \left\{\begin{array}{cc}1\hfill & forx\in \Gamma \left(t\right)\hfill \\ 0\hfill & forx\notin \Gamma \left(t\right)\hfill \end{array}\right.\hfill \end{array}$$

    (29)

    $$\begin{array}{ccc}\hfill \mathbf{N}& =& -\frac{\nabla \varphi }{\left|\nabla \varphi \right|}\hfill \end{array}$$

    (30)

    $$\begin{array}{ccc}\hfill \kappa & =& -div\left(\frac{\nabla \varphi }{\left|\nabla \varphi \right|}\right)\hfill \end{array}$$

    (31)

    Note that the term $\sigma \kappa \mathbf{N}$ added to Equation (20) is used to model the surface tension forces [98].
    The function $\varphi$ can have different format [105,106,107], which helps to avoid some disadvantages of the LS method related to the discrete solution of transport equations, which may be affected by numerical error leading to loss or gain of mass.
    In standard level set methods, the level set function $\varphi$ is defined to be a signed distance function

    $${\varphi }_d\left(x,t\right)=\left\{\begin{array}{cc}d\left(x,t\right)\hfill & \mathrm{for}x\in {\Omega }_1\hfill \\ -d\left(x,t\right)\hfill & \mathrm{for}x\notin {\Omega }_1\hfill \end{array}\right.$$

    (32)

    where $d\left(x,t\right)={min}_{x_{\Gamma }\in \Gamma \left(t\right)}\left|x-x_{\Gamma }\right|$.
    To improve numerical robustness, “smoothed out” Heaviside functions (needed to represent density and viscosity discontinuities over the interface) are often used. For example [106],

    $$\begin{array}{ccc}\hfill H_{sm}\left(\varphi \right)& =& \varphi \hfill \end{array}$$

    (33)
  • Immersed Boundary Method
    The Immersed Boundary Method (IBM) can be useful to describe fluid–structure interaction when the structure is intended to be an Immersed Body. In the IBM, the fluid is represented in a Eulerian coordinate system, and the configuration of the Immersed Body is described by a Lagrangian approach [101].
    Consider the simulation of incompressible flow around the body in Figure 4a, which is described by Problem 1 where $\Omega ={\Omega }_f\cup {\Omega }_b$. ${\Omega }_f$ denotes the surrounding fluid domain occupied by the solid body domain ${\Omega }_b$ with the boundary denoted by ${\Gamma }_b$.
    In an IB method, the boundary condition would be imposed indirectly through a specification of Problem 1. In general, the specification takes the form of a source term (or forcing function) in the governing equations, which reproduces the effect of boundary [102].
    The original IBM, created by Peskin [58], is typically appropriate for flows with immersed elastic limits. It was designed for the combined modeling of blood flow and muscle contraction in a beating heart. A collection of massless points moves with the local fluid velocity $\mathit{v}$, and these points serve as a Lagrangian tracker of the location of the Immersed Body, represented by a set of elastic fibers.
    Thus, the coordinate X of the Lagrangian point is governed by the equation

    $$\frac{\delta X}{\delta t}=\mathit{v}\left(X,t\right)$$

    (34)
    The effect of the Immersed Body on the surrounding fluid is essentially captured by transmitting the fiber’s stress $\mathit{F}$ to the fluid through a localized forcing term ${\mathit{f}}^{IBM}\left(x,t\right)$ in the momentum equations, which is given by

    $${\mathit{f}}^{IBM}\left(x,t\right)={\int }_{{\Omega }_b}\mathit{F}\left(X,t\right)\delta \left(\left|x-X\right|\right)dX,$$

    (35)

    where $\delta \left(·\right)$ is the Dirac delta function. The Dirac delta function in ${\Re }^n$ is formally defined by the following:

    $$\delta \left(x\right)=\left\{\begin{array}{cc}+\infty \hfill & x=0\hfill \\ 0\hfill & x\ne 0\hfill \end{array}\right.$$

    and is also constrained to satisfy the identity

    $${\int }_{x\in {\Re }^n}\delta \left(x\right)dx=1.$$
    The function $\mathit{F}$ can be written as

    $$\mathit{F}\left(X,t\right)=-\frac{\wp E}{\wp X},$$

    (36)

    where $E\left[X\right]$ is a given functional called “the elastic potential energy of the material” in configuration X and where the notation $\frac{\wp E}{\wp X}$ is shorthand for the “Fréchet derivative” of E with respect to X. Thanks to the use of the Dirac delta function, Equation (34) can be rewritten as

    $$\frac{\delta X}{\delta t}={\int }_{{\Omega }_b}\mathit{v}\left(x,t\right)\delta \left(\left|x-X\right|\right)dx.$$

    (37)
    The forcing term is thought to be distributed over a band of cells surrounding each Lagrangian point (see Figure 4b), as the fiber locations typically do not coincide with the nodal points of the Eulerian grid. This distributed force is then imposed on the momentum equations of the surrounding nodes. As a result, a smoother distribution function, represented by d in this case, effectively replaces the delta function and may be applied to a discrete mesh. The d function allows for the rewriting of Equations (35) and (37) as follows:

    $$\begin{array}{ccc}\hfill \frac{\delta X}{\delta t}& =& {\int }_{{\Omega }_b}\mathit{v}\left(x,t\right)d\left(\left|x-X\right|\right)dx,\hfill \end{array}$$

    (38)

    $$\begin{array}{ccc}\hfill {\mathit{f}}^{IBM}\left(x,t\right)& =& {\int }_{{\Omega }_b}\mathit{F}\left(X,t\right)d\left(\left|x-X\right|\right)dX.\hfill \end{array}$$

    (39)
    The choice of the distribution function d is a key ingredient in this method. Several different distribution functions were employed in the past (i.e., see [102] for a list of employed d).
  Coupling the IB equations with problems related to the Navier–Stokes incompressible flow requires the reformulation of the Problem 1 as follows:

  Problem 3. 

  Equations of the Navier–Stokes incompressible flow defined in ${\Omega }_f\times \Theta$ surrounding an Immersed Body ${\Omega }_b$:

  $$\begin{array}{ccc}\hfill \frac{\partial }{\partial t}\left(\rho \mathit{v}\right)+\left(\rho \mathit{v}\right)\odot \nabla \mathit{v}& =& -\nabla p+div\mathit{T}+\rho \mathit{g}+{\mathit{f}}^{IBM}\hfill \end{array}$$

  (40)

  $$\begin{array}{ccc}\hfill div\mathit{v}& =& 0\hfill \end{array}$$

  (41)

  $$\begin{array}{ccc}\hfill {\mathit{v}}_{\left|t=0\right.}& =& {\mathit{v}}_0\hfill \end{array}$$

  (42)

  $$\begin{array}{ccc}\hfill p_{\left|t=0\right.}& =& p_0\hfill \end{array}$$

  (43)

  $$\begin{array}{ccc}\hfill \frac{\delta X}{\delta t}& =& {\int }_{{\Omega }_b}\mathit{v}\left(x,t\right)d\left(\left|x-X\right|\right)dx\hfill \end{array}$$

  (44)

  $$\begin{array}{ccc}\hfill {\mathit{f}}^{IBM}\left(x,t\right)& =& {\int }_{{\Omega }_b}\mathit{F}\left(X,t\right)d\left(\left|x-X\right|\right)dX\hfill \end{array}$$

  (45)

  $$\begin{array}{ccc}\hfill \mathit{F}\left(X,t\right)& =& -\frac{\wp E}{\wp X},\hfill \end{array}$$

  (46)

  where ρ is the density of the fluid, $\mathit{v}$ is the velocity field, p is the pressure field, $\mathit{g}$ is the gravitational acceleration, $\mathit{T}$ is the stress response to the deformation of the fluid, X is the Lagrangian coordinate of the generic point of the body, and E is the elastic potential energy of the body material.
  Equations (38) and (39) are interaction equations. They convert Lagrangian variables to Eulerian variables (Equation (38)) and vice versa (Equation (39)).
• Surface boundary conditions treatment
  Since the second derivative of a discontinuous function determines the curvature $\kappa$, using front-capturing methods to compute the surface tension force requires particular considerations. Various methods were devised to precisely calculate the surface tension force. The most well-known method is presumably the continuum surface force (CSF) model [61]. The model considers surface tension force as a continuous, 3D effect across an interface, rather than as a boundary value condition on the interface. The continuum method eliminates the need for interface reconstruction. A model of a diffuse interface is examined, in which the surface tension force is converted into a volume force that distributes across several cell layers.
  We note (see Equation (20)) that surface tension ${\mathit{F}}_{sa}$ per interfacial area at a point ${\mathit{x}}_{\Gamma }$ on the interface $\Gamma$ is given by

  $${\mathit{F}}_{sa}\left({\mathit{x}}_{\Gamma }\right)=\rho \kappa \left({\mathit{x}}_{\Gamma }\right)\mathbf{N}\left({\mathit{x}}_{\Gamma }\right)$$

  (47)

  where $\rho$ is the surface tension coefficient, $\kappa$ is the curvature of the interface, and $\mathbf{N}$ is the unit normal vector outward to $\Gamma \left(t\right)$.
  Suppose that the interface $\Gamma$ where the fluid changes from fluid 1 to fluid 2 discontinuously is replaced by a continuous transition. Applying a pressure jump at an interface brought on by surface tension is no longer appropriate. Instead, as can be shown in Figure 5, surface tension should be considered active everywhere in the transition area.
  As in [61], we choose a volume force ${\mathit{F}}_{sv}\left(\mathit{x}\right)$ at any point x of transition region as

  $${\mathit{F}}_{sv}\left(\mathit{x}\right)=\left\{\begin{array}{cc}\rho \kappa \left(\mathit{x}\right)\nabla \tilde{c}\left(\mathit{x}\right)\hfill & \mathrm{for}\left|\mathbf{N}\left(x_{\Gamma }\right)\left(\mathit{x}-{\mathit{x}}_{\Gamma }\right)\right|<h\hfill \\ 0\hfill & \mathrm{for}\left|\mathbf{N}\left(x_{\Gamma }\right)\left(\mathit{x}-{\mathit{x}}_{\Gamma }\right)\right|\ge h\hfill \end{array}\right.$$

  (48)

  where $\tilde{c}\left(\mathit{x}\right)$ is a smooth “color function” that approaches the “characteristic function”, $c\left(\mathit{x}\right)$, as the width h of the region approaches to zero,

  $$\underset{h\to 0}{lim}\tilde{c}\left(\mathit{x}\right)=c\left(\mathit{x}\right),$$

  (49)

  and where the “characteristic function” can be defined as

  $$c\left(\mathit{x}\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{for}x\in {\Omega }_1\hfill \\ 0\hfill & \mathrm{for}x\in {\Omega }_2\hfill \\ \frac{1}{2}\hfill & \mathrm{for}x\in \Gamma \hfill \end{array}\right.,$$

  (50)

  where ${\Omega }_1$ and ${\Omega }_2$ represent the domains occupied by fluids 1 and 2, respectively. We note that for the curvature $\kappa$ of the interface $\Gamma$ and for the unit normal vector $\mathbf{N}$ outward to $\Gamma$, the following equations are valid:

  $$\begin{array}{ccc}\hfill \mathbf{N}& =& -\frac{\nabla \tilde{c}}{\left|\nabla \tilde{c}\right|}\hfill \end{array}$$

  (51)

  $$\begin{array}{ccc}\hfill \kappa & =& -div\left(\frac{\nabla \tilde{c}}{\left|\nabla \tilde{c}\right|}\right)\hfill \end{array}$$

  (52)
  The function $\tilde{c}\left(\mathit{x}\right)$ can be defined as a convolution of the function $c\left(\mathit{x}\right)$ with an interpolation function $\mathcal{J}$ according to Brackbill et al. [61]:

  $$\tilde{c}\left(\mathit{x}\right)=\frac{1}{h^3}{\int }_{V}c\left(\mathit{y}\right)\mathcal{J}\left(y-x\right)dy$$

  (53)

  where $\mathcal{J}$ has to satisfy the following conditions

  (a)

  ${\int }_V\mathcal{J}\left(x\right)dx=h^3$,

  (b)

  $\mathcal{J}\left(x\right)=0for\left|x\right|\ge h/2$,

  (c)

  $\mathcal{J}\left(x\right)$ is differentiable and decreases monotonically with increasing $\left|x\right|$.

  Volume force ${\mathit{F}}_{sv}\left(\mathit{x}\right)$ will result in the same total force as ${\mathit{F}}_{sa}\left(x_{\Gamma }\right)$, but spread over the finite interface width, i.e.,

  $$\underset{h\to 0}{lim}{\int }_{∆V}{\mathit{F}}_{sv}\left(\mathit{x}\right)d{x}^n={\int }_{∆A}{\mathit{F}}_{sa}\left({\mathit{x}}_{\Gamma }\right)dA$$

  (54)
  Coupling the continuum surface force (CSF) model with problems related to two-phase Navier–Stokes incompressible flow requires the reformulation of the Problem 1 as follows:

  Problem 4. 

  Equations of two-phase Navier–Stokes incompressible flow defined in $\Omega \times \Theta$ using the CSF model:

  $$\begin{array}{ccc}\hfill \frac{\partial }{\partial t}\left(\rho \mathit{v}\right)+\left(\rho \mathit{v}\right)\odot \nabla \mathit{v}& =& -\nabla p+div\mathit{T}+\rho \mathit{g}+{\mathit{F}}_{sv}\left(\mathit{x}\right)\hfill \end{array}$$

  (55)

  $$\begin{array}{ccc}\hfill div\mathit{v}& =& 0\hfill \end{array}$$

  (56)

  $$\begin{array}{ccc}\hfill {\mathit{v}}_{\left|t=0\right.}& =& {\mathit{v}}_0\hfill \end{array}$$

  (57)

  $$\begin{array}{ccc}\hfill p_{\left|t=0\right.}& =& p_0\hfill \end{array}$$

  (58)

  where $\mathit{v}$ is the velocity field, p is the pressure field, $\mathit{g}$ is the gravitational acceleration, and $\rho \left(x\right)$ and $\mathit{T}\left(x\right)$ are the piecewise constant fluid densities and stress tensors described by the Heaviside function.

  The following equations hold [61]:

  $$\begin{array}{ccc}\hfill \mathit{T}\left(x\right)& =& {\mathit{T}}_1{H}_1\left(x\right)+{\mathit{T}}_2\left(1-H_1\left(x\right)\right)\hfill \end{array}$$

  (59)

  $$\begin{array}{ccc}\hfill \rho \left(x\right)& =& {\rho }_1{H}_1\left(x\right)+{\rho }_2\left(1-H_1\left(x\right)\right)\hfill \end{array}$$

  (60)

  $$\begin{array}{ccc}\hfill H_1\left(x\right)& =& \left\{\begin{array}{cc}1\hfill & forx\in {\Omega }_1\left(t\right)\hfill \\ 0\hfill & forx\notin {\Omega }_1\left(t\right)\hfill \end{array}\right.\hfill \end{array}$$

  (61)

  Note that the term ${\mathit{F}}_{sv}\left(\mathit{x}\right)$ (as defined in Equation (48)) is added to Equation (55) to model the surface tension forces [61].

5.1.3. Transport and Response of Biological and Chemical Species

This subsection will describe models that are used to study how the tissue reacts to the chemical and mechanical environment in the culture medium. Although there are not many studies on this topic, this aspect is crucial for models that seek to create better culture protocols [64]. Creating a strict framework of conservation laws for mixes of interacting continua has been the subject of countless investigations [108]. According to this theory, the characteristics of composite biological tissues are assessed in relation to the volume percentage of the constituent parts (i.e., extracellular matrix, culture medium, and cells). A system of conservation equations that control the transfer of mass and momentum between the component phases is the foundation of the multi-phase models.

For a system comprising incompressible phases, if inertial effects and body forces are negligible, then the equations governing the i-th phase may be expressed as

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial t}\left({\rho }_i{\theta }_i\right)+{\rho }_{i}div\left({\theta }_i\mathit{v}\right)-D_i{\nabla }^2{\theta }_i& =& S_i,\hfill \end{array}$$

(62)

$$\begin{array}{ccc}\hfill div\left({\theta }_i{\mathit{T}}_i\right)+\sum _{j\ne i}{\mathit{F}}_{ij}& =& \mathbf{0}\hfill \end{array}$$

(63)

where ${\rho }_i$, ${\theta }_i$, $\mathit{v}$, $D_i$, and ${\mathit{T}}_i$ are, respectively, the density, the volume fraction (a value in $\left[01\right]$ interval), the velocity, the diffusive coefficient, and the stress tensor characterizing the i-th phase, where $S_i$ is the net rate of mass transfer into the i-th phase and ${\mathit{F}}_{ij}$ denotes the force acting on phase i as a result of interactions with phase j.

The sum of all phases adds up to 1 (${\sum }_i{\theta }_i=1$) following the conservation of mass. The biological component can also be modeled as a single-phase material, for instance, cells only, where Equation (62) can be used to estimate ${\theta }_c$ as the cell density; in such a case, the conservation of mass law above does not apply; for example, Equation (63) does not apply.

Depending on the type of biological phase, diffusive and convective transport are taken into account in Equation (62). For example, cells are typically thought of as a single phase that diffuses slowly and does not undergo convection. The force balance between the various phases is captured by the equation for the conservation of momentum (see Equation (63)). Interactions between cell populations, between the cells and the scaffold, and between the cells and the media are included in Equation (63). More complex processes, including tissue evolution through cell proliferation or ECM deposition (in biology, the extracellular matrix (ECM), also called the intercellular matrix (ICM), is a network consisting of extracellular macromolecules and minerals that provide structural and biochemical support to surrounding cells [109]), can be considered by using the term $S_i$. The introduction of functions controlling cell proliferation or ECM formation is possible. Proliferation and death terms are typically included in cell growth functions, and they depend on the available cell volume fraction ${\theta }_c$ and the concentration of nutrients like oxygen. Indeed, it is widely known that the migration, differentiation, and proliferation of cells are influenced by growth factors, waste products like lactate, and nutrients like oxygen (called “chemical species”).

Mass conservation is used to describe the distribution of any solute, assuming Fickian diffusion and setting the initial and boundary conditions.

$$\begin{array}{ccc}\hfill \frac{\partial C_i}{\partial t}+div\left(C_i\mathit{v}\right)-D_i{\nabla }^2{C}_i& =& M_i,\hfill \end{array}$$

(64)

where $\mathit{v}$ is the media flow velocity, $D_i$ is the appropriate coefficient of diffusion, $M_i$ is a term indicating the metabolic reaction, and $C_i$ is the concentration of the i-th species (in, for example, oxygen).

See [64,108] for more details about models related to the transport of chemical species and the response of biological species (cells and tissues) to the chemical and mechanical environment.

5.2. Models at the Meso Scale

The proposed models are based on a stochastic approach (the Monte Carlo-based method) on a deterministic particle-dynamics-based approach (the Cellular Particle Dynamics), and on a potential hybrid stochastic–deterministic approach (the cellular automata method). In

Table 6. Summary of “Pros and Cons” of the models at the meso-scale.

Model	“Pros”	“Cons”
Monte Carlo-based methods (MCMs) (Section 5.2.1)	MCMs are a broad category of computing methods for assessing sums and integrals or estimates of energy or other macro-parameters in a physical or biological system. They are universally appreciated for their power and easy implementation beyond their versatility and flexibility.	They can only be used to assess equilibrium states.
The cellular automata method (Section 5.2.3)	An approach where the behaviors of the species under study are determined by rules which can be probabilistic or deterministic. CA could be particularly significant in managing the behaviors of systems where the traditional, differential-equation-based methodologies are either useless or difficult to adopt due to the numerous complex interconnections between the system constituents.	They can only be used to assess equilibrium states. Either synchronous or asynchronous rule applications lead to CA development but serious practical implementations limit the use of CA to an asynchronous rule application.
The cellular particle dynamics (CPD) (Section 5.2.2)	CPD is based on the idea of constructing a cell from a cloud of sub-cellular entities called elements. All the elements, through their relative positions, and in response to cell internal and external forces, determine the dynamics of all the cells.	The definition of model parameters could be very difficult.

, we provide a summary of the “Pros and Cons” of the described models.

5.2.1. Monte Carlo-Based Methods

The collection of algorithms, based on Markov Chain Monte Carlo (MCMC) methods, are universally appreciated for their power and easy implementation beyond their versatility and flexibility. Such relevance inspired Dongarra et al. [110] to call the original Metropolis algorithm [111], which is the first implementation of an MCMC method, one of the ten most important algorithms of the twentieth century. For the story of the evolution of MCMC methods, we suggest reading the paper by Matthew Richey [112] that is also relevant for the accessible and synthetic introduction to MCMC methods. MCMC is a broad computing method for assessing sums and integrals or estimates of energy or other macro-parameters in a physical or biological system [113].

The original MCMC method, the so-called “Metropolis algorithm”, was inspired by nature and arose in physics. Its creators were aware that they had defined

...a general method, suitable for fast electronic computing machines, of calculating the properties of any substance which may be considered as composed of interacting individual molecules. [111]

The original Metropolis algorithm proposed an approach to solve the basic problem of averaging in classical equilibrium statistical mechanics. Such an approach was intended to compute averages of any physical observable F using an arithmetic mean of the values of F obtained in different sampled states of a system composed of N particles. The sampling schema described in [111] was based on an approach that later was discovered to be a Marchov Chain stochastic process [112]. Let us briefly describe some of the fundamental ideas behind the Marchov Chain.

Definition 1. 

Markov Chains

Given a finite state space $S=\left\{1,2,...,N\right\}$, a Markov Chain is a stochastic process defined by a sequence of random variables, $X_i\in S,i\in \mathbb{N}$ such that

$$Prob\left(X_{k+1}=x_{k+1}|X_1=x_1,\dots ,X_k=x_k\right)=Prob\left(X_{k+1}=x_{k+1}|X_k=x_k\right).$$

(65)

Equation (65) expresses the fact that the probability of being in a particular state at the $(k+1)$-th step only depends on the state at the k-th step. When Markov Chains are considered for which this dependence is independent of k (that is, time-homogeneous Markov Chains), a $N\times N$ transition matrix $\mathit{P}=\left({\mathit{p}}_{i,j}\right)$ can be defined by

$${\mathit{p}}_{i,j}=Prob\left(X_{k+1}=j|X_k=i\right).$$

(66)

Note that

• ${\sum }_{i=1}^N{\mathit{p}}_{i,j}=1$.
• The $(i,j)$-entry of the K-th power of $\mathit{P}$ gives the probability of transitioning from state i to state j in K steps.

A Markov Chain is

• Irreducible if, for all states i and j, there exists K such that ${\left({\mathit{P}}^K\right)}_{i,j}\ne 0$.
• A-periodic if, for all states i and j, the Greatest Common Divisor of the set $\left\{K:{\left({\mathit{P}}^K\right)}_{i,j}>0\right\}$ is equal to 1.

An irreducible, a-periodic Markov Chain must have a unique distribution $\pi$ on the state space S with the property that

$$\pi =\pi \mathit{P}.$$

(67)

where $\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_N\right)$ and where ${\pi }_i$ is the probability of state i.

If Equation (67) is valid, the Markov Chain is said to be “stable on the distribution” $\pi$, or that $\pi$ is the stable distribution for the Markov Chain. The following two considerations are valid:

• If $\pi$ is the stable distribution for an irreducible, a-periodic Markov Chain, then the Markov Chain can be used to sample from $\pi$.
• Samples from $\pi$ can be used to approximate the properties of $\pi$. For example, suppose f is any real-valued function on the state space S, and suppose that $s_1,s_2,\dots ,s_M$ is a sample from $\pi$; then, the ergodic theorem [114] ensures that

  $$\underset{M\to \infty }{lim}\frac{1}{M}\sum _{i=1}^{M}f\left(s_i\right)=E_{\pi }\left[f\right],$$

  (68)

  where $E_{\pi }\left[f\right]$ is the expected value $E_{\pi }\left[f\right]={\sum }_{i=1}^{N}f\left(i\right){\pi }_i$.

So, if an irreducible, a-periodic Markov Chain can be built that is stable on $\pi$, that chain can be used to generate states from S that can be exploited to approximate, by Equation (68), the expected value $E_{\pi }\left[f\right]$ of any real-valued function on the state space S. The original Metropolis algorithm builds a Markov Chain that is stable on a particular type of distribution ${\pi }^{Boltzmann}$ that is the “Boltzmann distribution”.

Indeed, in the context of statistical mechanics, which is the branch of physics concerned with the average behavior of large systems of interacting particles, the Metropolis algorithm was introduced to study the properties of ${\pi }^{Boltzmann}$. In statistical mechanics, the state of the particles is described by a configuration $\omega$ taken from the configuration space $\Omega$. The physics of a configuration space is described by an energy function $E:\omega \in \Omega \leftarrow E\left(\omega \right)\in {\Re }^{+}$. Then, the value $E\left(\omega \right)$ is said to be the energy of the configuration $\omega$. The random organization of molecules in a limited space is governed by the fundamental principle of statistical physics assuming that Nature seeks low-energy configurations. For any $\omega \in \Omega$, its Boltzmann probability, ${\pi }^{Boltzmann}\left(\omega \right)$, is

$${\pi }^{Boltzmann}\left(\omega \right)=\frac{BoltzmannWeight\left(\omega \right)}{Z},$$

(69)

where T is the temperature and k is the Boltzmann’s constant,

$$BoltzmannWeight\left(\omega \right)=e^{-\frac{E\left(\omega \right)}{kT}}$$

(70)

and where Z is known as “Partition Function”, defined as

$$Z=\sum _{{\omega }^{\prime }}{e}^{-\frac{E\left({\omega }^{\prime }\right)}{kT}}.$$

(71)

The relationship between energy and probability leads to expressions for many interesting physical quantities. For example, the total energy of the system, $〈E〉$, is the expected value of the energy function $E\left(\omega \right)$ and is defined by

$$〈E〉=\sum _{\omega \in \Omega }E\left(\omega \right){\pi }^{Boltzmann}\left(\omega \right).$$

(72)

The brilliant idea behind the Metropolis algorithm is that it creates an easily computed Markov Chain that is stable on the Boltzmann distribution. The making of such a chain requires only the evaluation of the ratio between the values of Boltzmann probabilities of two consecutive states (i.e., see algorithm described in Section 6.2.1 where $g\left(·\right)={\pi }^{Boltzmann}\left(·\right)$) that is

$$\frac{{\pi }^{Boltzmann}\left(\omega \right)}{{\pi }^{Boltzmann}\left({\omega }^{\ast }\right)}=\frac{BoltzmannWeight\left(\omega \right)}{BoltzmannWeight\left({\omega }^{\ast }\right)}=e^{-\frac{E\left(\omega \right)-E\left({\omega }^{\ast }\right)}{kT}}$$

and not the full probabilities Z, avoiding the evaluation of the partition function which is analytically and computationally intractable in any realistic setting. All the steps in the Markov Chain can be computed easily, if the value of $E\left(\omega \right)$ is easily computable, and, most importantly, the value of $∆E=E\left(\omega \right)-E\left({\omega }^{\ast }\right)$. In many settings, E is extremely simple to compute; often, it is independent of $\left|\Omega \right|$ (the energy could reflect just the influence that neighboring particles exert on each other).

Finally, thanks to the use of the Metropolis algorithm, a sample ${\left\{{\omega }_i\right\}}_{i=1,\dots ,M}$ of $\Omega$ can be generated that can be used to compute an approximation (see Equation (68)) $\tilde{E}$ of $〈E〉$ by the following

$$\tilde{E}=\frac{1}{M}\sum _{i=1}^{M}E\left({\omega }_i\right).$$

(73)

Moreover, since naturally physical systems “seek” the state of minimal value for their energy, the succession of states ${\left\{{\omega }_i\right\}}_{i=1,\dots ,M}$ “converges” toward the equilibrium state characterizes the minimum energy state.

The generalization of the Metropolis algorithm is due to Hastings [115]. To generalize the Metropolis method for a given distribution $\pi$, a Markov Chain $P={\left(p_{i,j}\right)}_{i,j=1,\dots ,N}$ should be defined with $\pi$ as its stationary distribution.

Hastings assumes that $p_{i,j}$ has the form

$$p_{i,j}=\left\{\begin{array}{cc}{q}_{i,j}{\alpha }_{i,j}& \mathrm{if}i\ne j\hfill \\ 1-\sum _{k\ne i}{p}_{i,k}& \mathrm{if}i=j\hfill \end{array}\right.,$$

(74)

where $Q={\left(q_{i,j}\right)}_{i,j=1,\dots ,N}$ is the proposed transition matrix of an arbitrary Markov Chain on a finite state space S, and ${\alpha }_{i,j}$ is given by

$${\alpha }_{i,j}=\frac{s_{i,j}}{1+\frac{{\pi }_i}{{\pi }_j}\frac{q_{i,j}}{q_{j,i}}},$$

(75)

where $s_{i,j}$ is a symmetric function of i and j and is chosen so that $0\le {\alpha }_{i,j}\le 1$ for all $i,j=1,\dots ,N$. For such choices for $s_{i,j}$, the matrix P satisfies the following  reversibility condition

$${\pi }_i{p}_{i,j}={\pi }_j{p}_{j,i}.$$

(76)

From Equation (76), it follows that

$$\sum _{i=1,\dots ,N}{\pi }_i{p}_{i,j}={\pi }_j,\mathrm{for}\mathrm{all}j=1,\dots ,N,$$

(77)

and hence that $\pi$ is a stationary distribution of P.

The following two examples for $S={\left(s_{i,j}\right)}_{i,j=1,\dots ,N}$ are given:

$$\begin{array}{cc}{S}^{\left(M\right)}={\left(s_{i,j}^{\left(M\right)}\right)}_{i,j=1,\dots ,N},& \mathrm{where}{s}_{i,j}^{\left(M\right)}=\left\{\begin{array}{cc}1+\frac{{\pi }_i}{{\pi }_j}\frac{q_{i,j}}{q_{j,i}}& \left(\mathrm{if}\frac{{\pi }_i}{{\pi }_j}\frac{q_{i,j}}{q_{j,i}}\ge 1\right)\\ 1+\frac{{\pi }_j}{{\pi }_j}\frac{q_{j,i}}{q_{i,j}}& \left(\mathrm{if}\frac{{\pi }_i}{{\pi }_j}\frac{q_{i,j}}{q_{j,i}}<1\right)\end{array}\right.\hfill \\ S^{\left(B\right)}={\left(s_{i,j}^{\left(B\right)}\right)}_{i,j=1,\dots ,N},& \mathrm{where}{s}_{i,j}^{\left(B\right)}=1.\hfill \end{array}$$

(78)

With $q_{i,j}=q_{j,i}$ and S defined as $S^{\left(M\right)}$, a method devised by Metropolis et al. is obtained. If $q_{i,j}=q_{j,i}$ and S is defined as $S^{\left(B\right)}$, another classical MCMC method from Barker [116] is recalled.

In Section 6.2.1, the algorithm implementing a Metropolis–Hastings method is described.

5.2.2. Cellular Particle Dynamics

In the previous subsection, MCMC methods were introduced. While it is incredibly helpful for various statistical physics and biology problems, there are some situations that might not even be amenable to stochastic approaches [49,117].

CPD is based, as in the Subcellular Element Model (SEM) approach [118], on the idea to “construct a cell from a cloud of $N_{CellularElements}$ sub-cellular entities” called elements [49,118]. The elements, belonging to a single cell, all together define the same cell and, through their relative positions and in response to the cell’s internal and external forces, determine the cell’s shape (see Figure 6).

The SEM approach (that takes place in a Lagrangian framework) is based on a stochastic assumption using Langevin equations [119]. The stochastic variables are the position vectors ${\mathit{y}}_{{\alpha }_i}$ of each element, where the notation ${\alpha }_i$ denotes the $\alpha$-th element of the i-th cell, and where ${\mathit{y}}_{{\alpha }_i}=\left(y_{{\alpha }_i}^1,\dots ,y_{{\alpha }_i}^N\right)\in {\Re }^N$.

In the simplest case, in which the inertial terms in the equation of motion are discarded and in which the movement of elements is due to stochastic forces and biomechanical interactions, the following equation is valid for the generic element ${\alpha }_i$:

$$\mu \frac{\partial {\mathit{y}}_{{\alpha }_i}}{\partial t}={\mathit{\eta }}_{{\alpha }_i}\underset{\mathrm{Internal}\mathrm{forces}{\mathit{F}}_{{\alpha }_i}^{Intra}}{\underbrace{-{\nabla }_{{\mathit{y}}_{{\alpha }_i}}\left[\sum _{{\beta }_i\ne {\alpha }_i}{V}_{intra}\left(\left|{\mathit{y}}_{{\alpha }_i}-{\mathit{y}}_{{\beta }_i}\right|\right)\right]}}\underset{\mathrm{External}\mathrm{forces}{\mathit{F}}_{{\alpha }_i}^{Inter}}{\underbrace{-{\nabla }_{{\mathit{y}}_{{\alpha }_i}}\left[\sum _{j\ne i}\sum _{{\beta }_j}{V}_{inter}\left(\left|{\mathit{y}}_{{\alpha }_i}-{\mathit{y}}_{{\beta }_j}\right|\right)\right]}}$$

(79)

where $\mu$ is the friction coefficient, ${\mathit{\eta }}_{{\alpha }_i}=\left({\eta }_{{\alpha }_i}^1,\dots ,{\eta }_{{\alpha }_i}^N\right)\in {\Re }^N$ is a stochastic force modeled as a Gaussian white noise with zero mean and variance $〈{\eta }_{{\alpha }_i}^n\left(t\right),{\eta }_{{\beta }_j}^m\left(t^{\prime }\right)〉$ with

$$〈{\eta }_{{\alpha }_i}^n\left(t\right),{\eta }_{{\beta }_j}^m\left(t^{\prime }\right)〉=2D{\delta }_{n,m}{\delta }_{i,j}{\delta }_{{\alpha }_i,{\beta }_j}{\delta }_{t,t^{\prime }},$$

(80)

where D is the self-diffusion coefficient. The Equation (80) indicates that stochastic forces are uncorrelated in all the possible variables: between cells, between elements within a cell, between spatial dimensions, and between two different instants. The vectors ${\mathit{F}}_{{\alpha }_i}^{Intra}=\left({F^{Intra}}_{{\alpha }_i}^1,\dots ,{F^{Intra}}_{{\alpha }_i}^N\right)\in {\Re }^N$ and ${\mathit{F}}_{{\alpha }_i}^{Inter}=\left({F^{Inter}}_{{\alpha }_i}^1,\dots ,{F^{Inter}}_{{\alpha }_i}^N\right)\in {\Re }^N$ represent the internal and external biomechanical forces, respectively, acting on the generic element ${\alpha }_i$.

The functions $V_{intra}\left(d_{{\alpha }_i,{\beta }_j}\right)$ and $V_{inter}\left(d_{{\alpha }_i,{\beta }_j}\right)$, which depend on the distance

$$d_{{\alpha }_i,{\beta }_j}=\left|{\mathit{y}}_{{\alpha }_i}-{\mathit{y}}_{{\beta }_j}\right|={\left(\sum _{n=1}^N{\left(y_{{\alpha }_i}^n-y_{{\beta }_j}^n\right)}^2\right)}^{\frac{1}{2}}$$

between two elements, represent the intra- and inter-cellular interaction potential between two elements in the same and different cells, respectively. In [49], Flenner et al. propose the following expression for functions $V_{intra}\left(d\right)$ and $V_{inter}\left(d\right)$

$$\begin{array}{ccc}\hfill V_{intra}\left(d\right)& =& V_{LJ}\left(d,ϵ,\rho \right)+V_c\left(d,k,\gamma \right),\hfill \end{array}$$

(81)

$$\begin{array}{ccc}\hfill V_{inter}\left(d\right)& =& V_{LJ}\left(d,ϵ,\rho \right),\hfill \end{array}$$

(82)

where $V_{LJ}\left(d,ϵ,\rho \right)$ is the Lennard–Jones (LJ) potential energy

$$V_{LJ}\left(d,ϵ,\rho \right)=4ϵ\left[{\left(\frac{\rho }{d}\right)}^{12}-{\left(\frac{\rho }{d}\right)}^6\right],$$

(83)

and where $V_c\left(d,k,\gamma \right)$ represents a confining potential energy of the form

$$V_c\left(d,k,\gamma \right)=\left\{\begin{array}{cc}\frac{k}{2}{\left(d-\gamma \right)}^2,& \mathrm{for}d>\beta \hfill \\ 0& \mathrm{for}d<\beta \hfill \end{array}\right..$$

(84)

By tuning the values of k and $\gamma$ in the confining potential $V_c\left(d,k,\gamma \right)$, the stiffness and the size of a cell can be controlled, respectively, while the LJ parameters $ϵ$ and $\rho$ represent the energy required to separate the elements and the size (diameter) of the elements, respectively.

5.2.3. Cellular Automata Model

The previous sections discussed a number of standard categories of physical models. These models govern the evolution of physical systems formalized in terms of coupled sets of ordinary or partial differential equations. Most conventional models are deterministic, which means that the force fields that are employed and the initial conditions of the simulations fully determine the results of running simulations. This section offers an alternative approach where the behaviors of the species under study are determined by rules, which can be probabilistic or deterministic, as opposed to forces and energy.

The foundation of this approach is the use of cellular automata (CA) [120] postulated for the first time by the mathematical physicist John von Neumann [121] and the mathematician Stanislaw Ulam [122], more than 70 years ago.

Since then, different research teams have proven that CA is particularly significant in managing the behaviors of systems where the traditional, differential-equation-based methodologies are either useless or difficult to adopt due to the numerous complex interconnections between the system constituents.

According to the mathematician Stephen Wolfram, the CA model can be defined as follows:

CA are simple mathematical idealizations of natural systems. They consist of a lattice of discrete identical sites, each site taking on a finite set of ... values. The values of the sites evolve in discrete time steps according to ... rules that specify the value of each site in terms of the values of neighboring sites. CA may thus be considered as discrete idealizations of the partial differential equations often used to describe natural systems [123].

According to Kier et al. [120], a model made up of the following parts is referred to as a cellular automaton (singular):

• A grid of individual cells.
• A group of ingredients.
• A collection of local rules dictating how the constituents behave.
• Identified starting conditions.

A simulation can be run after the model’s aforementioned elements have been defined. The system in the simulation changes through a sequence of discrete time steps, or iterations, when all of the system’s components are subjected to the model’s rules, and the system’s configuration is updated as a result.

Either synchronous or asynchronous rule application might lead to CA development. In the first scenario, every automaton cell changes on each iteration; in the second scenario, the algorithm merely processes a subset of the cells or even just one cell. One way to model the complex behavior of macroscopic systems, which arises from the interaction of all components on a small scale, is through the synchronous application of transition rules. However, there are serious practical implementation limitations associated with this approach, such as the potential for conflicts in certain situations (e.g., assigning two ingredients to move to the same empty cell).

Due to this latter factor, the majority of automata currently in development can only be implemented asynchronously [124] (see algorithm described in Section 6.2.2 for the asynchronous version of the CA simulation). The above general CA framework can be used to construct a wide range of various models.

The components of this CA model are then described in more detail.

• The grid, the cells, and the ingredients: A cellular automaton consists of a regular grid of $N_{Cells}$ cells. The grid can be in any finite number of dimensions N. Every grid cell can typically exist in a limited number $N_{States}$ of “states”, which specify the cell’s occupancy. The cell may be empty or hold a specified ingredient, which, if it is present, may be a type of particle, a specific molecule, or some other relevant thing for the topic under consideration. An illustration of a bi-dimensional CA grid with $5\times 5$ cells each, whereby some cells are occupied by ingredient A (red cells) and other cells by ingredient B (blue cells), can be found in Figure 7a.
  Movements and other actions on the grid are controlled by rules that are only dependent on the characteristics of the cells that are closest to the ingredient. The neighborhood of a cell is its immediate surroundings. The “Von Neumann neighborhood” is the most often employed neighborhood in two-dimensional CA investigations (see Figure 7d, where the blue cell neighborhood is pictured by the four red cells).
  Another common neighborhood is the “Moore neighborhood”, pictured by the red cells surrounding the blue cell in Figure 7c. Another useful neighborhood is the “extended Von Neumann neighborhood”, shown in Figure 7d by the red and green cells surrounding the blue cell.
  Every grid cell’s value will be impacted by how its neighborhoods are managed, including cells that are on the edge of the CA grid. Keeping the values in those cells constant is one approach that might be taken. Using distinct neighborhood definitions for these cells is an additional strategy. They might have fewer neighbors, but it is also possible to define more elaborate strategies for the cells that are closest to the edges.
  These cells are typically handled in a toroidal manner; for instance, in a bi-dimensional grid, one cell “goes off the top” and “enters” at the corresponding position on the bottom, and one cell “enters on” the right when it “goes off” the left (this is sometimes referred to as periodic boundary conditions in the field of partial differential equations). To illustrate this, consider taping the rectangle’s left and right sides to create a tube, and then the top and bottom edges to create a torus (see Figure 7b).
• The rules: The behaviors of the ingredients on the grid, and consequently the evolutions of the CA systems, are governed by a variety of rules. According to what is described in [69], a list of potential rule types is provided below.
  • Movement rules: Movement rules define the condition that, during an iteration, the ingredients move to neighborhood cells. These rules take several forms [120]:
    • The “breaking” probability, $P_B\left(AB\right)\in \left[0,1\right]$, defines the condition that two adjacent ingredients A and B may remain linked to each other.
    • The “joining” parameter, $J\left(AB\right)\in {\Re }^{+}$, defines the condition that, if two ingredients A and B are separated by an empty cell, ingredient A moves toward or away from ingredient B.
    • The “free-moving” probability $P_m\left(A\right)\in \left[0,1\right]$ of an ingredient A defines the ingredient’s propensity to move on the grid more rapidly or slowly.
  • Transition rules: Transition rules define the condition that, during an iteration, an ingredient will transform to some other species. These rules take several forms [120]:
    • The “simple first-order transition” probability $P_T\left(AB\right)\in \left[0,1\right]$ defines the condition that an ingredient of species A will change to species B.
    • The “reaction” probability $P_R\left(AB\to CD\right)\in \left[0,1\right]$ defines the condition that ingredients A and B will transform into ingredients C and D, respectively, in case they “encounter” each other during their movements in the grid.
  Other types of rules may also be considered [69,120].
  The critical characteristics of all these rules are their local nature, focusing solely on the ingredient in question and potentially any nearby ingredients.
  All the probabilities listed above are enforced using a random-number generator in the CA algorithm. For example, suppose we use a random-number generator which generates numbers in the interval $\left[0,1\right]$. Suppose that one of the above rule probabilities $P_{·}\left(A\right)$ is set to $P_{·}\left(A\right)=p$ and that the random-number generator generates a number r; the ingredient A can make the “move” prescribed by the rule if $r<p$ and “cannot move” otherwise.
• The initial conditions: The remaining conditions of the simulation must be determined after the grid type, size, and governing rules have been established, the latter by giving particular values to the previously mentioned parameters. These consist of (1) the types and quantities of the beginning ingredients; (2) the setup of the system’s initial state; (3) the number of simulation runs that are to be performed; and (4) the duration of the runs or the number of iterations they should contain.
  About points (3) and (4) above, it needs to be emphasized that when the CA rules are stochastic, i.e., probabilistic, each simulation run is, in effect, an independent “experiment”. This implies that the outcomes of different runs could theoretically be different. A single ingredient’s activity is typically totally unpredictable. But for the majority of cases, we will look at which collective outcome—from a run with a lot of ingredients or from a lot of runs with few ingredients—tends to show a similar pattern. As a result, two further simulation-related details must be determined: the number of independent runs that must be completed and their duration (in iterations). These numbers will be heavily influenced by the type of simulation that will be run. In certain situations, it will be preferable to let the runs continue for a sufficient amount of time in order to reach an equilibrium or steady-state condition [120].

6. Numerical Methods and Algorithms

6.1. Models at the Macro-Scale

This section will describe the methods and techniques that can be used to discretize the continuous problems introduced in Section 5.1 and then compute the solution of the discretized problem as an approximation of the continuous one. The themes can be explored further by reading [125,126,127].

6.1.1. Discretization in the Space-Time Domain

Numerical techniques called the finite-element, finite-difference, and finite-volume methods (FEM, FDM, and FVM, respectively) are used to discretize Partial Differential Equations (PDEs) into the space domain.

Nevertheless, they differ from one another in different ways, and each of them has both advantages and disadvantages. Understanding these distinctions could help to select the method most appropriate for the purposes. In the following, a description of the ideas on the basis of each of those methods is given. See [127,128,129] for other discussions about finite elements versus finite volumes versus finite differences.

• Finite Elements A strong computational method for solving differential and integral equations that come up in many applied scientific and engineering domains is the Finite Element Method (FEM) [125].
  The fundamental idea behind the FEM is to consider a given domain as a collection of basic geometric shapes called finite elements, for which the approximation functions required to solve a differential equation can be generated systematically. Indeed, the solution u of a differential equation can be approximated, on each element e, by a linear combination of unknown parameters $c_j$ and appropriately selected functions ${\varphi }_j$:

  $$u^h\left(x\right)=\sum _{i=1}^l{c}_i{\varphi }_i\left(x\right),\mathrm{on}{\Omega }^e.$$

  (85)
  For a given differential equation, it is possible to develop different finite element models, depending on the choice of a particular type of approximation method (e.g., Galerkin, weak-form Galerkin, least-squares, subdomain, collocation, and so on). The finite element model is a set of algebraic relations among the unknown parameters $c_j$ of the approximation: so, solving the problem to find an approximation $u^h$ of u requires the solution of a system of algebraic equations with the unknown parameters $c_j$.
  The major steps in the finite element formulation and analysis of a typical problem are as follows.
  • Discretization of the domain into a set of selected finite elements. This is accomplished by subdividing the given domain $\overline{\Omega }=\Omega \cup \Gamma$ into a set of subdomains $\overline{{\Omega }^e}={\Omega }^e\cup {\Gamma }^e$ where $\Gamma =\partial \Omega$ and ${\Gamma }^e=\partial {\Omega }^e$, called finite elements (see Figure 8a). The phrase “finite element” often refers to both the geometry of the element and degree (or order) of approximation: The form of the element ${\Omega }^e$ can be either triangle or quadrilateral, and the degree of interpolation over it can take on various forms, including linear and quadratic. The finite element mesh of the domain $\Omega$ is the non-overlapping total (or assembly) of all elements used to represent the actual domain, and it is represented by ${\Omega }_h$. In general, ${\Omega }_h$ may not equal the actual domain $\Omega$ because of its potentially complex geometry. Nodal points, also known as nodes, are typically taken at appropriate positions in the structure, usually to simplify the element. They are used to characterize the individual elements and subsequently the complete mesh structure (see Figure 8b).
  • Construction of a statement, often a weighted-integral in a weak-form statement according to Weighted Residual Methods (WRMs) [130], which is equivalent (in some sense) to the differential equation to be analyzed over a typical element. A general class of techniques, called the Weighted Residual Method, was created to obtain an approximate solution to the problem of the form

    $$L\left(u\left(x\right)\right)=f\left(x\right),on\Omega$$

    (86)

    where $L\left(u\right)$ is a general linear differential operator. If $u^h$ is just an approximation of the true solution function u of (86), then an error or residual will exist such that

    $$R\left(x\right)=L\left(u^h\left(x\right)\right)-f\left(x\right)\ne 0$$

    (87)
    The idea at the basis of WRM is to force the residual to zero in some average sense over the domain $\Omega$, namely,

    $${\int }_{\Omega }R\left(x\right)w_i\left(x\right)dx=0,i=1,\dots ,l,$$

    (88)

    where $w_i\left(x\right)$ values are the so-called weight functions. Depending upon the nature of the weight function, different types of Weighted Residual Methods can be used. The Point Collocation Method, Subdomain Collocation Method, Least Square Method, and Galerkin Method are a few of the common ones. The trial functions themselves are selected as the weight functions in the Galerkin variant of the Weighted Residual Method. So, in the Galerkin method, we set $w_i={\varphi }_i,\forall i=1,\dots ,l$ and the weak-form Galerkin model, for each element e is the following set of algebraic relations obtained from (88) and (85):

    $$\sum _{j=1}^l{c}_j\left[{\int }_{\Omega }L\left({\varphi }_j\left(x\right)\right){\varphi }_i\left(x\right)dx\right]={\int }_{\Omega }f\left(x\right){\varphi }_i\left(x\right)dx,i=1,\dots ,l.$$

    (89)

    or, in matrix form,

    $${ }^e\mathit{M}{ }^e\mathit{c}={ }^e\mathit{f},$$

    (90)

    where

    $$\begin{array}{ccc}\hfill { }^e\mathit{M}& =& {\left[{\int }_{\Omega }L\left({\varphi }_j\left(x\right)\right){\varphi }_i\left(x\right)dx\right]}_{i,j=1,\dots ,n},\hfill \\ \hfill { }^e\mathit{c}& =& {\left[c_1,\dots ,c_l\right]}^T,\hfill \\ \hfill { }^e\mathit{f}& =& {\left[{\int }_{\Omega }f\left(x\right){\varphi }_i\left(x\right)dx\right]}_{i=1,\dots ,l}^T,\hfill \end{array}$$

    are, respectively, an $l\times l$ matrix and two vectors of length l. We assume that the vectors ${ }^e\mathit{f}$ also include both the “essential” and the “natural” boundary conditions (see [125] for details). The drawback of this model for second- and higher-order differential equations is that the approximation functions should be differentiable as many times as the actual solution $u\left(x\right)$.
    The weak form in Equation (86) requires that the approximation chosen for $u\left(x\right)$ should be at least linear in x. In addition, it requires that $u\left(x\right)$ be made continuous across the elements. The approximation functions ${\varphi }_i{\left(x\right)}_{i=1,\dots ,l}$, in each element e, are chosen to have the so-called interpolation property, that is,

    $${\varphi }_i\left({ }^e{x}_j\right)={\delta }_{ij},\forall i,j=1,\dots ,l,$$

    (91)

    where ${\left\{{ }^e{x}_j\right\}}_{j=1,\dots ,l}$ are the nodal points of the element e. The scalar ${\delta }_{ij}$ is defined as follows:

    $${\delta }_{ij}=\left\{\begin{array}{ccc}1& \mathrm{if}\hfill & i=j\hfill \\ 0& \mathrm{if}\hfill & i\ne j\hfill \end{array}\right.$$
    Then, the values of $c_j$ in Equation (85) coincide with the values that $u^h$ takes on at the same nodal points:

    $$c_j=u^h\left({ }^e{x}_i\right).$$
    Therefore, solving the Equation (90) allows one to identify the values that the approximation $u^h$ takes on in the nodal points of each element e. Functions that satisfy the interpolation property (91) are known as the Lagrange interpolation functions. The shape and number of nodes of the element are determined by the number l of linearly independent terms in the representation of $u^h$. Not all geometric shapes qualify as finite element domains. It turns out that triangle- and quadrilateral-like shapes, with an appropriate number of nodes, qualify as elements.
  • Finite element assembly to produce the global system of algebraic equations of the form

    $${ }^g\mathit{M}{ }^g\mathit{c}={ }^g\mathit{f},$$

    (92)

    where ${ }^g\mathit{M}$, ${ }^g\mathit{c}$, and ${ }^g\mathit{f}$ are, respectively, am $N_{nodes}\times N_{nodes}$ matrix and two vectors of length $N_{nodes}$ and where $N_{nodes}$ denotes the number of the nodes (the global grid node) of all the $N_{elements}$ elements composing the finite element mesh. Therefore, the assembly procedure represents the realization of the global displacement boundary conditions as well as the inter-element compatibility of the approximation functions and the displacement field. Indeed, each global grid node is shared by different finite elements which all contribute to the solution of the unknown values ${ }^g\mathit{c}$ on the involved nodes. Let us denote with

    $${\left\{{ }^g{x}_I\right\}}_{I=1,\dots ,N_{nodes}}$$

    the set of such global grid nodes; then, a transformation $\tau \left(i,e\right)$ exists that “maps” the “local” index i of the i-th node of the element e into the “global” index I of the same node (see Figure 8c). Thanks to the validity of the “superposition” property in FEM context due to the linear nature of the problem, for the global matrix ${ }^g\mathit{M}$ and the global vector ${ }^g\mathit{f}$, the following equalities are valid:

    $$\begin{array}{ccc}\hfill { }^g{\mathit{M}}_{IJ}& =& {\sum }_{e=1}^{N_{nodes}}\left({\sum }_{\begin{array}{cc}i:\hfill & \tau \left(i,e\right)=I\hfill \\ j:\hfill & \tau \left(j,e\right)=J\hfill \end{array}}{{ }^e\mathit{M}}_{ij}\right)\hfill \end{array}$$

    (93)

    $$\begin{array}{ccc}\hfill { }^g{\mathit{f}}_I& =& {\sum }_{e=1}^{N_{nodes}}\left({\sum }_{\begin{array}{cc}i:\hfill & \tau \left(i,e\right)=I\hfill \end{array}}{{ }^e\mathit{f}}_i\right)\hfill \end{array}$$

    (94)
  If u is a function defined both in space and time and the linear differential operator $L\left(u\right)$ has the following form:

  $$L\left(u\left(x\right)\right)=\frac{\partial u}{\partial t}+E\left(u\left(x\right)\right),$$

  (95)

  where $E\left(u\left(x\right)\right)$ does not contain terms with differentiation with respect to the time variable t, then Equation (92) can be rewritten as:

  $${ }^g\overline{M} { }^g\dot{\mathit{u}}+{ }^g\overline{K}\left({ }^g\mathit{u}\right){ }^g\mathit{u}={ }^g\mathit{f},$$

  (96)

  where $\mathit{u}$ is a vector of the values assumed by the approximation $u^h$ in all the nodal points of the finite elements mesh and where the symbol $\dot{u}$ denotes the partial derivative of the function u respect to time t: $\dot{u}=\frac{\partial u}{\partial t}$. The nonlinear Equation (96) represents an approximation to the original system of partial differential equations, which is discrete in space and continuous in time.
  See [131,132,133,134] for applications of FEM to some viscoelastic problems. See [125,135,136] for details about the Finite Element Method, including those details related to how the integrals used by these methods are computed.
• Finite Differences [127] The basic idea of FDM is to compute an approximation $u^h$ of the solution u of (86) by mean an approximation of all the differential operators defining $L\left(u\left(x\right)\right)$. As for FEM, the major steps in the finite element formulation and analysis of a typical problem are:
  • Discretization of the space domain into a set of nodes ${\left\{x_I=\left[x_1^I,\dots ,x_n^I\right]\right\}}_{I=1,\dots ,N_{nodes}}$. This is accomplished by subdividing the given domain $\overline{\Omega }=\Omega \cup \Gamma$ by mean of a grid of nodes which represent the finite difference mesh (see Figure 9a).
  • Construction of approximation statements, based on a finite difference, for all the differential operators present in $L\left(u\left(x\right)\right)$. Thanks to the use of Taylor expansion, the validity of the following relations can be demonstrated [137]:

    $$\begin{array}{cc}\hfill \frac{{\partial }^{n}u}{\partial {x_i}^n}\left(x\right)& =\frac{{ }_i{∆}_h^n\left[u\right]\left(x\right)}{h^n}+O\left(h\right)\hfill \\ \hfill & =\frac{{ }_i{\nabla }_h^n\left[u\right]\left(x\right)}{h^n}+O\left(h\right)\hfill \\ \hfill & =\frac{{{ }_i\kappa }_h^n\left[u\right]\left(x\right)}{h^n}+O\left(h^2\right),\hfill \end{array}$$

    (97)

    where ${{ }_i∆}_h^n\left[u\right]\left(x\right)$, ${{ }_i\nabla }_h^n\left[u\right]\left(x\right)$ and ${{ }_i\kappa }_h^n\left[u\right]\left(x\right)$ are, respectively, named the n-th order forward, backward, and central differences with respect to $x_i$ and are given by, respectively.
    • Forward difference

      $${ }_i{∆}_h^n\left[u\right]\left(x\right)=\sum _{j=0}^n{\left(-1\right)}^{n-j}\left(\begin{array}{c}n\\ j\end{array}\right)u\left(x+jh{\mathit{\delta }}_j\right)$$

      (98)
    • Backward difference

      $${ }_i{\nabla }_h^n\left[u\right]\left(x\right)=\sum _{j=0}^n{\left(-1\right)}^j\left(\begin{array}{c}n\\ j\end{array}\right)u\left(x-jh{\mathit{\delta }}_j\right)$$

      (99)
    • Central difference

      $${{ }_i\kappa }_h^n\left[u\right]\left(x\right)=\sum _{j=0}^n{\left(-1\right)}^j\left(\begin{array}{c}n\\ j\end{array}\right)u\left(x+\left(\frac{n}{2}-j\right)h{\mathit{\delta }}_j\right)$$

      (100)

      where ${\mathit{\delta }}_j={\left[{\delta }_{ij}\right]}_{i=1,\dots ,n}$.
  • By substituting one of the above approximations to each space derivative operator in $L\left(u\left(x\right)\right)$, we obtain the so-called space approximated operator $L_h\left(u\left(x\right)\right)$, which can be written generally as a nonlinear algebraic operator as follows:

    $$L_h\left(u\left(x\right)\right)=\sum _j^m{a}_j\left(u\right)u\left(x+\alpha jh{\mathit{\delta }}_j\right)$$

    (101)

    where $\alpha$ and m depend on the type of approximation chosen.
  • The set of the following $N_{nodes}$ algebraic equations

    $$L_h\left(u\left(x_I\right)\right)=f\left(x_I\right),I=1,\dots ,N_{nodes}$$

    (102)

    which are obtained by evaluating the equation $L_h\left(u\left(x\right)\right)=f\left(x\right)$ in each node of finite difference mesh can be written, after the inclusion of boundary conditions, in matrix form as

    $$\mathit{M}\mathit{u}=\mathit{z},$$

    (103)

    where $\mathit{M}$, $\mathit{u}$, and $\mathit{z}$ are, respectively, a matrix and two vectors whose generic elements are

    $$\begin{array}{ccc}\hfill {\mathit{M}}_{i,j}& =& a_j\left(u\left(x_i\right)\right)\hfill \\ \hfill {\mathit{z}}_i& =& z\left(x_i\right)\hfill \\ \hfill {\mathit{u}}_i& =& u_h\left(x_i\right)\hfill \end{array}$$

    where $i,j=1,\dots ,N_{nodes}$. Therefore, solving Equation (103) allows us to identify the values at which the approximation $u_h$ takes on the nodal points of the finite element mesh. It is obvious that the accuracy of $u_h$ depends on the type of approximation chosen for the derivative operators that are present in $L\left(u\left(x\right)\right)$.
• Finite Volumes [127,138] A space discretization technique called the finite volume approach works well for numerical modeling of several kinds of conservation laws that are represented by PDEs of the form (86) where the linear differential operator $L\left(u\right)$ can be expressed as

  $$L\left(u\left(x\right)\right)=div\mathit{F}\left(u\left(x\right)\right).$$

  (104)

  As for FEM and FDM, the major steps in the finite element formulation and analysis of a typical problem are
  • Discretization of the domain into a set of selected subdomains. This is accomplished by subdividing the given domain $\overline{\Omega }=\Omega \cup \Gamma$ into a set of $N_{cells}$ subdomains $\overline{{\Omega }^c}={\Omega }^c\cup {\Gamma }^c$ where $\Gamma =\partial \Omega$ and ${\Gamma }^c=\partial {\Omega }^c$, called control volumes or cells (see Figure 9b), forming the finite volume mesh. Planar surfaces in 3D or straight edges in 2D define the boundaries of the control volumes. Consequently, flat faces or straight edges are used to approximate the bounding surface, if it is curved. Cell faces, or just faces, are the names given to these bounding discrete surfaces.
  • Construction of approximation statements needed to identify the approximation function $u^h$ for the actual solution u of (104). Following the approach described for the FEM framework and based on Weighted Residual Methods, from (88), where the weight functions $w_i\left(x\right)$ are chosen to be constant and equal to identity, it follows that

    $${\int }_{{\Omega }^c}div\mathit{F}\left(u\left(x\right)\right)dx={\int }_{{\Omega }^c}f\left(x\right)dx$$

    (105)

    for each cell ${\Omega }^c$ of the finite volume mesh. By using the Divergence Theorem, Equation (105) can be rewritten as

    $${\oint }_{{\Gamma }^c}\mathit{F}\left(u\left(x\right)\right){\mathit{n}}_{{\Gamma }^c}d\gamma ={\int }_{{\Omega }^c}f\left(x\right)dx.$$

    (106)
    Thanks to the use of mean value theorem, in each subdomain ${\Omega }^c$, Equation (106) can be re-written as

    $${\oint }_{{\Gamma }^c}\mathit{F}\left(u\left(x\right)\right){\mathit{n}}_{{\Gamma }^c}d\gamma =\left|{\Omega }^c\right|{\overline{f}}_{{\Omega }^c},$$

    (107)

    where ${\overline{f}}_{{\Omega }^c}$ are the mean values respectively of functions f in subdomain ${\Omega }^c$ (which can be considered as values of f in cell center $x^c$) and where the symbol $\left|·\right|$ is used to represent the volume/area/length of respectively a 3D/2D/1D space domain.
    One can substitute the integral on the left side of Equation (107) with a summation over the faces enclosing the c-th cell, resulting in

    $$\sum _{f=1,\dots ,N_f^c}{\oint }_{{\Gamma }_f^c}\mathit{F}\left(u\left(x\right)\right){\mathit{n}}_{{\Gamma }_f^c}d\gamma =\left|{\Omega }^c\right|{\overline{f}}_{{\Omega }^c},$$

    (108)

    where $N_f^c$ is the number of faces of the c-th cell and ${\Gamma }_f^c$ represents the f-th face of the c-th cell. Details about the forms for the restriction of $u^h$ on each ${\Omega }^c$ are available in the literature (i.e., see [139,140]), but, for the sake of semplicity, assume, as in [139], that, on each cell, $u^h$ is a constant function whose value is $\overline{u^c}$; then, Equation (108) can be rewritten as

    $$\sum _{f=1,\dots ,N_f^c}{\oint }_{{\Gamma }_f^c}{\mathit{F}}^h\left(\overline{u^c}\right){\mathit{n}}_{{\Gamma }_f^c}d\gamma =\left|{\Omega }^c\right|{\overline{f}}_{{\Omega }^c},$$

    (109)

    where $F^h$ is an approximation of F that can be defined using a variety of approaches. For this purpose, Taylor series expansions have historically been employed [141]. The end result of approximating F by $F^h$ in terms of cell center values (that is, the constant values $\overline{u^c}$), followed by substitution into Equation (109) and by the boundary conditions incorporation, is a set of discrete linear algebraic equations of the form

    $$\mathcal{F}\mathit{u}=\mathit{h}$$

    (110)

    where $\mathcal{F}$, $\mathit{u}$, and $\mathit{h}$ are, respectively, an $N_{cells}\times N_{cells}$ matrix and two $N_{cells}$ vectors. The solution u of (110) allows us to calculate the values of $u_h\left(x^c\right)$ for all the $N_{cells}$ representing the finite volume mesh.
  For further information, we recommend reading [138,139,141].

The same considerations about partial differential equations that are discrete in space and continuous in time, which were already made for finite elements, can be repeated in the case of finite difference and finite volume discretizations. An attempt at comparison between the different discretization methods, according to [127,128,129], can lead to the considerations listed in

Table 7. Comparison, from the numerical point of view, between space discretization methods in terms of their “Pros and Cons”. The table contains, by way of example, references to the literature already introduced in Section 3 and regarding the use of discretization methods (where such information is available).

Discretization Methods	“Pros”	“Cons”	Refs.
Finite Element Method (FEM)	Possibility to increase the accuracy of the solution around a “critical” part of the domain: – By “increasing the order of the elements”, approximating the physics fields with higher order polynomials, – By adaptive mesh refinement. The finer the mesh, the more accurate approximation can be obtained. Irregular domain geometries can be considered.	The mathematics needs are advanced.	[11,30,33,35]
Finite Difference Method (FDM)	Increased “approximation order” and accuracy are made simple on each dimension basis. Space domain with box-shaped geometry can be discretized by a regular grid, useful for very large-scale simulations.	Managing curved boundaries or material discontinuities might be challenging.	[46]
Finite Volume Method (FVM)	Only the cell borders’ flux evaluation needs to be completed. Helpful for nonlinear problems (transport problems). Refining the mesh can improve local accuracy (around a corner of interest).	The functions that approximate the solution cannot be easily made of higher order.	[31]

. Table 7 also contains, by way of example, references to the literature already introduced in Section 3 and regarding the use of discretization methods (where such information is available).

It is noteworthy that each of the space discretization methods described above can be re-formulated via the other ones (i.e., see [127]).

6.1.2. Solution of the “Discrete in Space” Model

• The Time Integration algorithm The proposed space discretization methods allow the construction of similar discrete problems since they all lead to a generally non-linear system of algebraic equations in the following form:

  $$\mathit{M}\dot{\mathit{u}}+\mathit{K}\left(\mathit{u}\right)\mathit{u}=\mathit{f}$$

  (111)

  which represents an ordinary differential equation (ODE) with respect to the time variable.
  Depending on the type of the approximation of the time derivative present in the left term of Equation (111) and the discretization of the temporal domain $\Theta$, different methods are available to solve the problem described by Equation (111), which is discrete in space and continuous in time.
  Assuming that $\Theta$ is the time interval $\Theta =\left[0,T\right]$ and that it is discretized by a set of $N_T+1$ equally spaced points ${\left\{t_n\right\}}_{n=0,\dots ,N_T}$, where $t_n=∆t\ast n$ and $∆t=\frac{T}{N_T}$, then there exist different methods—called “Time Integration Schemes (TIS)”—to compute the approximation $u_{n+1}$ of u at the time $t_{n+1}$ as a function of just the approximation of u and $\dot{u}$ at the time $t_n$ (explicit methods) or as a function of u and $\dot{u}$ at the times $t_n$ and $t_{n+1}$ (implicit methods).

  $$\begin{array}{cccc}\hfill u_{n+1}& =f\left(u_n,{\dot{u}}_n,\right)\hfill & \hfill & \mathrm{explicit}\mathrm{schema}\hfill \\ \hfill u_{n+1}& =f\left(u_n,{\dot{u}}_n,u_{n+1}\right)\hfill & \hfill & \mathrm{implicit}\mathrm{schema}\hfill \end{array}$$
  See [142] for a classification of such schemes.
  Among the rich set of available TIS, a one-parameter family of methods, called the $\alpha$-family, is commonly used [143]. In this family of methods, a weighted average of time derivative $\dot{u}$ of a dependent variable u is approximated at two consecutive time steps by a linear interpolation of the values of the same variable u at the same two steps:

  $$\left(1-\alpha \right){\dot{u}}_n+\alpha {\dot{u}}_{n+1}=\frac{u_{n+1}-u_n}{∆t}.$$

  (112)
  By substituting Equation (112) into Equation (111), the following relation is obtained

  $${\widehat{\mathit{K}}}_{n+1}\left({\mathit{u}}_{n+1}\right){\mathit{u}}_{n+1}={\widehat{\mathit{f}}}_{n,n+1},$$

  (113)

  where

  $$\begin{array}{ccc}\hfill {\widehat{\mathit{K}}}_{n+1}\left({\mathit{u}}_{n+1}\right)& =& \mathit{M}+a_1{\mathit{K}}_{n+1}\hfill \end{array}$$

  (114)

  $$\begin{array}{ccc}\hfill {\widehat{\mathit{f}}}_{n,n+1}& =& ∆t\left[\alpha {\mathit{f}}_{n+1}+\left(1-\alpha \right){\mathit{f}}_n\right]+{\overline{\mathit{K}}}_n{\mathit{u}}_n\hfill \end{array}$$

  (115)

  $$\begin{array}{ccc}\hfill {\overline{\mathit{K}}}_n& =& \mathit{M}-a_2{\mathit{K}}_n\hfill \end{array}$$

  (116)

  and where

  $$\begin{array}{ccc}\hfill {\mathit{K}}_i& =& \mathit{K}\left({\mathit{u}}_i\right),\forall i=0,\dots ,N_T,\hfill \end{array}$$

  (117)

  $$\begin{array}{ccc}\hfill {\mathit{f}}_i& =& \mathit{f}\left(t_i\right),\forall i=0,\dots ,N_T,\hfill \end{array}$$

  (118)

  $$\begin{array}{ccc}\hfill a_1& =& \alpha ∆t,\hfill \end{array}$$

  (119)

  $$\begin{array}{ccc}\hfill a_2& =& \left(1-\alpha \right)∆t.\hfill \end{array}$$

  (120)
  Since at time $t=0$ (i.e., $n=0$), the right-hand side is computed using the initial values defined in the time boundary conditions, and since the vector $\mathit{f}$ is always known, for both times $t_n$ and $t_{n+1}$, then the approximation $u_{n+1}$ of u at time $t_{n+1}$ (when $n=0,\dots ,N_T-1$) can be computed by Algorithm 1.
  For different values of the parameter $\alpha$, several well-known time approximation schemes are obtained:
  • $\alpha =0$, the forward difference Euler explicit scheme. The problem (113) is linear, the schema is conditionally stable, and its order of accuracy (see Definition 2 for a definition of the “order of accuracy” terms) is $O\left(∆t\right)$;
  • $\alpha =0.5$, the Crank–Nicolson implicit scheme. The problem (113) is nonlinear, the schema is unconditionally stable, and its order of accuracy is $O\left({\left(∆t\right)}^2\right)$;
  • $\alpha =\frac{2}{3}$, the Galerkin implicit scheme. The problem (113) is nonlinear, the schema is unconditionally stable, and its order of accuracy is $O\left({\left(∆t\right)}^2\right)$;
  • $\alpha =1$, the backward difference Euler implicit scheme. The problem (113) is nonlinear, the schema is unconditionally stable, and its order of accuracy is $O\left(∆t\right)$.

    Algorithm 1 The algorithm implementing the $\alpha$-family methods for the approximated solution of the discrete in space and continuous in time problem described by Equation (111).

    1: procedure TimeIntegrator($\mathit{M}$, ${\mathit{K}}_0$, ${\mathit{K}}_1$, ${\mathit{u}}_0$, ${\mathit{u}}_1$, ${\mathit{f}}_0$, ${\mathit{f}}_1$, $\alpha$, T, $N_T$, ${\mathit{u}}_{N_T}$)   2: Input: $\mathit{M}$, ${\mathit{K}}_0$, ${\mathit{K}}_1$, ${\mathit{u}}_0$, ${\mathit{u}}_1$, ${\mathit{f}}_0$, ${\mathit{f}}_1$, T, $N_T$   3: Output:  ${\mathit{u}}_{N_T}$   4:     Compute $∆t$, $a_1$ and $a_2$   5:     for $n=1$ to $N_T-1$ do   6:           ${\widehat{\mathit{K}}}_n\left({\mathit{u}}_n\right)\leftarrow$Compute ${\widehat{\mathit{K}}}_n\left({\mathit{u}}_n\right)=\mathit{M}+a_1{\mathit{K}}_n$   7:           ${\overline{\mathit{K}}}_{n-1}\leftarrow$Compute ${\overline{\mathit{K}}}_{n-1}=\mathit{M}-a_2{\mathit{K}}_{n-1}$   8:           ${\widehat{\mathit{f}}}_{n-1,n}\leftarrow$Compute ${\widehat{\mathit{f}}}_{n-1,n}=∆t\left[\alpha {\mathit{f}}_n+\left(1-\alpha \right){\mathit{f}}_{n-1}\right]+{\overline{\mathit{K}}}_{n-1}{\mathit{u}}_{n-1}$   9:           ${\mathit{u}}_{n+1}\leftarrow$Solve ${\widehat{\mathit{K}}}_n\left({\mathit{u}}_n\right){\mathit{u}}_{n+1}={\widehat{\mathit{f}}}_{n-1,n}$ 10:           ${\mathit{K}}_{n+1}\leftarrow$Compute ${\mathit{K}}_{n+1}=\mathit{K}\left({\mathit{u}}_{n+1}\right)$ 11:           ${\mathit{f}}_{n+1}\leftarrow$Compute ${\mathit{f}}_{n+1}=\mathit{f}\left(t_{n+1}\right)$ 12:     end for 13: end procedure

  For $\alpha \ge 0.5$, the scheme is stable, and for $\alpha <0.5$, the scheme is stable only if the time step meets the following restrictions (i.e., conditionally stable schemes) (see Reddy [136]):

  $$∆t<\frac{2}{(1-2\alpha ){\lambda }_{max}},\alpha <\frac{1}{2}$$

  (121)

  where the greatest eigenvalue of the eigenvalue problem related to the matrix Equation (113) is denoted by ${\lambda }_{max}$:

  $$\left|{\widehat{\mathit{K}}}_{n+1}\left({\mathit{u}}_{n+1}\right)-\lambda \mathit{I}\right|=0.$$
  See [144] for further definitions and descriptions of the “stability” and “accuracy” terms. Generally, Algorithm 1 requires the solution of a nonlinear problem (see line 9) that can be solved by methods described in the next point 2. If the problem degenerates into a linear one (i.e., when $a_1=0$) its solution can be obtained by methods described at next point 3.  
• The Non-linear problem solver Let us consider the following nonlinear algebraic system:

  $$\mathit{G}\left(\mathit{u}\right)=\mathbf{0},$$

  (122)

  where $\mathit{G}={\left[G_i\left(\mathit{u}\right)\right]}_{i=1,\dots ,M}^T$ and $\mathit{u}={\left[u_i\right]}_{i=1,\dots ,M}^T$ are vectors in ${\Re }^M$. Assuming the existence of a solution $\overline{\mathit{u}}$ for the system (122), an approximation of that solution can be obtained by different methods. The “Newton–Raphson methods”, and their derivatives, should be cited as the most commonly used [145]. Such methods iteratively compute an approximation ${\mathit{u}}_{n+1}$ of $\overline{\mathit{u}}$, and they are based on a Taylor series development of the left term of (122) at an already known state ${\mathit{u}}_n$

  $$\mathit{G}\left({\mathit{u}}_n+∆u\right)\approx \mathit{G}\left({\mathit{u}}_n\right)+D\left[\mathit{G}\left({\mathit{u}}_n\right)\right]∆\mathit{u},$$

  (123)

  where $D\left[\mathit{G}\left({\mathit{u}}_n\right)\right]$ is the Jacobi matrix of $\mathit{G}$ at ${\mathit{u}}_n$, i.e.,

  $${\mathit{J}}_n=D\left[\mathit{G}\left({\mathit{u}}_n\right)\right]={\left[\frac{\partial G_i\left({\mathit{u}}_n\right)}{\partial u_j}\right]}_{i,j=1,\dots ,M}.$$
  Let ${\mathit{u}}_{n+1}={\mathit{u}}_n+∆u$; then

  $$\mathit{G}\left({\mathit{u}}_{n+1}\right)\approx \mathbf{0}\iff \mathit{G}\left({\mathit{u}}_n\right)+{\mathit{J}}_n∆\mathit{u}\approx \mathbf{0}.$$

  (124)
  Therefore, an approximation ${\mathit{u}}_N$ of $\overline{\mathit{u}}$ can be computed using the iterative algorithm represented in Algorithm 2. The rate of convergence of the Newton-Raphson scheme is characterized by the following inequality:

  $$∥{\mathit{u}}_{n+1}-\overline{\mathit{u}}∥\le C{∥{\mathit{u}}_n-\overline{\mathit{u}}∥}^2.$$

  Algorithm 2 The algorithm implementing the “Newton–Raphson methods” for the approximated solution of Equation (122).

  1: procedure NewtonSolver($\mathit{G}$, ${\mathit{u}}_0$, N, ${\mathit{u}}_N$)   2: Input: $\mathit{G}$, ${\mathit{u}}_0$, N   3: Output:  ${\mathit{u}}_N$   4:      for $n=0$ to $N-1$ do   5:            $\mathit{G}\left({\mathit{u}}_n\right)\leftarrow$Compute $\mathit{G}\left({\mathit{u}}_n\right)$   6:            ${\mathit{J}}_n\leftarrow$Compute ${\mathit{J}}_n$   7:            $∆{\mathit{u}}_n\leftarrow$Solve ${\mathit{J}}_n∆{\mathit{u}}_n=-\mathit{G}\left({\mathit{u}}_n\right)$   8:            ${\mathit{u}}_{n+1}\leftarrow$Compute ${\mathit{u}}_{n+1}={\mathit{u}}_n+∆{\mathit{u}}_n$   9:      end for 10: end procedure

  Depending on the approach used to approximate/compute the Jacobi-matrix (or tangent matrix) ${\mathit{J}}_n$ of $\mathit{G}\left({\mathit{u}}_n\right)$, different derivations of the scheme can be obtained:

  Table 8. The Newton–Raphson-like schemes obtained from the original formulation by using different approaches to approximate/compute the Jacobi-matrix of $\mathit{G}$.

  Discrete Newton	The matrix ${\mathit{J}}_n$ is approximated by ${\mathit{J}}_n^{DN}$ where $${\mathit{J}}_n^{DN}={\left[{ }^n{\mathit{j}}_k^{DN}\right]}_{k=1,\dots ,M},$$ and where $${ }^n{\mathit{j}}_k^{DN}=\frac{1}{h_k}\left(\mathit{G}\left({\mathit{u}}_n+{\mathit{ϵ}}_k{\mathit{\delta }}_k\right)-\mathit{G}\left({\mathit{u}}_n\right)\right).$$ ${ }^n{\mathit{j}}_k^{DN}$ should be considered an approximation of the k-th column of ${\mathit{J}}_n$ obtained by a difference finite method with step size ${\mathit{ϵ}}_k$. The value of ${\mathit{ϵ}}_k$ must be selected so that the tangent matrix approximation is as accurate as feasible. Its value should be very tiny.
  Modified Newton	The matrix ${{\mathit{J}}^{MN}}_n$ substitutes ${\mathit{J}}_n$. It is considered to be equal to ${\mathit{J}}_0$ for all steps n, i.e., $${{\mathit{J}}^{MN}}_n={\mathit{J}}_0,\forall n=1,\dots ,N$$
  Quasi-Newton	The tangent matrix ${\mathit{J}}_n$ is approximated by the “secant matrix” ${\mathit{J}}_n^{QN}$ for which the following relation is valid: $${\mathit{J}}_n^{QN}\left({\mathit{u}}_n-{\mathit{u}}_{n-1}\right)=-\left(\mathit{G}\left({\mathit{u}}_n\right)-\mathit{G}\left({\mathit{u}}_{n-1}\right)\right).$$ Letting ${\mathit{K}}_n^{QN}={\left({\mathit{J}}_n^{QN}\right)}^{-1}$, then ${\mathit{K}}_n^{QN}$ can be considered an approximation of the inverse of the tangent matrix ${\mathit{J}}_n$ for which the following relation is valid: $${\mathit{K}}_n^{QN}{\mathit{g}}_n={\mathit{w}}_n$$ where ${\mathit{w}}_n={\mathit{u}}_n-{\mathit{u}}_{n-1}$ and ${\mathit{g}}_n=-\left(\mathit{G}\left({\mathit{u}}_n\right)-\mathit{G}\left({\mathit{u}}_{n-1}\right)\right)$. A number of update algorithms exist for the explicit determination of ${\mathit{K}}_n^{QN}$. Here, the BFGS method will be presented since it has been observed to be successfully applied in the solution of problems from Finite Element Methods. It was based on the following update for ${\mathit{K}}_n^{QN}$: $${\mathit{K}}_n^{QN}=\left(\mathbf{1}+{\mathit{a}}_n{\mathit{b}}_n^T\right){\mathit{K}}_{n-1}^{QN}\left(\mathbf{1}+{\mathit{b}}_n{\mathit{a}}_n^T\right),$$ where $$a_n=\frac{1}{{\mathit{g}}_n^T{\mathit{w}}_n}{\mathit{w}}_n,$$ $$b_n=-\left\{{\mathit{g}}_n-{\left[-\frac{{\mathit{w}}_n^T{\mathit{g}}_n}{{\mathit{w}}_n^T\mathit{G}\left({\mathit{u}}_{n-1}\right)}\right]}^{-\frac{1}{2}}\mathit{G}\left({\mathit{u}}_{n-1}\right)\right\}.$$

  lists some of them. Also, when the application of the BFGS-update is considered, Algorithm 2 can be used. Only the equation system at the line 7 of Algorithm 2 has to be reformulated by introducing the BFGS-update of the inverse of the secant matrix. See Algorithm 3 for details on the BFGS reformulation of Algorithm 2.
  Algorithms 2 and 3 require the solution of a linear problem (see, respectively, lines 6 and 7) that can be solved by methods described in the next point 3.
  Details about convergence and other numerical issues are available in the literature (see [145]).

  Algorithm 3 The algorithm implementing the “BFGS” reformulation of the Newton–Raphson method.

  1: procedure BFGSNewtonSolver($\mathit{G}$, ${\mathit{u}}_0$, N, ${\mathit{u}}_N$)   2: Input: $\mathit{G}$, ${\mathit{u}}_0$, N   3: Output:  ${\mathit{u}}_N$   4:       ${\mathit{g}}_1\leftarrow$Compute ${\mathit{g}}_1=-\mathit{G}\left({\mathit{u}}_0\right)$   5:       ${\mathit{J}}_0^{QN}\leftarrow$Compute ${\mathit{J}}_0^{QN}$   6:       ${\mathit{K}}_0^{QN}\leftarrow$Solve ${\mathit{J}}_0^{QN}{\mathit{K}}_0^{QN}=\mathit{I}$   7:       ${\mathit{w}}_1\leftarrow$Compute ${\mathit{w}}_1={\mathit{K}}_0^{QN}{\mathit{g}}_1$   8:       for $n=1$ to $N-1$ do   9:             ${\mathit{u}}_n\leftarrow$Compute ${\mathit{u}}_n={\mathit{u}}_{n-1}+{\mathit{w}}_n$ 10:             ${\mathit{G}}_n\leftarrow$Compute ${\mathit{G}}_n=\mathit{G}\left({\mathit{u}}_{n-1}\right)$ 11:             ${\mathit{a}}_n\leftarrow$Compute ${\mathit{a}}_n=\frac{1}{{\mathit{g}}_n^T{\mathit{w}}_n}{\mathit{w}}_n$ 12:             ${\mathit{b}}_n\leftarrow$Compute ${\mathit{b}}_n=-\left\{{\mathit{g}}_n-{\left[-\frac{{\mathit{w}}_n^T{\mathit{g}}_n}{{\mathit{w}}_n^T{\mathit{G}}_n}\right]}^{-\frac{1}{2}}{\mathit{G}}_n\right\}.$ 13:             ${\mathit{K}}_n^{QN}\leftarrow$Compute ${\mathit{K}}_n^{QN}=\left(\mathbf{1}+{\mathit{a}}_n{\mathit{b}}_n^T\right){\mathit{K}}_{n-1}^{QN}\left(\mathbf{1}+{\mathit{b}}_n{\mathit{a}}_n^T\right),$ 14:             ${\mathit{w}}_{n+1}\leftarrow$Compute ${\mathit{w}}_{n+1}={\mathit{K}}_n^{QN}{\mathit{g}}_{n+1}$ 15:       end for 16:       ${\mathit{u}}_N\leftarrow$Compute ${\mathit{u}}_N={\mathit{u}}_{N-1}+{\mathit{w}}_N$ 17: end procedure

• The linear problem solver: Let us consider the linear problem of the form

  $$\mathit{A}\mathit{x}=\mathit{y},$$

  (125)

  where $\mathit{A}={\left[a_{ij}\right]}_{i,j=1,\dots ,M}^T$, $\mathit{y}={\left[y_i\right]}_{i=1,\dots ,M}^T$ and $\mathit{x}={\left[x_i\right]}_{i=1,\dots ,M}^T$ are, respectively, a matrix and two vectors in ${\Re }^{M\times M}$ and ${\Re }^M$. Since the matrices coming from the discretization of problems of interest for this review are sparse if not even structured—for example, see matrices generated for the solution of Navier–Stokes equations for viscoelastic fluids in Section 6.1.3, which have a lot of zero elements (or have zero blocks)—particular attention will be spent in this work to describe methods to be used for the solution of (125) when $\mathit{A}$ is sparse (or structured) (see [146] for a more precise definition of the sparsity concept).
  There are three fundamental classes of methods that are used to solve Equation (125):
  • Direct methods The direct solution of (125) is generally based on a technique called “$LU$ decomposition” [147]. This technique consists of factoring the matrix $\mathit{A}$ as in Equation (126) (by algorithms whose computational complexity is generally on the order of $M^3$), and it is based on the concept that triangular systems of equations are “easy” to solve:

    $$\mathit{P}\mathit{A}{\mathit{Q}}^T=\mathit{L}\mathit{U},$$

    (126)

    where $\mathit{L}$ and $\mathit{U}$ are, respectively, lower triangular and upper triangular matrices and where $\mathit{P}$ and $\mathit{Q}$ are permutation matrices. A permutation matrix is used to represent row or column interchanges in matrix $\mathit{A}$, and it is obtained from the identity matrix $\mathit{I}$ by applying on it the same sequence of row or column interchanges. The row or column permutations on $\mathit{A}$ are often required for both numerical and sparsity issues [146].
    The $LU$ decomposition facilitates the solution of (125) effectively, particularly when solving several systems that share the same matrix $\mathit{A}$. In fact, if a matrix $\mathit{A}$ already has a $\mathit{L}\mathit{U}$ decomposition available, the linear system in (125) can be solved by (1) the solution of $\mathit{L}\mathit{z}=\mathit{P}\mathit{y}$ (forward substitution), followed by (2) the solution of $\mathit{U}\mathit{w}=\mathit{z}$ (backward substitution) where $\mathit{w}={\mathit{Q}}^T\mathit{x}$. Then, the values of all the unknowns $x_i,\forall i=1,\dots ,M$ can be obtained by permuting the rows of $\mathit{w}$ based on the permutation matrix $\mathit{Q}$. The execution of both a forward and a backward substitution has a general computational complexity of $O\left(2M^2\right)$).
    If the matrix $\mathit{A}$ is symmetric and positive definite (that is, if i${\mathit{x}}^T\mathit{A}\mathit{x}>0$ for all nonzero vectors x), then the $\mathit{L}\mathit{U}$ decomposition of $\mathit{A}$ becomes $\mathit{P}\mathit{A}{\mathit{P}}^T=\mathit{L}{\mathit{L}}^T$ and is called “Cholesky” factorization.
    Numerical issues related to the stability and accuracy of the presented direct solver can be found in [146].
    With the aim to present and discuss methods useful in computing the $LU$ decomposition of sparse matrices, we propose the “Frontal methods”. Such methods have their origin in the solution of finite-element problems, but they are not restricted to this application. They are of great interest in their own right since the extension of these ideas to multiple fronts, regardless of the origin of the problem, will offer a great opportunity for parallelism [146,148,149].
    In the context of “Frontal methods” the matrix $\mathit{A}$ can be considered as the sum of submatrices of varying sizes, as happens for the matrices that are the result of the assembly process underlying the discretization using finite elements. That is, each entry is computed as the sum of one or more values:

    $${\mathit{A}}_{ij}=\sum _{l=1,\dots ,L}{\mathit{A}}_{ij}^{\left[l\right]},\forall i,j=1,\dots ,M.$$

    (127)
    Entry ${\mathit{A}}_{ij}$ in (127) is said to be fully summed (or fully assembled) if all the operations (127) have been performed, and the index of the unknown k is said to be fully summed if all the $\mathit{A}$ entries in its row (i.e., ${\mathit{A}}_{kj},\forall j=1,\dots ,M$) and its column (i.e., ${\mathit{A}}_{ik},\forall i=1,\dots ,M$) are fully summed.
    The fundamental notion of frontal techniques is to limit decomposition operations to a frontal matrix $\mathit{F}$, on which Level 3 BLAS [150] is used to execute dense matrix operations. (The BLAS (Basic Linear Algebra Subprograms) are routines that provide optimized standard building blocks for performing primary vector and matrix operations. BLAS routines can be classified depending on the type of operands: Level 1: operations involving just vector operands; Level 2: operations between vectors and matrices; and Level 3: operations involving just matrix operands.) In the frontal scheme, the factorization proceeds as a sequence of partial factorization on frontal matrices ${\mathit{F}}^{\left[l\right]}$, which can be represented, up to some row or column permutations, by a $2\times 2$ blocks structure:

    $${\mathit{F}}^{\left[l\right]}=\left[\begin{array}{cc}{\mathit{F}}_{11}^{\left[l\right]}& {\mathit{F}}_{12}^{\left[l\right]}\\ {\mathit{F}}_{21}^{\left[l\right]}& {\mathit{F}}_{22}^{\left[l\right]}\end{array}\right].$$

    (128)

    where ${\mathit{F}}_{11}^{\left[l\right]}$ is related to the unknown indices that are fully summed.
    The $LU$ decomposition by Frontal methods can be described by Algorithm 4.
    A variation of frontal solvers is the Multifrontal method. It could be considered an improvement of the frontal solver, which, based on an appropriate order or coupling of the addends in the summation in (127), can lead to the use of several independent fronts paving the way, naturally, to the development of parallel algorithms. Efficient parallel and sequential decomposition algorithms based on Frontal and Multifrotal methods should be able to perform an analysis of the structure of matrix $\mathit{A}$ to define all the needed reorganizations of its row and columns (i.e., by the definition of appropriate permutations) to better exploit sparsity, preserve “good” numerical properties, and define the order of front assembly. See [146] for details about techniques to be used in such analyses.

    Algorithm 4 The algorithm implementing the “Frontal method”.

    1: procedure FrontalDecomposition(${\left\{{\mathit{A}}^{\left[l\right]}\right\}}_{l=1,\dots ,L}$, L)   2: Input: ${\left\{{\mathit{A}}^{\left[l\right]}\right\}}_{l=1,\dots ,L}$, L   3: Output: $\mathit{L}$, $\mathit{U}$   4:     ${\mathbf{S}}^{\left[l-1\right]}\leftarrow \mathbf{0}$   5:     for $l=1$ to L do   6:           ${\mathit{F}}^{\left[l\right]}\leftarrow$Compute ${\mathbf{S}}^{\left[l-1\right]}+{\mathit{A}}^{\left[l\right]}$   7:           ${\mathit{L}}_{11}^{\left[l\right]},{\mathit{U}}_{11}^{\left[l\right]}\leftarrow$Factorize ${\mathit{F}}_{11}^{\left[l\right]}={\mathit{L}}_{11}^{\left[l\right]}{\mathit{U}}_{11}^{\left[l\right]}$   8:           ${\mathit{U}}_{12}^{\left[l\right]}\leftarrow$Solve ${\mathit{L}}_{11}^{\left[l\right]}{\mathit{U}}_{12}^{\left[l\right]}={\mathit{F}}_{12}^{\left[l\right]}$   9:           ${\mathit{L}}_{21}^{\left[l\right]}\leftarrow$Solve ${\mathit{L}}_{21}^{\left[l\right]}{\mathit{U}}_{11}^{\left[l\right]}={\mathit{F}}_{21}^{\left[l\right]}$ 10:           ${\mathbf{S}}^{\left[l\right]}\leftarrow$Compute ${\mathbf{S}}^{\left[l\right]}={\mathit{F}}_{22}^{\left[l\right]}-{\mathit{L}}_{21}^{\left[l\right]}{\mathit{U}}_{12}^{\left[l\right]}$ 11:     end for 12: $$\begin{array}{c}\mathit{L}\leftarrow \mathbf{Form}\mathit{L}=\left[\begin{array}{cccc}{\mathit{L}}_{11}^{\left[1\right]}& \mathbf{0}& \dots & \mathbf{0}\\ & {\mathit{L}}_{11}^{\left[2\right]}& & \mathbf{0}\\ & & \ddots & ⋮\\ {\mathit{L}}_{21}^{\left[1\right]}& {\mathit{L}}_{21}^{\left[2\right]}& \dots & {\mathit{L}}_{11}^{\left[L\right]}\end{array}\right]\hfill \end{array}$$ 13: $$\begin{array}{c}\mathit{U}\leftarrow \mathbf{Form}\mathit{U}=\left[\begin{array}{cccc}{\mathit{U}}_{11}^{\left[1\right]}& & & {\mathit{U}}_{12}^{\left[1\right]}\\ \mathbf{0}& {\mathit{U}}_{11}^{\left[2\right]}& & {\mathit{U}}_{12}^{\left[2\right]}\\ ⋮& & \ddots & ⋮\\ \mathbf{0}& \mathbf{0}& & {\mathit{U}}_{11}^{\left[L\right]}\end{array}\right]\hfill \end{array}$$ 14: end procedure

  • Iterative methods: Direct methods could exploit the sparse linear system structure as much as possible to avoid computations with zero elements and zero-element storage. However, these methods are often too expensive for large systems, except where the matrix $\mathit{A}$ has a special structure. For many problems, the direct solution methods will not lead to solutions in a reasonable amount of time. So, researchers have long tried to iteratively approximate the solution $\mathit{x}$ starting from a “good ” initial guess ${\mathit{x}}_0$ for the solution [151].
    The Krylov subspace iteration methods (KSMs) are among the most important iterative techniques for solving linear systems because they have significantly altered how users approach huge, sparse, non-symmetric matrix problems. The techniques in question were recognized as one of the top ten algorithms that had the biggest impact on the advancement and application of science and engineering during the 1900s [110], and they seem to have also confirmed their importance in the current century due to the effort spent to improve their effectiveness and efficiency in up-to-date computing architecture.
    Like the majority of feasible iterative methods now in use for resolving large linear equation systems, KSM makes use of a projection process. A projection process is the standard method for taking an approximation from a subspace to the solution of a linear system [152].
    Let ${\mathcal{K}}_N$ and ${\mathcal{L}}_N$ be two N-dimensional subspaces of ${\Re }^M$. Finding an approximate solution ${\mathit{x}}_N$ to (125) requires a projection technique onto the subspace ${\mathcal{K}}_N$ and orthogonal to ${\mathcal{L}}_N$. This is carried out by imposing the conditions that ${\mathit{x}}_N$ belongs to ${\mathcal{K}}_N$ and that the new residual vector is orthogonal to ${\mathcal{L}}_N$, where ${\mathit{x}}_0$ is the initial guess to the solution; that is,

    $$\mathrm{Find}{\mathit{x}}_N\in {\mathit{x}}_0+{\mathcal{K}}_N,\mathrm{such}\mathrm{that}\mathit{y}-\mathit{A}{\mathit{x}}_N\perp {\mathcal{L}}_N.$$

    Note that if ${\mathit{x}}_N$ is written in the form ${\mathit{x}}_N={\mathit{x}}_0+\mathit{\delta }$, and the initial residual vector ${\mathit{r}}_0$ is defined as ${\mathit{r}}_0=\mathit{y}-\mathit{A}{\mathit{x}}_0$, then the above condition becames,

    $$\mathrm{Find}\mathit{\delta }\in {\mathcal{K}}_N,\mathrm{such}\mathrm{that}{\mathit{r}}_0-\mathit{A}\mathit{\delta }\perp {\mathcal{L}}_N.$$

    (129)

    In other words, the approximate solution can be defined as

    $$\begin{array}{ccccc}\hfill \mathrm{Find}\mathit{\delta }\in {\mathcal{K}}_N& \mathrm{such}\mathrm{that}\hfill & \hfill 〈{\mathit{r}}_0-\mathit{A}\mathit{\delta },\mathit{w}〉& =0,\hfill & \hfill \forall \mathit{w}\in {\mathcal{L}}_N\end{array}$$

    (130)

    $$\begin{array}{cccc}\hfill \mathrm{Compute}& \hfill & \hfill {\mathit{x}}_N& ={\mathit{x}}_0+\mathit{\delta },\hfill \end{array}$$

    (131)

    where the symbol $〈·,·〉$ denotes the Euclidean inner product. In the most general form, this is a basic projection step. A series of these projections is used in the majority of common techniques. A new pair of subspaces, ${\mathcal{K}}_{N+1}$ and ${\mathcal{L}}_{N+1}$, are used in a new projection step to compute ${\mathit{x}}_{N+1}$. An initial guess, ${\mathit{x}}_N$, is equal to the most recent solution estimate obtained from the previous projection step. Many well-known techniques in scientific computing, such as the more complex KSM procedure or the simpler Gauss–Seidel step, have a unifying framework that is provided by projection methods.
    A Krylov subspace method is a method for which the subspace ${\mathcal{K}}_N={\mathcal{K}}_N\left(\mathit{A},{\mathit{r}}_0\right)$ is the Krylov subspace [152]

    $${\mathcal{K}}_N\left(\mathit{A},{\mathit{r}}_0\right)=span{\left\{{\mathit{A}}^i{\mathit{r}}_0\right\}}_{i=0,\dots ,N-1}.$$

    (132)
    Variations in the choices of subspace ${\mathcal{L}}_N$ give rise to different Krylov subspace methods. The most popular methods arise from two general options for ${\mathcal{L}}_N$.
    The first is simply ${\mathcal{L}}_N={\mathcal{K}}_N$ and the minimum residual variation ${\mathcal{L}}_N=A{\mathcal{K}}_N$. The most representative among the KSMs is related to the choice ${\mathcal{L}}_N=A{\mathcal{K}}_N$, and it will be described in this work: the “Generalized Minimum Residual Method (GMRES)”. The second class of methods is based on defining ${\mathcal{L}}_N$ to be a Krylov subspace method associated with ${\mathit{A}}^T$, namely, ${\mathcal{L}}_N={\mathcal{K}}_N\left({\mathit{A}}^T,{\mathit{r}}_0\right)$ (see [152] for details).
    The GMRES method can be used to solve sparse linear systems whose matrix $\mathit{A}$ is general (i.e., not necessarily symmetric or positive definite). Let ${\mathit{V}}_N$ matrices whose columns are an orthonormal basis for space ${\mathcal{K}}_N={\mathcal{K}}_N\left(\mathit{A},{\mathit{r}}_0\right)$, then $\mathit{\delta }$ can be expressed as as $\mathit{\delta }={\mathit{V}}_N\mathit{\vartheta }$ where $\mathit{\vartheta }\in {\mathcal{K}}_N$. Furthermore, let ${\mathcal{L}}_N=A{\mathcal{K}}_N$ and define

    $$\begin{array}{ccc}\hfill J_N\left(\mathit{\vartheta }\right)={∥{\mathit{r}}_N∥}_2& =\hfill & \hfill {∥\mathit{y}-\mathit{A}{\mathit{x}}_N∥}_2,\\ \hfill & =\hfill & \hfill {∥\mathit{y}-\mathit{A}\left({\mathit{x}}_0+{\mathit{V}}_N\mathit{\vartheta }\right)∥}_2,\\ \hfill & =\hfill & \hfill {∥{\mathit{r}}_0-\mathit{A}{\mathit{V}}_N\mathit{\vartheta }∥}_2,\end{array}$$

    (133)

    where ${∥·∥}_2$ is the spectral norm, that is, the matrix norm induced by the Euclidean vector’s inner product.
    Then, condition (129) (or equivalently the condition (131)) is satisfied by ${\tilde{\mathit{x}}}_N={\mathit{x}}_0+{\mathit{V}}_N\tilde{\mathit{\vartheta }}$ if, and only if, $\tilde{\mathit{\vartheta }}$ is the solution to the following minimum problem (see Proposition $5.3$ in Saad [152]):

    $$\tilde{\mathit{\vartheta }}=argmi{n}_{\mathit{\vartheta }\in {\mathcal{K}}_N}{J}_N\left(\mathit{\vartheta }\right).$$

    (134)
    A schema for the GMRES iterative algorithm, useful to compute the approximation ${\mathit{x}}_N$ of the solution for problem (125), is represented in Algorithm 5.

    Algorithm 5 The GMRES iterative algorithm.

    1: procedure GMRES($\mathit{A}$, $\mathit{y}$, ${\mathit{x}}_0$, N, ${\mathit{x}}_N$)   2: Input: $\mathit{A}$, ${\mathit{x}}_0$, N   3: Output:  ${\mathit{x}}_N$   4:       ${\mathit{r}}_0\leftarrow$Compute ${\mathit{r}}_0=\mathit{A}{\mathit{x}}_0-\mathit{y}$   5:       ${\mathit{V}}_0\leftarrow$Assign $\left[{\mathit{r}}_0\right]$   6:       for $n=1$ to N do   7:             ${\mathit{V}}_n\leftarrow$Compute an orthonormal basis $\left[{\mathit{V}}_{n-1},{\mathit{v}}_n\right]$ for ${\mathcal{K}}_n\left(\mathit{A},{\mathit{r}}_0\right)$   8:             ${\mathit{\vartheta }}_n\leftarrow$Compute ${\tilde{\mathit{\vartheta }}}_n=argmi{n}_{\mathit{\vartheta }\in {\mathcal{K}}_N}{J}_n\left(\mathit{\vartheta }\right)$   9:             ${\mathit{u}}_n\leftarrow$Compute ${\mathit{x}}_n={\mathit{x}}_{n-1}+{\mathit{V}}_n{\tilde{\mathit{\vartheta }}}_n$ 10:     end for 11: end procedure

    Different methods can be used to perform steps at line 7 of Algorithm 5. The most common ones are related with the Gram–Schmidt or Householder procedures, which both compute, at the step n, both the matrix ${\mathit{V}}_n$ and the $\left(n+1\right)\times n$ upper Hessenberg matrix (see [153] for the definition of upper Hessenberg matrix) ${\overline{\mathit{H}}}_n$, for which the following relation is valid [152]:

    $$\mathit{A}{\mathit{V}}_n={\mathit{V}}_{n+1}{\overline{\mathit{H}}}_n.$$

    (135)
    Thanks to Equation (135) (see Proposition $6.5$ in Saad [152]), the function $J_n\left(\mathit{\vartheta }\right)$ can be rewritten as

    $$J_n\left(\mathit{\vartheta }\right)={∥\mathit{c}-{\overline{\mathit{H}}}_n\mathit{\vartheta }∥}_2,$$

    (136)

    where $\mathit{c}$ is the constant $n+1$ vector defined as $\mathit{c}={\left[{∥{\mathit{r}}_0∥}_2,0,\dots ,0,0\right]}^T$. The solution to the minimum problem at line 8 of Algorithm 5 is easy and inexpensive to compute since it requires the solution of an $\left(n+1\right)\times n$ least-squares problem where n is typically small and where the coefficient matrix ${\overline{\mathit{H}}}_n$ is an upper triangular matrix.
    Although KS methods are well founded theoretically, they are all likely to experience slow convergence when dealing with problems arising from applications where the numerical features of the coefficient matrix $\mathit{A}$ are not “good”. The matrix’s “condition number” is the primary parameter that describes its numerical “goodness”. The condition number of a square matrix $\mathit{A}$ is defined as

    $$\kappa \left(\mathit{A}\right)={∥\mathit{A}∥}_2{∥{\mathit{A}}^{-1}∥}_2$$
    In numerical analysis, the condition number of a matrix $\mathit{A}$ is a way of describing how well or badly a linear system with $\mathit{A}$ as a coefficient matrix could be numerically solved in an effective way: if $\kappa \left(\mathit{A}\right)$ is small, the problem is well-conditioned; otherwise, the problem is rather ill-conditioned and it can hardly be solved numerically in an effective way [147].
    In these situations, preconditioning is essential to the effectiveness of Krylov subspace algorithms. Preconditioners, in general, are modifications to initial linear systems that make them “easier” to solve.
    Identifying a preconditioning matrix $\mathit{M}$ is the first stage in the preconditioning process. The matrix $\mathit{M}$ can be defined in a variety of ways, but it must meet a few minimal requirements. The most crucial one is that linear system $\mathit{M}\mathit{x}=\mathit{b}$ must be cheap to solve because the preconditioned algorithms will need to solve linear systems with coefficient matrix $\mathit{M}$ at each stage. Furthermore, $\mathit{M}$ should be nonsingular and “close” to matrix $\mathit{A}$ in some sense (i.e., ${\mathit{M}}^{-1}\mathit{A}\approx \mathit{I}$) [152].
    For example, if the preconditioner $\mathit{M}$ is applied from the left, the modification leads to the preconditioned linear system

    $${\mathit{M}}^{-1}\mathit{A}\mathit{x}={\mathit{M}}^{-1}\mathit{b},$$

    (137)

    whose coefficients matrix ${\mathit{M}}^{-1}\mathit{A}$ should have better numerical features than $\mathit{A}$ because it is “near” the identity matrix $\mathit{I}$. An incomplete factorization of the original matrix $\mathit{A}$ is one of the easiest ways to define a preconditioner. This involves taking into account the formula $\mathit{A}=\mathit{L}\mathit{U}-\mathit{R}$, in which $\mathit{R}$ represents the residual or factorization error and $\mathit{L}$ and $\mathit{U}$ represent the lower and upper portions of $\mathit{A}$, respectively, with the same nonzero structure. Calculating this incomplete factorization, often known as $ILU\left(0\right)$, is not too difficult or expensive.
    More sophisticated preconditioners are available, such as the preconditioners based on Domain Decomposition Methods and those based on Multigrid Methods (i.e., see [152] for the basic panoramic of such tools).
    It is worth mentioning that if the matrix $\mathit{A}$ is symmetric and positive definite, then specialized algorithms exist in the context of KSM: the Conjugate Gradient (CG) method is the most famous among those [152].  
  • Hierarchical/Multilevel methods: When solving linear systems resulting from discretized partial differential equations (PDEs), preconditioned Krylov subspace algorithms tend to converge more slowly as the systems get bigger. There is a significant loss of efficiency because of this decline in the convergence rate. The convergence rates that the Multigrid method class can achieve, on the other hand, are theoretically independent of the mesh size. Discretized elliptic PDEs are the primary focus of Multigrid methods, which distinguish them significantly from the preconditioned Krylov subspace approach. Later, the technique was expanded in many ways to address nonlinear PDE problems as well as problems not described by PDEs [152]. The most important contribution in this sense is related to the Algebraic Multigrid (AMG) [154]. The main ingredients of multilevel methods to solve (125) are as follows.

    (a)

    A hierarchy

    $${\mathit{A}}_h{\mathit{x}}^h={\mathit{y}}^h,h=0,\dots ,H$$

    (138)

    of H problems along with restriction ${\mathit{I}}_{h+1}^h$ and prolongation ${\mathit{I}}_h^{h+1}$ operators to move between levels, where

    $$\begin{array}{c}\hfill {\mathit{A}}_H=\mathit{A},{\mathit{x}}^H=\mathit{x},\mathrm{and}{\mathit{y}}^H=\mathit{y},\end{array}$$

    (139)

    and where

    $$\begin{array}{cc}\hfill {\mathit{A}}_h& ={\mathit{I}}_{h+1}^h{\mathit{A}}_{h+1}{\mathit{I}}_h^{h+1},\hfill \end{array}$$

    (140)

    $$\begin{array}{cc}\hfill {\mathit{x}}^h& ={\mathit{I}}_{h+1}^h{\mathit{x}}^{h+1},\hfill \end{array}$$

    (141)

    $$\begin{array}{cc}\hfill {\mathit{y}}^h& ={\mathit{I}}_{h+1}^h{\mathit{y}}^{h+1},\hfill \end{array}$$

    (142)

    with

    $${\mathit{I}}_{h+1}^h={\left({\mathit{I}}_h^{h+1}\right)}^T$$

    for each $h=0,\dots ,H-1$;

    (b)

    A relaxation operator, $S\left({\mathit{A}}_h,{\mathit{f}}^h,{\mathit{f}}_0^h,{\mathit{b}}^h\right)$, for “smoothing”. $S\left({\mathit{A}}_h,{\mathit{f}}^h,{\mathit{f}}_0^h,{\mathit{b}}^h\right)$ is in general a tool to compute iteratively an approximation for the solution of the linear system ${\mathit{A}}_h{\mathit{f}}_h={\mathit{b}}_h$ from the initial guess ${\mathit{f}}_0^h$.

    Algorithm 6 describes the recursive implementation schema of AMG methods based on the ingredients listed above, where ${\mathit{x}}_0^h$ is the initial guess for solution ${\mathit{x}}^h$.

    Algorithm 6 Implementation schema of AMG methods.

    1: function ${\mathit{x}}^h=$AMG (H, ${\mathit{A}}_h$, ${\mathit{x}}_0^h$, ${\mathit{y}}^h$, ${\nu }_1$, ${\nu }_2$, $\gamma$)   2: Input:H, ${\mathit{A}}_h$, ${\mathit{x}}_0^h$, ${\mathit{y}}^h$, ${\nu }_1$, ${\nu }_2$, $\gamma$   3: Output:  ${\mathit{x}}^h$   4:       ${\mathit{x}}^h\leftarrow$Pre-smooth ${\nu }_1$ times $S^{{\nu }_1}\left({\mathit{A}}_h,{\mathit{x}}^h,{\mathit{x}}_0^h,{\mathit{y}}^h\right)$   5:       ${\mathit{r}}^h\leftarrow$Get residual ${\mathit{r}}^h={\mathit{y}}^h-{\mathit{A}}_h{\mathit{x}}^h$   6:       ${\mathit{r}}^{h-1}\leftarrow$Restrict residual ${\mathit{r}}^{h-1}={\mathit{I}}_h^{h-1}{\mathit{r}}^h$   7:       ${\mathit{A}}_{h-1}\leftarrow$Restrict matrix ${\mathit{A}}_{h-1}={\mathit{I}}_h^{h-1}{\mathit{A}}_h{\mathit{I}}_{h-1}^h$   8:       if $h=1$ then   9:             ${\mathit{\delta }}^0\leftarrow$Solve Coarse System ${\mathit{A}}_0{\mathit{\delta }}^0={\mathit{r}}^0$ 10:       else 11:             ${\mathit{\delta }}^{h-1}=\mathbf{0}$ 12:             for $i=1$ to $\gamma$ do 13:                  ${\mathit{\delta }}^{h-1}\leftarrow$Recursive Solve ${\mathit{\delta }}^{h-1}=AMG\left(H-1,{\mathit{A}}_{h-1},{\mathit{\delta }}^{h-1},{\mathit{r}}^{h-1},{\nu }_1,{\nu }_2,\gamma \right)$ 14:           end for 15:       end if 16:       ${\mathit{x}}^h\leftarrow$Correct by prolongation ${\mathit{x}}^h={\mathit{x}}^h+{\mathit{I}}_{h-1}^h{\mathit{\delta }}^{h-1}$ 17:       ${\mathit{x}}^h\leftarrow$Post-smooth ${\nu }_2$ times $S^{{\nu }_2}\left({\mathit{A}}_h,{\mathit{x}}^h,{\mathit{x}}^h,{\mathit{y}}^h\right)$ 18:       return ${\mathit{x}}^h$ 19: end function

    The Multigrid schema is defined by the parameter $\gamma$, which controls the number of times AMG is iterated in line 13 of Algorithm 6. The V-cycle Multigrid is obtained for the situation $\gamma =1$. The W-cycle Multigrid is the case where $\gamma =2$. The illustrations in Figure 10 show how complicated the ensuing inter-grid up and down moves can be. Seldom is the case $\gamma =3$ used [152].
  Details about restriction, prolongation, and relaxation operators can be found in Stuben [154].
• The Saddle Point Problem solvers Let us consider the block $2\times 2$ linear systems of the form

  $$\left[\begin{array}{cc}A& B_1^T\\ B_2& -C\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=\left[\begin{array}{c}{y}_1\\ y_2\end{array}\right]$$

  (143)

  where $A\in {\Re }^{n\times n}$, $B_1,B_2\in {\Re }^{m\times n}$ and $C\in {\Re }^{m\times m}$. If one or more of the following requirements are met by the constituent blocks A, $B_1$, $B_2$, and C, then the linear system (143) defines a (generalized) “Saddle Point problem” [155]:
  • C1 A is symmetric ($A=A^T$);
  • C2 the symmetric part of A, $H=\frac{1}{2}\left(A+A^T\right)$, is positive semidefinite;
  • C3 $B_1=B_2=B$;
  • C4 C is symmetric and positive semidefinite;
  • C5 $C=O$.
  In addition to the standard differentiation between direct and iterative techniques, generalized Saddle Point problem-solving algorithms can be broadly classified into two groups: segregated and coupled methods [155].
  • Segregated methods $x_1$ and $x_2$, the two unknown vectors, are computed independently via segregated procedures. This method entails solving two linear systems (referred to as reduced systems), one for each of $x_i$, that are less in size than $n+m$. Each reduced system in segregated techniques can be addressed using an iterative or direct approach. The Schur complement reduction method, which is based on a block LU factorization of the block $2\times 2$ matrix in Equation (143) (also known as the global matrix $\mathcal{A}$), is one of the primary examples of the segregated approach. Indeed, if A is nonsingular, the Saddle Point matrix admits the following block triangular factorization:

    $$\mathcal{A}=\left[\begin{array}{cc}A& B_1^T\\ B_2& -C\end{array}\right]=\left[\begin{array}{cc}I& 0\\ B_2{A}^{-1}& I\end{array}\right]\left[\begin{array}{cc}A& B_1^T\\ 0& S\end{array}\right],$$

    (144)

    where $S=-\left(C+B_2{A}^{-1}{B}_1^T\right)$ is the Schur complement of A in $\mathcal{A}$. It follows that if $\mathcal{A}$ is nonsingular, S is also nonsingular. Using Equation (144), the linear system (143) can be transformed into

    $$\left[\begin{array}{cc}I& 0\\ -B_2{A}^{-1}& I\end{array}\right]\left[\begin{array}{cc}A& B_1^T\\ B_2& -C\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=\left[\begin{array}{cc}I& 0\\ -B_2{A}^{-1}& I\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\end{array}\right],$$

    (145)

    or equivalently,

    $$\left[\begin{array}{cc}A& B_1^T\\ 0& S\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=\left[\begin{array}{c}{y}_1\\ y_2-B_2{A}^{-1}{y}_1\end{array}\right].$$

    (146)

    Algorithm 7 The algorithm of the segregated approach to the solution of a generalized Saddle Point system.

    1: procedure SaddlePointSegregatedMethod(A, $B_1$, $B_2$, C, $y_1$, $y_2$, $x_1$, $x_2$) 2: Input:A, $B_1$, $B_2$, C, $y_1$, $y_2$ 3: Output: $x_1$, $x_2$ 4:       $w\leftarrow$Solve $Aw=y_1$ 5:       $v\leftarrow$Compute $v=y_2-B_{2}w$ 6:       $x_2\leftarrow$Solve $S{x}_2=v$ 7:       $z\leftarrow$Compute $z=y_1-B_1{x}_2$ 8:       $x_1\leftarrow$Solve $A{x}_1=z$ 9: end procedure

    Then, the solution of the linear system (143) by the segregated method can be performed using Algorithm 7. Solution of linear systems at the lines 4, 6 and 8 of Algorithm 7 can be solved either directly or iteratively. This approach is attractive if the order m of the reduced system at line 6 of Algorithm 7 is small and if linear systems with coefficient matrix A can be solved efficiently. Since the solution of two linear systems with coefficient matrix A is required, it should be convenient to perform just one LU factorization of A to be used twice. The two main drawbacks are that A must be nonsingular and that the Schur complement S can be full, making it prohibitively expensive to factor or compute. When creating S, numerical instability could potentially be an issue, particularly if A is not well conditioned [155]. If S is too costly to form or factor, a Schur complement reduction can still be used by solving related linear systems using iterative techniques like KSM, which only require S in the form of matrix-vector products that involve the matrices $B_1^T$, $B_2$, and C, as well as by solving a linear system with matrix A. These techniques do not require access to individual entries of S. It is possible that the Schur complement system is not well conditioned, in which case preconditioning will be necessary.
  • Coupled methods Coupled methods deal with the system (143) as a whole, computing $x_1$ and $x_2$ (or approximations to them) simultaneously. Among these approaches are direct solvers based on triangular factorizations of the global matrix $\mathcal{A}$, as well as iterative algorithms such as KSM applied to the system as a whole (143), usually preconditioned in some way.
    For direct methods based on triangular factorizations of $\mathcal{A}$, we give a brief overview limited to the symmetric case (that is, where $A=A^T$, $B_1=B_2=B$ and $C=C^T$), since no specialized direct solver exists for a nonsymmetric Saddle Point problem. Furthermore, we assume that A is positive definite and B has full rank, then the Saddle Point matrix $\mathcal{A}$ admits the following factorization,

    $$\mathcal{A}=\mathcal{L}\mathcal{D}{\mathcal{L}}^T$$

    (147)

    where $\mathcal{D}$ is a diagonal matrix and $\mathcal{L}$ is a unit lower triangular matrix. To be more precise, A is positive definite, so its decomposition is $A=L_A{D}_A{L}_A^T$, where $L_A$ is the unit lower triangular and $D_A$ is the diagonal (also positive definite); additionally, the Schur complement $S=-(C+B{A}^{-1}{B}^T)$ is negative definite, so its decomposition is $S=-L_S{D}_S{L}_S^T$. Thus, we are able to write

    $$\mathcal{A}=\left[\begin{array}{cc}A& B^T\\ B& -C\end{array}\right]=\left[\begin{array}{cc}{L}_A& 0\\ L_B& L_S\end{array}\right]\left[\begin{array}{cc}{D}_A& 0\\ 0& -D_S\end{array}\right]\left[\begin{array}{cc}{L}_A^T& L_B^T\\ 0& L_S^T\end{array}\right]=\mathcal{L}\mathcal{D}{\mathcal{L}}^T,$$

    (148)

    where $L_B=B{L}_A^{-T}{D}_A^{-1}$. Note that $B{A}^{-1}{B}^T=L_B{D}_A{L}_B^T$. However, in practice, with the original ordering, the factors will be fairly dense, and sparsity preservation requires the employment of symmetric permutations $\mathcal{Q}$. Not every sparsity-preserving permutation is suitable, though. It is demonstrated that there are permutation matrices $\mathcal{Q}$ such that the factorization of $\mathcal{L}\mathcal{D}{\mathcal{L}}^T$, where $\mathcal{D}$ is a diagonal matrix, is not available for $\mathcal{Q}\mathcal{A}{\mathcal{Q}}^T$. Moreover, some permutations might result in issues with numerical instability [155].
    Iterative methods are preferred for solving the linear system (143) using coupled methods, as sparse $\mathcal{Q}\mathcal{A}{\mathcal{Q}}^T$ factorization methods are not completely foolproof, especially for Saddle Point systems arising from PDE problems.
    About the usage of iterative algorithms, we wrote some words about the use of preconditioned KSM for iterative solution of the entire linear system (143). Various preconditioning strategies designed for (generalized) Saddle Point systems are explained. To develop high-quality preconditioners for Saddle Point problems, one must take advantage of the problem’s block structure and possess a comprehensive understanding of the origin and structure of each block [155].
    We consider block diagonal and block triangular preconditioners for KSM applied to the coupled system $\mathcal{A}x=y$ as in (143). The basic block diagonal preconditioner is given by

    $${\mathcal{P}}_d=\left[\begin{array}{cc}\widehat{A}& 0\\ 0& -\widehat{S}\end{array}\right]$$

    (149)

    where both $\widehat{A}$ and $\widehat{s}$ are approximations of A and S, respectively. Several different approximations have been considered in the literature. A fairly general framework considers a splitting of A into

    $$A=D-E,$$

    (150)

    where D is invertible and easy to invert. Then $\widehat{A}=D$ and $\widehat{S}=-\left(C+B_2{D}^{-1}{B}_1^T\right)$ are chosen.
    The basic block (upper) triangular preconditioner has the form

    $${\mathcal{P}}_t^{upper}=\left[\begin{array}{cc}\widehat{A}& B_1^T\\ 0& \widehat{S}\end{array}\right],$$

    (151)

    and, on the other hand, the form of a (lower) triangular preconditioner is

    $${\mathcal{P}}_t^{lower}=\left[\begin{array}{cc}\widehat{A}& 0\\ B_2& \widehat{S}\end{array}\right],$$

    (152)

    where, as before, both $\widehat{A}$ and $\widehat{s}$ are approximations of A and S, respectively.
    For the coupled solution of a linear system (143) using KSM, the availability of reasonable approximations for the block A and the Schur complement S is necessary for the development of excellent block diagonal and block triangular preconditioners. Such approximations are difficult to build and heavily rely on the specific problem at hand.
  For other details about the treatment of Saddle Point problems, we suggest reading the monograph by Benzi et al. [155].

6.1.3. The Discrete Model of the Navier–Stokes Equations for Viscoelastic Fluids

Considering the high flexibility of FEM (see considerations listed in Table 7), we propose the description of FEM discretization of the vscoelastic fluid model.

According to [125], and for the sake of simplicity, we choose to use an Oldroyd-B model for $T_P$ and a Newtonian viscosity model for $T_s$. So, a viscoelastic fluid’s behavior can be described by the following system of equations:

$$\begin{array}{ccc}\hfill \mathrm{div}\mathit{v}& =& 0\hfill \end{array}$$

(153)

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial t}\left(\rho \mathit{v}\right)+\left(\rho \mathit{v}\right)\odot \nabla \mathit{v}& =& -\nabla p+\mathrm{div}\left(2{\mu }_s\mathit{D}\right)+\mathrm{div}{\mathit{T}}_p+\rho \mathit{g}+\mathit{f}\hfill \end{array}$$

(154)

$$\begin{array}{ccc}\hfill \lambda T_p+\stackrel{▽}{T_p}& =& 2{\mu }_p\mathit{D}.\hfill \end{array}$$

(155)

$$\begin{array}{ccc}\hfill {\mathit{v}}_{\left|t=0\right.}& =& {\mathit{v}}^0\hfill \end{array}$$

(156)

$$\begin{array}{ccc}\hfill p_{\left|t=0\right.}& =& p^0\hfill \end{array}$$

(157)

$$\begin{array}{ccc}\hfill {T_p}_{\left|t=0\right.}& =& {T_p}^0\hfill \end{array}$$

(158)

The finite element form of the above formulation is often termed the MIX (mixed method) formulation, and it is the basis for many other advanced formulations.

The system of Equations (153)–(155) can also be written in Cartesian component form in a Eulerian reference frame $x=\left(x_1,\dots ,x_n\right)$ as follows:

$$\begin{array}{ccc}\hfill Continuity equation& & \hfill \end{array}$$

$$\begin{array}{ccc}\hfill \sum _{i=1,\dots ,n}\frac{\partial v_i}{\partial x_i}& =& 0\hfill \\ \hfill Momentum equation& & \hfill \end{array}$$

(159)

$$\begin{array}{ccc}\hfill \rho \left(\frac{\partial v_i}{\partial t}+\sum _{j=1,\dots ,n}{v}_j\frac{\partial v_i}{\partial x_j}\right)& =& -\frac{\partial p}{\partial x_i}+\sum _{j=1,\dots ,n}\frac{\partial }{\partial x_j}\left(2{\mu }_s{D}_{ij}\right)+\sum _{j=1,\dots ,n}\frac{\partial }{\partial x_j}{\left(T_p\right)}_{ij}+h_i\hfill \\ \hfill Constitutiv eequation& & \hfill \end{array}$$

(160)

$$\begin{array}{ccc}\hfill \lambda {T_p}_{ij}+{\stackrel{▽}{T_p}}_{ij}& =& 2{\mu }_p{D}_{ij}.\hfill \end{array}$$

(161)

for all $i=1,\dots ,n$ where $\mathit{h}=\rho \mathit{g}+\mathit{f}$, $D_{ij}=\frac{1}{2}\left(\frac{\partial v_i}{\partial x_j}+\frac{\partial v_j}{\partial x_i}\right)$. The equations in (159)–(161) provide $n+1+n^2$ relations for the unknowns ${\left[v_i\left(x,t\right)\right]}_{1,\dots ,n},p\left(x,t\right),{\left[{\left(T_p\right)}_{ij}\left(x,t\right)\right]}_{i,j=1,\dots ,n}$.

The application of the finite element procedure to the partial differential equations in (159)–(161) follows the standard format. The velocity components $v_i^e$, pressure $p^e$, and extra-stress components ${\left(T_p^e\right)}_{ij}$ are approximated, in each element e, by expansions of the form

$$\begin{array}{ccccc}\hfill v_i^e& =\hfill & \hfill \sum _{m=1}^M{\mathbf{\Psi }}_m{v}_i\left({ }^e{x}_m^{\mathbf{\Psi }},t\right)& =\hfill & \hfill {\mathbf{\Psi }}^T{ }^e{\mathit{v}}_i\end{array}$$

(162)

$$\begin{array}{ccccc}\hfill p^e& =\hfill & \hfill \sum _{n=1}^N{\mathbf{\Phi }}_{n}p\left({ }^e{x}_n^{\mathbf{\Phi }},t\right)& =\hfill & \hfill {\mathbf{\Phi }}^T{ }^e\mathit{p}\end{array}$$

(163)

$$\begin{array}{ccccc}\hfill T_{ij}^e& =\hfill & \hfill \sum _{k=1}^K{\mathbf{\Pi }}_k{T}_{ij}\left({ }^e{x}_k^{\mathbf{\Pi }},t\right)& =\hfill & \hfill {\mathbf{\Pi }}^T{ }^e{\mathit{T}}_{ij}\end{array}$$

(164)

where

• $\mathbf{\Psi }$, $\mathbf{\Phi }$, and $\mathbf{\Pi }$ are vectors of basis functions (the symbol ${\left(·\right)}^T$ denotes the transposition operation);
• M, N, and K indicate the number of nodal points ${ }^e{x}_m^{\mathbf{\Psi }}$, ${ }^e{x}_m^{\mathbf{\Phi }}$ and ${ }^e{x}_m^{\mathbf{\Pi }}$ at which the various unknowns are defined in each element e;
• ${ }^e{\mathit{v}}_i$, ${ }^e\mathit{p}$ and ${ }^e{\mathit{T}}_{ij}$ are vectors of unknowns into the nodal points in each element e, i.e.,

  $$\begin{array}{ccc}\hfill { }^e{\mathit{v}}_i& =& {\left[v_i\left({ }^e{x}_m^{\mathbf{\Psi }},t\right)\right]}_{m=1,\dots ,M}^T,\hfill \end{array}$$

  (165)

  $$\begin{array}{ccc}\hfill { }^e\mathit{p}& =& {\left[p\left({ }^e{x}_n^{\mathbf{\Phi }},t\right)\right]}_{n=1,\dots ,N}^T,\hfill \end{array}$$

  (166)

  $$\begin{array}{ccc}\hfill { }^e{\mathit{T}}_{ij}& =& {\left[T_{ij}\left({ }^e{x}_k^{\mathbf{\Pi }},t\right)\right]}_{k=1,\dots ,K}^T;\hfill \end{array}$$

  (167)

and where the underscript p for the extra stress has been omitted for clarity.

We recall that

$${\stackrel{▽}{T}}_{ij}=\frac{\partial T_{ij}}{\partial t}+\sum _{k=1,\dots ,n}{v}_k\frac{\partial T_{ij}}{\partial x_k}-\sum _{k=1,\dots ,n}{T}_{ik}\frac{\partial v_j}{\partial x_k}-\sum _{k=1,\dots ,n}{T}_{kj}\frac{\partial v_i}{\partial x_k}$$

(168)

Then, by using the finite element approximations (162)–(164) in standard weighted-integral statements (i.e., weak forms) of the system in (159)–(161), the following system of finite element equations can be obtained.

• Continuity equation

  $$\sum _{i=1,\dots ,n}\left[{\int }_{{\Omega }_e}\mathbf{\Phi }\frac{\partial {\mathbf{\Psi }}^T}{\partial x_i}dx\right]{ }^e{\mathit{v}}_i=0$$

  (169)
• Momentum equation

  $$\begin{array}{c}\hfill \begin{array}{cc}\left[{\int }_{{\Omega }_e}\rho \mathbf{\Psi }{\mathbf{\Psi }}^{T}dx\right]{ }^e\dot{{\mathit{v}}_i}\hfill & +\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}\rho \mathbf{\Psi }{\mathbf{\Psi }}^T{ }^e{\mathit{v}}_j\frac{\partial {\mathbf{\Psi }}^T}{\partial x_j}dx\right]\right]{ }^e{\mathit{v}}_i\hfill \\ \hfill & +\left[{\int }_{{\Omega }_e}\frac{\partial \mathbf{\Psi }}{\partial x_i}{\mathbf{\Phi }}^{T}dx\right]{ }^e\mathit{p}\hfill \\ \hfill & -\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_j}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_j}dx\right]\right]{ }^e{\mathit{v}}_i\hfill \\ \hfill & -\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_j}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_i}dx\right]{ }^e{\mathit{v}}_j\right]\hfill \\ \hfill & -\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}\frac{\partial \mathbf{\Psi }}{\partial x_j}{\mathbf{\Pi }}^{T}dx\right]{ }^e{\mathit{T}}_{ij}\right]\hfill \\ \hfill & ={\int }_{{\Omega }_e}\mathbf{\Psi }{ }^e{\mathit{h}}_{i}dx\hfill \end{array}\end{array}$$

  (170)
• Constitutive equation

  $$\begin{array}{c}\hfill \begin{array}{cc}\left[{\int }_{{\Omega }_e}\mathbf{\Pi }{\mathbf{\Pi }}^{T}dx\right]{ }^e{\dot{\mathit{T}}}_{ij}\hfill & +\left[{\int }_{{\Omega }_e}\lambda \mathbf{\Pi }{\mathbf{\Pi }}^{T}dx\right]{ }^e{\mathit{T}}_{ij}\hfill \\ \hfill & -\left[{\int }_{{\Omega }_e}{\mu }_p\mathbf{\Pi }\frac{\partial {\mathbf{\Psi }}^T}{\partial x_j}dx\right]{ }^e{\mathit{v}}_i-\left[{\int }_{{\Omega }_e}{\mu }_p\mathbf{\Pi }\frac{\partial {\mathbf{\Psi }}^T}{\partial x_i}dx\right]{ }^e{\mathit{v}}_j\hfill \\ \hfill & +\left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Psi }}^T{ }^e{\mathit{v}}_k\frac{\partial {\mathbf{\Pi }}^T}{\partial x_k}dx\right]\right]{ }^e{\mathit{T}}_{ij}\hfill \\ \hfill & -\left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Pi }}^T{ }^e{\mathit{T}}_{ik}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_k}dx\right]\right]{ }^e{\mathit{v}}_j\hfill \\ \hfill & -\left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Pi }}^T{ }^e{\mathit{T}}_{kj}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_k}dx\right]\right]{ }^e{\mathit{v}}_i\hfill \end{array}\end{array}$$

  (171)

where ${ }^e\mathit{n}={\left[{ }^e{n}_i\right]}_{i=1,\dots ,n}$ is the vector normal to boundary $\partial {\Omega }_e$.

The algebraic relations system (169)–(171), in each element e, can be written in nonlinear matrix form as follows:

$$\begin{array}{cc}\hfill \left[\begin{array}{ccc}{ }^e{\mathit{M}}_{\mathbf{\Psi }\mathbf{\Psi }}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& { }^e{\mathit{M}}_{\mathbf{\Pi }\mathbf{\Pi }}\end{array}\right]\left[\begin{array}{c}{ }^e\dot{\mathit{V}}\\ { }^e\dot{\mathit{p}}\\ { }^e\dot{\mathit{T}}\end{array}\right]& +\left[\begin{array}{ccc}{ }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Psi }}\left({ }^e\mathit{V}\right)& { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Phi }}& { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}\\ { }^e{\mathit{K}}_{\mathbf{\Phi }\mathbf{\Psi }}& \mathbf{0}& \mathbf{0}\\ { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}& \mathbf{0}& { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Pi }}{+}^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}\left({ }^e\mathit{V}\right)\end{array}\right]\left[\begin{array}{c}{ }^e\mathit{V}\\ { }^e\mathit{p}\\ { }^e\mathit{T}\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{c}{ }^e\mathit{H}\\ \mathbf{0}\\ \mathbf{0}\end{array}\right]\hfill \end{array}$$

(172)

where

$$\begin{array}{cccccc}\hfill { }^e{\mathit{M}}_{\mathbf{\Psi }\mathbf{\Psi }}& =\mathit{I}\otimes { }^e{\mathit{M}}_{\mathbf{\Psi }},\hfill & \hfill { }^e{\mathit{M}}_{\mathbf{\Pi }\mathbf{\Pi }}& =\mathit{I}\otimes \left(\mathit{I}\otimes { }^e{\mathit{M}}_{\mathbf{\Pi }}\right),\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Phi }\mathbf{\Psi }}& =\left[{ }^e{\mathit{K}}_{\mathbf{\Phi }\mathbf{\Psi }}^1\cdots {}_e\mathit{K}_{\mathbf{\Phi }\mathbf{\Psi }}^n\right],\hfill \end{array}$$

$$\begin{array}{cccc}\hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Phi }}& ={\left[{ }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Phi }}^1\cdots {}_e\mathit{K}_{\mathbf{\Psi }\mathbf{\Phi }}^n\right]}^T,\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Pi }}& =\mathit{I}\otimes \left(\lambda \mathit{I}\otimes { }^e{\mathit{M}}_{\mathbf{\Pi }}\right),\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}& =\mathit{I}\otimes \left[{ }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}^1\cdots {}_e\mathit{K}_{\mathbf{\Psi }\mathbf{\Pi }}^n\right],\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Phi }\mathbf{\Psi }}^i& =\left[{\int }_{{\Omega }_e}\mathbf{\Phi }\frac{\partial {\mathbf{\Psi }}^T}{\partial x_i}dx\right],\hfill \\ \hfill { }^e\mathit{T}& ={\left[{ }^e{\mathit{T}}_{11}^T\cdots {}_e\mathit{T}_{1n}^T{\cdots }^e{\mathit{T}}_{n1}^T{\cdots }^e{\mathit{T}}_{nn}^T\right]}^T,\hfill & \hfill { }^e\mathit{V}& ={\left[{ }^e{\mathit{v}}_1^T\cdots {}_e\mathit{v}_n^T\right]}^T,\hfill \\ \hfill { }^e{\overline{\mathit{K}}}_{\mathbf{\Psi }\mathbf{\Psi }}& =-\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_j}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_j}dx\right]\right],\hfill & \hfill { }^e{\mathit{M}}_{\mathbf{\Pi }}& =\left[{\int }_{{\Omega }_e}\mathbf{\Pi }{\mathbf{\Pi }}^{T}dx\right],\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^{i\ast }\left(T\right)& =\left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Pi }}^T{ }^e{\mathit{T}}_{ik}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_k}dx\right]\right],\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}^j& =-\left[{\int }_{{\Omega }_e}\frac{\partial \mathbf{\Psi }}{\partial x_j}{\mathbf{\Pi }}^{T}dx\right],\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Pi }}\left({ }^e\mathit{V}\right)& =\mathit{I}\otimes \left(\mathit{I}\otimes \left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Psi }}^T{ }^e{\mathit{v}}_k\frac{\partial {\mathbf{\Pi }}^T}{\partial x_k}dx\right]\right]\right),\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^j& =\left[{\int }_{{\Omega }_e}{\mu }_p\mathbf{\Pi }\frac{\partial {\mathbf{\Psi }}^T}{\partial x_i}dx\right],\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^{\ast j}\left(T\right)& =\left[\sum _{k=1}^n\left[{\int }_{{\Omega }_e}{\mathbf{\Pi }}^T{ }^e{\mathit{T}}_{kj}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_k}dx\right]\right],\hfill & \hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Phi }}^i& =\left[{\int }_{{\Omega }_e}\frac{\partial \mathbf{\Psi }}{\partial x_i}{\mathbf{\Phi }}^{T}dx\right],\hfill \\ \hfill { }^e{\overline{\mathit{K}}}_{\mathbf{\Psi }\mathbf{\Psi }}^i\left({ }^e\mathit{V}\right)& =\left[\sum _{j=1,\dots ,n}\left[{\int }_{{\Omega }_e}\rho \mathbf{\Psi }{\mathbf{\Psi }}^T{ }^e{\mathit{v}}_j\frac{\partial {\mathbf{\Psi }}^T}{\partial x_j}dx\right]\right],\hfill & \hfill { }^e{\mathit{M}}_{\mathbf{\Psi }}& =\left[{\int }_{{\Omega }_e}\rho \mathbf{\Psi }{\mathbf{\Psi }}^{T}dx\right],\hfill \end{array}$$

$$\begin{array}{ccc}\hfill { }^e\mathit{H}& =& {\left[\left({\int }_{{\Omega }_e}{\mathbf{\Psi }}^T{ }^e{\mathit{h}}_{1}dx\right)\cdots \left({\int }_{{\Omega }_e}{\mathbf{\Psi }}^T{ }^e{\mathit{h}}_{n}dx\right)\right]}^T,\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}& =& -\left(\mathit{I}\otimes diag{\left({ }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^j\right)}_{j=1,\dots ,n}\right)-\left(diag{\left({ }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^i\right)}_{i=1,\dots ,n}\otimes I\right)\hfill \\ & & -\left(\mathit{I}\otimes diag{\left({ }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^{\ast j}\left(T\right)\right)}_{j=1,\dots ,n}\right)-\left(diag{\left({ }^e{\mathit{K}}_{\mathbf{\Pi }\mathbf{\Psi }}^{i\ast }\left(T\right)\right)}_{i=1,\dots ,n}\otimes I\right),\hfill \\ \hfill { }^e{\mathit{K}}_{\mathbf{\Psi }\mathbf{\Psi }}\left({ }^e\mathit{V}\right)& =& \left(diag{\left({ }^e{\overline{\mathit{K}}}_{\mathbf{\Psi }\mathbf{\Psi }}^i\left({ }^e\mathit{V}\right)\right)}_{i=1,\dots ,n}+\left(\mathit{I}\otimes { }^e{\overline{\mathit{K}}}_{\mathbf{\Psi }\mathbf{\Psi }}\right)\right)+{ }^e{\overline{\overline{\mathit{K}}}}_{\mathbf{\Psi }\mathbf{\Psi }},\hfill \\ \hfill { }^e{\overline{\overline{\mathit{K}}}}_{\mathbf{\Psi }\mathbf{\Psi }}& =& -\left[\begin{array}{ccc}\left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_1}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_1}dx\right]& \cdots & \left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_n}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_1}dx\right]\\ ⋮& \ddots & ⋮\\ \left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_1}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_n}dx\right]& \cdots & \left[{\int }_{{\Omega }_e}{\mu }_s\frac{\partial \mathbf{\Psi }}{\partial x_n}\frac{\partial {\mathbf{\Psi }}^T}{\partial x_n}dx\right]\end{array}\right],\hfill \end{array}$$

and where $\mathit{I}$ is the identity matrix of dimension $n\times n$. The operator ⊗ represents the Kronecker products of two matrices: if $A={\left(a_{ij}\right)}_{i,j=1,\dots ,n}$ and $B={\left(b_{ij}\right)}_{i,j=1,\dots ,m}$ are respectively matrices of dimensions $n\times n$ and $m\times m$, $A\otimes B$ is a matrix of dimensions $mn\times mn$, where

$$A\otimes B=\left[\begin{array}{ccc}{a}_{11}B& \dots & a_{1n}B\\ ⋮& \cdots & ⋮\\ a_{n1}B& \dots & a_{nn}B\end{array}\right].$$

The operator $\mathrm{diag}{\left(B_i\right)}_{i=1,\dots ,n}$ represents the matrix of dimensions $mn\times mn$

$$diag{\left(B_i\right)}_{i=1,\dots ,n}=\left[\begin{array}{ccc}{B}_1& \dots & 0\\ ⋮& \cdots & ⋮\\ 0& \dots & B_n\end{array}\right].$$

where $B_i$ is a matrix of dimension $m\times m$ for all $i=1,\dots ,n$.

The assembly process of all the elements of the finite element mesh generates an algebraic system of the type in Equation (111) where the matrices and vectors have the structure described in Equation (172), that is,

$${ }^{NSVF}\mathit{M}{ }^{NSVF}\dot{\mathit{u}}+{ }^{NSVF}\mathit{K}\left({ }^{NSVF}\mathit{u}\right)\mathit{u}={ }^{NSVF}\mathit{f},$$

(173)

where

$$\begin{array}{ccc}\hfill { }^{NSVF}\mathit{M}& =& \left[\begin{array}{ccc}{\mathit{M}}_V& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathit{M}}_T\end{array}\right],\hfill \end{array}$$

(174)

$$\begin{array}{ccc}\hfill { }^{NSVF}\mathit{K}\left(\mathit{u}\right)& =& \left[\begin{array}{ccc}{\mathit{K}}_{V,V}\left(\mathit{u}\right)& {\mathit{K}}_{V,p}& {\mathit{K}}_{V,T}\\ {\mathit{K}}_{p,V}& \mathbf{0}& \mathbf{0}\\ {\mathit{K}}_{T,V}& \mathbf{0}& {\mathit{K}}_{T,T}\left(\mathit{u}\right)\end{array}\right],\hfill \end{array}$$

(175)

$$\begin{array}{cc}\hfill { }^{NSVF}\mathit{u}& =\left[\begin{array}{c}\mathit{V}\\ \mathit{p}\\ \mathit{T}\end{array}\right],{ }^{NSVF}\mathit{f}=\left[\begin{array}{c}\mathit{H}\\ \mathbf{0}\\ \mathbf{0}\end{array}\right].\hfill \end{array}$$

(176)

During a Time Integration Scheme, the solver of a nonlinear problem always requires the solution to linear systems (see the line 6 of Algorithm 3) whose coefficient matrix $\mathit{A}$ (see Equation (114)) has the form of a Saddle Point problem. Indeed, the block structure of matrix $\mathit{A}$ can be reassembled as

$$\mathit{A}=\left[\begin{array}{cc}\left[\begin{array}{cc}{\mathit{M}}_V+a_1{\mathit{K}}_{V,V}\left({\mathit{u}}_0\right)& {\mathit{K}}_{V,p}\\ {\mathit{K}}_{p,V}& \mathbf{0}\end{array}\right]& \left[\begin{array}{c}{\mathit{K}}_{V,T}\\ \mathbf{0}\end{array}\right]\\ \left[\begin{array}{cc}{\mathit{K}}_{T,V}& \mathbf{0}\end{array}\right]& {\mathit{M}}_T+a_1{\mathit{K}}_{T,T}\left({\mathit{u}}_0\right)\end{array}\right],$$

(177)

or equivalently

$$\mathit{A}=\left[\begin{array}{cc}{ }^{NSVF}{\mathit{K}}_{11}& { }^{NSVF}{\mathit{K}}_{12}\\ { }^{NSVF}{\mathit{K}}_{21}& { }^{NSVF}{\mathit{K}}_{22}\end{array}\right],$$

(178)

where the matrix ${ }^{NSVF}{\mathit{K}}_{11}$ is, in turn, a Saddle Point problem where its upper-left block ${\mathit{M}}_V+a_1{\mathit{K}}_{V,V}\left({\mathit{u}}_0\right)$ is a symmetric matrix, where its left-bottom and upper-right blocks are each the transpose of the other (i.e., ${\mathit{K}}_{p,V}={\mathit{K}}_{V,p}^T$), and where its bottom-right one is a zero block. Such a system could then be solved by an opportune combination of the algorithms described in Section 6.1.2 on the basis of numerical and structural properties of matrices ${ }^{NSVF}\mathit{M}$ and ${ }^{NSVF}\mathit{K}\left(\mathit{u}\right)$. The number of system unknowns is $N_{nodes}\times \left(n+n^2+1\right)$, but, in consideration of the symmetric nature of the matrix ${\mathit{T}}_p$, just $\frac{n\left(n+1\right)}{2}$ elements of ${\mathit{T}}_p$ have to be computed, so the real number of system unknowns should be $N_{nodes}\times \left[\left(n+1\right)+\frac{n\left(n+1\right)}{2}\right]=\frac{N_{nodes}}{2}\left(n+1\right)\left(n+2\right)$.

We note that the solution of Problems 2 and 3 related to Two-Phase Fluids requires the solution of the equations describing Problem 1 coupled with other equations describing, respectively, interface motion (see Equation (22)) and Immersed Body motion (see Equation (44)).

The discretization of such equations adds row and column blocks, related, respectively, with values chosen to approximate $\varphi$ and X on the discretization mesh, to the algebraic system in Equation (173). The resulting system, which preserves its block sparse structure, can then be solved using algorithms described in Section 6.1.2.

The same considerations could be made about the Equations (62)–(64) if they are coupled with the equations describing Problem 1 (or Problems 2 and 3).

6.2. Models at the Meso Scale

6.2.1. The Markov Chain Monte Carlo Algorithms

Let $\Omega$ be a finite configuration space of a physical system and assume that a suitable proposal transition method $\mathcal{M}\left(\omega \right)$ from the state $\omega \in \Omega$ to the state ${\omega }^{\ast }\in \Omega$ has been selected: in general, $\mathcal{M}$ is defined by a transition probabilities $P\left({\omega }^{\ast }|\omega \right)$. Let $\pi$ be a given distribution for which a function $g:\Omega \leftarrow \Omega$ exists such that

$$\frac{g\left(\omega \right)}{g\left({\omega }^{\ast }\right)}=\frac{\pi \left(\omega \right)}{\pi \left({\omega }^{\ast }\right)}.$$

If ${\omega }_0$ is the initial state, the Metropolis–Hastings algorithm generates the set of M states ${\left\{{\omega }_i\right\}}_{i=1,\dots ,M}\subset \Omega$ that asymptotically reaches the stationary $\pi$ distribution, and it is depicted in Algorithm 8.

Algorithm 8 The MCMC Metropolis–Hastings algorithm.

1: procedure MCMCMetropolisHastings($\Omega$, ${\omega }_0$, M)   2: Input: $\Omega$, ${\omega }_0$, M   3: Output:  ${\left\{{\omega }_i\right\}}_{i=1,\dots ,M}$   4:       for $n=1$ to M do   5:             ${\omega }^{\ast }\leftarrow$$\mathbf{Select}{\omega }^{\ast }=\mathcal{M}\left({\omega }_{i-1}\right)$   6:             $\xi \leftarrow$$\mathbf{Generate}\mathbf{a}\mathbf{uniform}\mathbf{random}\mathbf{number}\xi \in \left[0,1\right]$   7:             $\Xi \leftarrow$$\mathbf{Compute}\Xi =min\left(1,\frac{g\left({\omega }^{\ast }\right)}{g\left({\omega }_{i-1}\right)}\frac{P\left({\omega }_{i-1}|{\omega }^{\ast }\right)}{P\left({\omega }^{\ast }|{\omega }_{i-1}\right)}\right)$ ▹Acceptance probability $\Xi$   8:             if $\xi \le \Xi$ then                 ▹Allow the new state   9:                  ${\omega }_i\leftarrow {\omega }^{\ast }$ 10:           else                      ▹Reject the inew state 11:                  ${\omega }_i\leftarrow {\omega }_{i-1}$ 12:             end if 13:       end for 14: end procedure

6.2.2. The Cellular Particle Dynamics Algorithms

Let $\mathcal{C}$ ne a finite set of $N_{Cells}$ cells, and assume that each cell is composed of $N_{CellularElement}$ elements. Let ${\mathit{y}}_{{\alpha }_i}=\left(y_{{\alpha }_i}^1,\dots ,y_{{\alpha }_i}^N\right)\in {\Re }^N$, the position vectors of each element ${\alpha }_i$ where $\alpha =1,\dots ,N_{CellularElement}$ and $i=1,\dots ,N_{Cells}$.

For the generic element ${\alpha }_i$, Equation (79) can be rewritten as

$$\mu \frac{\partial {\mathit{y}}_{{\alpha }_i}}{\partial t}={\mathit{\eta }}_{{\alpha }_i}-\sum _{{\beta }_i\ne {\alpha }_i}{\nabla }_{{\mathit{y}}_{{\alpha }_i}}\left[V_{intra}\left(\left|{\mathit{y}}_{{\alpha }_i}-{\mathit{y}}_{{\beta }_i}\right|\right)\right]-\sum _{j\ne i}\sum _{{\beta }_j}{\nabla }_{{\mathit{y}}_{{\alpha }_i}}\left[V_{inter}\left(\left|{\mathit{y}}_{{\alpha }_i}-{\mathit{y}}_{{\beta }_j}\right|\right)\right]$$

(179)

Thanks to the properties of the gradient ${\nabla }_y$ of a composite real function $f\left(g\left(y\right)\right)$, we have

$${\nabla }_{y}f\left(g\left(y\right)\right)=\frac{df}{dg}\left(g\right){\nabla }_{y}g\left(y\right).$$

Then, for each component n and for each element ${\alpha }_i$, Equation (179) can rewritten as

$$\mu \frac{\partial y_{{\alpha }_i}^n}{\partial t}={\eta }_{{\alpha }_i}^n-\sum _{{\beta }_i\ne {\alpha }_i}\left[\frac{y_{{\alpha }_i}^n-y_{{\beta }_i}^n}{d_{{\alpha }_i,{\beta }_i}}\frac{d{V}_{intra}}{d{d}_{{\alpha }_i,{\beta }_i}}\left(d_{{\alpha }_i,{\beta }_i}\right)\right]-\sum _{j\ne i}\sum _{{\beta }_j}\left[\frac{y_{{\alpha }_i}^n-y_{{\beta }_j}^n}{d_{{\alpha }_i,{\beta }_j}}\frac{d{V}_{intra}}{d{d}_{{\alpha }_i,{\beta }_j}}\left(d_{{\alpha }_i,{\beta }_j}\right)\right]$$

(180)

The set of all $N\times N_{CellularElement}\times N_{Cells}$ equations of type (180) represents a generally non-linear system of algebraic equations in the form represented by Equation (111) where the vector of unknowns $\mathit{y}$ is given by

$$\mathit{y}={\left[y_{{\alpha }_i}^n\right]}_{\begin{array}{c}\alpha =1,\dots ,N_{CellularElement}\hfill \\ i=1,\dots ,N_{Cells}\hfill \\ n=1,\dots ,N.\hfill \end{array}}$$

This system could then be solved by an opportune combination of the algorithms described in Section 6.1.2 on the basis of numerical and structural properties of the matrices involved.

6.2.3. The Cellular Automata Model Algorithms

Let $\mathcal{A}$ bea Cellular Automata defined by

• A N-dimensional grid $\mathcal{G}$ of $N_{Cells}={\prod }_{n=1}^N{N}_{Cells}^n$ cells $\mathcal{C}={\left\{{\mathfrak{c}}^n\right\}}_{n=1,\dots ,N_{Cells}}$ where, for each of them, the set of neighborhood ${\mathcal{N}}^n$ is defined;
• A set $\mathcal{I}$ of $N_{ingredients}$ ingredients, i.e., $\mathcal{I}={\left\{{\mathfrak{i}}^l\right\}}_{l=1,\dots ,N_{ingredients}}$;
• A set $\mathcal{R}$ of $N_{rules}$ rules, i.e., $\mathcal{R}={\left\{{\mathfrak{r}}^n\left(\mathcal{I}\right)\right\}}_{n=1,\dots ,N_{rules}}$.

Given the number M of the iteration defining the length of each run and the number $N_{run}$ defining the number of the runs to be executed, the $M\times N_{run}$ states ${\left\{{\mathcal{G}}_{i,j}\right\}}_{i=1,\dots ,M;j=1,\dots ,N_{run}}$ of asynchronous simulation based on the automata $\mathcal{A}$ can be computed by Algorithm 9.

Algorithm 9 The asynchronous cellular automata algorithm.

1: procedure AsynchronousCellularAutomata($\mathcal{A}$, ${\mathcal{G}}_0$, N, M)   2: Input: $\mathcal{A}$, ${\mathcal{G}}_0$, N, M   3: Output:  ${\left\{{\mathcal{G}}_{i,j}\right\}}_{i=1,\dots ,M;j=1,\dots ,N_{run}}$   4:  for $j=1$ to $N_{run}$ do   5:    Generate an initial condition state ${\mathcal{G}}_{0,j}$ for the grid $\mathcal{G}$;   6:    for $i=1$ to M do   7:     for $n=1$ to $N_{Cells}$ do   8:      if (cell ${\mathfrak{c}}_{i-1,j}^n$ of the grid ${\mathcal{G}}_{i-1,j}$ is occupaied by some ingredient ${\mathfrak{i}}^l\in \mathcal{I}$) then   9:       for $m=1$ to $N_{rules}$ do 10:         Apply rule ${\mathfrak{r}}^m\left(\mathcal{I}\right)$ to ${\mathfrak{c}}_{i-1,j}^n$ to define ${\mathfrak{c}}_{i,j}^n$, and its neighborhood ${\mathcal{N}}_{i,j}^n$, on grid ${\mathcal{G}}_{i,j}$ 11:       end for 12:      end if 13:     end for 14:    end for 15:  end for 16: end procedure

7. Algorithms for HPC Systems in the Exascale Era

The extremes of temporal and spatial scales, as well as the intricate nonlinear interactions of several physical processes, are two recurring themes in computer science. Although it is still beyond the capabilities of present petascale systems, high-fidelity simulation of this kind of physical system may become affordable on approaching Exascale computer resources. Exploiting the capabilities of impending Exascale architectures will need significant new model creation, algorithm reform, and reimplementation of research application code. A few of these problems have already been documented [156,157,158].

If solvers are a mainstay of scientific computing and justify a great investment in CS research for Exascale resources, the discretization and mathematical formulation of the problem are also critical stages in the simulation process that provide new difficulties for both linear and nonlinear solvers.

In computational materials sciences, Exascale computing is needed to predict and understand the behavior of new and heterogeneous materials at disparate lengths and time scales. It is worth noting that to describe such systems, characterized by a significant structural disorder and chemical complexity, a multiscale approach should be pursued. To this aim, creating links between continuum and microscale models is essential for bridging different length scales. Providing precise up-scaling interaction for coarse-grained approaches and down-scaling interactions to treat nanoscale and electronic environments is how this is achieved [156]. For example, linking a microscopic model to an atomistic or continuum model using a microscopic simulation to fit or estimate parameters contained in a higher-level model is among the CS challenges. To this aim, a system of closely coupled nonlinear differential/integral equations should be solved. Additionally, gathering data from several parallel instances of microscopic simulations might be necessary. For solving both coupling equations and nonlinear equations employed in a microscopic model, it is extremely desirable to apply iterative solver acceleration approaches that can leverage a physics-motivated preconditioner. Time-evolution systems that are both stable and efficient are required to explain the material’s dynamic behavior [156].

7.1. Models for Exascale Computing

As described in Section 5, mathematical models often involve coupled physical phenomena being inherently multiphysics-based. In addition, there are many potential levels of description from the atomic up to macroscopic scales.

Although it is severely constrained by the computer resources at hand, the degree of fine-grained physical accuracy in models is contingent upon the significance of the fine-scale processes’ characteristics on the macroscopic time and length scales. In theory, classical molecular models may try to perform simulations at this size. Even with Exascale resources, such a model has far too many unknowns to compute at macroscopic (engineering) scales. To this aim, the approach is to divide and conquer the scales by creating simplified (or coarse-grained) models able to correctly describe a particular set of dynamics. In such a way, it can also be possible to control the complexity and scale disparity of first-principles-based models. Naturally, using this method produces multiscale or scale-bridging models [156,159,160,161] in which a coupled hierarchy of models is considered.

These models avoid using a brute-force approach that would lead to an infeasible solution by making many attempts to bridge disparate time and length scales. By definition, these scale-bridging algorithms employ the split of scales to generate optimal formulations at different levels of issue description. This clearly leads to a hierarchical or layered issue description, which is useful in conjunction with the expected future Exascale computer systems’ hierarchical nature [156] (see Section 4).

Two common mathematical representations are utilized, as explained in Section 5: (1) continuum models, which use a Eulerian approach to numerically represent functions on a continuous domain by means of a finite-dimensional representation linked to a mesh, and (2) particle models, which follow the dynamics of individual or groups of discrete entities in space in a Lagrangian fashion. The particle models can explain a wider range of (nonequilibrium) behavior, but they are noisy and can become costly at macroscopic scales. Despite their lack of noise and better-understood convergence properties, continuum models may become too expensive to use at small scales. Particle-based models (see Section 5.2 for examples of such approaches) are suitable for Exascale computing applications in some cases.

It is possible to process particles individually until they synchronize at a later time-step level. This allows for arbitrary fine-grained parallelism and the exploitation of asynchrony. Particle models can efficiently use single precision due to their statistical character, where the accuracy impact of a lower precision is significantly smaller than the associated statistical noise. It is also easier to manage particles’ fault tolerance due to their statistical character.

Exascale computing generally offers a chance to reconsider and enhance the way that models in multiphysics/multiscale models should be integrated across scales and physical processes. An effective substitute for operator splitting has been made possible by the development of nonlinear implicit solvers. The shift toward more closely coupled techniques presents a great possibility for adaptation to the Exascale, with its memory bandwidth stringencies, despite some unsolved mathematical issues [156,162].

This article makes multiple references to the phrases “coupled” (also known as “splitted” or “segregated”), and since coupling exists at least on an algorithmic and physical level, some definitions are needed. According to Keyes et al. [162], “the terms “strong (versus weak) coupling” of physical models mean that physical components of the modeled system are coupled by a strong (versus weak) interaction. Correspondingly, the term “tight (versus loose) coupling of numerical models means that algorithmic components are coupled by a tight (versus loose) interaction”.

In the management of multiscale multiphysics problems, the strategy described in Keyes et al. can therefore be considered appropriate

“Because we strongly advocate examining coupling strength before pursuing a decoupled or split strategy for solving a multiphysics problem, we propose “coupled until proven decoupled” as a perspective worthy of 21st-century simulation purposes and resources.” [162]

Description of multiphysics modeling, associated implementation issues such as coupling or decoupling approaches, and example applications are available in Keyes et al. [162].

A contemporary multiscale technique, the multiscale high-order/low-order (HOLO) strategy [161], is appropriate for issues where a microscopic description must be connected to a macroscopic one. The resulting technique will be referred to as HOLO if the macroscopic system is called the low-order (LO) description and the microscopic one the high-order (HO) description. Descriptions of some applications of the HOLO method are available in Chacon et al. [161].

7.2. Parallel Solvers for Exascale Computing

To create and implement effective solvers on Exascale systems, it will be necessary to overcome many obstacles. The architectural features of these computers, which include additional memory levels, more concurrency with less memory per core, and heterogeneous cores, will affect the performance of all the proposed algorithms. As explained in [157], in fact, the following considerations should guide the development and use of new Exascale solvers.

• Extreme-scale computers might consist of millions of nodes with thousands of light cores or hundreds of thousands of nodes with more aggressive cores, according to technological advancements. In any case, solvers will need to enable billion-way concurrence with small thread states [157].
• Extreme-scale systems will probably encounter a high number of hardware malfunctions since they will be constructed on a vast number of components, highlighting a resilient and non-deterministic behavior. Applications should therefore anticipate a high frequency of soft mistakes (such as data value changes due to logic latch faults) and hard interruptions (device failure). Current fault-tolerance methods (those based on traditional, full checkpoint/restart) may cause significant overhead and be impractical for these systems. Solvers must therefore be equipped with methods for locating and resolving software errors and interruptions to reduce any detrimental consequences on an application [157].
• Compared to memory technologies, the cost of devices designed for floating-point operations is decreasing more quickly. Recent system trends indicate that the memory per node is already modestly increasing while memory per core is dropping. Therefore, to take advantage of parallelism, solvers will need to reduce synchronization, increase computation on local data, and shift their focus from the typical scenario of weak scaling, which involves using more resources to solve problems of larger sizes, to one that favors strong scaling, which involves using more resources to solve a problem of fixed size in order to reduce the time to solution (see [163] for a definition of weak vs. strong scaling) [157].
• Power consumption will be a major issue for future HPC systems. Although low-energy accelerators and other hardware design elements can lower the needed power, more steps must be taken to further minimize energy consumption. An approach that could be used is algorithmic study [157].

Several methods and procedures [156,157] have been proposed in response to these factors to help achieve algorithms that make use of the opportunities provided by Exascale computing platforms. A summary of some of these strategies and techniques is provided below.

• Communication- and synchronization-avoiding algorithms The expected costs of computation at the Exascale (also in terms of energy consumption) will be more conditioned by movements from/to the memory or between different memories (i.e., “a communication”) than by the execution of floating point operations. So, when solving huge problems on parallel architectures with extreme levels of concurrency, the most significant concern becomes the cost related to communication and synchronization overheads. This is especially the case for preconditioned Krylov methods (see Section 6.1.2). New versions of such algorithms “minimizing communication” should be investigated to address the critical issue of such communication costs (see Section 7.2.3). Likewise, such popular algorithms execute tasks requiring synchronization points, for example, the computation of BLAS operators and the preconditioner applications. Then, additive algorithmic variants (with no synchronization among components) may be more attractive compared to their multiplicative counterparts [164]. Such considerations are taken into account also by “Time integrators” in the Exascale context (see Section 7.2.1). Furthermore, the linear solution process, which is at the base of the solution of almost all described problems in the previous sections, requires constantly moving the sparse matrix entries from the main memory into the CPU, without register or cache reuse. Therefore, methods that demand less memory bandwidth need to be researched. There are promising linear techniques, such as coefficient-matrix-free representations [165], that totally avoid sparse matrices, or Fast Multipole Methods (see Section 7.2.4) [157].
• Mixed-Precision Arithmetic Algorithms Using a combination of single- and double-precision arithmetic in a calculation is known as mixed-precision arithmetic. Single-precision arithmetic is typically faster than double-precision arithmetic, but some computations may be so susceptible to roundoff errors that the entire computation is typically not possible to complete with just a single precision, so mixed-precision arithmetic can be used. In order to quantify the situations in which using mixed-precision arithmetic is advantageous for a given class of algorithms, further research is required, with a particular focus on the numerical behavior of this technique. In addition to speeding up problem-solving, mixed-precision arithmetic might also need less memory. When working with mixed-precision arithmetic, it is crucial to understand how round-off errors spread and affect computed value accuracy [157]. Section 7.2.3 provides examples of mixed-precision techniques for solving linear problems.
• Fault-tolerant and resilient algorithms Allowing for nondeterminism in both data and operations is a beneficial—and perhaps essential—feature that makes fault-tolerant and robust algorithms possible. Algorithms based on pseudorandom number generators offer a multitude of possibilities at the extreme scale. Monte Carlo-like methods, or more in general, stochastic-based algorithms such as Cellular Automata, can identify resilient implementations by using stochastic replications, which can improve concurrency by several orders of magnitude by employing stochastic replications to find robust implementations [157] (see Section 7.2.5). Regarding fault-tolerant algorithms, a possible solution to the problem of global synchronous checkpoint/restart on extreme-scale systems is the use of localized checkpoints and asynchronous recovery. Recent work is focused on algorithm-based fault tolerance (ABFT): for example, much of the ABFT work on linear solvers is based on the use of checksum [157,166].

7.2.1. “Implicit-Explicit” and “Parallel in Time” Solvers

Different problem components frequently develop on various temporal scales. Thus, in real-world applications, it is frequently not possible to integrate PDE discretizations in time efficiently using a single time-stepping technique. By utilizing the benefits of several strategies and mitigating the drawbacks of employing a single set strategy, partitioned methods get around these challenges. A class of partitioned time integration methods called implicit–explicit (IMEX) methods, commonly referred to as semi-implicit schemes, expand explicit and implicit schemes that accommodate variable redundant calculations in favor of more scalable algorithms.

Proposed IMEX methods are based on an approach that uses extrapolation methods [167] for the solution of ODE [168,169]. Extrapolation techniques are a subset of numerical analysis that are associated with convergence acceleration techniques, the application of which has been researched and used in numerous contexts for many years. Sequences and series are frequently dealt with in numerical analysis, applied mathematics, and engineering. They are created using approximation techniques based on parameters, iterative processes, and so on. In actuality, certain series or sequences frequently converge so slowly that it seriously hinders their usefulness. In such a context, extrapolation methods can play a role. In the specific context of the ODE solution, extrapolation methods can help to improve the rate of convergence of a “time integrator” method.

Definition 2. 

The extrapolation method for ordinary differential equations:

The aim is to compute an approximated solution to the the following ODE problem

$$\dot{u}\left(t\right)=F\left(t,u\left(t\right)\right)$$

(181)

where $u\left(t_0\right)=u_0$, $u\left(t\right)\in {\Re }^d$ and where $t\in \Theta =\left[t_0{t}_n\right]$. As described in Section 6.1.2 about the Time Integration Algorithms, to compute an approximation of the solution in the considered time domain Θ, a discretization ${\Theta }_{N_n}$ of Θ should be considered. For this purpose, consider ${\Theta }_{N_n}$ as the set made up of the $N_n+1$ equally spaced points ${\left\{t_k\right\}}_{k=0,\dots ,N_n}$ where $t_k=t_0+{∆}_n\ast k,k=1,\dots ,N_n$ and where ${∆}_n=\frac{t_n}{N_n}$.

The approximation $u_{{∆}_n}$ in ${\Theta }_{N_n}$ is constructed recursively with the operator ${\mathbf{\Psi }}_{{∆}_n}^{k,k-1}$ (a so-called “one step method”) with step size ${∆}_n$ and order of accuracy p. The rate at which a numerical approximation of a differential equation converges to the actual solution is measured in numerical analysis by the order of accuracy. Consider a differential equation’s exact solution, u, and its numerical approximation, $u_h$, where h is a parameter that describes the approximation, and the numerical solution $u_h$ is said to be pth-order accurate if the error $E\left(h\right):=∥u-u_h∥$ is limited above by a value proportional to h to the pth power:

$$E\left(h\right):=∥u-u_h∥\le C{h}^p$$

$$\begin{array}{ccc}\hfill u_{{∆}_n}\left(t_0\right)& =& u_0,\hfill \end{array}$$

(182)

$$\begin{array}{ccc}\hfill u_{{∆}_n}\left(t_k\right)& =& {\mathbf{\Psi }}_{{∆}_n}^{k,k-1}{u}_{{∆}_n}\left(t_{k-1}\right).\hfill \end{array}$$

(183)

Let ${\left\{n_j\right\}}_{j=1,\dots ,M}$ be a set of positive integers representing a sequence such that $n<n_j<n_{j+1}$, and let

$$T_{l,j}=\prod _{k=1}^l{\mathbf{\Psi }}_{{∆}_{n_j}}^{k,k-1}{u}_{{∆}_{n_j}}\left(t_0\right),0<l\le n_j.$$

(184)

Given the M couples $\left({∆}_{n_j},T_{n_j,j}\right)$ and an integer value $\omega \ge 1$, there exists a unique polynomial $\pi \in \mathbf{\Pi }$ where

$$\mathbf{\Pi }=\left\{∆\mapsto a_0+\sum _{l=0}^{M-1}{a}_{p+l}{∆}^{p+\omega l}:a_l\in {\Re }^d\right\},$$

such that interpolates all the above M couples, that is, $\pi \left({∆}_{n_j}\right)=T_{n_j,j},\forall j=1,\dots ,M$ [169]. The value $T_{n,0}=u_{{∆}_n}\left(t_n\right)$ can then be obtained by evaluating π in 0, that is, $T_{n,0}=\pi \left(0\right)$. The method described above, used to compute $T_{n,0}$, can be considered a“one step method”, and it can be proved [169] that, under some specific conditions, it is a method of order $p+\omega M$.

The dynamics of processes that have multiple physics and multiscale components can be categorized into relatively fast and slow terms. The informal expressions “stiff” and “nonstiff” are commonly associated with the fast and slow evolution, respectively. So we are concerned with solving the following problem:

$$\dot{u}\left(t\right)=f\left(t,u\left(t\right)\right)+g\left(t,u\left(t\right)\right)=F\left(t,u\left(t\right)\right)$$

(185)

where the functions f and g represent the nonstiff and the stiff components of the problem, respectively, and where $u\left(t_0\right)=u_0$, $u\left(t\right)\in {\Re }^d$ and where $t\in \Theta =\left[t_{0}T\right]$.

In Constantinescu et al. [170,171], the following base “one step methods” (with step size h) ${\left\{{ }^{\left(•\right)}{\mathbf{\Psi }}_h^{n,n-1}i\right\}}_{•=[\mathrm{W}-\mathrm{IMEX},\mathrm{Pure}-\mathrm{IMEX},\mathrm{Split}-\mathrm{IMEX}]}$ are used to solve (185) by an approach based on extrapolation methods:

$$\begin{array}{ccc}\hfill u^{n+1}=& u^n+{\left[I-h\frac{\partial g}{\partial u}\left(t_n,u^n\right)\right]}^{-1}\left[hF\left(t_n,u^n\right)\right]\hfill & \hfill [\mathrm{W}-\mathrm{I}\mathrm{M}\mathrm{E}\mathrm{X}]\end{array}$$

(186)

$$\begin{array}{ccc}\hfill u^{n+1}=& u^n+hf\left(t_n,u^n\right)+{\left[I-h\frac{\partial g}{\partial u}\left(t_n,u^n\right)\right]}^{-1}\left[hg\left(t_n,u^n\right)\right]\hfill & \hfill [\mathrm{Pure}-\mathrm{I}\mathrm{M}\mathrm{E}\mathrm{X}]\end{array}$$

(187)

$$\begin{array}{ccc}\hfill u^{n+1}=& u^{\ast }+{\left[I-h\frac{\partial g}{\partial u}\left(t_n,u^n\right)\right]}^{-1}\left[hg\left(t_n,u^{\ast }\right)\right];u^{\ast }=u^n+hf\left(t_n,u^n\right)\hfill & \hfill [\mathrm{Split}-\mathrm{I}\mathrm{M}\mathrm{E}\mathrm{X}]\end{array}$$

(188)

The computation of each $T_{n_j,j}$ during the implementation of an extrapolation method (see Definition 2) can be executed independently from the others, so the wall set of tasks to compute all the $T_{n_j,j}$ values can be executed concurrently. Then, extrapolation methods, as well as improving the convergence order of a “one step method”, can be considered as a first approach to implement “parallelism“ in a naturally serial process as the ODE solution by time steppers. Indeed, current methods for transient simulations are almost exclusively sequential in time. Time progresses by employing one or more solutions from past timesteps to compute the next, regardless of how spatial domains are discretized. As a result, the number of timesteps multiplied by the speed at which one timestep can be computed determines the time to solution for transient applications.

To overcome such limitations on concurrency levels exploited by traditional time integrators, parallel-time algorithms have been studied for many years [172]. Based on Gander et al., who very recently created a unified framework for Parallel-In-Time algorithms [173], we describe them as the iterative updating of the approximation for the solution of the “all-at-once global problem” (see Equation (191) in Definition 3). In particular, following the approach described in Gander et al., the Definition 3 for the Parallel-In-Time method is given as follows,

Definition 3. 

The Parallel-In-Time (PInt) iterative solution of time-dependent ODE

The aim is to compute an approximate solution of the following ODE problem

$$\dot{u}\left(t\right)=A\left(t,u\left(t\right)\right)$$

(189)

where $u\left(t_0\right)=u_0$, $u\left(t\right)\in {\Re }^d$ and where $t\in \Theta =\left[t_{0}T\right]$. As described in Section 6.1.2 about the time integration algorithms, to compute an approximation of the solution in the considered time domain Θ, a discretization ${\Theta }_N$ of Θ should be considered. For this purpose, consider ${\Theta }_N$ as the set made up of the $N+1$ equally spaced points ${\left\{t_n\right\}}_{n=0,\dots ,N}$ where $t_n=t_0+∆\ast n,n=1,\dots ,N$ and where $∆=\frac{T}{N}$.

Let a time block (or simply block) denote the discretization of a time subinterval $\left[t_n,t_{n+1}\right]$ using $M>0$ grid points,

$${\tau }_{n,m}=t_n+{∆}_m{\tau }_m,m\in \left\{1,...,M\right\},$$

where ${\tau }_m\in \left[0,1\right]$ denotes normalized grid points in time used for all N time subintervals.

Let a block variable be a vector

$${\mathit{u}}_n={\left[u_{n,1},\dots ,u_{n,M}\right]}^T$$

where $u_{n,m}$ is an approximation of $u\left({\tau }_{n,m}\right)$ on the time block for the time subinterval $\left[t_n,t_{n+1}\right]$.

Finally, let us denote as block operators the two functions $\varphi :{\Re }^M\to {\Re }^M$ and $\chi :{\Re }^M\to {\Re }^M$ for which the block variables of a numerical solution of (189) satisfy

$$\begin{array}{ccc}\hfill \varphi \left({\mathit{u}}_1\right)& =& \chi \left(u_0\mathbf{1}\right),\hfill \\ \hfill \varphi \left({\mathit{u}}_{n+1}\right)& =& \chi \left({\mathit{u}}_n\right),n=1,\dots ,N-1\hfill \end{array}$$

(190)

where $\mathbf{1}={\left[1,\dots ,1\right]}^T$, where the time integration operator ϕ is bijective and where χ is a transmission operator. The time propagator ψ updating ${\mathit{u}}_n$ to ${\mathit{u}}_{n+1}$ is given by

$$\psi ={\varphi }^{-1}\chi .$$

The numerical approximation (190) of (189) can be described as the following all-at-once global problem

$$\left(\begin{array}{cccc}\varphi & & & \\ -\chi & \varphi & & \\ & \ddots & \ddots & \\ & & -\chi & \varphi \end{array}\right)\left[\begin{array}{c}{\mathit{u}}_1\\ {\mathit{u}}_2\\ ⋮\\ {\mathit{u}}_N\end{array}\right]=\left[\begin{array}{c}\chi \left(u_0\mathbf{1}\right)\\ \mathbf{0}\\ ⋮\\ \mathbf{0}\end{array}\right]$$

(191)

Iterative PinT algorithms solve (191) by updating a vector ${\mathit{u}}^k={\left[u_1^k,\dots ,u_N^k\right]}^T$ to ${\mathit{u}}^{k+1}$ until some stopping criterion is satisfied. If the global iteration can be written as a local update for each block variable separately, the local update formula is called a block iteration.

Let a primary block iteration be an updating formula of the form

$${\mathit{u}}_{n+1}^{k+1}={\mathbf{B}}_1^0\left({\mathit{u}}_{n+1}^k\right)+{\mathbf{B}}_0^1\left({\mathit{u}}_n^{k+1}\right)+{\mathbf{B}}_0^0\left({\mathit{u}}_n^k\right),{\mathit{u}}_0^k=u_0\mathbf{1},\forall k\in \mathbb{N},$$

(192)

where $n\ge 0$, and where ${\mathbf{B}}_1^0$, ${\mathbf{B}}_0^1$, ${\mathbf{B}}_0^0$ are operators that satisfy the consistency condition

$$\left({\mathbf{B}}_1^0-\mathit{I}\right)\psi +{\mathbf{B}}_0^1+{\mathbf{B}}_0^0=0.$$

(193)

The solution of the all-at-once global problem (191) by an iterative method based on a primary block iteration of the type (192) represents the Parallel-In-Time approach for the solution of time-dependent ODE (189).

In

Table 9. Summary of the the most popular PInt methods and their block iteration operators, where $\omega >0$ is a relaxation parameter, ${\mathit{T}}_C^F$ and ${\mathit{T}}_F^C$ are transfer operators from “coarse to fine” and “fine to coarse” time blocks, respectively, and where ${\varphi }_C$ is the time integration operator on a coarse time block.

Methods	${\mathbf{B}}_1^0\left({\mathit{u}}_{\mathit{n}+1}^{\mathit{k}}\right)$	${\mathbf{B}}_0^1\left({\mathit{u}}_{\mathit{n}}^{\mathit{k}+1}\right)$	${\mathbf{B}}_0^0\left({\mathit{u}}_{\mathit{n}}^{\mathit{k}}\right)$
DBJ	$\mathit{I}-\omega \mathit{I}$	$\omega {\varphi }^{-1}\chi$	−
ABJ	$\mathit{I}-{\tilde{\varphi }}^{-1}\varphi$	${\tilde{\varphi }}^{-1}\chi$	−
ABGS	$\mathit{I}-{\tilde{\varphi }}^{-1}\varphi$	−	${\tilde{\varphi }}^{-1}\chi$
Parareal	−	$\left({\varphi }^{-1}-{\tilde{\varphi }}^{-1}\right)\chi$	${\tilde{\varphi }}^{-1}\chi$
TMG	$\left(1-\omega \right)\left(\mathit{I}-{\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\varphi \right)$	$\omega \left({\varphi }^{-1}-{\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\right)\chi$	${\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\chi$
TMGc	−	$\left({\varphi }^{-1}-{\mathit{T}}_C^F{\tilde{\varphi }}_C^{-1}{\mathit{T}}_F^C\right)\chi$	${\mathit{T}}_C^F{\tilde{\varphi }}_C^{-1}{\mathit{T}}_F^C\chi$
TMGf	$\left(\mathit{I}-{\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\varphi \right)\left(\mathit{I}-{\tilde{\varphi }}^{-1}\varphi \right)$	$\left({\tilde{\varphi }}^{-1}-{\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\varphi {\tilde{\varphi }}^{-1}\right)\chi$	${\mathit{T}}_C^F{\varphi }_C^{-1}{\mathit{T}}_F^C\chi$
PFASST	$\left(\mathit{I}-{\mathit{T}}_C^F{\tilde{\varphi }}_C^{-1}{\mathit{T}}_F^C\varphi \right)\left(\mathit{I}-{\tilde{\varphi }}^{-1}\varphi \right)$	$\left({\tilde{\varphi }}^{-1}-{\mathit{T}}_C^F{\tilde{\varphi }}_C^{-1}{\mathit{T}}_F^C\varphi {\tilde{\varphi }}^{-1}\right)\chi$	${\mathit{T}}_C^F{\tilde{\varphi }}_C^{-1}{\mathit{T}}_F^C\chi$

, a summary of the the most popular PInt methods, and their block iteration operators, are reported (see Gander et al. [173] for details).

Algorithm 10 describes an implementation of the PInt approach. We observe that Algorithm 10 lends itself well to parallelization: indeed, if we imagine dividing the N vectors ${\mathit{u}}_n$ among P concurrent tasks, then

• The initial approximation vectors ${\mathit{u}}_n^0$ (see lines 5–7 of Algorithm 10) can be computed concurrently and independently of each other in an “embarrassingly parallel” approach (no communications are needed among tasks);
• The initial vectors ${\mathit{u}}_0^k$ (see lines 8–10 of Algorithm 10) can be computed by all the P tasks concurrently and independently of each other;
• In each iteration of the iterative method that is needed to refine the approximation of the vectors ${\mathit{u}}_n$ (see lines 11–17 of Algorithm 10), the computation of vectors ${\mathit{z}}_n^k$ can be performed concurrently and locally to the assigned task p. To complete the computation of ${\mathit{u}}_{n+1}^{k+1}$, task p should eventually send the vector ${\mathit{w}}_n^k$ to its right neighbor task $p+1$ and should receive the vector ${\mathit{z}}_{n-1}^k$ from its left neighbor task $p-1$.

Algorithm 10 The algorithm of the Parallel-In-Time iterative solution of ODE.

1: procedure Pint($\varphi$, $\chi$, $u_0$, ${\mathbf{B}}_0^0$, ${\mathbf{B}}_1^0$, ${\mathbf{B}}_0^1$, N, K, ${\left\{{\mathit{u}}_n^K\right\}}_{n=1,\dots ,N}$)   2: Input: $\varphi$, $\chi$, $u_0$, ${\mathbf{B}}_0^0$, ${\mathbf{B}}_1^0$, ${\mathbf{B}}_0^1$, N, K   3: Output:  ${\left\{{\mathit{u}}_n^K\right\}}_{n=1,\dots ,N}$   4:   ▹ Compute initial approximation vectors ${\mathit{u}}_n^0$ by a one step method applied M times   5:       for $n=1$ to N do   6:           Compute ${\mathit{u}}_n^0={\left[u_{n,1}^0,\dots ,u_{n,M}^0\right]}^T$   7:       end for   8:       for $k=1$ to Kdo                    ▹Compute vectors ${\mathit{u}}_0^k$   9:           Compute ${\mathit{u}}_0^k=u_0\mathbf{1}$ 10:       end for 11:       for $k=0$ to $K-1$do   ▹Compute approximation vectors ${\mathit{u}}_n^k$ by iterating on k 12:           for $n=0$ to $N-1$ do 13:              ${\mathit{z}}_n^k\leftarrow$Compute ${\mathbf{B}}_0^1\left({\mathit{u}}_n^{k+1}\right)+{\mathbf{B}}_0^0\left({\mathit{u}}_n^k\right)$ 14:              ${\mathit{w}}_n^k\leftarrow$Compute ${\mathbf{B}}_1^0\left({\mathit{u}}_{n+1}^k\right)$ 15:              ${\mathit{u}}_{n+1}^{k+1}\leftarrow$Compute ${\mathit{u}}_{n+1}^{k+1}={\mathit{z}}_n^k+{\mathit{w}}_n^k$ 16:           end for 17:       end for 18: end procedure

7.2.2. Composite Nonlinear Solver

An key part of contemporary simulations is large-scale algebraic solvers for nonlinear problems. Newton’s techniques begin with the nonlinear operator’s global linearization. After linearization, a huge sparse linear system is produced, and the matrix can be implicitly represented by several “matrix-free” techniques or explicitly represented by storing the nonzero coefficients [165]. As a stand-alone solver, Newton’s approach has a number of shortcomings. Memory bandwidth and communication limitations are caused by the repetitive linearization construction and solution [174].

According to Brune et al. [174] we consider a collection of solvers for nonlinear equations of the type

$$\mathit{F}\left(\mathit{u}\right)=\mathit{b}.$$

(194)

A small set of algorithmic building blocks is shown, which, through composite combinations or nonlinear preconditioning, yield a wide range of solvers with varying convergence and performance characteristics. This is an easily understood comparison to the situation of linear solvers. Implementations of this kind of solvers are available in the PETSc software library and may be brought to bear on many relevant applications.

The history of the “Portable, Extensible Toolkit for Scientific computing (PETSc)” [27,175] began in the early 1990s and has always focused on the largest-scale parallel systems available at the time. PETSc had the original goal to develop efficient algebraic solvers and provide the ability to use any appropriate solver for any suitable set of PDEs and can be considered one of the most important and widely used mathematical software libraries. New advances in PETSc software library are now available to accomplish the needs of the emerging extreme-scale architecture [176].

Let us express the action of a stationary nonlinear solver that employs the solution technique M as follows: ${\mathit{u}}_{i+1}=M\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right)$. A sequence or series of two (or more) solution techniques M and N, which both provide an approximate solution to (194), make up a nonlinear composite combination. On the other hand, by using inner technique N, nonlinear preconditioning can be expressed as a modification of the function $\mathit{F}$. After that, an outer solver M receives the modified function and uses it to solve the preconditioned system. In general, composite and preconditioning combinations of different solution methods can be considered as an effective acceleration strategy to be used in approximate solutions of Problem (194).

A compact representation for nonlinear composite combinations and preconditioning is described in

Table 10. Nonlinear composite combinations and preconditioning given outer and inner solver M and N.

Composite
Type	Statement	Abbreviation
Additive	${\mathit{u}}_{i+1}={\mathit{u}}_i+{\alpha }_M\left(M\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right)-{\mathit{u}}_i\right)+{\alpha }_N\left(N\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right)-{\mathit{u}}_i\right)$	$M+N$
Multiplicative	${\mathit{u}}_{i+1}=M\left(\mathit{F},N\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right),\mathit{b}\right)$	$M\ast N$
Preconditioning
Type	Statement	Abbreviation
Left	${\mathit{u}}_{i+1}=M\left({\mathit{u}}_i-N\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right),{\mathit{u}}_i,\mathit{b}\right)$	$M{-}_{L}N$
Right	${\mathit{u}}_{i+1}=M\left(\mathit{F}\left(N\left(\mathit{F},{\mathit{u}}_i,\mathit{b}\right),{\mathit{u}}_i,\mathit{b}\right)\right)$	$M{-}_{R}N$

. A list of solution methods M that can be considered as building blocks for different combinations are listed in Brune et al. [174]. In these works, we describe just the Newton–Krylov solution methods (“ NEWT-K”) since they are “... often the workhorse of simulations requiring solution of nonlinear equations...” [174].

In Algorithm 11, the Newton–Krylov solution methods are described: details about the cited (see line 7 in Algorithm 11) “line search” algorithm (or other “globalization strategies”) can be found in Brune et al. [174] or in Pawlowski et al. [177]. Newton-Krylov’s basic method is very sensitive to the initial iterate and may diverge if it is not sufficiently close to the solution. Generally, globalization strategies seek directions $∆\mathit{u}$ and step lengths $\lambda$ in such a way that such convergence problems can be reduced [177].

An extremely important class of methods comprises nonlinear variants of domain decomposition and multilevel algorithms. The convergence of the decomposition solvers is not guaranteed. Although they may converge, since the decomposition solvers provide additional opportunities for parallelism, the logical extension is to employ them in conjunction with the global solvers as nonlinear preconditioners or accelerators. Among these kinds of methods, we introduce the “Nonlinear Additive Schwarz Method (NASM)” [178] for its properties concerning computational “granularity,” which is desirable in a parallel computing context.

Algorithm 11 The algorithm implementing the “Newton–Krylov methods” for the approximate solution of Equation (194).

1: procedure NewtonKrylovSolver($\mathit{F}$, $\mathit{b}$, ${\mathit{u}}_0$, N, ${\mathit{u}}_N$)   2: Input: $\mathit{G}$, $\mathit{b}$, ${\mathit{u}}_0$, N   3: Output:  ${\mathit{u}}_N$   4:       for $n=0$ to $N-1$ do   5:             $\mathit{r}\left({\mathit{u}}_n\right)\leftarrow$Compute $\mathit{F}\left({\mathit{u}}_n\right)-\mathit{b}$   6:             ${\mathit{J}}_n\leftarrow$Compute ${\mathit{J}}_n$   7:             $∆{\mathit{u}}_n\leftarrow$Solve ${\mathit{J}}_n∆{\mathit{u}}_n=\mathit{r}\left({\mathit{u}}_n\right)$▹ The linear system is solved by a Krylov method   8:             ${\mathit{u}}_{n+1}\leftarrow$Compute ${\mathit{u}}_{n+1}={\mathit{u}}_n+\lambda ∆{\mathit{u}}_n$      ▹$\lambda$ is determined by line search   9:       end for 10: end procedure

Definition 4. 

The Nonlinear Additive Schwarz Method (NASM)

Let us consider the solution of problem (194) where $\mathit{u}\in {\Re }^n$, and let ${\left\{{\mathfrak{I}}_d\right\}}_{d=1,\dots ,N}$ be a set of subsets of $\mathfrak{I}=\left\{1,\dots ,n\right\}$ whose overall union coincides with $\mathfrak{I}$ (i.e., ${\bigcup }_{d=1,\dots ,N}{\mathfrak{I}}_d=\mathfrak{I}$). Each index subset $\mathfrak{I}$ defines a subdomain ${\mathfrak{D}}_d\subseteq {\Re }^{n_d}$ of dimension $n_d=\left|{\mathfrak{I}}_d\right|$ of ${\Re }^n$.

Let us consider the two sets ${\left\{{\mathit{R}}_d\right\}}_{d=1,\dots ,N}$ and ${\left\{{\mathit{P}}_d\right\}}_{d=1,\dots ,N}$ of“restriction”and“prolongation”operators, respectively, with the assumption that ${\mathit{R}}_d$ and ${\mathit{P}}_d$ satisfy ${\sum }_{d=1,\dots ,N}{\mathit{P}}_d{\mathit{R}}_d=I$.

Examples for ${\mathit{R}}_d$ and ${\mathit{P}}_d$, related with disjoint subsets ${\mathfrak{I}}_j$ (i.e., ${\mathfrak{I}}_j\cap {\mathfrak{I}}_i=\varnothing ,\forall i\ne j$), are, respectively,

$$\begin{array}{cccc}\hfill {\mathit{R}}_d:\mathit{u}\in {\Re }^n& \mapsto {\mathit{u}}^d\in {\Re }^{n_d}\hfill & \hfill where{\mathit{u}}^d& ={\left[{\mathit{u}}_k\right]}_{k\in {\mathfrak{I}}_d}^T\hfill \\ \hfill {\mathit{P}}_d:{\mathit{u}}^d\in {\Re }^{n_d}& \mapsto \mathit{u}\in {\Re }^n\hfill & \hfill where{\mathit{u}}_k& =\left\{\begin{array}{cc}{\mathit{u}}_k^d\hfill & ifk\in {\mathfrak{I}}_d\hfill \\ 0\hfill & ifk\notin {\mathfrak{I}}_d\hfill \end{array}\right.\hfill \end{array}$$

Moreover, let ${\mathit{F}}_d\left({\mathit{u}}^d\right)$ denote the“restriction”of operator $\mathit{F}\left(\mathit{u}\right)$ to the d-th subdomain ${\mathfrak{D}}_d$ defined as

$${\mathit{F}}_d:{\mathit{u}}^d\in {\mathfrak{D}}_d\mapsto {\mathit{F}}_d\left({\mathit{u}}^d\right)={\mathit{R}}_d\left(\mathit{F}\left({\mathit{P}}_d\left({\mathit{u}}^d\right)\right)\right)\in {\mathfrak{D}}_d.$$

If a set of solver method ${\left\{M_d\right\}}_{d=1,\dots ,N}$, one for each subdomain ${\mathfrak{D}}_d$, is given, the Nonlinear Additive Schwarz solver Method (NASM) is based on the following statement:

$${\mathit{u}}_{i+1}=\sum _{d=1}^N{\mathit{P}}_d\left(M_d\left({\mathit{F}}_d,{\mathit{u}}_i^d,{\mathit{b}}^d\right)\right).$$

NASM can be used combined with a Newton–Krylov method: or (1) as its left preconditioner (i.e., see Dolean et al. [178]), or (2) by using “ NEWT-K” as a local solver on each subdomain of an NASM.

7.2.3. Parallel Iterative and Direct Linear Solver

A sparse linear system solution is sometimes the most computationally demanding step in large-scale research and engineering simulations. Even though iterative approaches are increasingly being used to tackle many of these issues, direct solvers are still frequently used and will remain essential at the Exascale [156,179,180].

The fundamental advantage of direct solutions is their robustness. In fact, when solving extremely ill-conditioned linear systems, direct solvers are often the method of choice, that is, once the system matrix has been rearranged. When compared to iterative methods, where the complexity is related to the primary kernel in an iteration (which is typically a matrix-vector multiplication, the cost of which is typically proportionate to the number of nonzeros in the matrix) and the number of iterations required for convergence, the main drawbacks of direct approaches are their computational expense and memory usage [156].

Exascale computing will continue to require multilevel techniques because of their high concurrency properties. Some multi-level techniques exhibit a “natural resilience to faults” [156], which may serve as a useful foundation for the development of fault-tolerant algorithms. While multilevel approaches offer many advantageous characteristics, the applicability of any one algorithm is usually very limited. Exascale computing does, in fact, constrain algorithmic selections of multilevel approaches, conditioning selections for relaxation operators and the Multigrid schema structure (e.g., W-cycles ought to be disregarded due to their substantial communications costs) [156].

In recent years, the effort spent by some scientific communities around the world (for example, see the USA Exascale Computung Project (ECP) Multiprecision Effort Team [181]) has obtained great results in developing and deploying new mixed-precision technology. Some of the most promising and impactful achievements concern:

• Mixed-precision dense and sparse LU factorization based on Iterative Refinement (IR);
• Mixed-Precision GMRES based on IR;
• Compressed-Basis (CB) GMRES.

One of the most crucial extreme scale needs is the relaxation and lowering of synchronization, which may also help to relieve load balancing pressure. It is probably difficult to completely eliminate periodic synchronization in solvers, although many algorithms could be designed in a way that requires a lot less synchronization.

Within the category of standard iterative methods, examples of such a type of algorithm are the “s-step” CA-GMRES iterative method [21,182], which uses a redesign of the KSM algorithms based on BLAS 2 and 3 operations (products of a vector by a matrix, matrices products, etc.), which have a higher computational granularity (i.e., a higher ratio of computation time to communication time) than the original KSM formulation based on BLAS 1 operations.

Let $M_{j=0,\dots ,m-1}\left(\mathit{A}\right)$ be a suitable set of monomials of the matrix $\mathit{A}$; then, the “s-step” version of Algorithm 5, which solves problem (125), is listed in Algorithm 12, where

$$\begin{array}{ccc}\hfill v_{m+j}& =& \prod _{i=0}^j{M}_i\left(\mathit{A}\right)v_m,j=1,\dots ,s\hfill \\ \hfill {\mathfrak{Q}}_0& =& \left[v_0,\dots ,v_{s-1},v_s\right],\hfill \\ \hfill {\mathfrak{Q}}_{m+1}& =& \left[q_0,\dots ,q_{ms-1},q_{ms},q_{ms+1}\right],\hfill \\ \hfill {\overline{\mathfrak{Q}}}_{m+1}& =& \left[q_0,\dots ,q_{ms-1},q_{ms}\right],\hfill \end{array}$$

$$\begin{array}{cccccc}\hfill V_{m+1}& \in {\mathbb{R}}^{n\times s},{\mathfrak{Q}}_{m+1}\hfill & \hfill \in {\mathbb{R}}^{n\times \left(ms+1\right)},Q_{m+1}& \in {\mathbb{R}}^{n\times s},R_{m+1}\hfill & \hfill \in {\mathbb{R}}^{s\times s},{\mathfrak{H}}_{m+1}& \in {\mathbb{R}}^{\left(ms+1\right)\times ms}.\hfill \end{array}$$

In the “s-step” version of GMRES (see line 5 of Algorithm 12), multiple column vectors $V_{m+1}=\left[v_{m+1},\dots ,v_{m+s}\right]$ are calculated at the same step m, and the new orthonormal basis is computed by two normalizations steps:

• Orthonormalize $V_{m+1}$ against the orthonormal basis ${\mathfrak{Q}}_m$ and compute ${\overline{V}}_{m+1}$,
• Orthonormalize columns of ${\overline{V}}_{m+1}$ by a QR factorization [147].

More details on Algorithm 12 can be found in Hoemmen et al. [182].

Algorithm 12 The m-th step of block-based GMRES Method.

1: procedure BB-GMRES(A, y, $x_0$, m, $V_m$, ${\mathit{x}}_{m+1}$)   2: Input:A, y, $x_0$, m   3: Output:  $x_{m+1}$   4:       OrthoBegin▹ Build an orthonormal basis of column vectors ${\mathfrak{Q}}_{m+1}=\left[{\mathfrak{Q}}_m,Q_{m+1}\right]$ of ${\mathcal{K}}_{m+s}$   5:                Compute $V_{m+1}=\left[v_{m+1},\dots ,v_{m+s-1},v_{m+s}\right]$   6:                Compute ${\mathfrak{R}}_{m+1}={\mathfrak{Q}}_m^T{V}_{m+1}$   7:                Compute ${\overline{V}}_{m+1}=V_{m+1}-{\mathfrak{Q}}_m{\mathfrak{R}}_{m+1}$   8:                Compute the QR factorization of ${\overline{V}}_{m+1}=Q_{m+1}{R}_{m+1}$   9:                Compute the Hessenberg matrix ${\underline{\mathfrak{H}}}_{m+1}$ (from $R_{m+1}$ and ${\mathfrak{R}}_{m+1}$) such that $A{\overline{\mathfrak{Q}}}_{m+1}={\mathfrak{Q}}_{m+1}{\underline{\mathfrak{H}}}_{m+1}$ 10:       OrthoEnd 11:       SolBegin           ▹ Extract a suitable vector from a subspace $x_0+{\mathcal{K}}_{m+s}$ 12:                compute $y_{m+1}={argmin}_y{∥\beta e_1-{\underline{\mathfrak{H}}}_{m+1}y∥}_2$ 13:                compute $x_{m+1}=x_0+{\mathfrak{Q}}_{m+1}{y}_{m+1}$ 14:       SolEnd 15:       PrepBegin                        ▹ Prepare for the next step 16:                Assign $v_{m+1}\leftarrow q_{ms+1}$ 17:     PrepEnd 18: end procedure

In exact arithmetic, s steps of Algorithm 5 are equivalent to one step of Algorithm 12, but Algorithm 12 suffers from some instabilities when finite precision is used. A lot of work was spent in recent years to identify the origin of these phenomena and to mitigate their consequences (e.g., see Bai et al. [183] about the role of monomials $M_j\left(\mathcal{A}\right)$ on the “condition number” of $V_{m+1}$ and then on the convergence of Algorithm 12). Also, to temper instability effects, s is chosen to be $s<<n$, and the execution of Algorithm 12 is restarted [152] just after a few steps $m<<n$ [21].

7.2.4. Fast Multipole Methods and “Hierarchical” Matrices

Algorithms listed in the 20th century have included the Fast Multipole Methods (FMM), developed by Rokhlin Jr. and Greengard [184] ranked in the top ten. They are initially discussed in the context of particle simulations, where they lower the computational cost from $O\left(N^2\right)$ to $O\left(N\right)$ to $O\left(Nlog\left(N\right)\right)$ operations for every pairwise interaction in a system of N particles. During its history [185,186,187], FFM was the basis for other useful applications such as the fast multiplication of vectors with fully populated special matrices. In the context of linear algebra, FFM takes the name of fast multiplication by “$\mathcal{H}$-matrices” which are a combination of the “Panel Clustering Method”, and the “mosaic skeleton matrices” approaches [188,189].

According to [188,189], let us introduce “Hierarchical” matrices (also called $\mathcal{H}$-matrices). First of all, it is necessary to introduce the concepts of the $\mathit{R}k$-matrix and the “block cluster quad-tree”.

Definition 5. 

Definition of $\mathit{R}k$-matrix

Let $R\in {\Re }^{M\times N}$ be a matrix of the form

$$R=A{B}^T,A\in {\Re }^{M\times k},B\in {\Re }^{k\times N}.$$

Then, R is called an $\mathit{R}k$-matrix. Any matrix of a rank of at most k can be represented as an $\mathit{R}k$-matrix, and each $\mathit{R}k$-matrix has at most a rank of k.

Definition 6. 

Definition of “block cluster quad-tree” of a matrix A

Let ${\mathcal{T}}_I\left(L\right)$ be a binary tree [190] with the $L+1$ levels $l=0,\dots ,L$ and denote by $T_I$ the set of its nodes. ${\mathcal{T}}_I$ is called a binary cluster tree corresponding to an index set I if the following conditions hold:

• Each node of $T_I$ is a subset of the index set I;
• I is the root of ${\mathcal{T}}_I\left(L\right)$ (i.e., the node at the 0-th level of ${\mathcal{T}}_I\left(L\right)$);
• If $\tau \in T_I$ is a leaf (i.e, a node at the L-th level of ${\mathcal{T}}_I\left(L\right)$), then $\left|\tau \right|\le C_{leaf}$;
• If $\tau \in T_I$ is not a leaf whose set of sons is represented by $S\left(\tau \right)\subseteq T_I$, then $\left|S\left(\tau \right)\right|=2$ and $\tau ={\dot{\bigcup }}_{{\tau }^{\prime }\in S\left(\tau \right)}{\tau }^{\prime }$

Let I be an index and let $AC\left(\tau \times \sigma \right)=0,1$ a logical value representing an“Admissibility Condition”on $\tau \times \sigma$. Moreover, let ${\mathcal{T}}_I\left(L\right)$ be a binary cluster tree on the index set I. The block cluster quad-tree ${\mathcal{T}}_{I\times I}$ corresponding to ${\mathcal{T}}_I\left(L\right)$ and to the admissibility condition $AC\left(\tau \times \sigma \right)$ could be built by the procedure represented in Algorithm 13.

Algorithm 13 Procedure for building the block quad-tree ${\mathcal{T}}_{I\times I}\left(L\right)$ corresponding to a cluster tree ${\mathcal{T}}_I\left(L\right)$ and an admissibility condition $AC$. The index set $\tau \times \sigma$ and the value l to be used in the first call to the recursive BlockClusterQuadTree procedure are such that $\tau =\sigma =I,l=0$.

1: procedure BlockClusterQuadTree(${\mathcal{T}}_{I\times I}\left(L\right)$, L, l, $AC\left(\tau \times \sigma \right)$, $\tau \times \sigma$)   2: Input: ${\mathcal{T}}_{I\times I}\left(L\right)$, $AC\left(\tau \times \sigma \right)$, L, l, $\tau \times \sigma$   3:     if ($AC\left(\tau \times \sigma \right)=0$ and $l<L$) then   4:         $S\left(\tau \times \sigma \right)=\left\{{\tau }^{\prime }\times {\sigma }^{\prime }:{\tau }^{\prime }\in S\left(\tau \right),{\sigma }^{\prime }\in S\left(\sigma \right)\right\}$   5:         for ${\tau }^{\prime }\times {\sigma }^{\prime }\in S\left(\tau \times \sigma \right)$ do   6:              BlockClusterQuadTree(${\mathcal{T}}_{I\times I}\left(L\right)$, L, $l+1$, $AC\left({\tau }^{\prime }\times {\sigma }^{\prime }\right)$, ${\tau }^{\prime }\times {\sigma }^{\prime }$)   7:         end for   8:     else   9:         $S\left(\tau \times \sigma \right)=\varnothing$ 10:     end if 11: end procedure

Definition 7. 

Definition of $\mathcal{H}$-matrix

Let $A\in {\Re }^{N\times N}={\left(a_{i,j}\right)}_{i,j=1,\dots ,N}$ be a matrix, let $I=\left\{1,\dots ,N\right\}$ the index set of A, and let $k\in \mathcal{N}$. Let us assume that, for a vector v and a subset $\tau \subset I$, $v_{\left|\tau \right.}$ represents the restriction vector ${\left(v_j\right)}_{j\in \tau }$ of v to τ, and that, for a matrix A and subsets $\tau ,\sigma \subset I$, the notation $A_{\left|\tau \times \sigma \right.}$ represents the block ${\left(A_{ij}\right)}_{i\in \tau ,j\in \sigma }$. And let ${\mathcal{T}}_{I\times I}\left(L\right)$ be the block cluster quad-tree on the index set I whose admissibility condition $AC\left(\tau \times \sigma \right)$ is defined as

$$AC\left(\tau \times \sigma \right)=\left\{\begin{array}{cc}1\hfill & if{A}_{\left|\tau \times \sigma \right.}can be approximated by an \mathit{R}k-matrix in a specified norm∥·∥\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

Then, the matrix A is called the $\mathcal{H}$-matrix of blockwise rank k defined on block cluster quad-tree ${\mathcal{T}}_{I\times I}\left(L\right)$.

Let us remember that, given a matrix A, the matrix $\tilde{A}$ is said to be an approximation of A in a specified norm $∥·∥$ if there exists $ϵ\in \Re$ such that $∥A-\tilde{A}∥<ϵ$.

Different approaches can be used to compute an $\mathit{R}k$-matrix approximation $\tilde{M}=\widetilde{M_1}\widetilde{M_2}$ of an arbitrary matrix M [188]; as an example, we cite the one based on the “truncated singular value decomposition,” which represents the best approximation of an arbitrary matrix in the spectral and Frobenius norm (see [189] for details).

As an example of the use of the $\mathcal{H}$-matrices to obtain algorithms of reduced complexity, we propose the algorithm for the calculation of the matrix-vector product (see Algorithm 14): the complexity for that matrix-vector multiplication algorithm is $O\left(knlog\left(n\right)\right)$. Other algorithms of interest from basic matrix algebra that use $\mathcal{H}$-matrices are described in Hackbusch et al. [189].

Algorithm 14 Matrix-vector multiplication $y=y+Ax$ of the $\mathcal{H}$-matrix $A\in {\Re }^{I\times I}$ (defined on block cluster quad-tree ${\mathcal{T}}_{I\times I}\left(L\right)$) with vector $x\in {\Re }^I$. The index sets $\tau \times \sigma$ to be used in the first call to the recursive HMatrix-MVM procedure are such that $\tau =\sigma =I$.

1: procedure HMatrix-MVM(A, y, x, ${\mathcal{T}}_{I\times I}\left(L\right)$, $\tau \times \sigma$)   2: Input: A, y, x, ${\mathcal{T}}_{I\times I}\left(L\right)$, $\tau \times \sigma$   3: Output:  y   4:     if $S\left(\tau \times \sigma \right)\ne \varnothing$ then               ▹$\tau \times \sigma$ is not a leaf of ${\mathcal{T}}_{I\times I}\left(L\right)$   5:         for each ${\tau }^{\prime }\times {\sigma }^{\prime }\in S\left(\tau \times \sigma \right)$ do   6:              HMatrix-MVM(A, y, x, ${\mathcal{T}}_{I\times I}\left(L\right)$, ${\tau }^{\prime }\times {\sigma }^{\prime }$)   7:         end for   8:     else                       ▹$\tau \times \sigma$ is a leaf of ${\mathcal{T}}_{I\times I}\left(L\right)$   9:         $y_{\left|\tau \right.}\leftarrow y_{\left|\tau \right.}+A_{\left|\tau \times \sigma \right.}{x}_{\left|\sigma \right.}$    ▹$A_{\left|\tau \times \sigma \right.}$ could be an unstructured or an $\mathit{R}k$-matrix 10:     end if 11: end procedure

7.2.5. Parallel Monte Carlo Based Methods

Monte Carlo-like algorithms seem to have all the characteristics that make them suitable for Exascale computing systems. Taking inspiration from [191,192], we can observe that it is easy enough to implement strategies that make them algorithms with the following attributes.

• Communication- and synchronization-avoiding: To implement parallelism in an MC-like algorithm, the named dynamic bag-of-work model can be used [191]. Using such a strategy, a large task, related to the computation of M states, is split into smaller independent P subtasks, where each task computes $\frac{M}{P}$ states and where all the P subtasks are executed independently of each other in an “embarrassingly parallel” approach.
• Fault-Tolerant and Resilient: To make an MC-like algorithm resilient to hard failures, an P-out-of-Q strategy can be used [191]. Since MC-based applications “accuracy” depend only on the number M of computed states and not on which random sample set is estimated (provided that all the random samples are independent in a statistical sense), the actual size of the computation is obtained by increasing the number of subtasks from P to Q, where $Q>P$. The whole task is considered to be completed when P partial results, from each subtask of size $\frac{M}{P}$ states, are ready. In the P-out-of-Q approach, more subtasks are needed than are actually scheduled. Therefore, none of these subtasks will become a “key” subtask and at most $Q-P$ faulted subtasks can be tolerated. The critical aspect is properly choosing the value Q since a “good” value for Q may prevent a few subtasks from delaying or halting the whole computation. On the other hand, an excessive value for Q could result in a much higher calculation workload with little gain to the computation outcomes. The proper choice of Q in the P-out-of-Q strategy can be determined by considering the average job-completion rate in the computing system. Suppose c is the completion probability of subtasks up to time t in the computing system. Clearly, $Q\ast c$ should be approximately P, i.e., the fraction of the Q subtasks finished should equal to P. Thus, a good choice is $Q=⌈\frac{P}{c}⌉$ [191].
  To make an MC-like algorithm resilient to a soft fault causing an error in the computation, for example, due to a change in data value in memory, strategies such as the “duplicate checking” or the “majority vote” [191] could be considered. Subtasks are repeated and executed on separate nodes in these algorithms. Comparing the outcomes of the same subtask carried out on other nodes might help identify inaccurate computational results (e.g., by ensuring that the results fall within a suitable confidence interval). When performing duplicate checking, an incorrect result must be found by doubling computations. It takes at least three times as much calculation to find an incorrect computation result in a majority vote.

8. Conclusions and Future Work

This work aims at providing a comprehensive summary of the models and methods that can be employed for “in silico” experimentation of the physicochemical processes underlying the process of 3D bioprinting of cell-laden materials by using the computational resources available in the Exascale Era.

Until now, the only way to optimize the properties of cell-laden hydrogels and the 3D bioprinting process has been to replicate multiple expensive and time-consuming trials. The advent of computational models has allowed us to better understand the different phases of the process. However, there are still many challenges to address. Massive data sets, appropriate method selection, and the identification of relevant inputs and outputs are all necessary for the modeling. Large-scale data collection from bioprinting research involving expensive cells and supplies, as well as labor-intensive procedures, could not be feasible. The challenge relating to the selection of numerical methods concerns the development of fully coupled multiscale models, now more affordable in the Exascale Era, that (1) allow the chemical/physical processes of interest to be described more fully and precisely and (2) are solvable effectively and efficiently by new algorithms on the complex computing architectures available in the Exascale Era (for example, consider the definition of the most appropriate preconditioners for iterative methods that are built based on the nature of the reference chemical/physical problem).

This work is the result of an interdisciplinary collaboration since the two authors, respectively, carry out research activities in the fields of material sciences (with particular regard to bioprinting) and computational sciences (with particular regard to algorithms in the HPC context). Then, this work can be considered the synthesis of two inventory processes, the first relating to the numerical techniques already used for the simulation of bioprinting processes, the second relating to the availability of versions of these techniques capable of fully exploiting the new HPC resources in the Exascale Era.

Furthermore, this work intends to lay the foundations for the ambitious project of building an inventory of “digital skills” and “mathematical knowledge” both necessary for European “twin green-digital” transitions based on the case study of advanced materials for health. This case study is taken from the Materials Innovation Markets (MIMs) (defined by the Materials 2030 roadmap of the AMI2030 [193] initiative) which are the markets of primary interest for Europe in terms of impacts (on people, planet, and prosperity), in which advanced materials play a key enabling role. To achieve these goals, Europe intends to maximize the sustainability characteristics of new advanced materials and their visibility using advanced digital technologies (the so-called “Materials Digitalization”) in the MIMs mentioned above.

Unfortunately, due to a mere question of space, in the drafting of this work, some of the most promising tools in the field of “digitalization of materials” were not included. Among these tools should be cited the techniques relating to (1) the construction of models starting from data (for example through the use of tools for “learning from data”, see [56,57,194,195]) or (2) the possibility of correcting/defining first-principles-based models using data (see for instance the data assimilation approach [196,197]). Combinations of both the techniques listed above should be investigated to implement hybrid approaches. For example, Bradley et al. [8] describe some hybrid approaches to combine first-principles models with data-driven methods, and Carlberg et al. [198] describe an approach to combine the data-driven method with the Parallel-In-Time method. Furthermore, another important topic to consider is that 3D bioprinting is evolving toward 4D Bioprinting, in which 3D bioprinted tissues can adapt in terms of shape, size, or pattern over time, according to external stimuli, adding another level of complexity to the process [199]. These important aspects will be the main objective of our future work.