An Alternate GPU-Accelerated Algorithm for Very Large Sparse LU Factorization

Abstract

The LU factorization of very large sparse matrices requires a significant amount of computing resources, including memory and broadband communication. A hybrid MPI + OpenMP + CUDA algorithm named SuperLU3D can efficiently compute the LU factorization with GPU acceleration. However, this algorithm faces difficulties when dealing with very large sparse matrices with limited GPU resources. Factorizing very large matrices involves a vast amount of nonblocking communication between processes, often leading to a break in SuperLU3D calculation due to the overflow of cluster communication buffers. In this paper, we present an improved GPU-accelerated algorithm named SuperLU3D_Alternate for the LU factorization of very large sparse matrices with fewer GPU resources. The basic idea is “divide and conquer”, which means dividing a very large matrix into multiple submatrices, performing LU factorization on each submatrix, and then assembling the factorized results of all submatrices into two complete matrices L and U. In detail, according to the number of available GPUs, a very large matrix is first divided into multiple submatrices using the elimination tree. Then, the LU factorization of each submatrix is alternately computed with limited GPU resources, and its intermediate LU factors from GPUs are saved to the host memory or hard disk. Finally, after finishing the LU factorization of all submatrices, these factorized submatrices are assembled into a complete lower triangular matrix L and a complete upper triangular matrix U, respectively. The SuperLU3D_Alternate algorithm is suitable for hybrid CPU/GPU cluster systems, especially for a subset of nodes without GPUs. To accommodate different hardware resources in various clusters, we designed the algorithm to run in the following three cases: sufficient memory for GPU nodes, insufficient memory for GPU nodes, and insufficient memory for the entire cluster. The results from LU factorization test on different matrices in various cases show that the larger the matrix is, the more efficient this algorithm is under the same GPU memory consumption. In our numerical experiments, SuperLU3D_Alternate achieves speeds of up to 8× that of SuperLU3D (CPU only) and 2.5× that of SuperLU3D (CPU + GPU) on the hybrid cluster with six Tesla V100S GPUs. Furthermore, when the matrix is too big to be handled by SuperLU3D, SuperLU3D_Alternate can still utilize the cluster’s host memory or hard disk to solve it. By reducing the amount of data exchange to prevent exceeding the buffer’s limit of the cluster MPI nonblocking communication, our algorithm enhances the stability of the program.

1. Introduction

Numerical problems often require the solution of large sparse systems of linear equations, such as those found in finite difference approximations of three-dimensional partial differential equations [1]. These computations can be time-consuming, and the most important part of the process is solving these large systems of equations. There are two main categories of methods for solving such systems: direct and iterative. Direct solvers include MUMPS (Multifrontal Massively Parallel Sparse Direct Solver) [1,2], PARDISO [3], and SuperLU (Supernodal Lower-Upper Factorization) [4,5,6,7]. The common iterative solvers are PETSc (Portable, Extensible Toolkit for Scientific Computation) [8], Trilinos [9], and HYPRE (High-Performance Preconditioners) [10]. However, iterative methods may suffer from slow convergence or divergence issues when dealing with complex matrices, whereas direct methods require high computing resources. In certain simulations, such as three-dimensional geodynamical simulations [11,12,13,14], solving a system of linear partial differential equations comprising the Stokes equations and mass continuity equations can result in matrices with hundreds of millions of dimensions. If the number of model grids is 500 × 500 × 100, it will require over 1 TB of host memory and 1 TB of GPU memory for traditional direct solvers with GPU acceleration. As the number of grids and computing accuracy increases, so does the demand for computing resources, making it difficult for small- and medium-sized clusters to solve such problems.

The solution of $\mathit{Ax}=\mathit{b}$ using the LU factorization method is divided into two logical steps. The first step is to factorize matrix A into the product of a lower triangular matrix L and an upper triangular matrix U. The second step is to obtain the solution x of the system of equations by solving $\mathit{L}\mathit{y}=\mathit{b}$ and then solving $\mathit{Ux}=\mathit{y}$. Since a large number of matrix operations are needed in LU factorization, GPU acceleration is a great way to significantly improve the computational efficiency.

In solving a very large system of linear equations with GPU acceleration, the first step takes much more time than the second step. Calculating matrices L and U requires a large amount of memory and communication among GPUs. It is challenging to factorize very large sparse matrices efficiently when the computing resources, such as the GPU memory and MPI communication buffer, cannot meet the requirements.

Among LU factorization solvers, the SuperLU solver is a supernode factorization method using a hybrid MPI + OpenMP programming model. The three libraries within SuperLU are Sequential SuperLU, Multithreaded SuperLU (SuperLU_MT), and Distributed SuperLU (SuperLU_DIST). The SuperLU_DIST library contains a 2D and 3D SuperLU_DIST algorithm. The 3D SuperLU_DIST (SuperLU3D) algorithm with GPU acceleration achieves an increased speed of 10~54× and 4.2~6.6× for solving 2D and 3D PDE problems, respectively, compared with the PARDISO parallel algorithm based on MKL (Math Kernel Library) [4,5]. Therefore, making full use of GPU acceleration can greatly improve computational efficiency. Based on SuperLU_DIST, Gaihre introduced the GSOFA algorithm to enable fine-grained symbolic factorization. This algorithm redesigned a supernode detection to expose massive parallelism, balanced workload and minimal inter-node communication, as well as introduced a three-pronged space optimization to handle large sparse matrices [15]. All the SuperLU algorithms are implemented with parallel extensions using OpenMP for shared-memory programming, or MPI + CUDA for distributed-memory programming. The 3D SuperLU_DIST (SuperLU3D) algorithm with GPU acceleration is the most efficient one among these algorithms but faces the difficulty of not being able to solve very large sparse matrices with limited GPU resources, and factorizing very large matrices makes a vast amount of nonblocking communication between processes that can cause a break in the SuperLU3D calculation due to cluster communication buffer overflow.

To address the issues above, we developed a new algorithm called SuperLU3D_Alternate. It uses an approach similar to a time-sharing operating system, where the matrix is divided into submatrices and LU factorization is performed on each submatrix alternately using limited GPU resources. Intermediate LU factors are saved to the host memory or hard disk, and the factorized submatrices are then assembled to form two triangle matrices L and U. In our numerical experiments, SuperLU3D_Alternate achieves speeds of up to 8× that of SuperLU3D (CPU only) and 2.5× that of SuperLU3D (CPU + GPU) on a hybrid cluster with six Tesla V100S GPUs. The new algorithm reduces the amount of data exchange to prevent the cluster’s MPI nonblocking communication buffer from being overloaded, thereby improving the stability of the algorithm. Additionally, if the GPU memory is insufficient for the LU factorization of the entire matrix, SuperLU3D_Alternate can still utilize the cluster’s host memory or hard disk to save the intermediate LU factors of submatrices.

2. SuperLU3D_Alternate Algorithm

2.1. Background

A communication-avoiding 3D algorithm for sparse LU factorization named SuperLU3D in this paper is based on the 2D SuperLU_DIST algorithm. The 2D SuperLU_DIST algorithm is a supernodal factorization parallel algorithm using a hybrid MPI + OpenMP programming model. A supernode is a set of consecutive columns of matrix L with a dense triangular block below the diagonal and with the same nonzero structure below the triangular block (Figure 1) [5,7,15]. Despite its efficiency, 2D SuperLU_DIST has two major limitations that impact its parallel efficiency: the sequential Schur-complement update and fixed latency cost [4]. To address these limitations, the SuperLU3D algorithm uses a 3D logical process grid, which greatly improves its computational efficiency, achieving ~27 times higher efficiency than 2D SuperLU_DIST. SuperLU3D also supports a hybrid architecture with both a multicore CPU and coprocessors such as GPU or Xeon-Phi, with GPU acceleration providing 1.4–3.5 times faster performance on smaller 2D process grids [4]. Furthermore, this algorithm is also applicable to unsymmetrical matrices [6,7].

Table 1. List of symbols mentioned.

Symbol	Description
P	number of MPI processes
$P_x,P_y,P_z$	3D process grid dimensions
$P_{xy}$	number of processes in xy plane
$P\{k,:\}$	groups of processes assigned to the k-th supernodal row
$P\{i:j,:\}$	i~j-th group of processes assigned to the k-th supernodal row (i < j)
$P\{:,k\}$	groups of processes assigned to the k-th supernodal column
$P\{:,i:j\}$	i~j-th groups of processes assigned to the k-th supernodal column (i < j)
$P_{i,j}$	No. $mod\left(i,P_x\right)\times P_y+mod\left(j,P_y\right)$ process
pid	process ID
${\mathit{p}}_i$	the i-th submatrix
${grid}_i$	the i-th 2D process grid
n	dimension of a matrix
nnz	the number of nonzero elements in a matrix
E	elimination tree
l	level of elimination tree ($l={\log }_2{P}_z$)
m	number of compute nodes in the cluster
${Mnode}_i$	memory size of the i-th compute node
$M_{gpu}$	GPU DRAM
$n_{pi}$	number of processes of the i-th node
$k_i$	number of GPU of the i-th node
$GPUnodenum$	number of nodes with GPU ($k_i>0$)
$P_{z\left(lvl\right)}$	number of submatrices at lvl-th level
$M_{ihost}\left({\le Mnode}_i\right)$	memory occupied by LU factors per process
$M_{igpu}\left(\le M_{gpu}\right)$	GPU memory occupied by LU factors per process
$M_{dev2host}$	host memory copied from GPU per process
${M_{g2h}}_j^i$	host memory of j-th process in ${grid}_i$ copied from j-th GPU
grid3d.comm	the 3D MPI communicator

summarizes the symbols used in the subsequent sections.

2.1.1. Elimination Trees

A fundamental concept in sparse matrix factorization is an elimination tree [16]. An n × n symmetric matrix has an n-node elimination tree. The relative position of the matrix elements determines the parent–child relationship among the tree nodes. If the first nonzero element in the j-th column is in the i-th row and i > j, node-i in the tree will be the parent node of node-j. We define this parent–child relationship by connecting node-i and node-j with a line (Figure 2). We express this relationship by

$$i=parent\left(j\right)=\left\{\begin{array}{c}\min \{k\in ro{w}_{index\left(j\right)}\},ro{w}_{index\left(j\right)}\ne NULL\\ j ,ro{w}_{index\left(j\right)}=NULL\end{array}\right.$$

where $row\_index\left(j\right)$ is the number set of rows where nonzero elements in the j-th column are located. We define a 16 × 16 sparse matrix A to illustrate the establishment process of the elimination tree (Figure 2). The blank position of matrix A is zero, and $a_{ij}$ is a nonzero element at the i-th row and j-th column. Sparse LU factorization by columns can be represented by an elimination tree. The dependency of column update is determined by the elimination tree. Therefore, we can utilize elimination trees to compute different subtrees with same parent node concurrently.

$$\mathit{A}=\left(\begin{array}{cccccccccccccccc}1& & & & & \times & \times & & & & & & & & \times & \times \\ & 2& & \times & & & \times & & & & & & & & \times & \times \\ & & 3& \times & & & \times & & & & & & & & \times & \times \\ & \times & \times & 4& & \times & \times & & & & & & & & \times & \times \\ & & & & 5& \times & \times & & & & & & & & \times & \times \\ \times & & & \times & \times & 6& \times & & & & & & & & \times & \times \\ \times & \times & \times & \times & \times & \times & 7& & & & & & & & \times & \times \\ & & & & & & & 8& & & & & & \times & \times & \times \\ & & & & & & & & 9& & \times & & & \times & \times & \times \\ & & & & & & & & & 10& \times & & & \times & \times & \times \\ & & & & & & & & \times & \times & 11& & \times & \times & \times & \times \\ & & & & & & & & & & & 12& \times & \times & \times & \times \\ & & & & & & & & & & \times & \times & 13& \times & \times & \times \\ & & & & & & & \times & \times & \times & \times & \times & \times & 14& \times & \times \\ \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & 15& \times \\ \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & \times & 16\end{array}\right)$$

2.1.2. Block Algorithm for LU Factorization

We describe the block algorithm for LU factorization in a simplest case. Suppose matrix A is decomposed into 2 × 2 blocks and the equation A = LU can be written as

$$\left(\begin{array}{cc}{\mathit{A}}_{11}& {\mathit{A}}_{12}\\ {\mathit{A}}_{21}& {\mathit{A}}_{22}\end{array}\right)=\left(\begin{array}{cc}{\mathit{L}}_{11}& \\ {\mathit{L}}_{21}& {\mathit{L}}_{22}\end{array}\right)\left(\begin{array}{cc}{\mathit{U}}_{11}& {\mathit{U}}_{12}\\ & {\mathit{U}}_{22}\end{array}\right),$$

(1)

where A is a n × n sparse matrix and ${\mathit{A}}_{11}$ is a d × d sparse matrix (d < n, mod(n, d) = 0).

Based on matrix multiplication, ${\mathit{A}}_{11}$, ${\mathit{A}}_{12}$, ${\mathit{A}}_{12}$, and ${\mathit{A}}_{22}$ can be formulated as

$${\mathit{A}}_{11}={\mathit{L}}_{11}{\mathit{U}}_{11},$$

(2)

$${\mathit{A}}_{12}={\mathit{L}}_{11}{\mathit{U}}_{12},$$

(3)

$${\mathit{A}}_{21}={\mathit{L}}_{21}{\mathit{U}}_{11},$$

(4)

$${\mathit{A}}_{22}={\mathit{L}}_{21}{\mathit{U}}_{12}+{\mathit{L}}_{22}{\mathit{U}}_{22},$$

(5)

The steps of the block algorithm for LU factorization are as follows:

(1)

Diagonal factorization: ${\mathit{L}}_{11}$ and ${\mathit{U}}_{11}$ are obtained from LU factorization of block matrix ${\mathit{A}}_{11}$ in Equation (2);

(2)

Panel updates: ${\mathit{U}}_{12}={\mathit{L}}_{11}^{-1}{\mathit{A}}_{12}$ and ${\mathit{L}}_{21}={\mathit{A}}_{21}{\mathit{U}}_{11}^{-1}$;

(3)

Schur-complement update: ${\mathit{L}}_{22}{\mathit{U}}_{22}={\mathit{A}}_{22}-{\mathit{L}}_{21}{\mathit{U}}_{12}$, and update ${\mathit{A}}_{22}={\mathit{L}}_{22}{\mathit{U}}_{22}$;

Then, we continue the LU partition factorization of ${\mathit{A}}_{22}$ until the dimension of ${\mathit{A}}_{22}$ is equal to d.

2.1.3. The 2D SuperLU_DIST Algorithm

The 2D SuperLU_DIST algorithm was designed for distributed memory parallel processors, using MPI for interprocess communication. It is able to utilize hundreds of parallel processors effectively on sufficiently large matrices [6,17]. It makes one-dimensional MPI processes into a new two-dimensional communicator. The algorithm distributes the supernodal blocks in a two-dimensional $P_x\times P_y$ block-cyclic fashion. This means that block $(I,J)$ ($0\le I,J\le N-1$, $N$ is the number of supernodal blocks) is mapped into the process at coordinate $\left(mod\left(I,P_x\right),mod\left(j,P_y\right)\right)$ of the 2D $P_x\times P_y$ process grid. The size of each supernodal block can be determined from the dimension of the diagonal matrix in the supernodes. The matrix in Figure 3 illustrates such a partition [18]. More specifically, the algorithm for the k-th supernode involves the following steps [4]:

(1)

Diagonal factorization: The process $P_{k,k}$ factorizes ${\mathit{A}}_{kk}$ into ${\mathit{L}}_{kk}{\mathit{U}}_{kk}$. The matrix block ${\mathit{A}}_{kk}$ belongs to $P_{k,k}$;

(2)

Diagonal broadcast (blue arrows in Figure 3): the process $P_{k,k}$ broadcasts ${\mathit{L}}_{kk}$ along its process row $P\{k,:\}$ to block ${\mathit{A}}_{k,:}$, and broadcasts ${\mathit{U}}_{kk}$ along its process column $P\{:,k\}$ to block ${\mathit{A}}_{:,k}$;

(3)

Panel solve: Each process in the $P\{k,:\}$ calculates ${\mathit{U}}_{k,:}={\mathit{L}}_{kk}^{-1}{\mathit{A}}_{k,:}$, and converts the ${\mathit{U}}_{k,:}$, which is stored in a sparse format, into a dense format, ${\stackrel{\mathrm{~}}{\mathit{U}}}_{k,:}$ (because matrix ${\mathit{L}}_{:,k}$ is a dense supernode mentioned in Section 2.1). Each process in the $P\{:,k\}$ calculates ${\mathit{L}}_{:,k}={\mathit{A}}_{:,k}{\mathit{U}}_{kk}^{-1}$;

(4)

Panel broadcast: each process in $P\{k,:\}$ or $P\{:,k\}$ broadcasts blocks of ${\stackrel{\mathrm{~}}{\mathit{U}}}_{k,:}$ or ${\mathit{L}}_{:,k}$ to its process column or row, respectively;

(5)

Schur-complement update: the process $P_{i,j}$ updates ${\mathit{A}}_{ij}$ by ${\mathit{A}}_{ij}={\mathit{A}}_{ij}-{\mathit{L}}_{ik}{\stackrel{\mathrm{~}}{\mathit{U}}}_{kj}$, where the process $P_{i,j}$ is No. $mod\left(i,P_x\right)\times P_y+mod\left(j,P_y\right)$ of process.

2.1.4. The SuperLU3D Algorithm

The SuperLU3D algorithm, developed from the 2D SuperLU_DIST algorithm, assigns multiple leaf nodes of an elimination tree to different processes to make smaller independent submatrices, which can reduce the communication cost during the factorization of each submatrix [4]. This is achieved by dividing the elimination tree into independent subtrees and a common ancestor tree of all the subtrees. Each subtree is assigned to a different 2D process grid, while the common ancestor tree is replicated across all process grids to obtain global matrices L and U [4]. As an example, the matrix A in Section 2.1.1 can be divided into two subtrees and a common ancestor tree involving node-16 and node-15.

According to the SuperLU3D algorithm, the matrix A in Section 2.1.1 can be written as

$$\mathbf{A}=\left(\begin{array}{ccc}{\mathit{A}}_{11}& & {\mathit{A}}_{13}\\ & {\mathit{A}}_{22}& {\mathit{A}}_{23}\\ {\mathit{A}}_{31}& {\mathit{A}}_{32}& {\mathit{A}}_{33}\end{array}\right).$$

Using the elimination tree (Figure 2), we divide the matrix A into two independent submatrices ${\mathit{A}}_0=\left(\begin{array}{cc}{\mathit{A}}_{11}& {\mathit{A}}_{13}\\ {\mathit{A}}_{31}& {\mathit{A}}_{33}^0\end{array}\right)$ and ${\mathit{A}}_1=\left(\begin{array}{cc}{\mathit{A}}_{22}& {\mathit{A}}_{23}\\ {\mathit{A}}_{32}& {\mathit{A}}_{33}^1\end{array}\right)$. ${\mathit{A}}_{11}$ and ${\mathit{A}}_{22}$ are 7 × 7 matrices, and ${\mathit{A}}_{33}^0={\mathit{A}}_{33}$, ${\mathit{A}}_{33}^1=0$. ${\mathit{A}}_0$ and ${\mathit{A}}_1$ are factorized on 2D process grid-0 and grid-1, respectively, which has the same number of processes (Figure 3). This algorithm considers a collection of 2D grids as a 3D process grid. We refer to the z-dimension of the 3D process grid as $P_z$, which is also equal to the number of submatrices. Based on a block algorithm for LU factorization (Section 2.1.2), after the Schur-complement update in parallel, ${\mathit{A}}_{33}^0={\mathit{A}}_{33}^0-{\mathit{L}}_{31}{\mathit{U}}_{13}$ and ${\mathit{A}}_{33}^1=-{\mathit{L}}_{32}{\mathit{U}}_{23}$. Then, we reduce ${\mathit{A}}_{33}^1$ from grid-1 to grid-0 and update ${\mathit{A}}_{33}={\mathit{A}}_{33}^0+{\mathit{A}}_{33}^1={\mathit{A}}_{33}^0-{\mathit{L}}_{31}{\mathit{U}}_{13}-{\mathit{L}}_{32}{\mathit{U}}_{23}$. Finally, we factorize ${\mathit{A}}_{33}$ to ${\mathit{L}}_{33}$ and ${\mathit{U}}_{33}$. Eventually, the L and U of A are distributed among the two process grids. Using the 3D process grid, the SuperLU3D algorithm performs the Schur-complement update of different subtrees or submatrices in parallel [4].

2.2. The SuperLU3D_Alternate Algorithm

In the context of very large matrices, the SuperLU3D algorithm with GPU acceleration faces two major computing problems. Firstly, if there is insufficient GPU memory, all submatrices cannot be decomposed simultaneously. Secondly, the simultaneous decomposition of each submatrix results in extensive nonblocking communication between processes, which may overload the cluster communication buffer and cause program failure. To address these issues, we propose a new algorithm called SuperLU3D_Alternate. The algorithm performs the LU factorization of each submatrix using GPU acceleration alternately, similar to the time-sharing operating system. Another algorithm, GSOFA, also tackles this problem by dividing the submatrix and temporarily buffering some data into the host memory. But all submatrices are still executed in parallel [15].

The SuperLU3D_Alternate algorithm is designed to address the challenges of LU factorization for very large sparse matrices. The algorithm divides the matrix into multiple submatrices using the elimination tree and processes the LU factorization of each submatrix on the same GPUs alternately. During the intermediate process, we synchronize the intermediate L and U of each submatrix in the GPU memory to the host memory or hard disk, and free GPU memory. Once the LU factorization of each submatrix is completed in turn, the submatrices are assembled to form a complete matrix L and a complete matrix U. By decomposing independent submatrices with GPU acceleration, the SuperLU3D_Alternate algorithm provides an efficient and scalable approach to solving very large systems of linear equations.

The implementation details of the SuperLU3D_Alternate algorithm are slightly different in various types of computing environments. To better describe the algorithm, we have the following settings:

(1)

Cluster computing resources: We assume that the hybrid CPU/GPU cluster is composed of m compute nodes. The i-th node has memory ${Mnode}_i$ and $k_i\left(k_i\ge 0\right)$ GPUs with the same video memory size ($M_{gpu}$), where $1\le i\le m$. The number of compute nodes of $k_i>0$ is defined as $GPUnodenum$, and the nodes are sorted from the largest of $k_i$ to the smallest;

(2)

Matrix blocking, process binding, and data structures: For simplicity, it is assumed that the leaf nodes of the elimination tree E are load-balanced on each level, in other words, the amount of computation and storage on each leaf node is almost the same. Moreover, we assume that the levels in E are indexed in the top-down order. Therefore, the parent node index of the tree is zero, while the leaf node index is from 1 to l. At each level, we define $E_f[lvl,i]$ as the activation state of subtree-i. If the leaf subtrees have been merged to form the parent trees, we will make $E_f\left[lvl,i\right]=NULL$. We define $P=P_{xy}\times P_z$, and $P_z$ is the number of 2D process grids, where $P_z=2^l$. We map each submatrix $p_i$ to a 2D process grid (${grid}_i$), so it is equivalent to dividing the total matrix into $P_z$ submatrices. Therefore, we consider that the processes are organized in a three-dimensional virtual processor grid with $P_x$ rows, $P_y$ columns, and $P_z$ depths using the MPI Cartesian grid topology construct. Each process has a 3D process coordinate $\left({pid}_x,{pid}_y,{pid}_z\right)$. A 2D process grid contains those processes with the same ${pid}_z$. It is convenient for processes with the same coordinates $\left({pid}_x,{pid}_y\right)$ to communicate between different 2D process grids. Each process invokes a piece of GPU memory, which is less than $M_{gpu}$. We bind processes with the same $mod\left(pid,P_{xy}\right)$ to the same GPU, where $pid$ is the MPI rank of process. We use a data structure that consists of sparse matrices and elimination trees, which is the same as the data structure in the SuperLU3D algorithm. The SuperLU3D algorithm uses a principal data structure SuperMatrix to represent a general matrix, sparse or dense. The SuperMatrix structure contains two levels of fields. The first level defines all the properties of a matrix that are independent of how it is stored in memory. The properties contain the storage type, the data type, the number of rows, the number of columns, and a pointer to the actual storage of the matrix (*Store). The second level (*Store) points to the actual storage of the matrix. The *Store points to a structure of four elements that consists of the number of nonzero elements in the matrix, a pointer to an array of nonzero values (*nzval), a pointer to an array of row indices of the nonzeros elements, and a pointer to the array of beginning of columns in nzval[]. The elimination tree is a vector of parent pointers for a forest;

(3)

The per-process host memory and GPU memory, $M_{ihost}$ and $M_{igpu}$, required to store all the LU factors: For LU factorization of the matrices arising from 2D PDEs, $M_{ihost}\approx M_{igpu}\approx \frac{1}{P}\left({\sum }_{i=0}^l{n}_i^2{2}^l+{\sum }_{i=l+1}^{\log n}{n}_i^2{2}^i\right)$; for LU factorization of those arising from 3D PDEs, $M_{ihost}\approx M_{igpu}={\sum }_{i=0}^{l-1}\frac{n_i^2}{P_{xy}}+{\sum }_{i=l}^{\log n}\frac{2^i{n}_i^2}{P}$ [4], where $n_i$ is the dimension of the supernodes in the level-i. However, these two equations are not suitable for the matrix with a more complex structure. We can obtain these two memories using the SuperLU3D_Alternate program output instead.

In the case of limited cluster host memory and GPU memory, we must first select the appropriate $P_x$, $P_y$, $P_z$, and $n_i$ to guarantee $M_{igpu}\le M_{gpu}$. In addition, the GPU memory size of one submatrix needs to be less than the cluster’s GPU memory size, i.e., $P_{xy}{M}_{igpu}\le {\sum _{i=1}^m{k}_{i}M}_{gpu}$. Moreover, our algorithm will store the LU factors from GPU device memory in the host memory or hard disk in the intermediate computational steps. Therefore, according to different types of the hybrid CPU/GPU cluster, the algorithm has to face three cases: sufficient memory for GPU nodes ($P\left(M_{ihost}+M_{igpu}\right)<\sum _{i=1}^n{Mnode}_i$), insufficient memory for GPU nodes ($\sum _{i=1}^n{Mnode}_i<P\left(M_{ihost}+M_{igpu}\right)\le \sum _{i=1}^m{Mnode}_i$), and insufficient memory for the entire cluster ($P\left(M_{ihost}+M_{igpu}\right)>\sum _{i=1}^m{Mnode}_i$), where $P\left(M_{ihost}+M_{igpu}\right)$ is the memory required by LU factorization, $\sum _{i=1}^n{Mnode}_i$ is the memory of the compute nodes with GPUs, and $\sum _{i=1}^m{Mnode}_i$ is the memory of the cluster. We define these three cases as Case A, Case B, and Case C.

2.2.1. Case A: Sufficient Memory for GPU Nodes

We first consider the simplest case: when all the memory of the nodes with GPUs, which include the host memory and GPU device memory, satisfies the requirement of LU factorization, for which the SuperLU3D_Alternate algorithm needs to save the LU factors from the GPU device memory to the host memory in the intermediate computational steps. Here, we give a simple example to illustrate this case. According to the SuperLU3D algorithm, for example, if the elimination tree only has $l=2$ levels, i.e., $P_{z\left(l\right)}=2^l=4$, the matrix will be divided Into four submatrices, denoted as $p_0$, $p_1$, $p_2$, and $p_3$ (Figure 4a). The i-th submatrix is factorized in ${grid}_i$ with $P_x\times P_y$ processes (Figure 4b) in compute Node 0 and Node 1 (Figure 5). Moreover, the host memory and GPU memory occupied by each process are resident on the same compute node. For example, block₀ of Node 0 only occupies gpu₀ of Node 0, rather than GPUs of Node 1. Coupled with the host memory copied from GPU, this case satisfies that $P\left(M_{ihost}+M_{igpu}\right)<\sum _{i=1}^n{Mnode}_i$. The detailed description of the algorithm is as follows:

I.

Starting with the elimination tree of lvl = 2, we initialize the GPU data structure (function InitSparseLU2D_GPU() in Figure 5b, see Algorithm 1). We divided the submatrix $p_3$ into ${block}_0~{block}_{P_x{P}_y-1}$ by ${grid}_3$, and factorize it with ${gpu}_0~{gpu}_{P_x{P}_y-1}$ acceleration (① in Figure 5a);

II.

After factorization of $p_3$, we synchronize the GPU data structure into the host memory, denoted as $M{gpu2host}_0^3~{Mgpu2host}_{P_x{P}_y-1}^3$ (function copyLU2D_GPU2CPU() in Figure 5b, see Algorithm 1). Then, we free the GPU data structure in the device memory;

III.

Like steps I and II, we decompose other submatrices from $p_2$ to $p_0$ with ${gpu}_0~{gpu}_{P_x{P}_y-1}$ acceleration (②, ③, and ④ in Figure 5a) and save ${Mgpu2host}_0^i~{Mgpu2host}_{P_x{P}_y-1}^i(i=\mathrm{0,1},2)$ into the host memory in turn. Finally, we reduce all the common ancestor nodes (function Ancestor_Reduction(), see Algorithm 1) to finish the LU factorization of the l-th level;

IV.

In the next level l-1, according to $P_{z(l-1)}=2$, only submatrices $p_0$ and $p_2$ are active (Figure 4a). We first restore the ${gpu2host}_0^2~{gpu2host}_{P_x{P}_y-1}^2$ of $p_2$ to ${gpu}_0~{gpu}_{P_x{P}_y-1}$ (function copyLU2D_CPU2GPU() in Figure 5b, see Algorithm 1). Then, we follow the steps I to III to complete the rest of the levels.

We extend the algorithm to the more general $l\ge 2$ case. The pseudocode of Case A is as follows Algorithm 1:

Algorithm 1: Case A

1 for $lvl\leftarrow l$ to 0 do     /* grid3d has a 3D MPI communicator and lower-dimensional subcommunicator. */     /* $A$ is the input matrix. */     /* $A_{gpu}$ stored in the GPU memory has a similar data structure to the matrix $A$. */     /* $A_{gpu2host}$ stored in the host memory is copied from the modified $A_{gpu}$. */     /* $E_f\left[lvl\right]$ are the subtrees at the lvl-th level. The elimination tree $E$ can have multiple     disjoint subtrees as a node. So, the final partition of the elimination tree is a tree of     forests, which we call the elimination tree-forest $E_f$ [4]. If the leaf subtrees are merged     to form the parent trees, then $E_f\left[lvl\right]=NULL$. */ 2    $dSparseLU2D\_GPUAlternate\left(P_z-\mathrm{1,0},l,lvl,A,A_{gpu},E_f\left[lvl\right],A_{gpu2host},grid3d\right);$ 3    if $lvl$ then       /* reduce the nodes of the ancestor matrix before factorizing of the next level [4]*/ 4      $Ancestor\_Reduction\left(A,A_{gpu2host},A_{gpu}\right);$ 5   end 6 end

7 $\mathbf{F}\mathbf{u}\mathbf{n}\mathbf{c}\mathbf{t}\mathbf{i}\mathbf{o}\mathbf{n}$   $dSparseLU2D\_Alternate\left(pi{d}_{zmax},pi{d}_{zmin},l,lvl,A,A_{gpu},E_f\left[lvl\right],A_{gpu2host},grid3d\right)$     Data: $pi{d}_{zmax},pi{d}_{zmin},l,lvl,A,A_{gpu},E_f\left[lvl\right],A_{gpu2host},grid3d$     Result: $A,A_{gpu},A_{gpu2host}$     /* Perform LU factorization of grid-$pi{d}_{zmax}$ to grid-$pi{d}_{zmin}$ with GPU acceleration */ 8    for $pi{d}_z\leftarrow pi{d}_{zmax}$ to $pi{d}_{zmax}$ do       /* grid3d.comm is the 3D MPI communicator. */ 9      $MPI\_Barrier(grid3d.comm);$       /* grid3d.zscp.rank is the z-coordinate in the 3D MPI communicator. */ 10     rank $\leftarrow$ grid3d.zscp.rank; 11     if $rank==pi{d}_z$ then 12       if $E_f\left[lvl\right]!=NULL$ then 13         if $lvl==l$ then             /* initialize the GPU data structures $A_{gpu}$ */ 14           $InitSparseLU2D\_GPU\left(A,A_{gpu},E_f\left[lvl\right]\right);$ 15         else             /* restore the GPU data structures $A_{gpu}$ from the host             memory $A_{gpu2host}$ */ 16           $copyLU2D\_CPU2GPU\left(A_{gpu2host},A_{gpu}\right);$ 17         end           /* dSparseLU2D_GPU() is a call to the modified factorization routine           of 2D SuperLU DIST with GPU for the sub-block of a matrix [4]. */ 18         $dSparseLU2D\_GPU(A,A_{gpu},E_f\left[lvl\right]);$           /* Based on 3D Sparse LU factorization algorithm, if $pi{d}_z\ne 0$, then           $E_f\left[0\right]=NULL$. */ 19         if $lvl$ and $E_f\left[lvl-1\right]!=NULL$ then             /* copy GPU data structures to the host memory */ 20           $copyLU2D\_GPU2CPU\left(A,A_{gpu},A_{gpu2host}\right);$ 21         end           /* freeLU2D_GPU() is a call to free the GPU memory of $A$. */ 22         if $pi{d}_z>0$ then 23           if $lvl$ then 24             $freeLU2D\_GPU\left(A_{gpu}\right);$ 25           end 26         else 27           if $lvl>1$ then 28             $freeLU2D\_GPU\left(A_{gpu}\right);$ 29           end 30         end 31       end 32     end 33   end 34 end

2.2.2. Case B: Insufficient Memory for GPU Nodes

In this case, all the memory of the nodes with GPUs, which include the host memory and all of the GPU device memory, is unable to satisfy the requirement of LU factorization, i.e., $\sum _{i=1}^n{Mnode}_i<P\left(M_{ihost}+M_{igpu}\right)\le \sum _{i=1}^m{Mnode}_i$. The main difference with Case A is that a part of the compute nodes has no GPUs. If we factorize the submatrices in these nodes with GPU acceleration, we must exchange the LU factors data of the compute node without GPU with the compute node with GPU. In addition, we can hide the data exchange behind the LU factorization of the other submatrix with GPU calculation. Taking eight submatrices instead of four as an example, the compute nodes (Node 2 and Node 3) of the last four submatrices do not have GPU (Figure 6a). At the lowest level, i.e., lvl = l, the SuperLU3D_Alternate algorithm of Case B is described as follows:

I.

$p_7$ exchanges the LU factors with $p_3$ (① in Figure 6a,c). By now, the LU factors of $p_7$ are in ${grid}_3$ and the LU factors of $p_3$ are in ${grid}_0$. Then, we initialize GPU in Node 1 and 2 and factorize $p_7$ with GPU acceleration (① in Figure 6b,c), while we exchange the other submatrices’ data independently (②→③→④ in Figure 6a,c) until the exchange of $p_4$ and $p_0$ is finished. The LU factors of $p_{4~7}$ are in ${grid}_{0~3}$ and the LU factors of $p_{0~3}$ are in ${grid}_{4~7}$;

II.

After the LU factorization of $p_7$ is finished in ${grid}_3$, we synchronize the LU factors from GPU to $M{gpu2host}_0^7~M{gpu2host}_{P_x{P}_y-1}^7$, which are in ${grid}_3$, to prepare for the next level, and free the GPU memory occupied by the LU factors (① in Figure 6a,c);

III.

We factorize $p_6$, $p_5$, and $p_4$ with GPU acceleration and synchronize the LU factors from GPU to $M{gpu2host}_0^i~{Mgpu2host}_{P_x{P}_y-1}^i\left(i=\mathrm{4,5},6\right)$ in turn (②→③→④ in Figure 6b,c). At this moment, $M{gpu2host}_0^{4~7}~{Mgpu2host}_{P_x{P}_y-1}^{4~7}$ are in ${grid}_{0~3}$;

IV.

The first four submatrices and the last four submatrices exchange data including ${Mgpu2host}_0^i~{Mgpu2host}_{P_x{P}_y-1}^i$ and LU factors again in turn after finishing the LU factorization (①→②→③→④ in Figure 6b,c). The LU factors of $p_{0~3}$ are in ${grid}_{0~3}$ and the LU factors of $p_{4~7}$ are in ${grid}_{4~7}$. $M{gpu2host}_0^{0~3}~{Mgpu2host}_{P_x{P}_y-1}^{0~3}$ are in ${grid}_{0~3}$ and $M{gpu2host}_0^{4~7}~{Mgpu2host}_{P_x{P}_y-1}^{4~7}$ are in ${grid}_{4~7}$;

V.

After we factorize the first four submatrices following the same procedures as Case A, we reduce all the common ancestor nodes to finish the LU factorization of the lowest level;

VI.

At the next level l-1, according to the step IV of Case A, submatrix $p_0$, $p_1$, $p_2$, and $p_3$ are active. We treat the remaining levels in a similar way to Case A. $M{gpu2host}_0^{4~7}~{Mgpu2host}_{P_x{P}_y-1}^{4~7}$ are kept in the host memory for Ancestor_Reduction() in the rest of levels.

The process blocks with the same color in Figure 6c can be performed at the same time to hide the data exchange behind the LU factorization of other submatrices with GPU calculation. In addition, the memory size of the i-th compute node meets $n_{pi}\left(M_{ihost}+M_{igpu}\right)<{Mnode}_i$.

2.2.3. Case C: Insufficient Memory for the Entire Cluster

As the size of the matrix increases further, the memory required by the LU factorization is larger than the total memory of the cluster, i.e., $P\left(M_{ihost}+M_{igpu}\right)>\sum _{i=1}^m{Mnode}_i$. Therefore, we have to save part of the intermediate data on the hard disk instead of the cluster memory. In addition, Case C is basically the same as Case B. In this case, we can use the large-capacity, inexpensive, and fast SSD (Solid State Drive), especially NVMe SSD, to improve the speed of data read and write.

3. Time Complexity and Space Complexity

3.1. Time Complexity

To evaluate the computational efficiency of LU factorization using the SuperLU3D algorithm, we consider four different solutions, some with CPU and GPU and some without. The first solution, called “SuperLU3D (CPU)”, involves factorizing each submatrix without GPU acceleration. The second solution, called “SuperLU3D (GPU)”, factorizes each submatrix using GPU acceleration. The third solution, called “SuperLU3D (CPU + GPU)”, is a combination of the first two solutions, where some submatrices are factorized with GPU acceleration, and others without. The fourth solution, which is similar to the SuperLU3D_Alternate algorithm, involves alternating the factorization of each submatrix with GPU acceleration. In this section, we will analyze and compare the time complexity of these four solutions.

During LU factorization, the elapsed time at level-lvl is primarily composed of the time for LU factorization of submatrix-i, denoted as $T_{\left(lvl,i\right)}$, and the time of the ancestor-reduction step, denoted as $T_{anc\left(lvl\right)}$. We define the elapsed time for LU factorization without and with GPU acceleration of submatrix-i as $T_{cpu\left(lvl,i\right)}$ and $T_{gpu\left(lvl,i\right)}$, respectively. Correspondingly, the elapsed time of all submatrices at level-lvl are ${T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}$ and ${T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}$. In addition, we denote the elapsed time of the data exchange between different processes as $T_{p2p\left(lvl\right)}$ in the host memory (like ①, ②, ③, ④ and ①, ②, ③, ④ in Figure 6), and denote the elapsed time of the data exchange between device memory and host memory or hard disk as $T_{gpu2host\left(lvl\right)}$. The sum of $T_{p2p\left(lvl\right)}$ and $T_{gpu2host\left(lvl\right)}$ is $T_{exc\left(lvl\right)}$. In Case A, $T_{exc\left(lvl\right)}=0$. In the following experiments, we assume that one GPU card can only accelerate one submatrix. In addition, if there is only one submatrix left, we can factorize this submatrix with GPU acceleration completely. We further assume level-$l^{\prime }$ as a conversion marker layer. If $l^{\prime }\ge 2$ and $P_{z\left(l^{\prime }\right)}\ge 1$, $P_{z\left(l^{\prime }-1\right)}=1$. Therefore, there is one matrix left at level-(l’-1). After level-$l^{\prime }$, the elapsed time of the SuperLU3D (GPU), SuperLU3D (CPU + GPU), and SuperLU3D_Alternate algorithm is consistent. Based on these settings, we obtained the following results:

SuperLU3D (CPU):

$$T_0=\sum _{lvl=0}^l\left[{T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}+T_{anc\left(lvl\right)}\right]$$

(6)

SuperLU3D (GPU):

$$T_1=\sum _{lvl=0}^l\left[{T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}+T_{anc\left(lvl\right)}\right]$$

(7)

SuperLU3D (CPU + GPU):

$$T_2=\sum _{lvl=l^{\prime }}^l\left[{T_{\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}+T_{anc\left(lvl\right)}\right]+\sum _{lvl=0}^{l^{\prime }-1}\left[T_{gpu\left(lvl,0\right)}+T_{anc\left(lvl\right)}\right]$$

(8)

SuperLU3D_Alternate:

$$T_3=\sum _{lvl=l^{\prime }}^l\left[T_{exc\left(lvl\right)}+\sum _{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)}+T_{anc\left(lvl\right)}\right]+\sum _{lvl=0}^{l^{\prime }-1}\left[T_{gpu\left(lvl,0\right)}+T_{anc\left(lvl\right)}\right]$$

(9)

The larger the matrix size, the more obvious effect of the GPU acceleration for matrix multiplication. Thus, when we factorize the very large matrix, there are two inequalities as follows:

$${T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}<{T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max};$$

(10)

$${T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}<{\sum }_{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)};$$

(11)

$${T_{\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}=\left\{\begin{array}{c}{T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max},\mathrm{f}\mathrm{o}\mathrm{r} T_2\\ {T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}, \mathrm{f}\mathrm{o}\mathrm{r} T_1\end{array}\right..$$

(12)

By comparing Equations (6)–(9), the same part of elapsed time is $\sum _{lvl=0}^l{T}_{anc\left(lvl\right)}$. After excluding $\sum _{lvl=0}^l{T}_{anc\left(lvl\right)}$, we compare Equations (6)–(9) to find that $T_1$ is the least time-consuming solution, and $T_2<T_0$.

In summary, the optimal algorithm is SuperLU3D (GPU). However, if the GPU memory is not enough to accelerate the LU factorization of all submatrices, we will select the other three solutions instead. Because $T_2<T_0$, and the relationship between $T_2$ and $T_3$ is unknown, we will focus on the simplified minimum of $∆T=T_2-T_3$, denoted as ${∆T}_{min}$, in theory, and its influencing factors.

From Equations (8) and (9),

$$∆T=T_2-T_3=\sum _{lvl=l^{\prime }}^l\left[{T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}-T_{exc\left(lvl\right)}-\sum _{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)}\right]$$

(13)

We assume the speed-up ratio of i-th submatrix at level-lvl to be $f=\frac{T_{cpu\left(lvl,i\right)}}{T_{gpu\left(lvl,i\right)}}$. Through the LU factorization of the test matrices in the numerical experiments, we discover that the speed-up ratio at level-l is the smallest, i.e.,

$$f_{min}=\frac{T_{{cpu\left(l,0\le i\le P_{z\left(l\right)}-1\right)}_{max}}}{T_{{gpu\left(l,0\le i\le P_{z\left(l\right)}-1\right)}_{max}}}$$

(14)

From Equations (10) and (14),

$${T_{cpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}=f{T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{max}\ge f_{min}{T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{avg}=f_{min}\frac{\sum _{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)}}{P_{z\left(lvl\right)}},$$

(15)

where the average of $T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}$ is

$${T_{gpu\left(lvl,0\le i\le P_{z\left(lvl\right)}-1\right)}}_{avg}=\frac{\sum _{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)}}{P_{z\left(lvl\right)}}.$$

(16)

Combining Equations (13) and (15), the simplified minimum of $∆T$ is

$${∆T}_{min}=\sum _{lvl=l^{\prime }}^l\left[\left(\frac{f_{min}}{P_{z\left(lvl\right)}}-1\right)\sum _{i=0}^{P_{z\left(lvl\right)}-1}{T}_{gpu\left(lvl,i\right)}-T_{exc\left(lvl\right)}\right]$$

(17)

From Equation (17), we consider that ${∆T}_{min}$ is affected by $P_{z\left(lvl\right)}$ and $T_{exc\left(l\right)}$. And the per-process memory of LU factors in SuperLU3D algorithm is approximately equal to $\frac{n}{P_z\times P_{xy}}\log n$ [4]. Therefore, $T_{exc\left(l\right)}$ has positive correlation with n, and has negative correlation with $P_z$. In addition, the GPU acceleration can improve the performance by 1.4–3.5× as compared to the SuperLU3D algorithm without GPU acceleration [4]. In conclusion, $P_z$ is the primary factor affecting the performance of the SuperLU3D_Alternate algorithm, and the secondary factor is n. While alternating LU factorization seems to reduce parallelism, it can solve the very large matrix factorization problem with limited GPU resources. Moreover, in the case of GPU acceleration, choosing the appropriate $P_z$ is still possible to make ${∆T}_{min}>0$ or $∆T>0$.

3.2. Space Complexity

The host memory ($M_{host}$) and device memory ($M_{dev}$) of all the LU factors for the four solutions mentioned in Section 3.1 are given by the following expressions:

SuperLU3D (CPU):

$$M_{host\left(0\right)}={\sum }_{i=0}^{P_{z\left(l\right)}-1}{M}_{ihost}, M_{dev\left(0\right)}=0;$$

(18)

SuperLU3D (GPU):

$$M_{host\left(1\right)}={\sum }_{i=0}^{P_{z\left(l\right)}-1}{M}_{ihost}, M_{dev\left(1\right)}={\sum }_{i=0}^{P_{z\left(l\right)}{P}_x{P}_y-1}{M}_{igpu};$$

(19)

SuperLU3D (CPU + GPU):

$$M_{host\left(2\right)}={\sum }_{i=0}^{P_{z\left(l\right)}-1}{M}_{ihost}, M_{dev\left(2\right)}={\sum }_{i=0}^{P_x{P}_y-1}{M}_{igpu};$$

(20)

SuperLU3D_Alternate:

$$M_{host\left(3\right)}={\sum }_{i=0}^{P_{z\left(l\right)}-1}(M_{ihost}+M_{dev2host}), M_{dev\left(3\right)}={\sum }_{i=0}^{P_x{P}_y-1}{M}_{igpu};$$

(21)

where $M_{ihost}$ is the per-process memory, $M_{igpu}$ is the per-process GPU memory, and $M_{dev2host}$ is the per-process host memory synchronized with GPU. From the above formulas, we can obtain $M_{ihost}=M_{igpu}=M_{dev2host}$.

Comparing Equations (18)–(21), we further obtain that

$$M_{host\left(0\right)}=M_{host\left(1\right)}=M_{host\left(2\right)}=0.5M_{host\left(3\right)},$$

(22)

$$M_{dev\left(1\right)}=P_{z\left(l\right)}{M}_{dev\left(2\right)}=P_{z\left(l\right)}{M}_{dev\left(3\right)}.$$

(23)

The host memory required by the SuperLU3D_Alternate algorithm is twice that of the SuperLU3D algorithm, which is primarily due to the need to synchronize matrices L and U from the GPU memory in the intermediate processes. Conversely, the GPU memory required by the SuperLU3D_Alternate algorithm is only $1/P_{z\left(l\right)}$ that of the SuperLU3D algorithm. Therefore, SuperLU3D_Alternate is able to utilize limited GPU to factorize larger scale matrices than SuperLU3D.

Because the MPI nonblocking communications occur in step (2) and (4) in Section 2.1.2, at level-l, there is only one submatrix performing LU factorization for the SuperLU3D_Alternate algorithm, but there are $P_{z\left(l\right)}$ submatrices performing LU factorization in parallel for the SuperLU_DIST algorithm. Therefore, the amount of data in the MPI nonblocking communication of SuperLU3D_Alternate is $1/P_{z\left(l\right)}$ that of SuperLU3D algorithm.

4. Numerical Experiments and Results

Given a very large matrix, if the host memory satisfies the computational requirements but the GPU memory does not, there are three possible solutions: SuperLU3D (CPU), SuperLU3D (CPU + GPU), and SuperLU3D_Alternate, as discussed in Section 3.1. To evaluate the performance of the SuperLU3D_Alternate algorithm, we compared the elapsed times of these three solutions. Additionally, we calculated $∆T$ for test matrices in both Case A and Case B to confirm the accuracy of ${∆T}_{min}$ as mentioned in Section 3.1.

As the matrix size increases, the host memory on the cluster may not be sufficient to meet the computational requirements. Alternatively, the amount of nonblocking communication data between different processes may exceed the limit of the cluster communication buffer, leading to system memory overflow. In such situations, the SuperLU3D_Alternate algorithm is recommended.

4.1. Setup

For SuperLU3D (CPU), we combined one “AMD-server” node and three “GPU-server” nodes in a hybrid cluster. The specific configurations are shown in

Table 2. Cluster specification.

Parameter	GPU-Server	Intel-Server	AMD-Server
CPU	2 × Intel(R) Xeon(R) Gold 6226R CPU @ 2.90 GHz	2 × Intel(R) Xeon(R) Silver 4110 CPU @ 2.10 GHz	2 × AMD EPYC 7742 @2.25 GHz
Memory	256 GB	256 GB	512 GB
GPUs per node	2	0	0
Type of GPU	Tesla V100S “Volta”	--	--
GPU DRAM	32 GB HMB2	--	--
CUDA Cores per GPU	5120	--	--
NVMe SSD	4 TB	4 TB	4 TB
InfiniBand Bandwidth	2.5 Gb/s	2.5 Gb/s	2.5 Gb/s
Operating system	CentOS Linux release 7.4.1708
Compiler	NVIDIA HPC-X
Libraries	CUDA 10.2, Intel MKL

.

For SuperLU3D (CPU + GPU) and SuperLU3D_Alternate, we used multi-GPU on three “GPU-server” nodes. The GPU node configuration is shown in Table 2. We ensured that a graphics card was only used by one process. In particular, for the SuperLU3D_Alternate algorithm, while we performed LU factorization with GPU acceleration on the “GPU-server” node, a part of the LU factors was also stored in the “AMD-server” node and “Intel-server” node without any computational task. We tested this algorithm via the mixed use of multiple types of servers.

4.2. Test Matrices

We took 11 matrices from a number of numerical problems of two-dimensional (such as G3_circuit) and three-dimensional PDEs (such as nlpkkt80) for testing the SuperLU3D_Alternate algorithm (

Table 3. Eleven matrices for testing LU solvers.

Name	n a	nnz a	nnz/n	Application
CoupCons3D b	4.2 × 105	1.7 × 107	41.5	Structural Problem
Ldoor b	9.5 × 105	4.2 × 107	44.6	Structural Problem
Seren b	1.4 × 106	6.4 × 107	46.1	Structural Problem
audikw_1 b	9.4 × 105	7.8 × 107	82.0	Structural Problem
dielFilterV3real b	1.1 × 106	8.9 × 107	81.0	FEM/EM
G3_circuit b	1.6 × 106	7.7 × 106	4.8	Circuit Sim.
nlpkkt80 b	1.1 × 106	2.8 × 107	26.5	KKT matrices
geodynamics1 c	3.3 × 106	4.5 × 107	14.0	FDM 3D d
geodynamics2 c	4.5 × 106	6.3 × 107	14.0	FDM 3D d
geodynamics3 c	5.1 × 106	7.1 × 107	14.0	FDM 3D d
geodynamics4 c	5.4 × 106	7.5 × 107	14.0	FDM 3D d

Notes: a—n is the dimension of the matrix, and nnz is the total number of nonzero elements; b—these matrices were obtained from the University of Florida Sparse Matrix Collection [19]; c—the last four matrices were obtained from the finite difference numerical simulation of geodynamics based on the 3D Stokes equations and the incompressible continuity equations [20]. d—FDM 3D is the abbreviation of 3D finite difference method.

). The last four matrices (geodynamics1~4) in Table 3 were obtained from the finite difference numerical simulation problem of geodynamics based on the 3D Stokes equations and the incompressible mass continuity equations. The LU factorization of these four matrices consumes large amounts of host memory and device memory. Therefore, these matrices are very suitable for testing our algorithms. As can be seen from Table 3, the scale of computing from CoupCons3D to geodynamics4 increases sequentially.

4.3. Comparison of Results

The numerical experiments set $P_x=2$, $P_y=3$, $P_z=2,$ and $n_s=100$, i.e., the total number of processes was 12, and all the matrices were divided into two submatrices. We ran 12 MPI processes on the cluster, and each MPI process used four OpenMP threads. The number of GPUs involved in the calculation was six, which is half the number of processes, i.e., only half the processes were permitted to invoke GPU.

We obtained the elapsed times and speed-up ratios of 11 matrices in Case A (

Table 4. The elapsed time and speed-up ratio of the SuperLU3D (CPU + GPU) and SuperLU3D_Alternate algorithms as compared with the SuperLU3D (CPU) algorithm in Case A.

Name	Per-Process GPU Memory (GB)	SuperLU3D (CPU) ${\mathit{T}}_0$ (Second)	SuperLU3D (CPU+GPU) ${\mathit{T}}_2$ (Second)	SuperLU3D_ Alternate ${\mathit{T}}_3$ (Second)	${\mathit{T}}_0/{\mathit{T}}_2$	${\mathit{T}}_0/{\mathit{T}}_3$
ldoor	4.3	3.54	4.21	18	0.840855	0.196667
CoupCons3D	4.3	4	15	20	0.266667	0.2
G3_circuit	4.5	7	27	23	0.259259	0.304348
dielFilterV3real	5.0	31	9	18	3.444444	1.722222
audikw_1	6.7	45	37	31	1.216216	1.451613
Serena	10.8	252	67	106	3.761194	2.377358
nlpkkt80	11.8	333	108	88	3.083333	3.784091
geodynamics1	18.7	864	394	156	2.192893	5.538462
geodynamics2	30.3	1932	752	307	2.569149	6.29316
geodynamics3	30.3	2560	833	338	3.073229	7.573964
geodynamics4	30.3	--	--	896	--	--

). “geodynamics4” was unable to be factorized by the SuperLU3D algorithm on the cluster because of the large amount of nonblocking communication data. The bar graph based on the speed-up ratios, including $T_2/T_0$ and $T_3/T_0$, is shown in Figure 7. The results of $T_0$ show that the elapsed time of SuperLU3D (CPU) increased with increasing matrix size. In comparison with $T_2/T_0$ and $T_3/T_0$, SuperLU3D_Alternate achieved a speed of up to 8× that of SuperLU3D (CPU) and 2.5× that of SuperLU3D (CPU + GPU) on the hybrid cluster with six Tesla V100S GPUs. The SuperLU3D_Alternate algorithm was proved to be more efficient than SuperLU3D (CPU + GPU) under the same device memory consumption in our experiments. This was thanks to the fact that the LU factorization of each submatrix was able to take advantage of GPU acceleration. It is worth mentioning that, due to the increasing amount of nonblocking communication data, the SuperLU3D algorithm will be unstable or even break in running programs.

In addition, in the numerical experiments using SuperLU3D (CPU) and SuperLU3D (CPU + GPU), there was a certain probability that “geodynamics3” failed in factorization, while “geodynamics4” was not always successful. Therefore, the SuperLU3D_Alternate algorithm is more applicable for very large sparse matrices than SuperLU3D (CPU) and SuperLU3D (CPU + GPU).

Finally, we conducted experiments on Case C using data from geodynamics1, geodynamics2, and geodynamics3. All these submatrices, which were created during LU factorization, were initially stored on the hard drive, and each one was sequentially read into the memory for computation to reduce the memory requirements of very large sparse LU factorization. The times spent on the LU factorization of these three matrices were 1881 s, 3290 s, and 4988 s, respectively. In addition, the memory consumption of these matrices in Case C was about 60% of that in Case A.

4.4. Correctness of Time Complexity Analysis

In the second experiment, we compared the elapsed times of the SuperLU3D_Alternate and SuperLU3D (CPU + GPU) algorithms under Case A and Case B. The biggest difference from Case A was that the compute nodes on which a part of processes reside had no GPUs in Case B. Therefore, we needed to invoke GPUs across other nodes, resulting in increasing the time spent on data exchange ($T_{exc}$). We obtained $∆T$ and $∆T_{min}$ of the test matrices, including geodynamics1, geodynamics2, and geodynamics3 (

Table 5. The elapsed times of the SuperLU3D_Alternate and SuperLU3D (CPU + GPU) algorithms in Case A and Case B.

Name	${\mathit{f}}_{\mathit{m}\mathit{i}\mathit{n}}$ a	${\mathit{P}}_{\mathit{z}}$	${\mathit{T}}_{\mathit{e}\mathit{x}\mathit{c}\left(\mathit{l}\right)}$ b (Second)	${\mathit{T}}_2$ c (Second)	${\mathit{T}}_3$ c (Second)	$∆\mathit{T}$ c (Second)	${∆\mathit{T}}_{\mathit{m}\mathit{i}\mathit{n}}$ c (Second)
geodynamics1_case_A	4.63	2	0	394	156	238	136.92
geodynamics1_case_B	4.31	2	62.11	394	225	169	54.34
geodynamics2_case_A	4.34	2	0	752	307	445	247.83
geodynamics2_case_B	3.28	2	118.34	752	467	285	17.35
geodynamics3_case_A	3.64	2	0	833	338	495	275.47
geodynamics3_case_B	3.37	2	145.76	833	586	247	53.08

a—the speed-up ratio at level-l $f_{min}=\frac{T_{{cpu\left(l,0\le i\le P_{z\left(l\right)}-1\right)}_{max}}}{T_{{gpu\left(l,0\le i\le P_{z\left(l\right)}-1\right)}_{max}}}$; b—$T_{exc\left(l\right)}=T_{p2p\left(l\right)}+T_{gpu2host\left(l\right)}$ where $T_{p2p\left(l\right)}$ is the elapsed time of the memory data exchange between different processes and $T_{gpu2host\left(l\right)}$ is the elapsed time of the data exchange between device memory and host memory or hard disk at level-l; c—$T_2$, $T_3$, $∆T$, and ${∆T}_{min}$ are mentioned in Table 4. In addiction, ${∆T}_{min}$ is obtained by Equation (17) with $l^{\prime }=2$ and $P_z=2$.

), in Case A and Case B, where $∆T=T_2-T_3$, and $∆T_{min}$ is the simplified minimum of $∆T$. In Table 5, all the results of $∆T$ are bigger than $∆T_{min}$. These results verify the analysis of $∆T$ and $∆T_{min}$ in Section 3.1. Therefore, when we choose a small $P_z$ in the SuperLU3D_Alternate algorithm, we can ensure that $∆T>0$. In addition, the results of $∆T>0$ show that, even if the submatrix needs to be decomposed alternately and additional data need to be exchanged at each level, SuperLU3D_Alternate with GPU acceleration has better computational efficiency than SuperLU3D (CPU + GPU).

5. Conclusions and Future Work

The SuperLU3D_Alternate algorithm described in this paper is suitable for solving the problem of LU factorization for large sparse matrices by utilizing hardware resources of different scales, even in situations where there may be insufficient memory available. This includes cases where there is sufficient memory available on GPU nodes, cases where there is insufficient memory available on GPU nodes, and cases where there is insufficient memory available across the entire cluster. Results from numerical experiments demonstrate that this approach is not only feasible but also improves computational efficiency with the help of GPU acceleration compared to SuperLU3D (CPU) and SuperLU3D (CPU + GPU).

(1) In Table 3, we compared the LU factorization of test matrices using SuperLU3D (CPU), SuperLU3D (CPU + GPU), and SuperLU3D_Alternate. The results indicate that, as the SuperLU3D (CPU) algorithm takes longer, the SuperLU3D_Alternate algorithm becomes more efficient than SuperLU3D (CPU + GPU) while consuming the same GPU memory. The numerical experiments showed that, on a hybrid cluster with six Tesla V100S GPUs, SuperLU3D_Alternate achieved speeds of up to 8× that of SuperLU3D (CPU) and 2.5× that of SuperLU3D (CPU + GPU).

(2) The SuperLU3D_Alternate algorithm proposed in this study offers a solution to the problem of LU factorization for very large matrices when the GPU memory is insufficient. By dividing the matrix into submatrices and performing LU factorization alternately with GPU acceleration, the algorithm allows for the utilization of the cluster’s host memory or hard disk to store intermediate LU factors of submatrices. This approach enables the efficient processing of very large matrices.

(3) The SuperLU3D_Alternate algorithm reduces the amount of data exchange required, which prevents it from exceeding the limit of the cluster MPI nonblocking communication buffer. This, in turn, enhances the stability of the algorithm.

In the current work, we observed that the SuperLU3D_Alternate algorithm requires additional synchronization of GPU data transmission between different GPUs with the host memory, which can result in increased latency and reduced throughput. Therefore, direct communication between GPUs can improve the overall performance and efficiency of the algorithm. NVIDIA GPUDirect technologies can provide low-latency, high-bandwidth communication between NVIDIA GPUs. Therefore, future work will focus on optimizing interprocess communication efficiency using technologies such as GPUDirect RDMA [21].