In this section, we present our main algorithms, i.e., the breakdownfree block COCG algorithm (BFBCOCG) and the breakdownfree block COCR algorithm (BFBCOCR) in detail. We first introduce the block COCG (BCOCG) and block COCR (BCOCR) methods briefly and then give the derivation of BFBCOCG and BFBCOCR with their orthogonality properties, respectively. In the end, the preconditioned variants of BFBCOCG and BFBCOCR are also proposed, denoted by BFPBCOCG and BFPBCOCR, respectively. We use an underscore “_” to distinguish the breakdownfree and the nonbreakdownfree algorithms.
2.2. BFBCOCG and BFBCOCR
If the block size
p is large, then the vectors of the block search direction will inevitably be linearly dependent on the increasing of the iteration number for BCOCG, hence rank deficiency occurs. In the following, in order to overcome this problem, we consider applying the breakdownfree strategy to BCOCG and propose the breakdownfree block COCG algorithm (BFBCOCG). The rationale of BFBCOCG is extracting an orthogonal basis
${\underline{P}}_{k+1}$ from the current searching space by using the operation
$orth(\xb7)$. Thus, compared with (
2), the new recurrence relation becomes
Therefore, again using the orthogonality condition (
3), we can get the Lemma 1. Here, we denote
${\underline{U}}_{k}=A{\underline{P}}_{k}$.
Lemma 1. For all $0\le j<k$, ${R}_{j}^{T}{R}_{k}=0$, ${\underline{P}}_{j}^{T}{R}_{k}=0$, and ${\underline{P}}_{j}^{T}A{P}_{k}=0$.
Proof. Because
${R}_{j},{\underline{P}}_{j}\in {\mathcal{K}}_{k}(A,{R}_{0})$ for all
$0\le j<k$, and
${R}_{k}\perp {\mathcal{K}}_{k}(\overline{A},{\overline{R}}_{0})$ by (
3), thus
${R}_{j}^{T}{R}_{k}=0$ and
${\underline{P}}_{j}^{T}{R}_{k}=0$. Then
${\underline{P}}_{j}^{T}A{P}_{k}=0$ can be obtained by the second orthogonality condition in (
3). □
Similarly, the following Theorem 1 is obtained to update the parameters ${\underline{\alpha}}_{k}$ and ${\underline{\beta}}_{k}$.
Theorem 1. Under the orthogonality condition (3), the value of parameters ${\underline{\alpha}}_{k}$ and ${\underline{\beta}}_{k}$ in the recurrence relation (4) can be obtained by solving the following equations: Proof. From Lemma 1 and (
4), we have the following two equations:
and
So, solving (
6), we can easily get the
${\underline{\alpha}}_{k}$.
Premultiplying
${\underline{\alpha}}_{k}^{T}$ to (
7), then from the third equation of (
4), we have
From Lemma 1, we have
${R}_{k}^{T}{R}_{k+1}=0$, and by the first equation of (
5), one has
${\underline{\alpha}}_{k}^{T}{\underline{P}}_{k}^{T}{\underline{U}}_{k}={R}_{k}^{T}{\underline{P}}_{k}$. Thus the above equation can be rewritten as
which can be used to update matrix
${\underline{\beta}}_{k}$. □
The following Algorithm 1 is the breakdownfree block COCG.
Algorithm 1 Breakdownfree block COCG (BFBCOCG) 
 1.
Given the initial guess ${X}_{0}\in {\mathbb{C}}^{n\times p}$ and a tolerance $tol$; Compute: ${R}_{0}=BA{X}_{0}$, ${\underline{P}}_{0}=orth({R}_{0})$, ${\underline{U}}_{0}=A{\underline{P}}_{0}$;  2.
For $k=0,1,2,\cdots $ until $\parallel {R}_{k}{\parallel}_{F}/{\parallel {R}_{0}\parallel}_{F}\le tol$, do
Solve: $({\underline{P}}_{k}^{T}{\underline{U}}_{k}){\underline{\alpha}}_{k}={\underline{P}}_{k}^{T}{R}_{k}$; Update: ${X}_{k+1}={X}_{k}+{\underline{P}}_{k}{\underline{\alpha}}_{k}$, ${R}_{k+1}={R}_{k}{\underline{U}}_{k}{\underline{\alpha}}_{k}$, Solve: (${R}_{k}^{T}{\underline{P}}_{k}){\underline{\beta}}_{k}={R}_{k+1}^{T}{R}_{k+1}$; Update: ${\underline{P}}_{k+1}=orth({R}_{k+1}+{\underline{P}}_{k}{\underline{\beta}}_{k})$, ${\underline{U}}_{k+1}=A{\underline{P}}_{k+1}$;
End For

Similar to BFBCOCG, we can also easily get BFBCOCR by using
$\mathcal{L}=\overline{A}{\mathcal{K}}_{k+1}(\overline{A},{\overline{R}}_{0})$ in the orthogonality condition (
3). The following Algorithm 2 is the breakdownfree block COCR.
Algorithm 2 Breakdownfree block COCR (BFBCOCR) 
 1.
Given the initial guess ${X}_{0}\in {\mathbb{C}}^{n\times p}$ and a tolerance $tol$; Compute: ${R}_{0}=BA{X}_{0}$, ${\underline{P}}_{0}=orth({R}_{0})$, ${\underline{U}}_{0}=A{\underline{P}}_{0}$;  2.
For $k=0,1,2,\cdots $ until $\parallel {R}_{k}{\parallel}_{F}/{\parallel {R}_{0}\parallel}_{F}\le tol$, do
Solve: $({\underline{U}}_{k}^{T}{\underline{U}}_{k}){\underline{\alpha}}_{k}={\underline{U}}_{k}^{T}{R}_{k}$; Update: ${X}_{k+1}={X}_{k}+{\underline{P}}_{k}{\underline{\alpha}}_{k}$, ${R}_{k+1}={R}_{k}{\underline{U}}_{k}{\underline{\alpha}}_{k}$, Solve: $({R}_{k}^{T}{\underline{U}}_{k}){\underline{\beta}}_{k}={R}_{k+1}^{T}A{R}_{k+1}$; Update: ${\underline{P}}_{k+1}=orth({R}_{k+1}+{\underline{P}}_{k}{\underline{\beta}}_{k})$, ${\underline{U}}_{k+1}=A{\underline{P}}_{k+1}$;
End For

2.3. BFPBCOCG and BFPBCOCR
As we all know, if the coefficient matrix has poor spectral properties, then the Krylov subspace methods may not robust, while a preconditioning strategy can make it better [
11]. The trick is preconditioning (
1) with a symmetric positive matrix
M, which approximates to the inverse of matrix
A; we get the following equivalent system:
Let
$M=L{L}^{T}$ be the Cholesky decomposition of
M. Then system (
9) is equivalent to
We add a tilde “˜” on the variables derived from the new system. Then applying the BFBCOCG method and its recurrence relations (
4) to (
10), we have the orthogonality conditions
It is easy to see
${\tilde{R}}_{k}={L}^{T}{R}_{k}$,
${\tilde{\underline{P}}}_{k}=orth({\tilde{R}}_{k}+{\tilde{\underline{P}}}_{k1}{\underline{\beta}}_{k1})\in {\mathcal{K}}_{k+1}(\tilde{A},{\tilde{R}}_{0})$. The approximate solution
${X}_{k}=L{\tilde{X}}_{k}$ is from
$L{\tilde{X}}_{0}+L{\mathcal{K}}_{k}(\tilde{A},{\tilde{R}}_{0})={X}_{0}+{\mathcal{K}}_{k}(MA,M{R}_{0})$. Set
${\underline{P}}_{k}=L{\tilde{\underline{P}}}_{k}$, then
${\underline{P}}_{k}\in {\mathcal{K}}_{k+1}(MA,M{R}_{0})$. The new recurrence relation becomes
The orthogonality condition (
11) become
Under the recurrence relation (
12) and the orthogonality condition (
13), we can get the following Lemma 2 and Theorem 2 to update the matrices
${\underline{\alpha}}_{k}$ and
${\underline{\beta}}_{k}$. Here, we omit the proof because it is like the proof of Lemma 1 and Theorem 1. We denote
${Z}_{k}=M{R}_{k},{\underline{U}}_{k}=A{\underline{P}}_{k}$.
Lemma 2. For all $0\le j<k$, ${R}_{j}^{T}{Z}_{k}=0$, ${\underline{P}}_{j}^{T}{R}_{k}=0$, and ${\underline{P}}_{j}^{T}A{P}_{k}=0$.
Remark 1. Under the preconditioned strategy, the relations of the block residuals are changed from orthogonal for BFBCOCG to Morthogonal for BFPBCOCG. Here, two vectors x and y are Morthogonal, meaning $x{\perp}_{M}y$, i.e., ${y}^{H}Mx=0$.
Theorem 2. Under the orthogonality condition (13), the value of parameters ${\underline{\alpha}}_{k}$ and ${\underline{\beta}}_{k}$ in the recurrence relations (12) can be obtained by solving the following equations: The following Algorithm 3 is the breakdownfree preconditioned block COCG algorithm.
Algorithm 3 Breakdownfree preconditioned block COCG (BFPBCOCG) 
 1.
Given the initial guess ${X}_{0}\in {\mathbb{C}}^{n\times p}$ and a tolerance $tol$; Compute: ${R}_{0}=BA{X}_{0}$, ${Z}_{0}=M{R}_{0}$, ${\underline{P}}_{0}=orth({Z}_{0})$, ${\underline{U}}_{0}=A{\underline{P}}_{0}$;  2.
For $k=0,1,2,\cdots $ until $\parallel {R}_{k}{\parallel}_{F}/{\parallel {R}_{0}\parallel}_{F}\le tol$, do
Solve: $({\underline{P}}_{k}^{T}{\underline{U}}_{k}){\underline{\alpha}}_{k}={\underline{P}}_{k}^{T}{R}_{k}$; Update: ${X}_{k+1}={X}_{k}+{\underline{P}}_{k}{\underline{\alpha}}_{k}$, ${R}_{k+1}={R}_{k}{\underline{U}}_{k}{\underline{\alpha}}_{k}$, ${Z}_{k+1}=M{R}_{k+1}$; Solve: $({R}_{k}^{T}{\underline{P}}_{k}){\underline{\beta}}_{k}={R}_{k+1}^{T}{Z}_{k+1}$; Update: ${\underline{P}}_{k+1}=orth({Z}_{k+1}+{\underline{P}}_{k}{\underline{\beta}}_{k})$, ${\underline{U}}_{k+1}=A{\underline{P}}_{k+1}$;
End For

Change the orthogonality conditions (
11) to the following conditions:
The breakdownfree preconditioned block COCR (BFPBCOCR) can be deduced with the similar derivation of BFPBCOCG. Algorithm 4 shows the code of BFPBCOCR.
Algorithm 4 Breakdownfree preconditioned block COCR (BFPBCOCR) 
 1.
Given the initial guess ${X}_{0}\in {\mathbb{C}}^{n\times p}$ and a tolerance $tol$; Compute: ${R}_{0}=BA{X}_{0}$, ${Z}_{0}=M{R}_{0}$, ${\underline{P}}_{0}=orth({Z}_{0})$, ${\underline{U}}_{0}=A{\underline{P}}_{0}$;  2.
For $k=0,1,2,\cdots $ until $\parallel {R}_{k}{\parallel}_{F}/{\parallel {R}_{0}\parallel}_{F}\le tol$, do
Solve: $({\underline{U}}_{k}^{T}M{\underline{U}}_{k}){\underline{\alpha}}_{k}={\underline{U}}_{k}^{T}{Z}_{k}$; Update: ${X}_{k+1}={X}_{k}+{\underline{P}}_{k}{\underline{\alpha}}_{k}$, ${R}_{k+1}={R}_{k}{\underline{U}}_{k}{\underline{\alpha}}_{k}$, ${Z}_{k+1}=M{R}_{k+1}$; Solve: $({Z}_{k}^{T}{\underline{U}}_{k}){\underline{\beta}}_{k}={Z}_{k+1}^{T}A{Z}_{k+1}$; Update: ${\underline{P}}_{k+1}=orth({Z}_{k+1}+{\underline{P}}_{k}{\underline{\beta}}_{k})$, ${\underline{U}}_{k+1}=A{\underline{P}}_{k+1}$;
End For

At the end of this section, we will give the complexity for six algorithms. They are the block COCG algorithm, block COCR algorithm, breakdownfree block COCG algorithm, breakdownfree block COCR algorithm, breakdownfree preconditioned block COCG algorithm, and breakdownfree preconditioned block COCR algorithm, which are denoted by BCOCG, BCOCR, BFBCOCG, BFBCOCR, BFPBCOCG, BFPBCOCR, respectively. The pseudocodes of BCOCG and BCOCR are from [
1].
Table 1 shows the main costs per cycle of the six algorithms. Here, we denote as “block vector” the matrix with size
$n\times \times p$, “bdp” the dot product number of two block vectors, “bmv” the product number of a matrix with
$n\times p$ and a block vector, “bsaxpy” the number of two block vectors summed with one of the block vectors being from multiplying a block vector to a
$p\times p$ matrix, “LE” the number of solving linear equations with a
$p\times p$ coefficient matrix, and “bSC” the storage capacity of block vectors.
From
Table 1, we can see the last four algorithms need one more dot product of two block vectors than the original two algorithms, i.e., BCOCG and BCOCR. For the product number of a matrix with a block vector, BFBCOCR and BFPBCOCR are both twice BFBCOCG and BFPBCOCG, respectively. This may result in more time to spend in BFBCOCR and BFPBCOCR, especially for a problem with a dense matrix. We reflect on the phenomenon in Example 1.