On the Locally Polynomial Complexity of the Projection-Gradient Method for Solving Piecewise Quadratic Optimisation Problems

Abstract

This paper proposes a method for solving optimisation problems involving piecewise quadratic functions. The method provides a solution in a finite number of iterations, and the computational complexity of the proposed method is locally polynomial of the problem dimension, i.e., if the initial point belongs to the sufficiently small neighbourhood of the solution set. Proposed method could be applied for solving large systems of linear inequalities.

1. Introduction

Let us consider the following optimisation problem:

$$\underset{x\in R^n}{min}{∥{\left(A·x-b\right)}_{+}∥}^2,$$

(1)

where $c_{+}:=max\left\{c,0\right\}$, A is an $m\times n$ matrix, $A=\left\{a_{ij}\right\}$, $x\in R^n$, $x=\left\{x_i\right\}$, $b\in R^m$, $b=\left\{b_j\right\},$$i=1,\dots ,n$, $j=1,\dots m$ and $\parallel ·\parallel$ is the Euclidean norm of $R^n$.

In this paper, a method for solving the problem in (1) is proposed; moreover, the number of iterations (equivalent to the computational complexity) required by the proposed method with respect to m and n is locally polynomial, and in the worst-case scenario, it has a geometric convergence rate.

Let us define the set of solutions of (1) as follows

$$X^{*}:=\left\{x^{*}\left|x^{*}=arg\underset{x\in R^n}{min}{∥{\left(A·x-b\right)}_{+}∥}^2\right.\right\}.$$

(2)

If some point sufficiently close to the set $X^{*}$ of solutions to (1) is known, then it is possible to find a solution of (1) within a polynomial number of computational iterations; thus, the computational complexity is of the order of $O(m^3·n^3)$.

Many methods for solving (1) have been proposed (cf. Karmanov [1], Golikov and Evtushenko [2], Evtushenko and Golikov [3], Tretyakov [4], Tretyakov and Tyrtyshnikov [5] and Han [6]). All of these methods have reasonable computational complexity but, as mentioned above, to date, no strongly polynomial-time algorithm for solving (1) has been proposed. In studies by Tretyakov and Tyrtyshnikov [7] and Mangasarian [8], linear programming problems were solved by reducing them to the unconditional minimisation of strongly convex piecewise quadratic functions. A solution is obtained within a finite polynomial number of iterations if the starting point of the algorithm belongs to a sufficiently close neighbourhood of the unique solution to the problem. Unfortunately, the authors imposed severe limitations on the functions to be minimised: they should be strongly convex, the eigenvalues of the Hessian matrices should satisfy specific conditions, etc.

These results create significant limitations on the class of problems that can be solved: it is required that (1) has only one unique solution, etc. The solution method described by Tretyakov and Tyrtyshnikov [7] is based on exploiting information about the problem being solved by analysing a sufficiently small neighbourhood of an arbitrary solution of (1). Analogous methods were proposed by Facchinei et al. [9] for the forecasting (identification) of the active constraints in a sufficiently close neighbourhood of the solution to the problem. In papers by Tretyakov and Tyrtyshnikov [5] and Wright [10], locally polynomial methods for solving quadratic programming problems based on similar ideas were presented. Tretyakov [4] proposed the gradient projection method for solving (1); this method involves finding a solution of (1) in a finite number of iterations and is a combination of iterative and straightforward (e.g., Gaussian) methods.

This paper proposes a computational method for solving (1). When the starting point of the proposed method is sufficiently close to the set $X^{*}$ of solutions to (1), then its computational complexity is locally polynomial, i.e., it is of the order of $O(m^3·n^3)$.

We point out that solving a system of linear inequalities involves

$$A·x-b\le 0_m,$$

(3)

where the $0_m$ – m-dimensional vectors of zeroes can be reduced to solve the problem (1). This means that the number of computations required for establishing a solution (if a given system of linear inequalities has one) is locally polynomial.

Let us denote

$$X:=\left\{x\in R^n\left|A·x-b\le 0_m\right.\right\}.$$

(4)

It is obvious that the set X might be empty in general, but our method, presented in this paper, either determines this situation in a locally polynomial number of computations or provides a solution to the system (3). The proposed method could be applied when solving large systems of linear inequalities, which appear in many practical, industrial applications, e.g., the simplex method (Pan [11]), Karmarkar’s method (Wright, [12]), Chubanov’s method (Roos [13]), and Fourier–Motzkin elimination method (Khachiyan [14], I. Šimeček, R. Fritsch, D. Langr, R. Lórencz [15]).

2. Definitions and Theoretical Results

Let

$$\phi \left(x\right)=\frac{1}{2}·{∥{(A·x-b)}_{+}∥}^2.$$

(5)

Theorem 1.

The function $\phi \left(x\right)$ is convex and has a nonempty set of minimal values

$$X^{*}:=\left\{x^{*}\in R^n\left|\phi \left(x^{*}\right)=\underset{x\in R^n}{min}\phi \left(x\right)\right.\right\}.$$

(6)

Proof. 

Theorem 1 follows immediately from the well-known features of quadratic-type convex functions (see, e.g., [16]). □

It is obvious that the elements $x^{*}\in X^{*}$, cf. (6) satisfy

$${\phi }^{\prime }{\left(x^{*}\right)}^{\top }=\sum _{i=1}^m{\left(〈a_i,x^{*}〉-b_i\right)}_{+}·a_i=0_n=A^T·{\left(A·x^{*}-b\right)}_{+},$$

(7)

where $a_i^T$ is the ith row of matrix A.

Therefore, in the general case, our goal is to solve the following equation

$${\phi }^{\prime }{\left(x\right)}^{\top }=\sum _{i=1}^m{\left(〈a_i,x〉-b_i\right)}_{+}·a_i=A^T·{\left(A·x-b\right)}_{+}=0_n,\mathrm{where}x\in R^n.$$

(8)

In the sequel, $x^{*}$ stands for an arbitrary element of $X^{*}$ (the minimum point of $\phi$). If the minimum value of $\phi$ is equal to zero, then $X=X^{*}$, and if the minimum value of $\phi$ is positive, then $X=\varnothing$. Let us denote

$$f_i\left(x\right):=〈a_i,x〉-b_i,i\in D=\{1,\dots m\},$$

and

$$\begin{array}{cc}\hfill J_0\left(x\right)& :=\left\{i\in D\left|f_i\left(x\right)=0\right.\right\},J_{-}\left(x\right):=\left\{i\in D\left|f_i\left(x\right)<0\right.\right\},\hfill \\ \hfill J_{+}\left(x\right)& :=\left\{i\in D\left|f_i\left(x\right)>0\right.\right\},\hfill \end{array}$$

(9)

where $f_i\left(x\right)$ is introduced to simplify the definitions of the sets $J_0\left(x\right)$ and $J_{+}\left(x\right)$.

According to (7) and the above notations, $x^{*}$ should satisfy the formula

$$\sum _{i\in J_0\left(x^{*}\right)\cup J_{+}\left(x^{*}\right)}{\left(〈a_i,x^{*}〉-b_i\right)}_{+}·a_i=0_n.$$

(10)

The formula (10) is equivalent to a condition that should be satisfied at point $x^{*}$

$$\begin{array}{c}\hfill \sum _{i\in J_0\left(x^{*}\right)\cup J_{+}\left(x^{*}\right)}\left(〈a_i,x^{*}〉-b_i\right)·a_i=0_n,\end{array}$$

(11)

In (11), it is considered that

$${\left(〈a_i,x^{*}〉-b_i\right)}_{+}=〈a_i,x^{*}〉-b_i,i\in J_0\left(x^{*}\right)\cup J_{+}\left(x^{*}\right).$$

This, in turn, means that, in the general case, we should solve the following equations

$$\sum _{i\in J_0\left(x\right)\cup J_{+}\left(x\right)}{\left(〈a_i,x〉-b_i\right)}_{+}·a_i=0_n,$$

(12)

or

$$\begin{array}{cc}\hfill \sum _{i\in J_{+}\left(x\right)}& {\left(〈a_i,x〉-b_i\right)}_{+}·a_i=0_n,\hfill \\ \hfill & 〈a_i,x〉-b_i=0,i\in J_0\left(x\right).\hfill \end{array}$$

(13)

Without loss of generality, we may denote

$$J_{-}\left(x^{*}\right):=\left\{1,\dots ,l\right\},J_0\left(x^{*}\right):=\left\{l+1,\dots ,p\right\},J_{+}\left(x^{*}\right):=\left\{p+1,\dots ,m\right\},$$

where $l\le p\le m$.

The main idea exploited in this paper is based on the following Lemma. For $\epsilon >0$ we set $U_{\epsilon }\left(x^{*}\right):=\left\{x\in R^n:\parallel x-x^{*}\parallel \le \epsilon \right\}$.

Lemma 1.

Let $x^{*}$ be a solution to the problem (1). Then, there exists $\epsilon >0$, such that for any $x\in U_{\epsilon }\left(x^{*}\right)$, the inequality $f_i\left(x\right)\ge 0$ implies the inequality $f_i\left(x^{*}\right)\ge 0$.

Proof. 

If $i\in J_{-}\left(x^{*}\right),$ that is $f_i\left(x^{*}\right)<0$, then, by continuity of the function $f_i$, there exists ${\epsilon }_i>0$, such that $f_i\left(x\right)<0$ for all $x\in U_{{\epsilon }_i}\left(x^{*}\right)$. Set

$$\epsilon =\underset{i\in J_{-}\left(x^{*}\right)}{min}{\epsilon }_i.$$

Then, for all $i\in J_{-}\left(x^{*}\right)$ and for all $x\in U_{\epsilon }\left(x^{*}\right)$, we have $f_i\left(x\right)<0$. Consequently, if there exists $x\in U_{\epsilon }\left(x^{*}\right)$ such that $f_i\left(x\right)\ge 0$ with some $i\in \{1,\dots ,m\}$, then $i\notin J_{-}\left(x^{*}\right)$, that is $f_i\left(x^{*}\right)\ge 0$. □

By virtue of the above lemma, in a sufficiently small neighbourhood of some fixed point $x^{*}\in X^{*}$, for every $\overline{x}\in U_{\epsilon }\left(x^{*}\right)$, the following hold

$$J_0\left(\overline{x}\right)\subseteq J_0\left(x^{*}\right)\mathrm{and}{J}_{+}\left(\overline{x}\right)\subseteq J_0\left(x^{*}\right)\cup J_{+}\left(x^{*}\right),J_{-}\left(\overline{x}\right)\subseteq J_0\left(x^{*}\right)\cup J_{-}\left(x^{*}\right).$$

Now, our goal is to correctly define the sets $J_0\left(x^{*}\right)$ and $J_{+}\left(x^{*}\right)$ based on the information gained at point $\overline{x}\in U_{\epsilon }\left(x^{*}\right)$. Let us denote

$${\overline{J}}_0\left(\overline{x}\right):=J_0\left(\overline{x}\right),{\overline{J}}_{+}\left(\overline{x}\right):=J_{+}\left(\overline{x}\right),{\overline{J}}_{-}\left(\overline{x}\right):=J_{-}\left(\overline{x}\right).$$

Let $A\left(\overline{x}\right)$ and $b\left(\overline{x}\right)$ represent the matrix and vector obtained from A and b, respectively. The rows of $A\left(\overline{x}\right)$ and the coefficients of $b\left(\overline{x}\right)$ correspond to the index set, which is defined by ${\overline{J}}_0\left(\overline{x}\right)\cup {\overline{J}}_{+}\left(\overline{x}\right)$. In this case, Equations (12)–(13) may be rewritten as

$$\begin{array}{cc}\hfill A^T\left(\overline{x}\right)& ·\left(A\left(\overline{x}\right)·x-b\left(\overline{x}\right)\right)=0_n,\hfill \\ \hfill 〈a_i,x〉& -b_i=0,i\in {\overline{J}}_0\left(\overline{x}\right).\hfill \end{array}$$

(14)

Let $\overline{A}\left(\overline{x}\right)$ denote the matrix in the equations in (14) corresponding to the maximum set of linearly independent rows, and let $\overline{b}\left(\overline{x}\right)$ denote the corresponding vector of constant terms in (14).

The equations in (14) may be reformulated in the following way

$$\overline{A}\left(\overline{x}\right)·x-\overline{b}\left(\overline{x}\right)=0_n.$$

(15)

Let us observe that, at point $x^{*}$, the following holds

$$A^T\left(x^{*}\right)·{\left(A\left(x^{*}\right)·x^{*}-b\left(x^{*}\right)\right)}_{+}=0_n.$$

(16)

This, in turn, means that

$$\overline{A}\left(x^{*}\right)·x^{*}-\overline{b}\left(x^{*}\right)=0_n.$$

(17)

$$\begin{array}{cc}\hfill M\left(\overline{x}\right):=\{x\in {\mathbb{R}}^n\mid & \sum _{i\in {\overline{J}}_0\left(\overline{x}\right)\cup {\overline{J}}_{+}\left(\overline{x}\right)}\left(〈a_i,x〉-b_i\right)·a_i=0_n\hfill \\ \hfill & \left.\mathrm{and}〈a_j,x〉-b_j=0,\mathrm{where}j\in {\overline{J}}_0\left(\overline{x}\right)\right\}.\hfill \end{array}$$

(18)

If the rank of a matrix B of size $r\times n$ is equal to r, then the pseudoinverse matrix (operator) $B^{+}$ may be defined as $B^{+}:=B^T·{(B·B^T)}^{-1}$. We denote the quadratic matrix $n\times n$ orthogonally projected on the space containing the rows of matrix B as ${\left(B^T\right)}^{\mathrm{II}}:=B^T{\left(B·B^T\right)}^{-1}·B=B^{+}·B$, and its projection on the orthogonal complement of the matrix is denoted as ${\left(B^T\right)}^{\perp }:=$$I-{\left(B^T\right)}^{\mathrm{II}}$, where I is an all-ones matrix of size $n\times n.$

Let a point $z\left(\overline{x}\right)$ be the projection of point $\overline{x}$ on the set $M\left(\overline{x}\right)$. Let us observe that $x^{*}\in M\left(\overline{x}\right)$ if $\overline{x}\in U_{\epsilon }\left(x^{*}\right)$ and $\epsilon$ is sufficiently small.

Moreover, if the constraints at point $z\left(\overline{x}\right)$ are $f_i\left(z\left(\overline{x}\right)\right)\le 0$ for a certain $i\in {\overline{J}}_{+}\left(\overline{x}\right)$, then we define the set $I_{-}$ in the following way

$$I_{-}=\left\{i\in {\overline{J}}_{+}\left(\overline{x}\right)\mid f_i\left(z\left(\overline{x}\right)\right)\le 0\right\};I_{-}\subseteq J_0\left(x^{*}\right).$$

Otherwise, if the constraints at point $z\left(\overline{x}\right)$ are $f_i\left(z\left(\overline{x}\right)\right)\ge 0$ for a certain $i\in {\overline{J}}_{-}\left(\overline{x}\right)$, we define the set $I_{+}$ in an analogous way

$$I_{+}=\left\{i\in {\overline{J}}_{-}\left(\overline{x}\right)\mid f_i\left(z\left(\overline{x}\right)\right)\ge 0\right\};I_{+}\subseteq J_0\left(x^{*}\right).$$

Now, we redefine ${\overline{J}}_0\left(\overline{x}\right)$, ${\overline{J}}_{+}\left(\overline{x}\right)$ and ${\overline{J}}_{-}\left(\overline{x}\right)$ as follows

$${\overline{J}}_0\left(\overline{x}\right):={\overline{J}}_0\left(\overline{x}\right)\cup I_{-}\cup I_{+},{\overline{J}}_{+}\left(\overline{x}\right):={\overline{J}}_{+}\left(\overline{x}\right)\backslash I_{-},{\overline{J}}_{-}\left(\overline{x}\right):={\overline{J}}_{-}\left(\overline{x}\right)\backslash I_{+}.$$

(19)

Next, we project point $\overline{x}$ on the new set $M\left(\overline{x}\right)$, cf. (18), and a new point $z\left(\overline{x}\right)$ is obtained.

Let

$$z\left(x\right)=P_{M\left(\overline{x}\right)}\left(x\right)={\left(A^T\left(\overline{x}\right)\right)}^{\perp }·x+{\overline{A}}^{+}\left(\overline{x}\right)·\overline{b}\left(\overline{x}\right)$$

(20)

define the operator for the projection of point x on set $M\left(\overline{x}\right).$

3. Algorithm for Finding the Solution of (1)

In this section, the algorithm designed to find the solution to (1) is presented. The main idea of this algorithm is based on information related to a current point $\overline{x}$ belonging to a sufficiently small neighbourhood of the point $x^{*}\in X^{*}$. We also demonstrate how to find such a point. The proposed method comprises two algorithms. The starting point of the method can be arbitrary, because Algorithm 2 (gradient method with a special step selection) starts at an arbitrary point and, on a certain iteration, it will provide a point arbitrarily close to the solution set. Therefore, Algorithm 1 could start at the point specified by Algorithm 2.

Algorithm 1.

Initialisation Step: For the current point $\overline{x}$, the sets of indices $J_0\left(\overline{x}\right)$, $J_{-}\left(\overline{x}\right)$ and $J_{+}\left(\overline{x}\right)$ are defined according to (9). If set $J_{+}\left(\overline{x}\right)=\varnothing$, then $\overline{x}$ is the solution of (1) and Algorithm 1 is terminated. Otherwise, the Main Recursive Step is performed.

Main Recursive Step: Let$z\left(\overline{x}\right)$, the projection of point$\overline{x}$on the set$M\left(\overline{x}\right)$, be defined according to (20). We check if the following condition is satisfied

$$I_{+}=\varnothing \mathrm{and}{I}_{-}=\varnothing .$$

(21)

Checking Step: If (21) holds, then $z\left(\overline{x}\right)\in X^{*}$, and Equation (10) is satisfied; $z\left(\overline{x}\right)$ is the solution of (1), as defined in (2), and Algorithm 1 is terminated. Otherwise, if for certain values of $i\in D$ the condition (21) is violated and $i\in I_{+}\cup I_{-}$, we define ${\overline{J}}_0\left(\overline{x}\right)$, ${\overline{J}}_{+}\left(\overline{x}\right)$ and ${\overline{J}}_{-}\left(\overline{x}\right)$ according to (19), $M\left(\overline{x}\right)$ is redefined according to (18), and the Main Recursive Step is repeated.

The set $D$ is finite, and $\left|D\right|=m$; therefore, the number of changes to the index sets ${\overline{J}}_0\left(\overline{x}\right)$, ${\overline{J}}_{+}\left(\overline{x}\right)$ and ${\overline{J}}_{-}\left(\overline{x}\right)$ does not exceed m, and finally, the point $z\left(\overline{x}\right)$ fulfilling (12) is established. This means that $z\left(\overline{x}\right)$ is the solution of (1), as defined in (2).

It is of utmost importance that $\overline{x}$ belongs to a sufficiently small neighbourhood of the point $x^{*}$, because otherwise, $z\left(\overline{x}\right)$ may not satisfy (12). If this is not the case, it is necessary to find another point $\overline{x}$ that is closer to $x^{*}$. The process for accomplishing this is described below.

Theorem 2.

For a sufficiently small $\epsilon >0$ and for every $\overline{x}\in U_{\epsilon }\left(x^{*}\right)$, Algorithm 1 provides $z^{*}=z\left(\overline{x}\right)$ as the solution for

$${\phi }^{\prime }\left(x\right)=A^T\left(x\right)·{\left(A\left(x\right)·x-b\right)}_{+}=0_n,$$

(22)

 and this is equivalent to finding the solution for (12) within a number of iterations of the order of $0(m^3·n^3)$.

Proof. 

The proof is based on the observation that for $\overline{x}$ belonging to a sufficiently small neighbourhood of the point $x^{*}$, according to Lemma 1, the constraints $f_i\left(\overline{x}\right)\ge 0$ correspond to constraints $f_i\left(x^{*}\right)\ge 0$. Therefore,

$${\overline{J}}_0\left(\overline{x}\right)\cup {\overline{J}}_{+}\left(\overline{x}\right)\subseteq J_0\left(x^{*}\right)\cup J_{+}\left(x^{*}\right).$$

Let us determine $z\left(\overline{x}\right)$ as the projection of the point $\overline{x}$ on the set $M\left(\overline{x}\right)$, which is defined according to (18). It may happen that the set ${\overline{J}}_0\left(\overline{x}\right)$ becomes enlarged. However, the number of iterations required when ${\overline{J}}_0\left(\overline{x}\right)$ becomes enlarged does not exceed m, the number of elements in the set D. Therefore, at some iteration, (21) is satisfied. This means that $z\left(\overline{x}\right)$ satisfies (12) or, equivalently, ${\phi }^{\prime }\left(z\left(\overline{x}\right)\right)=0_n$. This demonstrates that $z\left(\overline{x}\right)$ is the solution for (1), as defined in (2). The computational complexity of establishing each projection $z\left(\overline{x}\right)$ is of order $0(m^2·n^3)$; this process takes the computational effort related to the multiplications of matrices into account. The number of iterations does not exceed m and, therefore, the overall computational complexity is of order $0(m^3·n^3)$. □

To complement the presentation of this chapter, the gradient method for establishing $\overline{x}$ belonging to the sufficiently small neighbourhood $U_{\epsilon }\left(x^{*}\right)$ of some fixed solution $x^{*}\in X^{*}$ to (1) is described. This gradient method has the following scheme

$$x_{k+1}=x_k-\alpha ·{\phi }^{\prime }\left(x_k\right)$$

(23)

where $\alpha =\frac{1}{L}$ and gradient ${\phi }^{\prime }\left(x_k\right)$ fulfils the Lipschitz condition

$$\left|{\phi }^{\prime }\left(x_{k+1}\right)-{\phi }^{\prime }\left(x_k\right)\right|\le L·\left|x_{k+1}-x_k\right|\mathrm{where}L=2·∥A^T·A∥.$$

The convergence of the gradient method (23) is considered in the following theorem, cf. Karmanov [1].

Theorem 3.

Let $x_0\in R^n$ and the sequence $\left\{x_k\right\}$, $k=0,1,2\dots$, be constructed according to (23). Then,

$$x_k\to x^{*},x^{*}\in X^{*},\mathrm{where}k\to \infty \mathrm{and}∥x_{k+1}-y∥<∥x_k-y∥\forall y\in X^{*}.$$

Proof. 

The scheme in (23) produces a sequence that converges to a certain $x^{*}\in X^{*}$. Moreover, for every sufficiently small $\epsilon >0$, there exists $\overline{k}=k\left(\epsilon \right)$ such that $\left\{x_k\right\}\in U_{\epsilon }\left(x^{*}\right)$, for all $k\ge \overline{k}$. This, in turn, means that at iteration $\overline{k}$, the hypothesis of Theorem 2 is satisfied, and we obtain a solution to (1). □

Now, we have all the necessary prerequisites to present the solution algorithm for (3).

Algorithm 2.

Initialisation Step: Let $k=0$, and let $x_0$ be an arbitrary point in $R^n$.

Main Recursive Step: Let

$$x_{k+1}=x_k-\alpha ·{\phi }^{\prime }\left(x_k\right).$$

Checking Step: If $z\left(x_k\right)$ is the solution for (3), then Algorithm 2 is terminated. Otherwise, we set $k:=k+1$, and the Main Recursive Step is repeated.

Theorem 4.

There exists a finite $\overline{k}$ such that $z\left(x_{\overline{k}}\right)\in X^{*}$ and $z\left(x_{\overline{k}}\right)$ is the solution for (3).

Proof. 

The sequence $\left\{x_k\right\}$ converges to a fixed $x^{*}\in X^{*}$ and therefore, at a certain iteration $\overline{k}$, the hypothesis of Theorem 2 is satisfied, and we obtain the solution $z^{*}=P_{M\left(x_{\overline{k}}\right)}\in X^{*}$. □

Theorem 4 allows us to establish whether (3) has a solution or not.

Corollary 1.

If

$$z^{*}\in X,$$

then $z^{*}$ is the solution of (3). Otherwise (3) has no solutions.

4. Conclusions and Appendix

As previously mentioned, the locally polynomial complexity estimate is valid only if the starting point of the proposed method belongs to a sufficiently small neighbourhood of the set of solutions $X^{*}$. To reach such a desired point, the gradient method (23) is used. There are accelerated gradient methods (see those of Nesterov [17] and Poliak [18]), but these methods do not guarantee monotonic convergence to a set of solutions $X^{*}$. The method presented in this paper monotonically converges to a certain point $x^{*}$, $x^{*}\in X^{*}$. It is obvious that the point $x^{*}$ depends on the initial point $x_0$ and, therefore, the number of iterations required by the gradient method for entering the proper neighbourhood of point $x^{*}$ depends on the position of the initial point $x_0$. Moreover, the $\epsilon$ radius of the neighbourhood of point $x^{*}$, which the gradient method should reach, is unknown in the general case and depends on the specific problem being considered. However, it appears that we can guarantee a geometric convergence rate for the gradient method (23) while minimising piecewise quadratic functions of the form (5).

Namely, for every strongly convex function $\psi \left(x\right),$ the gradient method (23) has a geometric convergence rate, i.e.,

$$\psi \left(x_k\right)-{\psi }^{*}\le c·{\delta }^k,\mathrm{where}0<\delta <1,c>0,$$

where c is a constant that is independent of the size of the problem but depends on the initial point $x_0$. In the general case, for functions that are not convex in the strongest sense, there is no proof of the geometric convergence of the gradient method (23). However, in the case where the function $\phi \left(x\right)$ is given by (5), it is possible to prove the geometric convergence of the gradient method (23). Let

$$\begin{array}{cc}\hfill l\left(x_k\right)& =\left\{x^{*}+\beta ·\left(x_k-x^{*}\right),\beta \ge 0\right\}\mathrm{and}M\left(s_k\right)=\left\{x^{*}+\beta ·s_k,\beta \ge 0\right\},\hfill \\ \hfill s_k& =\frac{x_k-x^{*}}{∥x_k-x^{*}∥}.\hfill \end{array}$$

The theorem presented below proves the strong convexity of the function $\phi \left(x\right)$ in the cone of convergence.

Theorem 5.

The elements of the sequence $\left\{x_k\right\},$ defined by (23), belong to the cone of strong convexity of the function $\phi \left(x\right)$, namely $\forall x$,$y\in l\left(x_k\right)$, the function $\phi \left(x\right)$ is uniformly strongly convex for the sequence $\left\{x_k\right\}$, i.e.,

$$\phi (\lambda ·x+(1-\lambda )·y)\le \lambda ·\phi \left(x\right)+(1-\lambda )·\phi \left(y\right)-\gamma ·\lambda ·(1-\lambda )·{∥x-y∥}^2$$

(24)

where $\lambda \in \left[0,1\right]$, $x,y\in l\left(x_k\right)$, $k=0,1\dots$, and $\gamma >0$.

Proof. 

First, it should be pointed out that because the second derivative of the function $\phi \left(x\right)$ has a finite number of points of discontinuity $\overline{S}\in R^n$ in every direction, i.e., on the ray $x^{*}+\lambda ·\overline{S}$, there exists $\sigma >0$ such that on the closed interval $[x^{*},x^{*}+\sigma ·\overline{S}]$, the function $\phi \left(x\right)$ has a continuous second derivative that obviously depends on $\overline{S}$. Let us assume that the theorem does not hold, i.e., there is not $\gamma >0$, such that (24) holds. This means that for

$$l\left(x_k\right)=\left\{x^{*}+\beta ·s_k,\beta \ge 0\right\}$$

the following must hold

$$\frac{{\partial }^2\phi \left(x^{*}\right)}{\partial s_k^2}={\gamma }_k\to 0\mathrm{when}k\to \infty ,$$

(25)

or

$$\frac{{\partial }^2\phi \left(x^{*}\right)}{\partial s_k^2}=〈A^T·A·s_k,s_k〉={\gamma }_k\to 0\mathrm{when}k\to \infty .$$

For vector $s=\underset{k\to \infty }{lim}{s}_k$, the following condition $〈A^T·A·s,s〉=0$ holds, or, due to the construction of $\phi \left(x\right)$,

$$\phi (x^{*}+\beta ·s)=0=\phi \left(x^{*}\right)=min{∥{\left(A·x-b\right)}_{+}∥}^2,$$

where $\beta \in [0,\overline{\beta }]$, $\overline{\beta }>0$ is a certain fixed constant. Let $x_k^{*}$ be (locally) the projection of $x_k$ on the set $M\left(s\right)\in X^{*}$. Then, due to $s_k\to s$, $k\to \infty$, we have

$$∥x_k-x_k^{*}∥={\delta }_k·∥x_k-x^{*}∥,\mathrm{where}{\delta }_k\to 0,k\to \infty .$$

(26)

Let us set ${\delta }_k$ sufficiently small and consider the points $x_{k+r}$, $r=1,2,$…. Then, according to Theorem 3, we have

$$∥x_{k+r}-x_k^{*}∥<∥x_k-x_k^{*}∥.$$

(27)

On the other hand, according to (26), when $r\to \infty$

$$\begin{array}{cc}\hfill ∥x_{k+r}-x_k^{*}∥& \ge ∥x_k^{*}-x^{*}∥-∥x_{k+r}-x^{*}∥\ge \hfill \\ \hfill ∥x_k-x^{*}∥& -∥x_k-x_k^{*}∥-∥x_{k+r}-x^{*}∥\ge \hfill \\ \hfill \frac{1}{{\delta }_k}∥x_k-x_k^{*}∥& -∥x_k-x_k^{*}∥-∥x_{k+r}-x^{*}∥>∥x_k-x_k^{*}∥.\hfill \end{array}$$

This is contradictory to (27), and therefore Theorem 5 holds. □

Theorem 5 allows for the estimation of the convergence rate of the gradient method (23).

Theorem 6.

Under the assumptions of Theorem 5 for the sequence $\left\{x_k\right\},$ constructed according to (23), the following convergence rates hold

$$\phi \left(x_k\right)-{\phi }^{*}\le c_1·{\tau }^k\mathrm{and}∥x_k-x^{*}∥\le c_2·{\tau }^{\frac{k}{2}},$$

(28)

where $\tau \in (0,1)$, and $c_1$ and $c_2$$>0$; the constants $c_1$, $c_2$ are independent of the value of k but depend on the initial point $x_0$.

Proof. 

Let us denote

$${\mu }_k=\phi \left(x_k\right)-{\phi }^{*}.$$

For the sequence $\left\{x_k\right\}$ and $q\in \left(\frac{1}{2},1\right)$ the following holds

$$\begin{array}{cc}\hfill \phi \left(x_k\right)-\phi \left(x_{k+1}\right)& \ge \alpha ·q·{∥{\phi }^{\prime }\left(x_k\right)∥}^2\ge \alpha ·q·{〈{\phi }^{\prime }\left(x_k\right),s_k〉}^2=\frac{{\partial }^2\phi \left(x_k\right)}{\partial s_k^2}\ge \hfill \\ \hfill & \ge \alpha ·q·{\gamma }^2·\left(\phi \left(x_k\right)-{\phi }^{*}\right)\hfill \end{array}$$

(29)

or, equivalently,

$${\mu }_k-{\mu }_{k+1}\ge \alpha ·q·{\gamma }^2{\mu }_k.$$

Therefore, for $\tau \in (0,1)$, the following holds

$${\mu }_k\le c_1·{\tau }^k\mathrm{or},\mathrm{equivalently},\phi \left(x_k\right)-{\phi }^{*}\le c_1·{\tau }^k.$$

This proves the first part of (28), while the latter part of (28) follows from the strong convexity of the function $\phi \left(x\right)$ in the cone of convergence. □

The conduction computational experiments and comparison of the presented method with other methods in the literature remains a topic for future research.