ASPDC: Accelerated SPDC Regularized Empirical Risk Minimization for Ill-Conditioned Problems in Large-Scale Machine Learning

Abstract

This paper aims to improve the response speed of SPDC (stochastic primal–dual coordinate ascent) in large-scale machine learning, as the complexity of per-iteration of SPDC is not satisfactory. We propose an accelerated stochastic primal–dual coordinate ascent called ASPDC and its further accelerated variant, ASPDC-i. Our proposed ASPDC methods achieve a good balance between low per-iteration computation complexity and fast convergence speed, even when the condition number becomes very large. The large condition number causes ill-conditioned problems, which usually requires many more iterations before convergence and longer per-iteration times in data training for machine learning. We performed experiments on various machine learning problems. The experimental results demonstrate that ASPDC and ASPDC-i converge faster than their counterparts, and enjoy low per-iteration complexity as well.

1. Introduction

In this paper, we consider a composite convex optimization problem, Regularized Empirical Risk Minimization (RERM), that can be solved by SPDC [1]. Our goal is to use our proposed ASPDC find the approximate solution of the following optimization problem:

$$\underset{w\in {\mathbb{R}}^d}{min}\{P\left(w\right)=\frac{1}{n}\sum _{i=1}^n{\varphi }_i(y_i,w^T{x}_i,b)+g\left(w\right)\}$$

(1)

where $x_i\in {\mathbb{R}}^d$ is a feature vector, $y_i$ is the corresponding label in a machine learning task, $\left\{(x_i,y_i)\right\},i=1,2,\dots ,n$ are n samples in the dataset, ${\varphi }_i$ is the proper convex function of the linear predictor $w^T{x}_i$, and $g\left(w\right)$ the simple convex regularization function.

RERM is one of the central problems in machine learning. It is now prevalent in the data mining and machine learning domain. More background information on RERM can be found in [2]. The following are four examples of RERM:

• Linear SVM, where ${\varphi }_i(y_i,w^T{x}_i,b)=max\{0,1-y_i(w^T{x}_i+b)\}$,  $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$
• Ridge Regression, where ${\varphi }_i(y_i,w^T{x}_i,b)=\frac{1}{2}{(y_i-(w^T{x}_i+b))}^2$, $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$
• Lasso, where ${\varphi }_i(y_i,w^T{x}_i,b)=\frac{1}{2}{(y_i-(w^T{x}_i+b))}^2$, $g\left(w\right)={\lambda \left|\right|w\left|\right|}_1$
• Logistic Regression, where ${\varphi }_i(y_i,w^T{x}_i,b)=log(1+exp(-y_i(w^T{x}_i+b)))$, $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$

Here, we focus on the scenario in which the number of samples n is very large, as the per-iteration complexity of SPDC is intolerable in this scenario. Computing a full gradient becomes extremely expensive in terms of time and space costs. Therefore, RERM algorithms with a lower per-iteration complexity are more attractive in large-scale machine learning applications.

General optimization methods to the RERM problem using gradients are categorized into two types, namely, first-order and second-order. Second-order methods such as the Newton algorithm employ a Hessian matrix at each iteration to decrease the objective value. The disadvantage of these second-order methods is that both obtaining and using a Hessian matrix is computationally expensive. On the other hand, while first-order optimization schemes are lightweight in gradient computation, they may converge slowly [3,4].

Among the algorithms for solving the RERM problem, we are more interested in dual algorithms such as stochastic dual coordinate ascent-SDCA, as the dual-gap is a clearer stopping criterion than gradients. In addition, they are capable of handling non-differentiable primal optimal functions more easily [5]. SDCA is a first-order optimization method and is widely used in the current machine learning domain. Dual coordinate methods have been implemented in open machine learning libraries [4].

The dual methods do not solve the primal problem directly. Instead, they solve the dual or saddle point problem of the primal problem. The corresponding dual problem of the primal problem in Equation (1) is formulated as follows:

$$\underset{\alpha \in {\mathbb{R}}^n}{max}\{D\left(\alpha \right)=\frac{1}{n}\sum _{i=1}^n-{\varphi }_i^{*}\left({\alpha }_i\right)-g^{*}(-\frac{1}{n}\sum _{i=1}^n{\alpha }_i{x}_i)\}$$

(2)

where $g^{*}\left(u\right)=\underset{w\in {\mathbb{R}}^d}{max}\{w^{T}u-g\left(w\right)\}$ and ${\varphi }_i^{*}$ are the convex conjugate functions of g and ${\varphi }_i$, respectively. Due to the structure of this dual problem, coordinate ascent methods can be more efficient than full gradient methods [4,6,7].

In the stochastic dual coordinate ascent method (SDCA) [5], a dual coordinate ${\alpha }_i$ is picked randomly at each iteration and then updated to increase the dual objective value. This helps SDCA to reach a low per-iteration computational complexity. Nevertheless, the convergency speed of SDCA becomes much slower as the condition number grows. A large condition number leads to an ill-conditioned problem. An ill-conditioned scenario refers to a case in which a small change in one of the values of the coefficient matrix causes a large change in the solution vector [8,9,10,11]. Hence, SDCA is not applicable to large-scale data processing in ill-conditioned scenarios. Unfortunately, many traning tasks involving large-scale data involve ill-conditioned scenarios. Ill-conditioned problems are particularly common in mathematics and geosciences [12].

Paper Organization. The rest of this paper is organized as follows. In Section 2, we describe related works.

In Section 3, we describe the relevant assumptions and preliminaries.

In Section 4, we discuss the accelerated stochastic primal–dual coordinate method. In this section, we present ASPDC in Algorithm 1 and its convergence analysis for the saddle point problem in Equation (3).

In Section 5, we extend ASPDC to ill-conditioned problems, in particular, those in which $\lambda \le \frac{4}{n\gamma }$. Our proposed extension method is called ASPDC-i, where i means “for ill-conditioned problems”.

In Section 6, we evaluate the performance of our proposed ASPDC algorithms with several state-of-art algorithms for solving machine learning problems, then discuss the experimental results.

In Section 7, we conclude the paper and discuss potential avenues for future work.

2. Related Work

Shalev-Shwartz and Zhang [13] developed an accelerated proximal stochastic dual coordinate ascent method (ASDCA), which converges faster than traditional methods when the condition number is large (Table 1). ASDCA can be regarded as a variant of a proximal point algorithm equipped with Nesterov’s accelerated technique [14,15,16]. ASDCA uses an inner–outer iteration procedure, where the outer loop is a minimization of an auxiliary problem with a regularized quadratic term. Then, the proximal SDCA starts to solve the auxiliary problem with a customized precision. At the end of each outer loop, Nesterov’s accelerated update is performed on the primal variable w. Nonetheless, ASDCA requires $\lambda$ to be limited to a range of low-level values, for example, $\lambda \le \frac{R^2}{10n\gamma }$, where $\gamma$ is the smooth parameter of ${\varphi }_i$, n is the number of samples, and $R^2=\underset{i}{max}\left|\right|x_i{\left|\right|}_2^2$.

Table 1. Abbreviations used in this study.

Complete Name	Abbreviation
Stochastic primal-dual coordinate ascent	SPDC
Stochastic dual coordinate ascent method	SDCA
Accelerated stochastic primal-dual coordinate ascent	ASPDC
Extended ASPDC to the ill-conditioned problem	ASPDC-i
Accelerated stochastic dual ascent	ASDCA

Studies have extended the inner–outer iteration method in order to derive more general accelerated proximal-point algorithms, e.g., Catalyst, [17,18]. Theoretically, one can replace the inner-loop proximal SDCA algorithm using other algorithms, such as SVRG [19] and Prox-SVRG [20], to obtain the same overall complexity concerning the number of outer loops.

More recently, Zhang and Xiao [1,21] proposed a stochastic primal–dual coordinate (SPDC) method to solve the RERM problem defined in Equation (1). SPDC achieves a faster convergence rate in reducing the dual-gap than ASDCA and other dual methods in general optimization problems with condition numbers that are not very large. The per-iteration computation complexity of SPDC is much higher than ASDCA and SDCA. Theoretically, the per-iteration complexity of SPDC is $\mathcal{O}\left(d\right)$. However, due to the auxiliary variable update and the momentum item, SPDC requires much more time to process one pass of a dataset, as verified in our experiments. When the condition number is large, the SPDC per-iteration computation complexity of SPDC is intolerable, which makes SPDC inapplicable to large-scale data processing. Our experiments verified that SPDC is more time-consuming than ASDCA and other low per-iteration complexity methods. Moreover, the dual-gap of SPDC is much larger when the data are sparse and have high dimensionality.

The above issue leads to the following key question: “Can we design an algorithm with both a low per-iteration complexity and a fast convergence rate, especially for ill-conditioned scenarios in large-scale data processing?” We propose the ASPDC and ASPDC-i algorithms as the answer to this question. ASPDC methods have the following three advantages:

• Simple structure at each iteration. In comparison with SPDC or other accelerated variants, ASPDC does not need to keep track of any other auxiliary variables; it only maintains the primal and dual variable. Each iteration only involves a dual update and primal update. This design makes its per-iteration complexity much lower than SPDC and other variants. The simple iteration design makes it easy to be implemented as well.
• Short running time. Our experiments show that to reach the same precision, our methods need far less time and fewer epochs (numbers of passes through the entire data) to satisfy the stop condition.
• Theoretical guarantee. ASPDC adopts Nesterov’s estimation technique [22,23]. We present a new proof onf the convergence of proposed methods.

3. Assumptions and Preliminary

Throughout this paper, the standard Euclidean is denoted as an equation such as ${\left|\right|w\left|\right|}_2=\sqrt{{\sum }_i{\left|w_i\right|}^2}$. We use E to denote the expectation that is taken with respect to the randomness of ${\alpha }_i$. For the sake of convenience, we use the new notation $x_i\leftarrow {(x_i^T,1)}^T,w\leftarrow {(w^T,b)}^T$. Without loss of generality, we continue to assume $w\in {\mathbb{R}}^d,x_i\in {\mathbb{R}}^d$. Then, we make the following assumptions to clearly specify the problem in Equation (1) as follows:

Assumption 1.

Each ${\varphi }_i$ is lower semi-continuous and convex, and its derivative is $\frac{1}{\gamma }$-Lipschitz continuous (or equivalently: ${\varphi }_i$ is $\frac{1}{\gamma }$-smooth), i.e., there exist $\gamma >0$ such that $|{\varphi }_i^{\prime }\left(a\right)-{\varphi }_i^{\prime }\left(b\right)|⩽\frac{1}{\gamma }|a-b|\forall a,b\in \mathbb{R}i=1,2,\dots ,n.$

It is widely known that Assumption 1 implies that ${\varphi }_i^{*}$ is $\gamma$-strongly convex (see Theorem 4.2.2 in the convex fundamental book [24]).

Assumption 2.

The primal function $P\left(w\right)$ is λ-strongly convex: There exists $\lambda >0$ such that $\forall w_1,w_2\in {\mathbb{R}}^d$,

$$\begin{array}{c}\hfill P\left(w_1\right)\ge P\left(w_2\right)+\nabla {P\left(w_2\right)}^T(w_1-w_2)+\frac{\lambda }{2}\left|\right|w_1-w_2{\left|\right|}_2^2\end{array}$$

The convexity of $P\left(w\right)$ may come from either ${\varphi }_i$ or $g\left(w\right)$ or both. For instance, if $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$, Assumption 2 holds.

Assumption 3.

$\left|\right|x_i{\left|\right|}_2⩽1,\forall i=1,2,\dots ,n.$

Assumption 3 is not a strict one, as when data are normalized, Assumption 3 holds.

Under the three assumptions above, the RERM problem defined in Equation (1) can be rewritten as the following convex–concave saddle point problem [1]:

$$\underset{w\in {\mathbb{R}}^d}{min}\underset{\alpha \in {\mathbb{R}}^n}{max}\{f(w,\alpha )=\frac{1}{n}\sum [{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]+g\left(w\right)\}$$

(3)

where ${\varphi }_i^{*}\left({\alpha }_i\right)=\underset{s\in \mathbb{R}}{sup}\{s{\alpha }_i-{\varphi }_i\left(s\right)\}$ is a convex conjugation function of ${\varphi }_i$. Lemma 1 demonstrates the relationship between the primal problem of Equation (1) with the problem of Equation (3).

Lemma 1.

Let $w^{*}=arg\underset{w\in {\mathbb{R}}^d}{min}P\left(w\right)$ and ${\alpha }^{*}=arg\underset{\alpha \in {\mathbb{R}}^n}{max}D\left(\alpha \right)$, then we have

(1) 

$P\left(w\right)=\underset{\alpha \in {\mathbb{R}}^n}{max}f(w,\alpha )$

(2) 

$D\left(\alpha \right)=\underset{w\in {\mathbb{R}}^d}{min}f(w,\alpha )$

(3) 

There exists a unique solution $(w^{*},{\alpha }^{*})$ such that $P\left(w^{*}\right)=D\left({\alpha }^{*}\right)=f(w^{*},{\alpha }^{*}).$

Proof. 

Presented in Appendix A. □

Lemma 1 implies that we can calculate the optimal solution of the primal problem in Equation (1) by solving the saddle point problem in Equation (3).

4. Accelerated Stochastic Primal–Dual Coordinate Method

In this section, we present ASPDC in Algorithm 1 and its convergence analysis for the saddle point problem in Equation (3).

Each iteration in ASPDC can be divided into two steps: the dual update step and the primal update step. The dual update step is executed first. As shown in lines 4–6 of Algorithm 1, a dual coordinate, ${\alpha }_i$, is picked randomly and updated to increase the objective value of $f(w,\alpha )$ while keeping the primal variable w and other ${\alpha }_j(j\ne i$) fixed. Then, the primal update step is executed later. As shown in line 7 of Algorithm 1, the primal variable w is updated to decrease the objective value of $f(w,\alpha )$ while keeping ${\alpha }_j(j=1,2,\dots ,n)$ fixed.

The update of the dual variable $\alpha$ is extremely simple. It can be simplified as a univariate optimal problem, which makes its per-iteration complexity much lower than traditional SPDC algorithms. Specifically, the local update of dual variable ${\alpha }_i$ is

$$\begin{array}{ccc}\hfill \Delta {\alpha }_i^{*}& =& \underset{\Delta {\alpha }_i\in \mathbb{R}}{arg\; max}f(w,\alpha +\Delta {\alpha }_i{e}_i)\hfill \\ \hfill & =& \underset{\Delta {\alpha }_i\in \mathbb{R}}{arg\; max}(\Delta {\alpha }_i{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i)),\hfill \end{array}$$

(4)

where $e_i\in {\mathbb{R}}^n$ is a unit vector with the $i-th$ element being one.

The update of primal variable w is shown in Equation (5) as follows:

$$\begin{array}{ccc}\hfill w^{*}& =& \underset{w\in {\mathbb{R}}^d}{arg\; min}f(w,{\alpha }^{(t+1)})\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill & =& \underset{w\in {\mathbb{R}}^d}{arg\; min}\{{\left(\frac{1}{n}\sum _{i=1}^n{\alpha }_i^{(t+1)}{x}_i\right)}^{T}w+g\left(w\right)\}\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill & =& \underset{w\in {\mathbb{R}}^d}{arg\; max}\{{(-\frac{1}{n}\sum _{i=1}^n{\alpha }_i^{(t+1)}{x}_i)}^{T}w-g\left(w\right)\}\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill & =& \nabla g^{*}(-\frac{1}{n}\sum _{i=1}^n{\alpha }_i^{(t+1)}{x}_i),\hfill \end{array}$$

(8)

where the last equation is derived from the conjugation sub-gradient theorem in [25]. In this way, we turn the optimization process into a derivative operation of $g^{*}\left(w\right)$. For instance, if $g\left(x\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$ the update of primal variable can be written as $w^{(t+1)}=-\frac{1}{\lambda n}{\sum }_{i=1}^n{\alpha }_i^{(t+1)}{x}_i$.

We compare the complexity of SPD1, SPD1-VR, and SVRG [19] with our methods in Table 2. In Table 2, r is the maximum number of non-zero elements in each sample, S is the number of non-zero elements in the whole data sets, d is the dimension of the dataset, and n the number of data samples. Usually, S is much smaller than $nd$ when the data are sparse and high-dimensional. Apparently, in most large-scale data applications the data sets are sparse have high dimensionality, i.e., most of the attributes are zeros. At each iteration, SPD1 and SPD1-VR choose $x_{ij}$ (the j-th value of sample $x_i$) to update the primal variable and dual variable regardless of whether $x_{ij}$ is 0 or not. This method enables the per-iteration complexity of SPD1 and SPD1-VR to be reduced to $\mathcal{O}\left(1\right)$. However, their complexity of pass-through data is $\mathcal{O}\left(nd\right)$, which is the same as SVRG. In contrast, ASPDC will not execute the update if $x_{ij}=0$. Thus, the complexity of its pass-through data is $\mathcal{O}\left(S\right)$, which is much lower than SPD1 and SVRG when the data are sparse and high-dimensional.

Table 2. Complexity comparison of per-iteration and pass through data.

	Per-Iteration	Pass through Data
ASPDC, ASPDC-i	$\mathcal{O}\left(r\right)$	$\mathcal{O}\left(S\right)$
SPD1, SPD1-VR	$\mathcal{O}\left(1\right)$	$\mathcal{O}\left(nd\right)$
SVRG [19]	$\mathcal{O}\left(d\right)$	$\mathcal{O}\left(nd\right)$

There are two major differences between SDCA and ASPDC, as follows. First, SDCA tries to solve the dual problem, while ASPDC tries to solve a saddle point problem. Second, the dual update of ASPDC is significantly simpler than the update of SDCA. The dual update of SDCA is shown in (9). In comparison with that of ASPDC in Equation (4), the dual update of SDCA involves the additional computation of $\frac{1}{2\lambda n}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i)}^2$:

$$\begin{array}{cc}\hfill & \Delta {\alpha }_i^{*}=\hfill \\ \hfill & \underset{\Delta {\alpha }_i\in \mathbb{R}}{arg\; max}(-\Delta {\alpha }_i{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}(-{\alpha }_i^{\left(t\right)}-\Delta {\alpha }_i))+\frac{1}{2\lambda n}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i)}^2\hfill \end{array}$$

(9)

We use the dual-gap metric as the stopping criterion, as shown in line 9 of Algorithm 1. The dual-gap is calculated by $P\left(w\right)-D\left(\alpha \right)$, and it is sufficient to say that $|P\left(w\right)-P\left(w^{*}\right)|\le ϵ$ if $P\left(w\right)-D\left(\alpha \right)\le ϵ$, as $|P\left(w\right)-p\left(w^{*}\right)|\le P\left(w\right)-D\left(\alpha \right)\le ϵ$. This stopping criterion is easier to implement than the other criteria, e.g., $|P\left(w\right)-P\left(w^{*}\right)|\le ϵ$. This is for the reason that $w^{*}$ is not known in advance in real-world machine learning applications.

Algorithm 1 ASPDC

1: Input $f(w,\alpha )$, ${\alpha }^{\left(0\right)},ϵ$   2: Initialize $w^{\left(0\right)}=\nabla g^{*}(-\frac{1}{n}{\sum }_{i=1}^n{\alpha }_i^{\left(0\right)}{x}_i)$   3: for$t=0,1,2,\dots$do   4:     pick $i\in \{1,2,\dots ,n\}$ under uniform distribution.   5:     $\Delta {\alpha }_i^{*}=\underset{\Delta {\alpha }_i\in \mathbb{R}}{arg\; max}(\Delta {\alpha }_i{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i))$   6:     ${\alpha }^{(t+1)}={\alpha }^{\left(t\right)}+\Delta {\alpha }_i^{*}{e}_i$   7:     $w^{(t+1)}=\nabla g^{*}(-\frac{1}{n}{\sum }_{i=1}^n{\alpha }_i^{(t+1)}{x}_i)$   8: end for   9: Stop condition: $P\left(w^{\left(T\right)}\right)-D\left({\alpha }^{\left(T\right)}\right)\le ϵ$ Output  $\phantom{1}$$w^{\left(T\right)},{\alpha }^{\left(T\right)}$, $P\left(w^{\left(T\right)}\right)-D\left({\alpha }^{\left(T\right)}\right)$

In the rest of this section, we show the proof for ASPDC’s convergence. We first present the following lemma.

Lemma 2.

On the basis of Assumptions 1–3, let $w^{\left(t\right)}$ and ${\alpha }^{\left(t\right)}$ be the sequence produced by ASPDC and let $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$. $\forall \lambda \ge \frac{4}{n\gamma }$; then, we have:

$$E(P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right))\le 2n{(1-\frac{1}{2n})}^t(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right))$$

(10)

Proof. 

The detailed proof can be found in the Appendix. In the proof, we assume that $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$ for convenience. Therefore, the theory only works for l2 regularization. The extension to l1 regularization is a topic for future work.

The skeleton of the proof in the Appendix can be described using the following three steps:

First, we obtain

$$E(D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right))\le {(1-\frac{1}{2n})}^t(D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(0\right)}\right)).$$

Second, we have

$$\begin{array}{cc}\hfill & \frac{1}{2n}E(P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right))\hfill \\ \hfill & \le E(D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right))\hfill \\ \hfill & \le E(D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{*}\right)+D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right))\hfill \\ \hfill & \le D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)-E(D\left({\alpha }^{*}\right)-D\left({\alpha }^{(t+1)}\right)\hfill \\ \hfill & \le D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\hfill \end{array}$$

Finally, using the weak duality we can obtain

$$E(P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right))\le 2n{(1-\frac{1}{2n})}^t(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right)).$$

   □

Theorem 1.

The total number of iterations needed to achieve the expected duality gap of $E(P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right))\le ϵ$ is

$$t\ge 2nlog\left(2n(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right))\frac{1}{ϵ}\right)$$

Proof. 

Using Lemma 2, we can obtain

$$\begin{array}{c}\hfill E(P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right))\le 2nexp\left(\frac{-t}{2n}\right)(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right)),\end{array}$$

(11)

where, in the inequality, we use the fact that ${(1-\frac{1}{2n})}^t\le exp\left(\frac{-t}{2n}\right)$. Let $2nexp\left(\frac{-t}{2n}\right)(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right))\le ϵ$; then, we finally obtain $t\ge 2nlog\left(2n(P\left(w^{\left(0\right)}\right)-D\left({\alpha }^{\left(0\right)}\right))\frac{1}{ϵ}\right)$. □

As shown by Equation (11), the complexity of ASPDC is $\mathcal{O}(nlog\left(n\frac{1}{ϵ}\right))$, In contrast, the complexity of SVRG is $\mathcal{O}\left(d\right(n+\kappa )log(1/ϵ\left)\right)$ and the complexity of SPDC is $\mathcal{O}(d(n+\sqrt{n\kappa })log(1/ϵ))$.

5. ASPDC for Ill-Conditioned Problems

According to convex theory [16], the value $Q_f=L/\mu$ is called the condition number of function f if f is $L-smooth$ and $\mu -stronglyconvex$. Under Assumptions 1–3, the condition number of the primal function in Equation (1) is $(1+\gamma \lambda )/\lambda =\frac{1}{\lambda \gamma }+1$. Suppose $\lambda$ becomes lower; then, the condition number, $Q_f$, will be larger. When $Q_f\gg 1$, the problem f is called ill-conditioned.

In this section, we extend ASPDC to the ill-conditioned problem, especially when $\lambda \le \frac{4}{n\gamma }$. The extension method is called ASPDC-i, in which the suffix i means “for ill-conditioned problems”.

As shown in Algorithm 2, the procedure of ASPDC-i can be divided into epochs, indexed $s=1,2,3,\dots ,S$. Each epoch uses ASPDC to solve the following problem with a decreasing precision parameter ${\xi }_s$:

$$\underset{w\in {\mathbb{R}}^d}{min}\underset{\alpha \in {\mathbb{R}}^n}{max}{\tilde{f}}_s(w,\alpha )=\frac{1}{n}\sum _{i=1}^n[{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]+\tilde{g}\left(w\right)$$

(12)

where $\tilde{g}\left(w\right)=g\left(w\right)+\frac{\kappa }{2}{\left|\right|w\left|\right|}_2^2-\kappa w^T{\tilde{w}}^s$, $\kappa \in \mathbb{R}$ is a constant throughout the procedure, and $\tilde{g}\left(w\right)$ is $g\left(w\right)$ plus an additional perturbation term. This additional term is employed to ensure that the strongly convex parameter $\lambda +\kappa$ of $\tilde{g}\left(w\right)$ satisfies $\lambda +\kappa \ge \frac{4}{n\gamma }$. Note that a smaller $\kappa$ is preferable, as a larger $\kappa$ leads to a severe bias between $f(w,\alpha )$ and ${\tilde{f}}_s(w,\alpha )$. Therefore, in the implementation of our ASPDC algorithms we simply use the smallest $\kappa$: $\kappa =\frac{4}{n\gamma }-\lambda$.

These calls of ASPDC produce a sequence ${\tilde{w}}^s,s=1,2,\dots$, which are the solutions of the corresponding approximate problem in Equation (12). Here, we need to prove that each running procedure of ASPDC from these calls can stop itself after finite epochs as well as that the output ${\tilde{w}}^S$ satisfies the condition $|P\left({\tilde{w}}^S\right)-P\left(w^{*}\right)|\le ϵ$. In this condition, the variable $w^{*}$ is the theoretical optimal solution of $P\left(w\right)$. These facts are illustrated in the following Theorem 2.

Theorem 2.

Algorithm 2 needs $S\ge 1+\frac{2}{\eta }log\left({\xi }_1\frac{1}{ϵ}\right)$ epochs to approach the approximate solution $w^{*}$, where $|P\left({\tilde{w}}^S\right)-P\left(w^{*}\right)|\le ϵ$.

The proof can be found in the Appendix A. The settings of the hyper parameters of Algorithm 2 are presented in the proof.

Algorithm 2 ASPDC-i

1:  Parameter: $\lambda \le \frac{4}{n\gamma }$, $\kappa =\frac{4}{n\gamma }-\lambda$, $\eta =\frac{\lambda }{\lambda +2\kappa }$,

${\xi }_1=(1+{\eta }^{-1})(P\left({\tilde{w}}^1\right)-D\left({\tilde{\alpha }}^1\right))$

2:  Initialize: ${\tilde{w}}^1=0$, ${\tilde{\alpha }}^1=0$

3:  for s = 1,2,3,... do

4:   $({\tilde{w}}^{s+1},{\tilde{\alpha }}^{s+1},{ϵ}_{s+1})$ = ASPDC$({\tilde{f}}_s(w,\alpha ),{\tilde{\alpha }}^s,\frac{\eta }{2(1+{\eta }^{-1})}{\xi }_s)$

5:   ${\xi }_{s+1}=(1-0.5\eta ){\xi }_s$

6:  end for

7:  stop condition: $S\ge 1+\frac{2}{\eta }log\left({\xi }_1\frac{1}{ϵ}\right)$

Output$\phantom{1}$${\tilde{w}}^S$, ${\tilde{\alpha }}^S$

To make for a fair comparison with other algorithms, we provide an realistic implementation version of Algorithm 2. This implementation version is shown in Algorithm 3. Here, the number of iterations in Algorithm 3 is set to be a constant m (e.g., $m=2n$). As be demonstrated in the experiment section, this approach works well.

Algorithm 3 Implemented version of ASPDC-i

1: Parameter: $\lambda \le \frac{4}{n\gamma }$, $\kappa =\frac{4}{n\gamma }-\lambda$   2: Initialize: ${\tilde{w}}^0=0$, ${\tilde{\alpha }}^0=0$   3: for$s=1,2,3,\dots ,S$do   4:    ${\alpha }^{\left(0\right)}={\tilde{\alpha }}^{(s-1)}$, $w^{\left(0\right)}=\nabla {\tilde{g}}^{*}(-\frac{1}{n}{\sum }_{i=1}^n{x}_i{\alpha }_i^{\left(0\right)})$   5:    for $t=0,1,2,\dots ,m-1$ do   6:        pick $i\in \{1,2,\dots ,n\}$ under uniform distribute   7:        $\Delta {\alpha }_i^{*}=\underset{\Delta {\alpha }_i\in \mathbb{R}}{arg\; max}(\Delta {\alpha }_i{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i))$   8:        ${\alpha }^{(t+1)}={\alpha }^{\left(t\right)}+\Delta {\alpha }_i^{*}{e}_i$   9:        $w^{(t+1)}=\nabla {\tilde{g}}^{*}(-\frac{1}{n}{\sum }_{i=1}^n{x}_i{\alpha }_i^{(t+1)})$   10:    end for   11:    ${\tilde{w}}^s=w^{\left(m\right)}$, ${\tilde{\alpha }}^s={\alpha }^{\left(m\right)}$   12: end for Output  $\phantom{1}$${\tilde{w}}^S$, ${\tilde{\alpha }}^S$

6. Experiments

In this section, we evaluate the performance of our ASPDC algorithms along with several state-of-art algorithms for solving machine learning problems such as SVM. All the algorithms were implemented in C++ and executed through a Matlab interface. The experiments were performed on a PC with an Intel i5-4690 CPU and 16.0 GB RAM. The source code and the detailed proofs can be downloaded from the GitHub website (https://github.com/lianghb6/ASPDC, accessed on 28 June 2022) and the datasets can be obtained from the LIBSVM website (https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/, accessed on 28 June 2022).

As the computation processes of the problems are similar, in these experiments we mainly evaluated the practical performance of ASPDC for solving the following SVM optimization problem:

$$\underset{w\in {\mathbb{R}}^d}{min}\{P\left(w\right)=\frac{1}{n}\sum _{i=1}^n{\varphi }_i\left(w^T{x}_i\right)+\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2\}$$

where ${\varphi }_i$ is a smooth hinge loss, and is used in [1,5] as well.

$${\varphi }_i\left(w^T{x}_i\right)=\left\{\begin{array}{c}0y_i{w}^T{x}_i\ge 1\hfill \\ \frac{1}{2}-y_i{w}^T{x}_i{y}_i{w}^T{x}_i\le 0\hfill \\ \frac{1}{2}{(1-y_i{w}^T{x}_i)}^{2}otherwise.\hfill \end{array}\right.$$

The corresponding convex–concave saddle point problem is as follows:

$$\underset{w\in {\mathbb{R}}^d}{min}\underset{\alpha \in {\mathbb{R}}^n}{max}\{f(w,\alpha )=\frac{1}{n}\sum _{i=1}^n[{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]+\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2\}$$

where

$${\varphi }_i^{*}\left({\alpha }_i\right)=\left\{\begin{array}{c}{y}_i{\alpha }_i+\frac{1}{2}{\alpha }_i^2-1\le y_i{\alpha }_i\le 0\hfill \\ +\infty otherwise.\hfill \end{array}\right.$$

Under Assumption 3, the smooth parameter $\gamma$ of ${\varphi }_i$ is 1. The strongly convex parameter of $P\left(w\right)$ is $\lambda$, which comes from the regularized function $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$.

In Figure 1 and Table 3, we show the cases when $\lambda$ is relatively large (e.g., ${10}^{-2},{10}^{-3},{10}^{-4}$). We compare ASPDC (Algorithm 1) with state-of-art dual methods: the stochastic dual coordinate ascent method (SDCA) [5] and stochastic primal–dual coordinate method (SPDC) [1]. Note that accelerated stochastic dual ascent (ASDCA) [13] cannot be applied to this scenario, as ASDCA requires $\lambda$ to be extremely small (i.e., $\lambda \le \frac{1}{10n\gamma })$. We omit the comparison between ASPDC and the stochastic gradient descent method and its variants (e.g., SVRG [19] and Katyusha [26]), as there have already been extensive experiments using SPDC and this situation performed in the literature.

Figure 1. Dual-gap (y-axis) vs, the number of epochs (x-axis). Comparing ASPDC with other methods for smooth hinge SVM on real-world datasets with regularization coefficient $\lambda \in \{0.1,0.01,0.001,0.0001\}$. The horizontal axis is the number of passes through the entire dataset, and the vertical axis is the logarithmic dual-gap.

Table 3. The running time for dual-gap approaches to the given precision (${10}^{-6}$) when $\lambda =0.01$.

	SDCA	SPDC	ASPDC
a9a	0.505 s	1.311 s	0.636 s
ijcnn	0.984 s	1.438 s	1.183 s
covtype	11.502 s	20.526 s	14.972 s

The horizontal axis in Figure 1 is the number of passes through the entire dataset, and the vertical axis is the logarithmic dual-gap. It can be seen from Figure 1 that ASPDC and SDCA have comparable performances on relatively large $\lambda$. With the same epoch, the dual-gap of ASPDC is lower than that of SPDC by two orders of magnitude after several epochs.

Figure 1 shows that both SDCA and ASPDC are faster than SPDC. This is because $\lambda$ in Figure 1 is relative large (e.g., 0.01). In this case, the condition number of problems is relatively small. When the condition number is large, ASPDC and SPDC perform better than SDCA. In total, ASPDC is faster and is well suited for ill-conditioned problems.

Table 3 lists the needed running time for the dual-gaps of different algorithms to decrease to the given precision (e.g., $dualgap\le {10}^{-6}$) for different algorithms and datasets. Table 3 demonstrates that ASPDC and SDCA need less time to approach the given precision, and verifies that the convergence of ASPDC and SDCA is faster than SPDC. Table 4 presents the total running time for the algorithms to go through the entire dataset once to measure the per-iteration computation complexity. An algorithm with a shorter running time indicates that the algorithm has a lower per-iteration computation complexity. Table 4 shows that ASPDC and SDCA have a lower per-iteration complexity than SPDC. Among all of the running time results, ASPDC demonstrates both fast convergence and low per-iteration complexity when $\lambda$ is large.

Table 4. The average running time for the algorithms to pass through the entire dataset once when $\lambda =0.01$.

	SDCA	SPDC	ASPDC
a9a	0.029 s	0.052 s	0.028 s
ijcnn	0.053 s	0.061 s	0.052 s
covtype	0.650 s	0.840 s	0.644 s

We then tested the case when $\lambda$ is relatively small (e.g., $\lambda \le \frac{4}{\gamma n}$) and compared ASPDC-i with SDCA, SPDC, and ASDCA. Figure 2 plots the convergence results. Figure 2 shows that the convergences of SDCA, ASDCA, and SPDC are significantly slower than those of the same algorithms in Figure 1. The reason for this is that the condition number of the problem in this test case is larger than that in Figure 1. ASPDC-i performs much better in this experiment, as can be seen from Figure 2. ASPDC-i needs far fewer epochs than other algorithms to approach the same level of dual-gap. Additionally, ASPDC can approach a significantly lower dual-gap than the others with the same epochs.

Figure 2. Dual-gap (y-axis) vs. the number of epochs (x-axis). Comparing ASPDC-i with other methods for smooth hinge SVM on real-world datasets with regularization coefficient $\lambda \in \{{10}^{-6},{10}^{-7},{10}^{-8}\}$. The horizontal axis is the number of passes through the entire dataset, and the vertical axis is the logarithmic dual-gap.

In addition, we compared ASPDC-i to a widely used non-dual-based algorithm, SVRG [19]. As SVRG is not dual-based, we directly compared its reduction speed of the primal value with ASPDC-i. Figure 3 shows that the convergence speed of ASPDC-i is faster than SVRG.

Figure 3. Optimal primal value (y-axis) vs. the number of epochs (x-axis): Comparing ASPDC-i with SVRG for smooth hinge SVM on real-world datasets with the regularization coefficient ${10}^{-6}$. The x-axis is the number of passes through the entire dataset, and the y-axis is the logarithmic dual-gap.

Note that ASDCA cannot be applied to cases in which the dataset is covtype and $\lambda ={10}^{-6}$, as ASDCA needs the extra condition $\lambda \le \frac{1}{10n\gamma }$. Table 5 illustrates the running time that different algorithms spend to decrease the dual-gap to the given precision (e.g., ${10}^{-4}$). Table 6 demonstrates the total running time for the algorithms to go through the entire dataset once. It shows that ASPDC and ASDCA have lower per-iteration complexity than SPDC. Although SDCA has low per-iteration complexity, its convergence is the slowest among these methods when $\lambda$ is relatively small. We did not list the corresponding results of SDCA in Table 5 and Table 6. In summary, the above experiments show that our proposed methods achieve both fast convergence and low per-iteration complexity.

Table 5. The running time for dual-gaps to approach the given precision (${10}^{-4}$) when $\lambda ={10}^{-6}$.

	ASDCA	SPDC	ASPDC-i
a9a	0.582 s	2.262 s	0.8464 s
ijcnn	0.994 s	3.127 s	2.033 s
covtype	8.407 s	91.132 s	47.734 s

Table 6. The average running time for the algorithms to pass through the entire dataset once when $\lambda ={10}^{-6}$.

	ASDCA	SPDC	ASPDC-i
a9a	0.0165 s	0.0857 s	0.0167 s
ijcnn	0.0305 s	0.0821 s	0.0302 s
covtype	0.208 s	1.253 s	0.408 s

7. Conclusions and Future Work

In this paper, we propose two stochastic primal–dual coordinate methods, ASPDC and its accelerated variant version, ASPDC-i. These two algorithms are designed for the regularized empirical risk minimization problem. We proved the theoretical convergence guarantee of the algorithms and performed a series of experiments. The results illustrate that our methods achieve a good balance between low per-iteration computation complexity and fast convergence. The new convergence proof presented here uses Nesterov’s estimation sequence technique and $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$. We believe that it is possible to extend this proof to the more general regularized function $g\left(w\right)$; however, we leave this as a possibility for future work.

Appendix A.1. Proof of Lemma 1

We prove the following equations: $P\left(w\right)=\underset{\alpha \in {\mathbb{R}}^n}{max}f(w,\alpha )$, $D\left(\alpha \right)=\underset{w\in {\mathbb{R}}^d}{min}f(w,\alpha )$ and $P\left(w^{*}\right)=D\left({\alpha }^{*}\right)=f(w^{*},{\alpha }^{*})$. We first prove $P\left(w\right)=\underset{\alpha \in {\mathbb{R}}^n}{max}f(w,\alpha )$.

Proof. 

$$\begin{array}{c}\underset{\alpha \in {\mathbb{R}}^n}{max}f(w,\alpha )\hfill \\ =\underset{\alpha \in {\mathbb{R}}^n}{max}\{\frac{1}{n}\sum _{i=1}^n[{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]+g\left(w\right)\}\hfill \\ =\underset{\alpha \in {\mathbb{R}}^n}{max}\left\{\frac{1}{n}\sum _{i=1}^n[{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]\right\}+g\left(w\right)\hfill \\ =\frac{1}{n}\sum _{i=1}^n\underset{{\alpha }_i\in \mathbb{R}}{max}\{{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)\}+g\left(w\right)\hfill \\ =\frac{1}{n}\sum _{i=1}^n{\varphi }_i\left(w^T{x}_i\right)+g\left(w\right)\hfill \\ =P\left(w\right)\hfill \end{array}$$

(A1)

In the last equation, we use the Conjugate Theorem (Convex Optimization Theory). Then, we prove that $D\left(\alpha \right)=\underset{w\in {\mathbb{R}}^d}{min}f(w,\alpha )$.

$$\begin{array}{cc}\hfill & \underset{w\in {\mathbb{R}}^d}{min}f(w,\alpha )\hfill \\ \hfill & =\underset{w\in {\mathbb{R}}^d}{min}\{\frac{1}{n}\sum _{i=1}^n[{\alpha }_i{w}^T{x}_i-{\varphi }_i^{*}\left({\alpha }_i\right)]+g\left(w\right)\}\hfill \\ \hfill & =\frac{-1}{n}\sum _{i=1}^n{\varphi }_i^{*}\left({\alpha }_i\right)+\underset{w\in {\mathbb{R}}^d}{min}\{{\left(\frac{1}{n}\sum _{i=1}^n{\alpha }_i{x}_i\right)}^{T}w+g\left(w\right)\}\hfill \\ \hfill & =\frac{-1}{n}\sum _{i=1}^n{\varphi }_i^{*}\left({\alpha }_i\right)-\underset{w\in {\mathbb{R}}^d}{max}\{{\left(\frac{-1}{n}\sum _{i=1}^n{\alpha }_i{x}_i\right)}^{T}w-g\left(w\right)\}\hfill \\ \hfill & =\frac{-1}{n}\sum _{i=1}^n{\varphi }_i^{*}\left({\alpha }_i\right)-g^{*}\left(\frac{-1}{n}\sum _{i=1}^n{\alpha }_i{x}_i\right)\hfill \\ \hfill & =D\left(\alpha \right)\hfill \end{array}$$

The proof of $P\left(w^{*}\right)=D\left({\alpha }^{*}\right)=f(w^{*},{\alpha }^{*})$ can be found in [1]. □

Appendix A.2. Proof of Lemma 2

Proof. 

When $g\left(w\right)=\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2$, the primal objective can be written as follows:

$$\begin{array}{cc}\hfill & P\left(w\right)=\frac{1}{n}\sum _{i=1}^n{\varphi }_i\left(w^T{x}_i\right)+\frac{\lambda }{2}{\left|\right|w\left|\right|}_2^2.\hfill \end{array}$$

(A2)

The corresponding dual objective is

$$\begin{array}{cc}\hfill & D\left(\alpha \right)=\frac{-1}{n}\sum _{i=1}^n{\varphi }_i^{*}\left({\alpha }_i\right)-\frac{\lambda }{2}\left|\right|\frac{-1}{\lambda n}\sum _{i=1}^n{\alpha }_i{x}_i{\left|\right|}_2^2.\hfill \end{array}$$

(A3)

Note that at through the algorithm, we can set

$$\begin{array}{cc}\hfill & w^{\left(t\right)}=\frac{-1}{\lambda n}\sum _{i=1}^n{\alpha }_i^{\left(t\right)}{x}_i.\hfill \end{array}$$

(A4)

Thus, the $D\left({\alpha }^{\left(t\right)}\right)$ can be written as

$$\begin{array}{cc}\hfill & D\left({\alpha }^{\left(t\right)}\right)=\frac{-1}{n}\sum _{i=1}^n{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2.\hfill \end{array}$$

(A5)

Suppose we have ${\alpha }^{\left(t\right)}$ and that the $i-th$ coordinate is chosen at iteration $t+1$:

$$\begin{array}{cc}\hfill & D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\hfill \\ \hfill & =\underset{R_1}{\underbrace{-\frac{1}{n}{\varphi }_i^{*}\left({\alpha }_i^{(t+1)}\right)-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}-\frac{1}{\lambda n}\Delta {\alpha }_i^{*}{x}_i{\left|\right|}_2^2}}\hfill \\ \hfill & -\underset{R_2}{\underbrace{\{-\frac{1}{n}{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\}}}.\hfill \end{array}$$

(A6)

The variables in the algorithm are as follows:

$$\begin{array}{c}\Delta {\alpha }_i^{*}=\underset{d\in \mathbb{R}}{arg\; max}(d{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+d))\hfill \\ =\underset{d\in \mathbb{R}}{arg\; max}(({\alpha }_t^{\left(t\right)}+d)x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+d))\hfill \\ =\underset{{\alpha }_t^{\left(t\right)}+d\in \mathbb{R}}{arg\; max}(({\alpha }_t^{\left(t\right)}+d)x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+d))\hfill \\ =\underset{\beta \in \mathbb{R}}{arg\; max}(\beta x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}\left(\beta \right))-{\alpha }_i^{\left(t\right)},\hfill \end{array}$$

(A7)

where in the last inequality we define $\beta ={\alpha }_i^{\left(t\right)}+d$, and correspondingly have ${\beta }^{*}={\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i^{*}$.

$$\begin{array}{c}{R}_1=-\frac{1}{n}{\varphi }_i^{*}\left({\alpha }_i^{(t+1)}\right)-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}-\frac{1}{\lambda n}\Delta {\alpha }_i^{*}{x}_i{\left|\right|}_2^2\hfill \\ =-\frac{1}{n}{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i^{*})+\frac{1}{n}\Delta {\alpha }_i^{*}{x}_i^T{w}^{\left(t\right)}\hfill \\ -\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ =\frac{1}{n}\left\{\underset{d\in \mathbb{R}}{max}(d{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+d))\right\}\hfill \\ -\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ \stackrel{①}{\ge }\frac{1}{n}\{q({\beta }^{*}-{\alpha }_i^{\left(t\right)})x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}({\alpha }_i^{\left(t\right)}+q({\beta }^{*}-{\alpha }_i^{\left(t\right)}))\}\hfill \\ -\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ =-\frac{1}{n}{\varphi }_i^{*}((1-q){\alpha }_i^{\left(t\right)}+q{\beta }^{*})+\frac{1}{n}q({\beta }^{*}-{\alpha }_i^{\left(t\right)})x_i^T{w}^{\left(t\right)}\hfill \\ -\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ \stackrel{②}{\ge }-\frac{1}{n}\{q{\varphi }_i^{*}\left({\beta }^{*}\right)+(1-q){\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-\frac{\gamma q(1-q)}{2}{({\beta }^{*}-{\alpha }_i^{\left(t\right)})}^2\}\hfill \\ +\frac{1}{n}q({\beta }^{*}-{\alpha }_i^{\left(t\right)})x_i^T{w}^{\left(t\right)}-\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ \ge \frac{q}{n}\{-{\varphi }_i^{*}\left({\beta }^{*}\right)+{\beta }^{*}{x}_i^T{w}^{\left(t\right)}\}-\frac{1-q}{n}{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)\hfill \\ +\frac{\gamma (1-q)q}{2n}{({\beta }^{*}-{\alpha }_i^{\left(t\right)})}^2-\frac{q}{n}{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\hfill \\ -\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \end{array}$$

(A8)

where $q\in (0,1)$ in the inequality ①, while in the inequality ② we use the fact that if ${\varphi }_i$ is $\frac{1}{\gamma }$ smooth, then ${\varphi }_i^{*}$ is $\gamma$ strong convex.

On the one hand, according to (A7), we obtain

$$\begin{array}{cc}\hfill & {\beta }^{*}=\underset{\beta \in \mathbb{R}}{arg\; max}(\beta x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}\left(\beta \right)).\hfill \end{array}$$

(A9)

This implies that

$$\begin{array}{cc}\hfill & x_i^T{w}^{\left(t\right)}=\nabla {\varphi }_i^{*}\left({\beta }^{*}\right).\hfill \end{array}$$

(A10)

On the other hand, by the definition of the convex conjugate function, we have ${\varphi }_i^{**}\left(x_i^T{w}^{\left(t\right)}\right)=\underset{\beta \in \mathbb{R}}{max}(\beta x_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}\left(\beta \right))$. According to the Fenchel conjugate sub-gradient theorem, we have

$$\begin{array}{c}{x}_i^T{w}^{\left(t\right)}=\nabla {\varphi }_i^{*}\left({\beta }^{*}\right)⟺{\beta }^{*}{x}_i^T{w}^{\left(t\right)}-{\varphi }_i^{*}\left({\beta }^{*}\right)\hfill \\ ={\varphi }_i^{**}\left(x_i^T{w}^{\left(t\right)}\right)\stackrel{③}{=}{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right),\hfill \end{array}$$

(A11)

where in ③ we apply the Fenchel Dual theorem.

Combined with (A8) and (A11), we obtain

$$\begin{array}{c}{R}_1\ge \frac{q}{n}\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\}\hfill \\ +\frac{\gamma (1-q)q}{2n}{({\beta }^{*}-{\alpha }_i^{\left(t\right)})}^2-\frac{1}{2\lambda n^2}\left|\right|x_i{\left|\right|}_2^2{(\Delta {\alpha }_i^{*})}^2\hfill \\ +\underset{R_2}{\underbrace{\{-\frac{1}{n}{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-\frac{\lambda }{2}\left|\right|w^{\left(t\right)}{\left|\right|}_2^2\}}}.\hfill \end{array}$$

(A12)

Combining ${\beta }^{*}={\alpha }_i^{\left(t\right)}+\Delta {\alpha }_i^{*}$ with (A6) and (A12), we have

$$\begin{array}{c}D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\hfill \\ \ge \frac{q}{n}\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\}\hfill \\ +\{\frac{\gamma (1-q)q}{2n}-\frac{1}{2\left(\lambda \right)n^2}\left|\right|x_i{\left|\right|}_2^2\}{(\Delta {\alpha }_i^{*})}^2\hfill \\ \ge \frac{q}{n}\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\}\hfill \\ +\{\frac{\gamma (1-q)q}{2n}-\frac{1}{2(\lambda n^2}\}{(\Delta {\alpha }_i^{*})}^2,\hfill \end{array}$$

(A13)

where in the last inequality we use the assumption $\left|\right|x_i{\left|\right|}_2^2\le 1$. Note that we have supposed that the $i-th$ coordinate of $\alpha$ is chosen, thus, we use the expectation of (A13) with respect to i, obtaining

$$\begin{array}{c}E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \ge \frac{q}{n}\frac{1}{n}\sum _{i=1}^n\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\}\hfill \\ +\{\frac{\gamma (1-q)q}{2n}-\frac{1}{2\lambda n^2}\}\frac{1}{n}\sum _{i=1}^n{(\Delta {\alpha }_i^{*})}^2.\hfill \end{array}$$

(A14)

Recall that

$$\begin{array}{c}P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\hfill \\ =\frac{1}{n}\sum _{i=1}^n\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)\}+\lambda \left|\right|w^{\left(t\right)}{\left|\right|}_2^2\hfill \\ \stackrel{④}{=}\frac{1}{n}\sum _{i=1}^n\{{\varphi }_i\left(x_i^T{w}^{\left(t\right)}\right)+{\varphi }_i^{*}\left({\alpha }_i^{\left(t\right)}\right)-{\alpha }_i^{\left(t\right)}{x}_i^T{w}^{\left(t\right)}\},\hfill \end{array}$$

(A15)

where in ④ we use the fact that $w^{\left(t\right)}=\frac{-1}{\lambda n}{\sum }_{i=1}^n{\alpha }_i^{\left(t\right)}{x}_i$.

Combined (A14) with (A15), we obtain

$$\begin{array}{c}E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \ge \frac{q}{n}\{P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}+\{\frac{\gamma (1-q)q}{2n}-\frac{1}{2\lambda n^2}\}\frac{1}{n}\sum _{i=1}^n{(\Delta {\alpha }_i^{*})}^2.\hfill \end{array}$$

(A16)

Using $q=1/2$ and $\lambda \ge \frac{4}{n\gamma }$, we have $\frac{\gamma (1-q)q}{2n}-\frac{1}{2\lambda n^2}\ge 0$, and

$$\begin{array}{cc}\hfill & E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\ge \frac{1}{2n}\{P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}.\hfill \end{array}$$

(A17)

Note that ${\alpha }^{*}={arg\; max}_{\alpha }D\left(\alpha \right)$; it is well known that $P\left(w^{\left(t\right)}\right)\ge D\left({\alpha }^{*}\right)\ge D\left({\alpha }^{\left(t\right)}\right)$.

Combined with (A17), we obtain

$$\begin{array}{c}\frac{1}{2n}\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \le \frac{1}{2n}\{P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \le E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ =E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{*}\right)+D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ =\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\}-E\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{(t+1)}\right)\}.\hfill \end{array}$$

(A18)

This further implies that

$$\begin{array}{cc}\hfill & E\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{(t+1)}\right)\}\le (1-\frac{1}{2n})\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right).\hfill \end{array}$$

(A19)

Until now, we have assumed that ${\alpha }^{\left(t\right)}$ is known and the expectation is for random variable i; if below we take this expectation with all the history i, we obtain

$$\begin{array}{cc}\hfill & E\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{(t+1)}\right)\}\le {(1-\frac{1}{2n})}^{(t+1)}\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(0\right)}\right).\hfill \end{array}$$

(A20)

In addition, it can be known from (A17) that

$$\begin{array}{cc}\hfill & \frac{1}{2n}E\{P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\le E\{D\left({\alpha }^{(t+1)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \hfill & =\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\}-E\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{(t+1)}\right)\}\hfill \\ \hfill & \le \{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\hfill \\ \hfill & \le {(1-\frac{1}{2n})}^t\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(0\right)}\right)\}\hfill \end{array}$$

This implies that $E\{P\left(w^{\left(t\right)}\right)-D\left({\alpha }^{\left(t\right)}\right)\}\le 2n{(1-\frac{1}{2n})}^t\{D\left({\alpha }^{*}\right)-D\left({\alpha }^{\left(0\right)}\right)\}$. □