Modified Accelerated Bundle-Level Methods and Their Application in Two-Stage Stochastic Programming

Abstract

The accelerated prox-level (APL) and uniform smoothing level (USL) methods recently proposed by Lan (Math Program, 149: 1–45, 2015) can achieve uniformly optimal complexity when solving black-box convex programming (CP) and structure non-smooth CP problems. In this paper, we propose two modified accelerated bundle-level type methods, namely, the modified APL (MAPL) and modified USL (MUSL) methods. Compared with the original APL and USL methods, the MAPL and MUSL methods reduce the number of subproblems by one in each iteration, thereby improving the efficiency of the algorithms. Conclusions of optimal iteration complexity of the proposed algorithms are established. Furthermore, the modified methods are applied to the two-stage stochastic programming, and numerical experiments are implemented to illustrate the advantages of our methods in terms of efficiency and accuracy.

1. Introduction

In the fields of production planning, finance risk, telecommunication, and electricity, decision makers need to take into consideration uncertainty about the information and the model itself. Lack of data, calculation errors as well as unpredictability, etc. lead to the uncertainty in information. The uncertainty of model is derived from the structure of problem, the features of constraints, as well as the risks and profiles of decisions. Stochastic programming (SP) is an effective tool for dealing with optimization problems under uncertainty. The expectation model in stochastic programming is widely used, which maximizes the expectation of the benefit or minimizes the loss under the expected constraints. In this paper, we are concerned with the two-stage stochastic programming with recourse. Next, through a practical problem, network planning with random demand, we introduce the mathematical model of two-stage stochastic programming.

Due to demand for higher bandwidth and dedicated lines, network capacity is becoming a scarce resource. Consider a situation where a network provider plans bandwidth allocation between network links, under total network capacity b that is available for allocation. In addition, there are n different links needed to be expanded in capacity. The extra capacity allocated to link j is $x_j$, where $j=1,2,\dots ,n$ and vector $x\in {\mathbb{R}}^n$ consists of elements $x_j$. In network planning, demands refer to the number of connections requested between point-to-point pairs provided by the network at a certain time. Here, the demand $D\in {\mathbb{R}}^m$ related to the m point-to-point pairs is modeled as a random variable.

Suppose that the extra capacities $x_j$, $j=1,2,\dots ,n$ are given and the demand D is observed. Then the capacity planning model introduced in [1] is to minimize the total number of unoffered requests. Let $i=1,\dots ,m$ be the indices of point-to-point pairs, and the paths that can be offered to connections related to point-to-point pair i be represented in $P\left(i\right)$. $c\in {\mathbb{R}}^n$ is a vector whose elements $c_j$ denote the capacity of current link j. $f_{ip}$ and $s_i$ denote the number of connections and unoffered requests related to pair i, respectively. For an observation $d=(d_1,\dots ,d_m)$ of the random variable D, one can obtain the optimal decision by solving the linear programming problem as follows

$$\begin{array}{cc}\hfill min& \sum _{i=1}^m{s}_i\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{i=1}^m\sum _{p\in P\left(i\right)}{A}_{ip}{f}_{ip}\le x+c,\hfill \\ \hfill & \sum _{p\in P\left(i\right)}{f}_{ip}+s_i=d_i,\hfill \\ & f_{ip}\ge 0,s_i\ge 0,i=1,2,\dots ,m,\mathrm{and}p\in P\left(i\right).\hfill \end{array}$$

(1)

Here, 0-1 vector$A_{ip}$ is introduced as an incidence whose j-th component, $a_{ipj}$, is 1 if link j lies in path p and 0 otherwise. Let $Q(x,d)$ denote the optimal value of the above linear programming (1) which depends on the observation d of D and capacity x. It is obvious that $Q(x,D)$ is also a random variable due to the randomness of D. Thus, the capacities $x_j$, $j=1,2,\dots ,n$ can be obtained from the following programming problem

$$\begin{array}{cc}\hfill min& \mathbb{E}\left[Q\right(x,D\left)\right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{j=1}^n{x}_j\le b,\hfill \\ & x\ge 0.\hfill \end{array}$$

(2)

Here $\mathbb{E}[·]$ refers to the expectation of the probability distribution of random variable D. In a word, the purpose of programming problem (2) is to minimize the expectation of unoffered requests, while satisfying the constraint within the total network capacity b. The above optimization model (1) and (2) is an instance of two-stage stochastic programming with recourse, in which (1) is the second-stage problem and (2) is the first-stage problem.

From the above practical instance we can see that the stochastic programming plays a key role in network planning with random demand. In addition to stochastic programming, the unit commitment problems in the field of energy systems usually involve mixed-integer quadratic programming. Recently, there have been some studies on solving such problems [2,3,4]. For solving the programming problems (1) to (2), Sen et al. [1] applied the stochastic decomposition method, which is one of the most efficient methods for stochastic programming. There are still many applications of stochastic programming in other fields, such as insurance and finance, valuation of electricity, telecommunications, hydrothermal power production planning and pollution control, etc. Stochastic models of these application problems can be found in [5]. In what follows, as an extension of the network planning problem, the standard mathematical model of two-stage stochastic programming with recourse is given, which is more convenient for generalization and theoretical analysis.

The two-stage stochastic programming with fixed recourse is in the following form

$$\begin{array}{cc}min& c^{T}x+\mathbb{E}\left[Q(x,D)\right]\hfill \\ \mathrm{s}.\mathrm{t}.& Ax=b,\hfill \\ & x\ge 0,\hfill \end{array}$$

(3)

where x is the decision variable, c is the the cost of production and $Q(x,D)$ denotes the optimal objective value of the second-stage problem

$$\begin{array}{cc}min& q^{T}y\hfill \\ \mathrm{s}.\mathrm{t}.& Wy=h-Tx,\hfill \\ & y\ge 0.\hfill \end{array}$$

(4)

Here, D is the demand vector with the elements $q^T$, $h^T$, and T. Let W be the fixed recourse and $D=(D_1,\dots ,D_n)$ be a random variable with the support of probability distribution $\Xi \subset {\mathbb{R}}^d$. $\mathbb{E}\left[Q\right(x,D\left)\right]$ in the first-stage problem (3) is the expectation with respect to D.

The mathematical model of the two-stage stochastic programming is given and next we will summarize the methods mainly used to solve it. Since the two and multi-stage stochastic programming problems have very large dimension and special structures, they can be solved by means of decomposition. In two-stage models (3) and (4), suppose that the probability space $\Theta$ is finite, ${\omega }^s,s=1,\dots ,S$ and that $p_s,s=1,\dots ,S$ represent all basic events and their probabilities. Then the first-stage problem can be rewritten in the form

$$\begin{array}{cc}min& c^{T}x+\sum _{s=1}^S{p}_s{Q}^s\left(x\right)\hfill \\ \mathrm{s}.\mathrm{t}.& Ax=b,\hfill \\ & x\ge 0.\hfill \end{array}$$

(5)

Here, $Q^s\left(x\right)$ is the optimal value of the second-stage problem

$$\begin{array}{cc}min& {\left(q^s\right)}^{T}y\hfill \\ \mathrm{s}.\mathrm{t}.& W^{s}y=h^s-T^{s}x,\hfill \\ & y\ge 0,\hfill \end{array}$$

(6)

where, $T^s=T\left({\omega }^s\right)$, $W^s=W\left({\omega }^s\right)$, $h^s=h\left({\omega }^s\right)$, $y^s=y\left({\omega }^s\right)$, $q^s=q\left({\omega }^s\right)$.

The main idea of the basic dual decomposition methods is to establish certain approximations to the first-stage problem by means of solving subproblems in structure (6). As an original decomposition-type method, the cutting-plane method establishes a linear model of ${\sum }_{s=1}^S{p}_s{Q}^s\left(x\right)$, and its main scheme can be found in [6,7]. Simplicity and cheapness in calculation are advantages of the cutting-plane model. With the increasing of cuts, however, the demand for storage growths which is a typical difficulty of such method. On the other hand, although there is a good initial iteration, such method may generate a longer stepsize. In order to tackle these difficulties, Ruszczynski [8] proposed the regularized decomposition method which is considered as an improvement of the basic one. Another way to avoid generating longer steps is the trust region method which was extended to two-stage stochastic programming by Linderoth and Wright [9]. In addition, bundle methods, which can be viewed as stabilized variants of the original cutting-plane methods, were developed in [10,11,12]. Algorithmic modifications based on bundle methods were introduced in [13,14]. The bundle level (BL) method was first proposed by Lemaréchal et al. [15] as another kind of bundle method. On this basis, in [16,17,18] the “restricted-memory” version of BL method is developed which performs well in numerical experiments. In recent years, there has been tremendous development in “asynchronous” and “partial” versions of BL methods, see references [19,20,21,22,23]. Considering that the research on BL methods is only focused on general non-smooth convex optimization (CP) problems, Lan [24] proposed an accelerated BL-type method, namely the accelerated bundle level (ABL) method and its restricted memory version, the accelerated prox-level (APL) method. Benefiting from the multi-step strategy introduced by Nesterov [25] and later applied in [26,27,28,29,30], both ABL and APL methods are uniformly optimal for solving non-smooth, weakly smooth and smooth CP problems. In addition, by incorporating Nesterov’s smoothing technique [16,17] into APL method, Lan [24] presented the uniform smoothing level (USL) method for solving structure non-smooth CP problems with optimal iteration complexity. In particular, the USL method does not require any input of problem parameters. Moreover, Lan [24] illustrated that the APL and USL methods, when applied to solving semidefine programming and two-stage stochastic programming, have obvious advantages in computation time and accuracy over the related gradient-type algorithms and some existing methods.

Our main work in this paper includes several aspects. First of all, on the basis of Lan’s work, we make further improvement and present the modified accelerated prox-level (MAPL) method. By selecting the proper proximal function, the MAPL method only need to solve one subproblem to update the prox-center and the lower bound synchronously, which improves the computational efficiency. In addition, MAPL method can achieve uniformly optimal complexity for solving smooth, weakly smooth, and non-smooth CP problems uniformly. Furthermore, we extend the MAPL method to solve the structure non-smooth CP problems and present the modified uniform smoothing level (MUSL) method. Finally, we apply the proposed methods to solve the two-stage stochastic programming problems with recourse. The numerical results show that the MAPL and MUSL methods have certain advantages in iterations and computation time.

The present paper is built up as follows. In Section 2, some related works about BL-type methods are reviewed. The MAPL method and its complexity analysis are presented in Section 3. The MUSL method and its complexity analysis are presented in Section 4. The application of MAPL and MUSL methods to two-stage stochastic programming as well as numerical experiments are shown in Section 5. Conclusions are presented in the final section.

2. Related Work

This section reviews some related work on BL methods. Specifically, the main ideas of BL methods and related notation are reviewed in Section 2.1. The APL method and its gap reduction procedure ${\mathcal{G}}_{MAPL}$ introduced in Section 2.2 are the basis of the main work in this paper.

2.1. The Bundle Level (BL) Method

Consider the general CP problem

$$f^{*}:=\underset{x\in X}{min}f\left(x\right),$$

(7)

where the constraint set $X\subseteq {\mathbb{R}}^n$ is convex and compact, and the objective function $f:X\to \mathbb{R}$ is closed and convex over X. The function f is known only through a first-order oracle returning the function value $f\left(x\right)$ and a subgradient $f^{\prime }\left(x\right)\in \partial f\left(x\right)$ for a given point $x\in X$, where $\partial f\left(x\right):=\{\xi \in {\mathbb{R}}^n|f\left(y\right)\ge f\left(x\right)+⟨\xi ,y-x⟩,\forall y\in {\mathbb{R}}^n\}$ is the subdifferential of f at x.

Given a sequence of iteration points, the first-order information $\left\{f\left(x_k\right)\right\}$ and $\left\{f^{\prime }\left(x_k\right)\right\}$ are provided by oracle. The cutting-plane approximation of f is generated by

$${\check{f}}_k\left(x\right):=max\{h(x_i,x):1\le i\le k\}\le f\left(x\right),$$

where $h(x_i,x):=f\left(x_i\right)+⟨f^{\prime }\left(x_i\right),x-x_i⟩.$ In addition, $⟨x,y⟩:={\sum }_i{x}_i{y}_i,x,y\in {\mathbb{R}}^n$ refers to the inner product. The level set $X_k$ of ${\check{f}}_k$ with level parameter ${\ell }_k$ is defined by $X_k:=\{x\in X:{\check{f}}_k\left(x\right)\le {\ell }_k\}$. The BL method [15] generates the next iteration point $x_{k+1}$ by

$$x_{k+1}={argmin}_{x\in X_k}{∥x-x_k∥}^2.$$

(8)

The main steps of the BL method are listed below.

Step 1. Set ${\overline{f}}_k:=min\left\{f\left(x_i\right),1\le i\le k\right\}$ and compute a lower bound ${\underline{f}}_k:={min}_{x\in X}{\check{f}}_k\left(x\right)$;

Step 2. Set the level parameter ${\ell }_k=\beta {\underline{f}}_k+(1-\beta ){\overline{f}}_k$ for some $\beta \in (0,1)$;

Step 3. Set $X_k:=\{x\in X:{\check{f}}_k\left(x\right)\le {\ell }_k\}$ and generate the new iterate by solving (8).

2.2. The Accelerated Prox-Level (APL) Method and Its Gap Reduction Procedure ${\mathcal{G}}_{MAPL}$

In this subsection we consider the CP problem (7), where f satisfies the following inequality

$$f\left(y\right)-f\left(x\right)-⟨f^{\prime }\left(x\right),y-x⟩\le \frac{M}{1+\rho }{\parallel y-x\parallel }^{1+\rho },\forall x,y\in X,$$

(9)

for some $M>0,\rho \in [0,1]$ and $f^{\prime }\left(x\right)\in \partial f\left(x\right).$ It is easy to show that non-smooth $(\rho =0)$, smooth $(\rho =1)$ and weakly smooth $(\rho \in (0,1\left)\right)$ CP problems are contained in this family of problems. Lan [24] generalized the BL method to accelerated bundle level (ABL) and accelerated prox-level (APL) methods, such that they achieve uniformly optimal complexity bounds for functions satisfying (9). Here, we mainly introduce the APL method and its principle.

Firstly, Lan [24] introduced three related iteration sequences, $\left\{x_k^l\right\},\left\{x_k^u\right\}$ and $\left\{x_k\right\}$ to establish the cutting-plane approximation ${\check{f}}_k\left(x\right)$, generate the upper bounds ${\overline{f}}_k$, and update prox-center $x_k$, respectively. Specifically, $\left\{x_k^l\right\}$ and $\left\{x_k^u\right\}$ are updated by $x_k^l=(1-{\alpha }_k)x_k^u+{\alpha }_k{x}_{k-1}$ and $x_k^u=(1-{\alpha }_k)x_{k-1}^u+{\alpha }_k{x}_k$. Secondly, Lan introduced an internal procedure ${\mathcal{G}}_{APL}$ to reduce the gap between the upper and lower bounds of $f^{*}$. The algorithmic framework of the APL method is as follows (Algorithm 1).

Algorithm 1: The APL method.

Input: Choose $p_0\in X$, stopping tolerance $\epsilon >0$ and parameters $\lambda ,\theta \in (0,1)$.   Step 0: (Initialization) Set $p_1\in {Argmin}_{x\in X}h\left(p_0,x\right)$, ${\mathrm{lb}}_1=h\left(p_0,p_1\right)$ and ${\mathrm{ub}}_1=f\left(p_1\right)$.   Let $s=1$.   Step 1: (Stopping test) If ${\mathrm{ub}}_s-{\mathrm{lb}}_s\le \epsilon$, terminate.   Step 2: (Call ${\mathcal{G}}_{APL}$ procedure) Set $\left(p_{s+1},{\mathrm{lb}}_{s+1}\right)={\mathcal{G}}_{APL}\left(p_s,{\mathrm{lb}}_s,\lambda ,\theta \right)$ and ${\mathrm{ub}}_{s+1}=f\left(p_{s+1}\right)$.   Step 3: (Loop) Set $s=s+1$, and return to Step 1.

Next, we focus on describing the gap reduction procedure ${\mathcal{G}}_{APL}$. Denote the level set $X_f(\ell ):=\{x\in X:f\left(x\right)\le \ell \}$. For a given iterate z, denote

$$\overline{f}:=min\{h(z,x):x\in X_f(\ell )\}.$$

(10)

Then, it can be verified that

$$min\{\ell ,\overline{f}\}\le f\left(x\right),\forall x\in X.$$

This implies that $min\{\ell ,\overline{f}\}$ a lower bound of $f^{*}$. However, the problem (10) is difficult to solve in general. To overcome the difficulty and obtain the lower bound in a convenient way, Lan used a compact and convex set $X^{\prime }$ to replace $X_f(\ell )$ in problem (10). Thus, one can solve the relaxation of (10)

$$\underline{f}:=min\{h(z,x):x\in X^{\prime }\}.$$

Here, the set $X^{\prime }$ is called the localizer of the level set $X_f(\ell )$ satisfying $X_f(\ell )\subseteq X^{\prime }\subseteq X$. Then we obtain a lower bound on $f^{*}$ as follow

$$min\{\ell ,\underline{f}\}\le f\left(x\right),\forall x\in X.$$

(11)

Indeed, as shown in [24], if $X^{\prime }\ne \varnothing$, then $\underline{f}\le min\{h(z,x):x\in X_f(\ell )\}\le f\left(x\right),$ for all $x\in X$. If $X^{\prime }=\varnothing$, we have that $\underline{f}=+\infty$ and $X_f(\ell )=\varnothing$, and therefore $f\left(x\right)\ge \ell$ for all $x\in X$. Thus, (11) holds.

Moreover, in order to make better use of the structure of the feasible set X, similar to the NERML algorithm in [16,17], Lan introduced the prox-function to replace the Euclidean distance function ${\parallel ·\parallel }^2$. Here, a function $\omega :X\to \mathbb{R}$ is served as a prox-function of a convex compact set $X\subseteq {\mathbb{R}}^n$ with coefficient ${\sigma }_{\omega }$, if it is a function with differentiability, strong convexity as well as coefficient ${\sigma }_{\omega }$ i.e.

$$⟨\nabla \omega \left(x\right)-\nabla \omega \left(z\right),x-z⟩\ge {\sigma }_{\omega }{\parallel x-z\parallel }^2,\forall x,z\in X.$$

(12)

Furthermore, one can redefine the diameter of X with $d_{\omega }$ by

$$D_{\omega ,X}^2:=max\{\omega \left(x\right)-\omega \left(z\right)-⟨\nabla \omega \left(z\right),x-z⟩,x,z\in X\}.$$

(13)

This lead to the following relation

$${\parallel x-z\parallel }^2\le (2/{\sigma }_{\omega })D_{\omega ,X}^2=:{\Omega }_{\omega ,X},\forall x,z\in X.$$

(14)

Furthermore, let

$$d_{\omega }\left(x\right):=\omega \left(x\right)-[\omega \left(x_0\right)+⟨{\omega }^{{}^{\prime }}\left(x_0\right),x-x_0⟩],\forall x\in X,$$

(15)

be the prox-function on X and $x_0:={argmin}_{x\in X}{d}_{\omega }\left(x\right)$ be the prox-center of $d_{\omega }\left(x\right)$. It follows that

$$d_{\omega }\left(x\right)\ge (1/2){\sigma }_{\omega }{\parallel x-x_0\parallel }^2,$$

(16)

and $d_{\omega }\left(x\right)$ is a strong convex function with coefficient ${\sigma }_{\omega }$.

The internal gap reduction procedure ${\mathcal{G}}_{APL}$ of APL method is as follows.

The APL gap reduction procedure:$(p^{+},{\mathrm{lb}}^{+})={\mathcal{G}}_{APL}(p,\mathrm{lb},\lambda ,\theta )$

• Step 0: (Initialization) Set $x_0^u=p,{\overline{f}}_0=f\left(x_0^u\right),{\underline{f}}_0=\mathrm{lb}$ and $\ell =\lambda {\underline{f}}_0+(1-\lambda ){\overline{f}}_0$. In addition, choose $x_0\in X$ and the initial localizer $X_0^{\prime }=X$. The prox-function $d_{\omega }\left(x\right)$ is defined in (15). Also let $k=1$.
• Step 1: (Update lower bound) Let $x_k^l={\alpha }_k{x}_{k-1}+(1-{\alpha }_k)x_{k-1}^u$, and $h(x_k^l,x)=f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),x-x_k^l⟩$.

  $${\underline{h}}_k:=\underset{x\in X_{k-1}^{\prime }}{min}\left\{h(x_k^l,x)\right\},\mathrm{and}{\underline{f}}_k:=max\left\{{\underline{f}}_{k-1},min\left\{\ell ,{\underline{h}}_k\right\}\right\}.$$
  If ${\underline{f}}_k\ge \ell -\theta (\ell -{\underline{f}}_0)$, then stop the procedure and output $p^{+}=x_{k-1}^u$ and ${\mathrm{lb}}^{+}={\underline{f}}_k$.
• Step 2: (Update prox-center) Set

  $$x_k:={argmin}_{x\in X_{k-1}^{\prime }}\left\{d_{\omega }\left(x\right):h(x_k^l,x)\le \ell \right\}.$$
• Step 3: (Update upper bound) Set ${\overline{f}}_k=min\{{\overline{f}}_{k-1},f({\alpha }_k{x}_k+(1-{\alpha }_k)x_{k-1}^u)\}$ and $x_k^u$ such that $f\left(x_k^u\right)={\overline{f}}_k$. If ${\overline{f}}_k\le \ell +\theta ({\overline{f}}_0-\ell )$, then stop the procedure and output $p^{+}=x_{k-1}^u$ and ${\mathrm{lb}}^{+}={\underline{f}}_k$.
• Step 4: (Update localizer) Choose $X_k^{\prime }$ such that ${\underline{X}}_k\subseteq X_k^{\prime }\subseteq {\overline{X}}_k$, where

  $${\underline{X}}_k:=\left\{x\in X_{k-1}^{\prime }:h(x_k^l,x)\le \ell \right\}\mathrm{and}{\overline{X}}_k=\left\{x\in X:⟨\nabla d_{\omega }\left(x_k\right),x-x_k⟩\ge 0\right\}.$$
• Step 5: (Loop) Set $k=k+1$ and return to Step 1.

3. The Modified Accelerated Prox-Level (MAPL) Method

In this section, we proposed the modified accelerated prox-level (MAPL) method which requires only one subproblem to be solved per iteration while achieving the uniformly optimal iteration complexity for solving the black-box CP problems. We first present the modified gap reduction procedure ${\mathcal{G}}_{MAPL}$, which generates a new search point $p^{+}$ and a new lower bound ${lb}^{+}$ of $f^{*}$ such that $f\left(p^{+}\right)-{lb}^{+}\le q[f\left(p\right)-lb]$ from the given input p and lb and some $q\in (0,1).$ Here, the value of q depends on the parameters $\lambda ,\theta \in (0,1)$.

The MAPL gap reduction procedure:$(p^{+},{\mathrm{lb}}^{+})={\mathcal{G}}_{MAPL}(p,\mathrm{lb},\lambda ,\theta )$

• Step 0: (Initialization) Set $x_0^u=p,{\overline{f}}_0=f\left(x_0^u\right),{\underline{f}}_0=\mathrm{lb}$ and $\ell =\lambda {\underline{f}}_0+(1-\lambda ){\overline{f}}_0$. In addition, choose $x_0\in X$ and the initial localizer $X_0^{\prime }=X$. The prox-function $d_{\omega }\left(x\right)$ is defined in (15). Let $k=1$.
• Step 1: (Update level set) Set

  $$x_k^l={\alpha }_k{x}_{k-1}+(1-{\alpha }_k)x_{k-1}^u,$$

  (17)

  $$h(x_k^l,x)=f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),x-x_k^l⟩,$$

  (18)

  $${\underline{X}}_k=\left\{x\in X_{k-1}^{\prime }:h(x_k^l,x)\le \ell \right\}.$$

  (19)
• Step 2: (Update prox-center and lower bound) Set

  $$x_k={argmin}_{x\in {\underline{X}}_k}{d}_{\omega }\left(x\right).$$

  (20)
  If ${\underline{X}}_k=\varnothing$, then stop the procedure and output $p^{+}=x_{k-1}^u,{lb}^{+}=\ell$.
• Step 3: (Update upper bound) Set

  $${\tilde{x}}_k^u={\alpha }_k{x}_k+(1-\alpha )x_{k-1}^u,$$

  (21)

  $$x_k^u=\left\{\begin{array}{cc}{\tilde{x}}_k^u,\hfill & iff\left({\tilde{x}}_k^u\right)<{\overline{f}}_{k-1},\hfill \\ x_{k-1}^u,\hfill & otherwise,\hfill \end{array}\right.$$

  (22)

  and ${\overline{f}}_k=f\left(x_k^u\right)$. If ${\overline{f}}_k\le \ell +\theta ({\overline{f}}_0-\ell )$, then stop the procedure (there is a significant improvement on the upper bound) and output $p^{+}=x_k^u,{lb}^{+}=lb$.
• Step 4: (Update localizer) Choose an arbitrary $X_k^{\prime }$ such that ${\underline{X}}_k\subseteq X_k^{\prime }\subseteq {\overline{X}}_k$, where

  $${\overline{X}}_k=\left\{x\in X:⟨\nabla d_{\omega }\left(x_k\right),x-x_k⟩\ge 0\right\}.$$

  (23)
• Step 5: (Loop) Set $k=k+1$ and return to Step 1.

The following are a few remarks on the MAPL gap reduction procedure. Firstly, the upper bound ${\overline{f}}_0=f\left(x_0^u\right)$ and lower bound ${\underline{f}}_0=\mathrm{lb}$ of $f^{*}$ in Step 0 are obtained from the outer iteration of the MAPL method (described below), and ${\overline{f}}_0$ and ${\underline{f}}_0$ are fixed throughout the entire progress of ${\mathcal{G}}_{MAPL}$. Furthermore, the level parameter ℓ is also fixed as a convex combination of ${\overline{f}}_0$ and ${\underline{f}}_0$. But in the original BL methods, the level parameter changes in each iteration. Secondly, Step 2 and Step 3 have two exits. When the procedure stops at Step 2, ℓ is the lower bound on $f^{*}$; when the procedure stops at Step 3, there is a significant progress on ${\overline{f}}_k$, which depends on the parameter $\theta$. Compared with the gap reduction procedure ${\mathcal{G}}_{APL}$ of APL method, the ${\mathcal{G}}_{MAPL}$ updates the lower bound in a easier way. Indeed, in Step 1 of ${\mathcal{G}}_{APL}$ see [24], in order to determine whether the lower bound lb needs to be updated, a linear programming subproblem is first solved. However, in the ${\mathcal{G}}_{MAPL}$ procedure, the prox-center $x_k$ and the lower bound lb are merged into one subproblem to be updated by selecting appropriate prox-function $d_{\omega }$. While solving the subproblem (20), it can be automatically checked that whether ${\underline{X}}_k=\varnothing$. If ${\underline{X}}_k=\varnothing$, the lower bound lb is directly updated to ℓ, which avoids solving the linear programming as ${\mathcal{G}}_{APL}$ and reduce the number of calculation. Thirdly, in Step 4, $X_k^{{}^{\prime }}$ can be selected arbitrarily under condition ${\underline{X}}_k\subseteq X_k^{{}^{\prime }}\subseteq {\overline{X}}_k$. For convenience, one can directly choose $X_k^{{}^{\prime }}={\underline{X}}_k$ or $X_k^{{}^{\prime }}={\overline{X}}_k$. However, the number of constraints in ${\underline{X}}_k$ increases with k, and ${\overline{X}}_k$ has only one more constraint than the feasible set X. In practice, we can choose $X_k^{{}^{\prime }}$ between these two sets, which can control the number of constraints in (19) and reduce the computation cost. Moreover, a proper selection of the stepsize sequences $\left\{{\alpha }_k\right\}$ is critical for ${\mathcal{G}}_{MAPL}$ to terminate after finite iterations and achieve optimal iteration complexity. Lan proposed a generalized selection rule for the sequence $\left\{{\alpha }_k\right\}$ in [24]. Chen et al. [31] proposed a more concise selection rule, i.e., the sequence $\left\{{\alpha }_k\right\}$ satisfies the following conditions:

$${\alpha }_1=1,0<{\alpha }_k\le 1,{\alpha }_k\le \frac{c}{k},\frac{1-{\alpha }_{k+1}}{{\alpha }_{k+1}^{1+\gamma }}\le \frac{1}{{\alpha }_k^{1+\gamma }},\forall k\ge 1,$$

(24)

for any $\gamma \in [0,1]$ and some $c>0$. Two examples for $\left\{{\alpha }_k\right\}$ are as follows [31]:

(1) ${\alpha }_k=2/(k+1),k=1,2,\dots ,$ with $c=2$;

(2) ${\alpha }_1=1,{\alpha }_{k+1}^2=(1-{\alpha }_{k+1}){\alpha }_k^2$, with $c=2$.

The following lemma shows the important properties of procedure ${\mathcal{G}}_{MAPL}$. These properties are similar to those of [24,31], whose proof can be found in Appendix A.

Lemma 1.

The following properties hold for the procedure ${\mathcal{G}}_{MAPL}$.

a. ${\left\{X_k^{\prime }\right\}}_{k\ge 0}$ is a collection of localizers for the level set $X_f(\ell )$;

b. ${\overline{f}}_0\ge {\overline{f}}_1\ge \cdots \ge {\overline{f}}_k\ge f^{*}$ holds for any $k\ge 1$;

c. $X_f(\ell )\ne \varnothing$, and hence $X_f(\ell )\subseteq {\underline{X}}_k\subseteq X_k^{\prime }\subseteq {\overline{X}}_k$ for any $k\ge 1$;

d. When ${\underline{X}}_k\ne \varnothing$ holds, the problem (20) has a unique solution. In addition, if the procedure ${\mathcal{G}}_{MAPL}$ stops at Step 2, we have $\ell \le f^{*}$.

e. When the procedure ${\mathcal{G}}_{MAPL}$ stops, the relation $f\left(p^{+}\right)-{\mathrm{lb}}^{+}\le q(f\left(p\right)-\mathrm{lb})$ holds, where

$$q\equiv q(\lambda ,\theta )=max\left\{\lambda ,1-\lambda (1-\theta )\right\}.$$

(25)

Referring to the theoretical analysis mode of [24], the following proposition will show that the gap between the upper bound $f\left(x_k^u\right)$ and the level parameter ℓ will decrease with k, and it is proven that when the algorithm stops, the total number of iterations does not exceed an upper bound.

Proposition 1.

If $\left\{{\alpha }_k\right\}$ satisfy the condition (24), take $\gamma =\rho$, let ℓ be the level parameter and $d_{\omega }\left(x\right)$ denotes the prox-function, then the number of iterations of the procedure ${\mathcal{G}}_{MAPL}$ does not exceed

$$N\left({\Delta }_0\right):=\left[{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{\lambda \theta (1+\rho ){\Delta }_0}\right)}^{\frac{2}{1+3\rho }}\right],$$

(26)

where ${\Delta }_0={\overline{f}}_0-{\underline{f}}_0$ and ${\Omega }_{\omega ,X}$ is defined in (14).

Proof. 

Assume that the gap reduction procedure ${\mathcal{G}}_{MAPL}$ does not stop at the K-th iteration. From (16) we have for $x_1\in X$

$$\frac{{\sigma }_{\omega }}{2}{∥x_1-x_0∥}^2\le d_{\omega }\left(x_1\right).$$

By the definition of ${\overline{X}}_k$ in (23) we have $⟨\nabla d_{\omega }\left(x_k\right),z-x_k⟩\ge 0,$ for any $z\in {\overline{X}}_k$. This together with $X_k^{\prime }\subseteq {\overline{X}}_k$ shows that

$$⟨\nabla d_{\omega }\left(x_k\right),z-x_k⟩\ge 0,\forall z\in X_k^{\prime }.$$

It follows from (19) and (20) that $x_{k+1}\in {\underline{X}}_{k+1}\subseteq X_k^{\prime }$, for all $1\le k\le K$. Moreover, due to the strong convexity of $d_{\omega }\left(x\right)$ and the optimality condition of subproblem (20), we have

$$\begin{array}{cc}\hfill \frac{{\sigma }_{\omega }}{2}{∥x_{k+1}-x_k∥}^2& \le d_{\omega }\left(x_{k+1}\right)-d_{\omega }\left(x_k\right)-⟨\nabla d_w\left(x_k\right),x_{k+1}-x_k⟩\hfill \\ & \le d_{\omega }\left(x_{k+1}\right)-d_{\omega }\left(x_k\right),\forall k\ge 1.\hfill \end{array}$$

It turns out that

$$\frac{{\sigma }_{\omega }}{2}{∥x_k-x_{k-1}∥}^2\le d_{\omega }\left(x_{k+1}\right)-d_{\omega }\left(x_k\right),\forall k\ge 1.$$

Summing up the above inequalities over k, we obtain that

$$\frac{{\sigma }_{\omega }}{2}\sum _{k=1}^K{∥x_{k+1}-x_k∥}^2\le d_{\omega }\left(x_K\right).$$

(27)

According to (22), (9) and (18), we have

$$\begin{array}{cc}\hfill f\left(x_k^u\right)\le f\left({\tilde{x}}_k^u\right)& \le f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),{\tilde{x}}_k^u-x_k^l⟩+\frac{M}{1+\rho }{\parallel {\tilde{x}}_k^u-x_k^l\parallel }^{1+\rho }\hfill \\ & =h(x_k^l,{\tilde{x}}_k^u)+\frac{M}{1+\rho }{\parallel {\tilde{x}}_k^u-x_k^l\parallel }^{1+\rho }.\hfill \end{array}$$

From (18), (21), the convexity of f, (19) and (22), we have

$$\begin{array}{cc}\hfill h(x_k^l,{\tilde{x}}_k^u)& =f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),(1-{\alpha }_k)x_{k-1}^u+{\alpha }_k{x}_k-x_k^l⟩\hfill \\ & =(1-{\alpha }_k)[f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),x_{k-1}^u-x_k^l⟩]+{\alpha }_k[f\left(x_k^l\right)+⟨f^{\prime }\left(x_k^l\right),x_k-x_k^l⟩]\hfill \\ & =(1-{\alpha }_k)h(x_k^l,x_{k-1}^u)+{\alpha }_{k}h(x_k^l,x_k)\hfill \\ & \le (1-{\alpha }_k)f\left(x_{k-1}^u\right)+{\alpha }_{k}h(x_k^l,x_k)\hfill \\ & \le (1-{\alpha }_k)f\left(x_{k-1}^u\right)+{\alpha }_k\ell .\hfill \end{array}$$

By the definition of ${\tilde{x}}_k^u$ (21) and $x_k^l$ (17), it is easy to show that

$$\begin{array}{cc}\hfill {\tilde{x}}_k^u-x_k^l& ={\alpha }_k{x}_k+(1-{\alpha }_k)x_{k-1}^u-{\alpha }_k{x}_{k-1}-(1-{\alpha }_k)x_{k-1}^u\hfill \\ & ={\alpha }_k{x}_k-{\alpha }_k{x}_{k-1}={\alpha }_k(x_k-x_{k-1}).\hfill \end{array}$$

It then follows that

$$f\left(x_k^u\right)\le (1-{\alpha }_k)f\left(x_{k-1}^u\right)+{\alpha }_k\ell +\frac{M{\alpha }_k^{1+\rho }}{1+\rho }{\parallel x_k-x_{k-1}\parallel }^{1+\rho }.$$

Subtracting both sides of the above inequality by ℓ and dividing both sides by ${\alpha }_k^{1+\rho }$, from (24), we obtain that

$$\begin{array}{cc}\hfill \frac{1}{{\alpha }_k^{1+\rho }}[f\left(x_k^u\right)-\ell ]& \le \frac{1-{\alpha }_k}{{\alpha }_k^{1+\rho }}[f\left(x_{k-1}^u\right)-\ell ]+\frac{M}{1+\rho }{\parallel x_k-x_{k-1}\parallel }^{1+\rho }\hfill \\ & \le \frac{1}{{\alpha }_{k-1}^{1+\rho }}\left[f\left(x_{k-1}^u\right)-\ell \right]+\frac{M}{1+\rho }{∥x_k-x_{k-1}∥}^{1+\rho },\forall k\ge 2.\hfill \end{array}$$

As a result,

$$\frac{1}{{\alpha }_1^{1+\rho }}\left[f\left(x_1^u\right)-\ell \right]\le \frac{M}{1+\rho }{∥x_1-x_0∥}^{1+\rho },$$

$$\frac{1}{{\alpha }_2^{1+\rho }}\left[f\left(x_2^u\right)-\ell \right]\le \frac{1}{{\alpha }_1^{1+\rho }}\left[f\left(x_1^u\right)-\ell \right]+\frac{M}{1+\rho }{∥x_2-x_1∥}^{1+\rho },$$

$$\dots$$

$$\frac{1}{{\alpha }_K^{1+\rho }}\left[f\left(x_K^u\right)-\ell \right]\le \frac{1}{{\alpha }_{K-1}^{1+\rho }}\left[f\left(x_{K-1}^u\right)-\ell \right]+\frac{M}{1+\rho }{∥x_K-x_{K-1}∥}^{1+\rho }.$$

Summing up the above inequality over m and from the fact that ${\alpha }_k>0,\forall k\ge 1,$ we have

$$f\left(x_K^u\right)-\ell \le \frac{{\alpha }_K^{1+\rho }M}{1+\rho }\sum _{k=1}^K{∥x_k-x_{k-1}∥}^{1+\rho }.$$

(28)

Applying the H$\ddot{\mathrm{o}}$lder inequality to the above inequality and using (27), we have

$$\begin{array}{cc}\hfill \sum _{k=1}^K{∥x_k-x_{k-1}∥}^{1+\rho }& \le K^{\frac{1-\rho }{2}}{\left(\sum _{k=1}^K{∥x_k-x_{k-1}∥}^2\right)}^{\frac{1+\rho }{2}}\hfill \\ & \le K^{\frac{1-\rho }{2}}{\left(\frac{2d_{\omega }\left(x_K\right)}{{\sigma }_{\omega }}\right)}^{\frac{1+\rho }{2}}\le K^{\frac{1-\rho }{2}}{\left(\frac{2D_{\omega ,X}}{{\sigma }_{\omega }}\right)}^{\frac{1+\rho }{2}}.\hfill \end{array}$$

Applying (24) and (14) to (28), as well as the above inequality, we have

$$f\left(x_K^u\right)-\ell \le \frac{c^{1+\rho }M}{K^{1+\rho }(1+\rho )}{K}^{\frac{1-\rho }{2}}{\left(\frac{2D_{\omega ,X}}{{\sigma }_{\omega }}\right)}^{\frac{1+\rho }{2}}\le \frac{c^{1+\rho }M}{1+\rho }{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}{K}^{-\frac{1+\rho }{2}}.$$

(29)

From the fact that ${\mathcal{G}}_{MAPL}$ does not stop at Step 3, we have

$${\overline{f}}_k-\ell =f\left(x_K^u\right)-\ell >\theta ({\overline{f}}_0-\ell )=\theta \lambda ({\overline{f}}_0-{\underline{f}}_0)=\theta \lambda (\mathrm{ub}-\mathrm{lb})=\lambda \theta {\Delta }_0.$$

This together with (29) shows that

$$\lambda \theta {\Delta }_0\le \frac{M{c}^{1+\rho }}{1+\rho }{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}{K}^{-\frac{1+3\rho }{2}}.$$

We finally conclude that

$$K<{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{\lambda \theta (1+\rho ){\Delta }_0}\right)}^{\frac{2}{1+3\rho }},$$

that is

$$N\left({\Delta }_0\right)=\left[{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{\lambda \theta (1+\rho ){\Delta }_0}\right)}^{\frac{2}{1+3\rho }}\right].$$

 □

After giving the relevant properties and complexity analysis of the procedure ${\mathcal{G}}_{MAPL}$, we are now in a position to introduce the so-called modified accelerated prox-level (MAPL) method, which repeatedly calls the procedure ${\mathcal{G}}_{MAPL}$ during iteration progress until it finds an approximate solution with the given accuracy. The algorithmic framework of MAPL method is as follows (Algorithm 2).

Algorithm 2: The MAPL method.

Parameters: Choose stopping tolerance $\epsilon >0$ and parameters $\lambda ,\theta \in (0,1)$.   Step 0: (Initialization) Choose initial point $p_0$, set $p_1\in {argmin}_{x\in X}h\left(p_0,x\right)$, ${\mathrm{lb}}_1=h\left(p_0,p_1\right)$ and ${\mathrm{ub}}_1=min\{f\left(p_0\right),f\left(p_1\right))\}$. Let $s=1$;   Step 1: (Stopping test) If ${\mathrm{ub}}_s-{\mathrm{lb}}_s\le \epsilon$, terminate;   Step 2: (Call ${\mathcal{G}}_{MAPL}$ procedure) Set $\left(p_{s+1},{\mathrm{lb}}_{s+1}\right)={\mathcal{G}}_{MAPL}\left(p_s,{\mathrm{lb}}_s,\lambda ,\theta \right)$ and ${\mathrm{ub}}_{s+1}=f\left(p_{s+1}\right)$;   Step 3: (Loop) Set $s=s+1$, and return to Step 1.

Since the procedure ${\mathcal{G}}_{MAPL}$ is called during the progress of the MAPL method, we consider that an iteration of ${\mathcal{G}}_{MAPL}$ is also an iteration of the MAPL method. Take this fact into consideration, the following theorem establishes the convergence and iteration complexity of the MAPL method. The principle of the proof comes from reference [24].

Theorem 1.

For given $\epsilon >0$, if ${\alpha }_k\in (0,1],k=1,2,\cdots$, in ${\mathcal{G}}_{MAPL}$ are chosen to satisfy condition (24) with some $c>0$, then

(1) The number of procedure ${\mathcal{G}}_{MAPL}$ called by the MAPL method can be bounded by

$$S\left(\epsilon \right):=\left[max\left\{0,{log}_{\frac{1}{q}}\frac{M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\epsilon }\right\}\right];$$

(2) The total number of iterations performed by the MAPL method does not exceed

$$S\left(\epsilon \right)+\frac{1}{1-q^{\frac{2}{1+3\rho }}}{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\lambda \theta \epsilon }\right)}^{\frac{2}{1+3\rho }}.$$

Proof. 

(1) Without loss of generality, we suppose that ${\Delta }_1>\epsilon ,$ where ${\Delta }_s={\mathrm{ub}}_s-{\mathrm{lb}}_s,s\ge 1.$ According to Step 1 of the MAPL method, (9) and (14), we have

$${\Delta }_1=f\left(p_1\right)-h(p_0,p_1)=f\left(p_1\right)-[f\left(p_0\right)+⟨f^{\prime }\left(p_0\right),p_1-p_0⟩]\le \frac{M\parallel p_1-p_0{\parallel }^{1+\rho }}{1+\rho }\le \frac{M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{1+\rho }.$$

From Lemma 1 and the fact that $u{b}_s=f\left(p_s\right)$, we have

$${\Delta }_{s+1}\le q{\Delta }_s,s\ge 1.$$

(30)

Furthermore, suppose that the MAPL method finds an $\epsilon$-solution after calling $S\left(\epsilon \right)$ times ${\mathcal{G}}_{MAPL}$, i.e., the MAPL method stops at $s:=S\left(\epsilon \right)+1$. Then we have

$${\Delta }_{S\left(\epsilon \right)+1}\le \epsilon \le {\Delta }_{S\left(\epsilon \right)},$$

(31)

and

$$\epsilon <q^{S\left(\epsilon \right)-1}{\Delta }_1\le q^{S\left(\epsilon \right)-1}M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}.$$

It is easy to obtain

$${\left(\frac{1}{q}\right)}^{S\left(\epsilon \right)-1}\le \frac{M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\epsilon },$$

that is

$$S\left(\epsilon \right):=\left[max\left\{0,{log}_{\frac{1}{q}}\frac{M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\epsilon }\right\}\right].$$

(2) Suppose that the procedure ${\mathcal{G}}_{MAPL}$ has been called $S\left(\epsilon \right)$ times in MAPL method. Then by (30) and (31) we have

$${\Delta }_s>\epsilon q^{s-S\left(\epsilon \right)},s=1,2,\dots ,S\left(\epsilon \right).$$

Due to ${\Delta }_{S\left(\epsilon \right)}>\epsilon$, we know that

$$\sum _{s=1}^{S\left(\epsilon \right)}{\Delta }_s^{-\frac{2}{1+3\rho }}<\sum _{s=1}^{S\left(\epsilon \right)}\frac{q^{\frac{2}{1+3\rho }(s-S\left(\epsilon \right))}}{{\epsilon }^{\frac{2}{1+3\rho }}}=\sum _{t=0}^{S\left(\epsilon \right)-1}\frac{q^{\frac{2}{1+3\rho }t}}{{\epsilon }^{\frac{2}{1+3\rho }}}\le \frac{1}{(1-q^{\frac{2}{1+3\rho }}){\epsilon }^{\frac{2}{1+3\rho }}}.$$

This together with Lemma 1 and Proposition 1 shows that the total number of iterations performed by the MAPL method does not exceed

$$\sum _{s=1}^{S\left(\epsilon \right)}{N}_s\le \sum _{s=1}^{S\left(\epsilon \right)}N\left({\Delta }_s\right)\le S\left(\epsilon \right)+\sum _{s=1}^{S\left(\epsilon \right)}{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\lambda \theta {\Delta }_s}\right)}^{\frac{2}{1+3\rho }}\le S\left(\epsilon \right)+\frac{1}{1-q^{\frac{2}{1+3\rho }}}{\left(\frac{c^{1+\rho }M{\Omega }_{\omega ,X}^{\frac{1+\rho }{2}}}{(1+\rho )\lambda \theta \epsilon }\right)}^{\frac{2}{1+3\rho }}.$$

Here, we denote the number of the internal iterations performed by the s-th procedure ${\mathcal{G}}_{MAPL}$ with $N_s$. □

We present a few remarks on the iteration complexity of MAPL method.

Remark 1.

According to the classic complexity theory [32] for CP problem (7), the number of iterations to find an ε-solution i.e., an approximate solution $\overline{x}\in X$ such that $f\left(\overline{x}\right)-f^{*}\le \epsilon$, does not exceed $\mathcal{O}(1/{\epsilon }^2)$, if f is a general non-smooth Lipschitz continuous convex function. For smooth convex optimization, the optimal iteration complexity bound is $\mathcal{O}(1/{\epsilon }^{\frac{1}{2}})$. Furthermore, in case that f is weakly smooth and its gradient is H$\ddot{\mathrm{o}}$lder continuous, the optimal iteration complexity is bounded by $\mathcal{O}\left(1/{\epsilon }^{\frac{2}{1+3\rho }}\right)$ for some $\rho \in (0,1)$. It follows from Theorem 1 that the iteration complexity bound of the MAPL method is

$$\mathcal{O}\left\{{\left(\frac{M{D}_X^{1+\rho }}{\epsilon }\right)}^{\frac{2}{1+3\rho }}\right\}.$$

In other words, the MAPL method can achieve uniformly optimal iteration complexity bounds for solving non-smooth ($\rho =0$), smooth ($\rho =1$) and weakly smooth ($\rho \in (0,1)$) CP problems.

4. The Modified Uniform Smoothing Level (MUSL) Method

In this section we consider the objective function f in (7) with the form of

$$f\left(x\right):=\widehat{f}+F\left(x\right),$$

(32)

where $\widehat{f}:X\to \mathbb{R}$ is Lipschitz continuous and simple. In addition, $F\left(x\right)$ has a special structure that

$$F\left(x\right):=\underset{y\in Y}{max}\{⟨Ax,y⟩-\widehat{g}\left(y\right)\}.$$

(33)

Here, the compact convex set $Y\subset {\mathbb{R}}^n$ is nonempty, $A:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is a linear operator and $\widehat{g}:Y\to \mathbb{R}$ is convex and continuous on Y.

Generally, the function F is convex and non-smooth. In this case, F can be approximated by constructing a series of smooth convex functions [16,17]. Let $v\left(y\right)$ denote the prox-function of compact convex set Y with coefficient ${\sigma }_v$. $c_v:=arg{min}_{y\in Y}v\left(y\right)$ refers to the prox-center of $v\left(y\right)$. Let

$$V\left(y\right):=v\left(y\right)-v\left(c_v\right)-⟨▿v\left(c_v\right),y-c_v⟩.$$

The functions $F\left(x\right)$ and $f\left(x\right)$ are approximated by $F_{\eta }\left(x\right)$ and $f_{\eta }\left(x\right)$ (with some smoothing parameter $\eta >0$), respectively, i.e.,

$$F_{\eta }\left(x\right):=\underset{y\in Y}{max}\{⟨Ax,y⟩-\widehat{g}\left(y\right)-\eta V\left(y\right)\},$$

$$f_{\eta }\left(x\right):=\widehat{f}+F_{\eta }\left(x\right).$$

(34)

As described in [17], from the first-order optimality conditions, the convexity of $\widehat{g}\left(y\right)$ and the strong convexity of $V\left(y\right)$, we know that the gradient of $F_{\eta }\left(x\right)$ is Lipschitz continuous with Lipschitz constant

$$L_{\eta }\equiv L\left(F_{\eta }\right):=\frac{{\parallel A\parallel }^2}{\eta {\sigma }_v}.$$

(35)

Furthermore, we have

$$F_{\eta }\le F\left(x\right)\le F_{\eta }+\eta D_{v,Y},$$

(36)

and

$$f_{\eta }\le f\left(x\right)\le f_{\eta }+\eta D_{v,Y},$$

(37)

for any $x\in X$.

Inspired by the smoothing technique of [16,17], Lan [24] proposed the uniform smoothing level (USL) method for solving structure non-smooth CP problem (7) with f being defined by (32). The advantage of the USL method is that the parameter $\eta$ and estimation of $D_{v,Y}$ can be automatically adjusted and obtained during the gap reduction procedure ${\mathcal{G}}_{USL}$, which makes the USL method free of parameters. However, similar to the APL method, each iteration of the USL method also involves two subproblems. Based on the USL method, combining with the analysis of the MAPL method in the previous section, we propose the modified uniform smoothing level (MUSL) method. Same as the USL method, the MUSL method can achieve the optimal iteration complexity when solving problem (7) with (32), but only one subproblem is required to solve in each iteration.

We next describe the gap reduction procedure ${\mathcal{G}}_{MUSL}$ of MUSL method. As the internal procedure of MUSL method, ${\mathcal{G}}_{MUSL}$ is called to compress the gap between upper and lower bounds of $f^{*}$.

The MUSL gap reduction procedure:$(p^{+},{\mathrm{lb}}^{+},{\tilde{D}}^{+})={\mathcal{G}}_{MUSL}(p,\mathrm{lb},\tilde{D},\lambda ,\theta )$

• Step 0: (Initialization) Set ${\begin{array}{c}\hfill errortypeTitlex\end{array}}_0^u:=p,{\overline{f}}_0:=f\left({\begin{array}{c}\hfill errortypeTitlex\end{array}}_0^u\right),{\underline{f}}_0:=\mathrm{lb}$ and $\ell :=\lambda {\underline{f}}_0+(1-\lambda ){\overline{f}}_0,x_0^u:={\begin{array}{c}\hfill errortypeTitlex\end{array}}_0^u$. Let

  $$\eta :=\theta ({\overline{f}}_0-\ell )/\left(2\tilde{D}\right).$$

  (38)
  Choose $x_0\in X$ and the initial localizer $X_0^{\prime }=X$. The prox-function $d_{\omega }\left(x\right)$ is defined in (15). Let $k=1$.
• Step 1: (Update level set) Let

  $$x_k^l:=\left(1-{\alpha }_k\right){\tilde{x}}_{k-1}^u+{\alpha }_k{x}_{k-1},$$

  (39)

  $$h_{\eta }(x_k^l,x):=\widehat{f}\left(x\right)+F_{\eta }\left(x_k^l\right)+⟨\nabla F_{\eta }\left(x_k^l\right),x-x_k^l⟩,$$

  (40)

  $${\underline{X}}_k=\left\{x\in X_{k-1}^{\prime }:h(x_k^l,x)\le \ell \right\}.$$

  (41)
• Step 2: (Update prox-center and lower bound) Let

  $$x_k={argmin}_{x\in {\underline{X}}_k}{d}_{\omega }\left(x\right).$$

  (42)
  If ${\underline{X}}_k=\varnothing$, then stop the procedure and output $p^{+}=x_{k-1}^u,{lb}^{+}=\ell ,{\tilde{D}}^{+}=\tilde{D}$.
• Step 3: (Update upper bound) Let

  $${\tilde{x}}_k^u={\alpha }_k{x}_k+(1-{\alpha }_k){\tilde{x}}_{k-1}^u,$$

  (43)

  $$x_k^u=\left\{\begin{array}{cc}{\tilde{x}}_k^u,\hfill & iff\left({\tilde{x}}_k^u\right)<{\overline{f}}_{k-1},\hfill \\ x_{k-1}^u,\hfill & otherwise,\hfill \end{array}\right.$$

  and ${\overline{f}}_k:=f\left(x_k^u\right)$. Check the following two possible stopping rules:
  • (3a) if ${\overline{f}}_k\le \ell +\theta ({\overline{f}}_0-\ell )$, stop the procedure with $p^{+}=x_k^u,{\mathrm{lb}}^{+}=\mathrm{lb},{\tilde{D}}^{+}=\tilde{D}$.
  • (3b) Otherwise, if $f_{\eta }\left({\tilde{x}}_k^u\right)\le \ell +\frac{\theta }{2}({\overline{f}}_0-\ell )$, stop the procedure with $p^{+}=x_k^u,1{\mathrm{b}}^{+}=\mathrm{lb},{\tilde{D}}^{+}=2\tilde{D}.$
• Step 4: (Update localizer) Choose an arbitrary $X_k^{\prime }$ such that ${\underline{X}}_k\subseteq X_k^{\prime }\subseteq {\overline{X}}_k$, where

  $${\overline{X}}_k=\left\{x\in X:⟨\nabla d_{\omega }\left(x_k\right),x-x_k⟩\ge 0\right\}.$$

  (44)
• Step 5: (Loop) Set $k=k+1$ and return to Step 1.

The following are a few remarks to the procedure ${\mathcal{G}}_{MUSL}$ including the differences from ${\mathcal{G}}_{MAPL}$ and some properties. Firstly, compared with ${\mathcal{G}}_{MAPL}$, the procedure ${\mathcal{G}}_{MUSL}$ needs to input one more parameter $\tilde{D}$ which is used to calculate the smoothing parameter $\eta$, so that the approximate function $f_{\eta }$ of f can be defined. Secondly, these two procedures approximate the objective function in different ways. ${\mathcal{G}}_{MAPL}$ approximates the objective function f by the linearization of f (18), while ${\mathcal{G}}_{MUSL}$ only linearizes the function $F_{\eta }$ and approximates the function f with $h_{\eta }$ (40). According to (40), convexity of $F_{\eta }$, (36) and (32), we have

$$h_{\eta }(x_k^l,x)\le \widehat{f}\left(x\right)+F_{\eta }\left(x\right)\le \widehat{f}\left(x\right)+F\left(x\right)=f\left(x\right),\forall x\in X,$$

(45)

which means that the function $h_{\eta }(x_k^l,x)$ is the lower estimate of $f\left(x\right)$. Thirdly, the procedure ${\mathcal{G}}_{MUSL}$ has one more exit than ${\mathcal{G}}_{MAPL}$ with three possible exits: Step 2, Step 3a, and Step 3b. If the procedure stops at Step 2, the lower bound lb is updated to ℓ; if it stops at Step 3a, we say that a significant improvement has been made on the upper bound of $f^{*}$ and update the upper bound ${\overline{f}}_k$. In addition, if the procedure stops at Step 3b, it is considered that there is no significant improvement on the upper bound of $f^{*}$, thus the parameter $\tilde{D}$ needs to be adjusted to estimate $D_{v,Y}$ and we update $\tilde{D}$ to $2\tilde{D}$.

Referring to the work of [24,31], the following lemma gives some simple observations of procedure ${\mathcal{G}}_{MUSL}$. Its proof can be found in Appendix A.

Lemma 2.

The following results hold for internal procedure ${\mathcal{G}}_{MUSL}$:

a. If ${\mathcal{G}}_{MUSL}$ terminates at Step 2 or Step 3a, we have $f\left(p^{+}\right)-{\mathrm{lb}}^{+}\le q(f\left(p\right)-\mathrm{lb})$. where $q\equiv q(\lambda ,\theta )=max\{\lambda ,1-\lambda (1-\theta \left)\right\}$;

b. If ${\mathcal{G}}_{MUSL}$ terminates at Step 3b, we have $\tilde{D}<D_{v,Y}$ and ${\tilde{D}}^{+}<2D_{v,Y}$.

Similar to the previous section, it is time to build the convergence results for procedure ${\mathcal{G}}_{MUSL}$. Referring to the theoretical analysis mode of [24], the detailed derivations of the iteration complexity are given as follows.

Proposition 2.

Let $\left\{{\alpha }_k\right\}$ satisfy (24) and take $\gamma =1$. Let $(x_k^l,x_k,x_k^u)\in X\times X\times X,k\ge 1$ be the search points, ℓ be the level parameter, $d_{\omega }(·)$ be the prox-function and η be the smoothing parameter. Then the number of iterations of internal procedure ${\mathcal{G}}_{MUSL}$ can be bounded by,

$$N\left({\Delta }_0,\tilde{D}\right):=\left[\frac{2c\parallel A\parallel }{\lambda \theta {\Delta }_0}\sqrt{\frac{D_{\omega ,X}\tilde{D}}{{\sigma }_{\omega }{\sigma }_v}}\right],$$

(46)

where ${\Delta }_0=\mathrm{ub}-1\mathrm{b}={\overline{f}}_0-{\underline{f}}_0,D_{\omega ,X}$ is defined in (14).

Proof. 

Suppose the gap reduction procedure ${\mathcal{G}}_{MUSL}$ does not stop at the K-th iteration. From (34) and inequality (45), we have $h_{\eta }(z,x)\le f_{\eta }\left(x\right),\forall z,x\in X$. Because of (34), (40) and the Lipschitz continuity of the subgradient of $F_{\eta }$, we have

$$f_{\eta }\left(x\right)-h_{\eta }(z,x)=F_{\eta }\left(x\right)-[F_{\eta }\left(z\right)+⟨\nabla F_{\eta }\left(z\right),x-z⟩]\le \frac{L_{\eta }}{2}{\parallel x-z\parallel }^2,\forall z,x\in X.$$

Hence,

$$f_{\eta }\left(x\right)\le h_{\eta }+\frac{L_{\eta }}{2}{\parallel x-z\parallel }^2,\forall x,z\in X.$$

From (40), the definition of ${\tilde{x}}_k^u$ (43), the convexity of $\widehat{f}$, (45) and (44), we have for all $k\ge 1$,

$$\begin{array}{cc}\hfill h_{\eta }(x_k^l,{\tilde{x}}_k^u)=& \widehat{f}\left({\tilde{x}}_k^u\right)+F_{\eta }\left(x_k^l\right)+⟨F_{\eta }^{\prime }\left(x_k^l\right),{\tilde{x}}_k^u-x_k^l⟩\hfill \\ \hfill =& \widehat{f}({\alpha }_k{x}_k+(1-{\alpha }_k){\tilde{x}}_{k-1}^u)+F_{\eta }\left(x_k^l\right)+⟨F_{\eta }^{\prime }\left(x_k^l\right),{\alpha }_k{x}_k+(1-{\alpha }_k){\tilde{x}}_{k-1}^u-x_k^l⟩\hfill \\ \hfill \le & (1-{\alpha }_k)h_{\eta }(x_k^l,{\tilde{x}}_{k-1}^u)+{\alpha }_k{h}_{\eta }(x_k^l,x_k)\hfill \\ \hfill \le & (1-{\alpha }_k)f_{\eta }\left({\tilde{x}}_{k-1}^u\right)+{\alpha }_k{h}_{\eta }(x_k^l,x_k)\hfill \\ \hfill \le & (1-{\alpha }_k)f_{\eta }\left({\tilde{x}}_{k-1}^u\right)+{\alpha }_k\ell .\hfill \end{array}$$

Moreover, by (39) and (43) we can easily obtain

$${\tilde{x}}_k^u-x_k^l={\alpha }_k{x}_k+(1-{\alpha }_k){\tilde{x}}_{k-1}^u-{\alpha }_k{x}_{k-1}-(1-{\alpha }_k){\tilde{x}}_{k-1}^u={\alpha }_k(x_k-x_{k-1}).$$

It then follows that

$$f_{\eta }\left({\tilde{x}}_k^u\right)\le (1-{\alpha }_k)f_{\eta }\left({\tilde{x}}_{k-1}^u\right)+{\alpha }_k\ell +\frac{L_{\eta }{\alpha }_k^2}{2}{\parallel x_k-x_{k-1}\parallel }^2.$$

Subtracting both sides of the above inequality by ℓ, then dividing both sides by ${\alpha }_k^2$, we have

$$\frac{f_{\eta }\left({\tilde{x}}_k^u\right)-\ell }{{\alpha }_k^2}\le \frac{1-{\alpha }_k}{{\alpha }_k^2}[f_{\eta }\left({\tilde{x}}_{k-1}^u\right)-\ell ]+\frac{L_{\eta }}{2}{\parallel x_k-x_{k-1}\parallel }^2.$$

Summing up the above inequalities over k, together with (24) and H$\ddot{\mathrm{o}}$lder inequality, we obtain

$$f_{\eta }\left({\tilde{x}}_k^u\right)-\ell \le \frac{L_{\eta }{\alpha }_k^2}{2}\sum _{k=1}^K{\parallel x_k-x_{k-1}\parallel }^2\le \frac{c^2{L}_{\eta }{d}_{\omega }\left(x_K\right)}{{\sigma }_{\omega }{K}^2}.$$

Thus, from the relation $d_{\omega }\left(x_K\right)\le D_{\omega ,X},$ (35) and (38), we have

$$f_{\eta }\left({\tilde{x}}_K^u\right)-\ell \le \frac{c^2{L}_{\eta }{D}_{\omega ,X}}{{\sigma }_{\omega }{K}^2}=\frac{c^2{\parallel A\parallel }^2{D}_{\omega ,X}}{\eta {\sigma }_{\omega }{\sigma }_v{K}^2}=\frac{2c^2{\parallel A\parallel }^2{D}_{\omega ,X}\tilde{D}}{\theta ({\overline{f}}_0-\ell ){\sigma }_{\omega }{\sigma }_v{K}^2}.$$

From the fact that the procedure does not stop at Step 3b at the K-th iteration, we obtain $f_{\eta }\left({\tilde{x}}_K^u\right)-\ell >\theta ({\overline{f}}_0-\ell )/2.$ Noticing that ${\Delta }_0=({\overline{f}}_0-{\underline{f}}_0)=({\overline{f}}_0-\ell )/\lambda$, thus

$$f_{\eta }\left({\tilde{x}}_K^u\right)-\ell >\frac{\lambda \theta {\Delta }_0}{2}.$$

In conclusion, we obtain

$$K<\frac{2c\parallel A\parallel }{\lambda \theta {\Delta }_0}\sqrt{\frac{D_{\omega ,X}\tilde{D}}{{\sigma }_{\omega }{\sigma }_v}}.$$

 □

Based on the results of the convergence of the procedure ${\mathcal{G}}_{MUSL}$ above, we next give the algorithm of the MUSL method. Similar to the MAPL method, the MUSL method is also implemented with outernal algorithm framework and internal gap reduction procedure ${\mathcal{G}}_{MUSL}$. The outernal algorithm of the MUSL method is mainly to determine whether the gap between the upper and lower bounds on $f^{*}$ has reached the given tolerance $\epsilon$ in current iteration k. If the given tolerance $\epsilon$ is reached, the algorithm terminates and output an approximate optimal solution of f, otherwise the outernal algorithm call the internal procedure ${\mathcal{G}}_{MUSL}$ continuously to compress the gap between the upper and lower bounds. The algorithmic framework of the MUSL method is as follows (Algorithm 3).

Algorithm 3: The MUSL method.

Step 0: (Input) Choose $p_0\in X$, stopping tolerance $\epsilon >0$, initial estimate ${\tilde{D}}_1\in (0,D_v]$ and parameters $\lambda ,\theta \in (0,1)$.   Step 1: (Initialization) Set $$\begin{array}{c}{p}_1\in {Argmin}_{x\in X}\left\{h_0(p_0,x):=\widehat{f}\left(x\right)+F\left(p_0\right)+⟨F^{\prime }\left(p_0\right),x-p_0⟩\right\},\hfill \end{array}$$ (47)   ${\mathrm{lb}}_1=h\left(p_0,p_1\right)$ and ${\mathrm{ub}}_1=min\{f\left(p_1\right),f\left({\tilde{p}}_1\right)\}$. Let $s=1$;   Step 2: (Stopping test) If ${\mathrm{ub}}_s-{\mathrm{lb}}_s\le \epsilon$, terminate;   Step 3: (Call ${\mathcal{G}}_{MUSL}$ procedure) Set $(p_{s+1},{\mathrm{lb}}_{s+1},{\tilde{D}}_{s+1})={\mathcal{G}}_{MUSL}\left(p_s,{\mathrm{lb}}_s,{\tilde{D}}_s,\lambda ,\theta \right)$ and ${\mathrm{ub}}_{s+1}=f\left(p_{s+1}\right)$;   Step 4: (Loop) Set $s=s+1$, and return to Step 2

We now turn to analyze the optimal complexity bound for the MUSL method. Please note that the following results are modifications of those in [24].

Lemma 3.

Suppose that F is defined in (33) and v is a prox-function of Y with strong convex modulus ${\sigma }_v$. Then, we have the following relation

$$F\left(x_0\right)-F\left(x_1\right)-⟨F^{\prime }\left(x_1\right),x_0-x_1⟩\le 2{\left(\frac{{2\parallel A\parallel }^2{D}_{v,Y}}{{\sigma }_v}\right)}^{\frac{1}{2}}\parallel x_0-x_1\parallel ,\forall x_0,x_1\in {\mathbb{R}}^n.$$

(48)

Here $F^{\prime }\left(x_1\right)\in \partial F\left(x_1\right)$ denotes the subgradient of F at $x_1$ and $D_{v,Y}$ is defined in (13).

Theorem 2.

Suppose that $\left\{{\alpha }_k\right\}$ in procedure ${\mathcal{G}}_{MUSL}$ is chosen to satisfy condition (24), and that $\gamma =1$. Then for given $\epsilon >0$, the following statements hold for the MUSL method.

(1) the number of non-significant phases can be bounded by

$$S_1\left({\tilde{D}}_1\right):=\left[{log}_2\frac{D_{v,Y}}{{\tilde{D}}_1}\right],$$

and the number of significant phases can be bounded by

$$S_2\left(\epsilon \right):=\left[max\left\{0,{log}_{\frac{1}{q}}\frac{2\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\epsilon }\right\}\right].$$

(2) the total number of iterations performed by the MUSL method does not exceed

$$S_1\left({\tilde{D}}_1\right)+S_2\left(\epsilon \right)+\frac{\overline{c}{\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\epsilon },$$

where $\overline{c}=\frac{c\parallel A\parallel }{\lambda \theta }(\sqrt{2}/(1-q)+\sqrt{2}+1)$, and ${\Omega }_{\omega ,X}$ and ${\Omega }_{v,Y}$ are defined in (14).

Proof. 

(1) Let ${\Delta }_s={\mathrm{ub}}_s-{\mathrm{lb}}_s,s\ge 1$. Without loss of generality, we suppose that ${\Delta }_s>\epsilon .$ From conclusion (2) of Lemma 2, we konw that ${\tilde{D}}_{S_1{\left(\tilde{D}\right)}_1+1}=2{\tilde{D}}_{S_1\left({\tilde{D}}_1\right)}<D_{v,Y}$, if non-significant phase occurs. From ${\tilde{D}}_{S_1+1}=2^{S_1\left({\tilde{D}}_1\right)}{\tilde{D}}_1$ and $2^{S_1\left({\tilde{D}}_1\right)}{\tilde{D}}_1<D_{v,Y}$, we have

$$S_1\left(\tilde{D}\right)<{log}_2\frac{D_{v,Y}}{{\tilde{D}}_1}.$$

Furthermore, by (47) and (48) in Lemma 3 we have

$$\begin{array}{cc}\hfill {\Delta }_1& =f\left(p_1\right)-h_0(p_0,p_1)\hfill \\ & =F\left(p_1\right)-\left[F\left(p_0\right)+⟨F^{\prime }\left(p_0\right),p_1-p_0⟩\right]\hfill \\ & \le 2\sqrt{\frac{{2\parallel A\parallel }^2{D}_{v,Y}}{{\sigma }_v}}\parallel p_1-p_0\parallel \hfill \\ & \le 2\parallel A\parallel \sqrt{\frac{2D_{v,Y}}{{\sigma }_v,Y}}\sqrt{\frac{2D_{\omega ,X}}{{\sigma }_{\omega }}}\hfill \\ & =2\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}.\hfill \end{array}$$

From (e) in Lemma 1 and the definition of ${\mathrm{ub}}_s$ and ${\mathrm{lb}}_s$, we know that $\epsilon <{\Delta }_{S_2\left(\epsilon \right)+1}\le q{\Delta }_{S_2\left(\epsilon \right)}\le q^{S_2\left(\epsilon \right)}{\Delta }_1$ and ${\left(\frac{1}{q}\right)}^{S_2\left(\epsilon \right)}\le \frac{{\Delta }_1}{\epsilon }$. In conclusion, we obtain that

$$S_2\left(\epsilon \right)\le {log}_{\frac{1}{q}}\frac{{\Delta }_1}{\epsilon }\le {log}_{\frac{1}{q}}\frac{2\parallel A\parallel {\Omega }_{w,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\epsilon }.$$

(2) Let $K_1=\{a_1,a_2,\dots ,a_m\}$ and $K_2=\{b_1,b_2,\dots ,b_m\}$ denote the index sets of non-significant and significant phases respectively. For any $1\le i\le m$ in the non-significant phases, we have ${\tilde{D}}_{a_{i+1}}=2{\tilde{D}}_{a_i}$. Then we know that ${\tilde{D}}_{a_i}=2^{i-1}{\tilde{D}}_{a_1}=2^{i-1}{\tilde{D}}_1$ and ${\Delta }_{a_i}>\epsilon .$ Due to Proposition 2, we know that K decreases about the first variable and increases about the second variable monotonously. In addition, we conclude that the total number of iterations performed by the significant phases can be bounded by

$$\begin{array}{cc}\hfill N_1=\sum _{i=1}^m{K}_{\mathrm{MUSL}}\left({\Delta }_{a_i},{\tilde{D}}_{a_i}\right)& \le \sum _{i=1}^m{K}_{\mathrm{MUSL}}(\epsilon ,{\tilde{D}}_{a_i})\hfill \\ & \le m+\sum _{i=1}^m\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{\tilde{D}}_{a_i}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ & \le m+\sum _{i=1}^m\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{2}^{i-1}{\tilde{D}}_{a_1}}{{\sigma }_w{\sigma }_v}}\hfill \\ \hfill & \le S_1\left({\tilde{D}}_1\right)+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{\tilde{D}}_1}{{\sigma }_{\omega },{\sigma }_v}}\sum _{i=1}^{S_1\left({\tilde{D}}_1\right)}{2}^{\frac{i-1}{2}}\hfill \\ \hfill & \le S_1\left({\tilde{D}}_1\right)+\frac{2c\parallel A\parallel }{(\sqrt{2}-1)\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ \hfill & =S_1\left({\tilde{D}}_1\right)+\frac{c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{(\sqrt{2}-1)\lambda \theta \epsilon }.\hfill \end{array}$$

(49)

From the fact that ${\Delta }_{b_{j+1}}\le q{\Delta }_{b_j},\forall 1\le j\le n$, we know that

$${\Delta }_{b_j}\ge q^{j-n}{\Delta }_{b_n}>\epsilon q^{j-n}.$$

It follows from the monotonic of K about variables and Proposition 2 that ${\tilde{D}}_{b_j}\le 2D_{v,Y}$. In addition, we obtain that the total number of iterations performed in the non-significant phases can be bounded by

$$\begin{array}{cc}\hfill N_2=\sum _{j=1}^n{K}_{\mathrm{MUSL}}({\Delta }_{b_j},{\tilde{D}}_{b_j})& \le \sum _{j=1}^n{K}_{\mathrm{MUSL}}(\epsilon q^{j-n},2D_{v,Y})\hfill \\ \hfill & \le n+\sum _{j=1}^n\frac{2c\parallel A\parallel }{\lambda \theta \epsilon q^{j-n}}\sqrt{\frac{D_{\omega ,X}2D_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ \hfill & \le n+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\sum _{j=1}^n{q}^{n-j}\hfill \\ \hfill & \le S_2\left(\epsilon \right)+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\sum _{j=1}^{S_2\left(\epsilon \right)}{q}^{n-j}\hfill \\ \hfill & \le S_2\left(\epsilon \right)+\frac{2c\parallel A\parallel }{(1-q)\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ \hfill & =S_2\left(\epsilon \right)+\frac{\sqrt{2}c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{(1-q)\lambda \theta \epsilon }.\hfill \end{array}$$

(50)

Combining (49) and (50), we conclude that the total number of iterations of the MUSL method does not exceed

$$\begin{array}{cc}\hfill N_1+N_2& =S_1\left({\tilde{D}}_1\right)+\frac{c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{(\sqrt{2}-1)\lambda \theta \epsilon }+S_2\left(\epsilon \right)+\frac{\sqrt{2}c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{(1-q)\lambda \theta \epsilon }\hfill \\ & =S_1\left({\tilde{D}}_1\right)+S_2\left(\epsilon \right)+\left(\frac{1}{\sqrt{2}-1}+\frac{\sqrt{2}}{1-q}\right)\frac{c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\lambda \theta \epsilon }.\hfill \end{array}$$

 □

In what follows, we apply the MUSL method to solve a special problem where $\widehat{f}$ in (32) is a smooth convex function. In addition, the complexity results of procedure ${\mathcal{G}}_{MUSL}$ and MUSL method are established in this special case. $\widehat{f}$ is a smooth convex function means that there exists a constant $L_1>0$ such that $\widehat{f}$ satisfies

$$\widehat{f}\left(z\right)-\widehat{f}\left(x\right)-⟨\nabla \widehat{f}\left(x\right),z-x⟩\le \frac{L_1}{2}{\parallel z-x\parallel }^2,\forall z,x\in X.$$

(51)

From the fact that $F_{\eta }$ has a Lipschitz continuous gradient with Lipschitz-constant $L_{\eta }>0$, we have

$$F_{\eta }\left(z\right)-F_{\eta }\left(x\right)-⟨\nabla F_{\eta }\left(x\right),z-x⟩\le \frac{L_{\eta }}{2}{\parallel z-x\parallel }^2,\forall z,x\in X.$$

Thus, we know that $f_{\eta }:=\widehat{f}\left(x\right)+F_{\eta }\left(x\right)$ is a smooth function on X and satisfies

$$f_{\eta }-f_{\eta }-⟨\nabla f_{\eta },z-x⟩\le \frac{L}{2}{\parallel z-x\parallel }^2,\forall z,x\in X,$$

(52)

where $L=L_1+L_{\eta }$. From the fact that F is non-smooth, we konw that f is non-smooth as well.

We make the following changes to the outernal algorithm of the MUSL method and internal procedure ${\mathcal{G}}_{MUSL}$. Replace (47) in outernal algorithm with

$$p_1\in \underset{x\in X}{argmin}\left\{h_0\left(p_0,x\right):=f_{\eta }\left(p_0\right)+⟨\nabla f_{\eta }\left(p_0\right),x-p_0⟩\right\},$$

(53)

and (40) in procedure ${\mathcal{G}}_{MUSL}$ with

$$h_{\eta }(x_k^l,x):=f_{\eta }\left(x\right)+f_{\eta }\left(x_k^l\right)+⟨\nabla f_{\eta }\left(x_k^l\right),x-x_k^l⟩.$$

The function $f_{\eta }$ is fixed as a result of the fact that $\eta$ is fixed in the internal procedure ${\mathcal{G}}_{MUSL}$. Similar to Propostion 2, we have the following results.

Theorem 3.

We assume that $\left\{{\alpha }_k\right\}$ satisfies condition (24) and take $\gamma =1$. Let ℓ be the level parameter, $d_{\omega }(·)$ denotes the prox-function and η is the smoothing parameter in procedure ${\mathcal{G}}_{MUSL}$. If $\widehat{f}$ is a smooth function, then the number of iterations of procedure ${\mathcal{G}}_{MUSL}$ can be bounded by,

$$\overline{N}:=c{\Omega }_{\omega ,X}^{\frac{1}{2}}\left(\sqrt{\frac{L_1}{\theta \lambda {\Delta }_0}}+\frac{\sqrt{2}\parallel A\parallel }{\theta \lambda {\Delta }_0}\sqrt{\frac{\tilde{D}}{{\sigma }_{v,Y}}}\right)+1,$$

where ${\Delta }_0=\mathrm{ub}-1\mathrm{b}={\overline{f}}_0-{\underline{f}}_0$, and ${\Omega }_{\omega ,X}$ is defined in (14).

Proof. 

Suppose that the procedure ${\mathcal{G}}_{\mathrm{MUSL}}$ does not stop in iteration $\overline{K}$. Because of the features of $\widehat{f}$ and $F_{\eta }$, we know that $f_{\eta }$ satisfies (52). Let $L_{\eta }=L$. From (14), we have

$$f_{\eta }\left({\tilde{x}}_{\overline{K}}^u\right)-\ell \le \frac{c^{2}L{\Omega }_{w,X}}{2{\overline{K}}^2},\forall \overline{K}\ge 1.$$

Then according to the stopping test in Step 3b in procedure ${\mathcal{G}}_{MUSL}$, we have

$$f_{\eta }\left({\tilde{x}}_K^u\right)-\ell >\frac{\lambda \theta {\Delta }_0}{2}.$$

Due to $L=L_1+L_{\eta }$, (35) and (38) we obtain that

$$\begin{array}{cc}\hfill \overline{K}<c{\Omega }_{\omega ,X}^{\frac{1}{2}}\sqrt{\frac{L}{\lambda \theta {\Delta }_0}}& <c{\Omega }_{\omega ,X}^{\frac{1}{2}}(\sqrt{\frac{L_1}{\lambda \theta {\Delta }_0}}+\sqrt{\frac{L_{\eta }}{\lambda \theta {\Delta }_0}})\hfill \\ =c{\Omega }_{\omega ,X}^{\frac{1}{2}}\left(\sqrt{\frac{L_1}{\lambda \theta {\Delta }_0}}+\frac{\sqrt{2}\parallel A\parallel }{\lambda \theta {\Delta }_0}\sqrt{\frac{\tilde{D}}{{\sigma }_{v,Y}}}\right).\hfill \end{array}$$

Thus, we have

$$\overline{N}=c{\Omega }_{\omega ,X}^{\frac{1}{2}}\left(\sqrt{\frac{L_1}{\lambda \theta {\Delta }_0}}+\frac{\sqrt{2}\parallel A\parallel }{\lambda \theta {\Delta }_0}\sqrt{\frac{\tilde{D}}{{\sigma }_{v,Y}}}\right)+1.$$

 □

Here we will also establish the complexity of the MUSL method under this special case where $\widehat{f}$ is smooth and convex.

Theorem 4.

Assume that $\left\{{\alpha }_k\right\}$ satisfy condition (24) and take $\gamma =1$. If $\widehat{f}$ is a smooth convex function, then the following statements hold for the MUSL method.

(1) the number of non-significant phases can be bounded by

$${\overline{S}}_1:=\left[{log}_2\frac{D_{v,Y}}{{\tilde{D}}_1}\right],$$

and the number of significant phases can be bounded by

$${\overline{S}}_2:=\left[max\left\{0,{log}_{\frac{1}{q}}\frac{2\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\epsilon }\right\}\right].$$

(2) the total number of iterations performed by the MUSL method does not exceed

$${\overline{S}}_1+{\overline{S}}_2+\left({\overline{S}}_1+\frac{1}{1-\sqrt{q}}\right)c{\Omega }_{w,X}^{\frac{1}{2}}\sqrt{\frac{L_1}{\lambda \theta \epsilon }}+\left(\frac{1}{\sqrt{2}-1}+\frac{\sqrt{2}}{1-q}\right)\frac{c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{\lambda \theta \epsilon }.$$

Proof. 

Let ${\Delta }_s={\mathrm{ub}}_s-{\mathrm{lb}}_s,s\ge 1.$ Similar to Theorem 2, when non-significant phase occurs we have $\overline{S}<{log}_2\frac{D_{v,Y}}{{\tilde{D}}_1}.$ By (53), (48) and (51) we have

$$\begin{array}{cc}\hfill {\Delta }_1=& f\left(p_1\right)-h_0(p_0,p_1)\hfill \\ \hfill \le & [F\left(p_1\right)-F\left(p_0\right)-⟨\nabla F\left(p_0\right),p_1-p_0⟩]+[\widehat{f}\left(p_1\right)-\widehat{f}\left(p_0\right)-⟨\nabla \widehat{f}\left(p_0\right),p_1-p_0⟩]\hfill \\ \hfill \le & 4\parallel A\parallel \sqrt{\frac{D_{v,Y}{D}_{\omega ,X}}{{\sigma }_v{\sigma }_{\omega }}}+\frac{L_1}{2}{D}_{\omega ,X}\hfill \\ \hfill =& 2\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}+\frac{L_1}{2}{D}_{\omega ,X}.\hfill \end{array}$$

From (e) in Lemma 1 and the definition of ${\mathrm{ub}}_s$ and ${\mathrm{lb}}_s$, we know that $\epsilon <{\Delta }_{{\overline{S}}_2+1}\le q{\Delta }_{{\overline{S}}_2}\le q^{{\overline{S}}_2}{\Delta }_1$ and ${\left(\frac{1}{q}\right)}^{{\overline{S}}_2}\le \frac{{\Delta }_1}{\epsilon }$. In conclusion, we obtain that

$${\overline{S}}_2\le {log}_{\frac{1}{q}}\frac{{\Delta }_1}{\epsilon }\le {log}_{\frac{1}{q}}\frac{2\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}+\frac{L_1}{2}{D}_{\omega ,X}}{\epsilon }.$$

(2) Let ${\overline{K}}_1=\{c_1,c_2,\dots ,c_m\}$ and ${\overline{K}}_2=\{d_1,d_2,\dots ,d_m\}$ denote the index sets of non-significant and significant phases respectively. For any $1\le i\le m$ in the non-significant phases, we have $Q_{c_{i+1}}=2Q_{c_i}$. Then we know that $Q_{c_i}=2^{i-1}{Q}_{c_1}=2^{i-1}{Q}_1$ and ${\Delta }_{c_i}>\epsilon .$ By Proposition 2, we know that $\overline{K}$ decreases about the first variable and increases about the second variable monotonously. It follows that the total number of iterations performed by the significant phases can be bounded by

$$\begin{array}{cc}\hfill {\overline{N}}_1=\sum _{i=1}^m{K}_{\mathrm{MUSL}}\left({\Delta }_{c_i},{\tilde{D}}_{c_i}\right)\le & \sum _{i=1}^m{K}_{\mathrm{MUSL}}\left(\epsilon ,{\tilde{D}}_{c_i}\right)\hfill \\ \hfill \le & m+\sum _{i=1}^m\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{\tilde{D}}_{c_i}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ \hfill \le & m+\sum _{i=1}^m\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{2}^{i-1}{\tilde{D}}_{c_1}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ \hfill \le & {\overline{S}}_1\left(1+c\sqrt{\frac{2D_{\omega ,X}{L}_1}{{\sigma }_{\omega }\lambda \theta \epsilon }}\right)+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{\tilde{D}}_1}{{\sigma }_{\omega }{\sigma }_v}}\sum _{i=1}^{{\overline{S}}_1}{2}^{\frac{i-1}{2}}\hfill \\ \hfill \le & {\overline{S}}_1\left(1+c\sqrt{\frac{2D_{\omega ,X}{L}_1}{{\sigma }_{\omega }\lambda \theta \epsilon }}\right)+\frac{2c\parallel A\parallel }{(\sqrt{2}-1)\lambda \theta \epsilon }\sqrt{\frac{D_{\omega ,X}{D}_{v,Y}}{{\sigma }_w{\sigma }_v}}\hfill \\ \hfill =& {\overline{S}}_1\left(1+c{\Omega }_{\omega ,X}^{\frac{1}{2}}\sqrt{\frac{L_1}{\lambda \theta \epsilon }}\right)+\frac{c\parallel A\parallel }{(\sqrt{2}-1)\lambda \theta \epsilon }{\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}.\hfill \end{array}$$

(54)

From the fact that ${\Delta }_{d_{j+1}}\le q{\Delta }_{d_j}\forall 1\le j\le n$, we know that

$${\Delta }_{d_j}\ge q^{j-n}{\Delta }_{d_n}>\epsilon q^{j-n}.$$

By the monotonic of $\overline{K}$ about variables and the fact that ${\tilde{D}}_{b_j}\le 2D_{v,Y}$, we know that the total number of iterations performed in the non-significant phases can be bounded by

$$\begin{array}{cc}\hfill {\overline{N}}_2=\sum _{j=1}^n{K}_{\mathrm{MUSL}}\left({\Delta }_{d_j},{\tilde{D}}_{d_j}\right)& \le \sum _{j=1}^n{K}_{\mathrm{MUSL}}\left(\epsilon q^{j-n},2D_{v,Y}\right)\hfill \\ & \le n+\sum _{j=1}^n\frac{2c\parallel A\parallel }{\lambda \theta \epsilon q^{j-n}}\sqrt{\frac{D_{\omega ,X}2D_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ & \le n+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\sum _{j=1}^n{q}^{n-j}\hfill \\ & \le {\overline{S}}_2+c\sqrt{\frac{2D_{\omega ,X}{L}_1}{{\sigma }_{\omega }\lambda \theta \epsilon }}\sum _{j=1}^{{\overline{S}}_2}{q}^{\frac{n-j}{2}}+\frac{2c\parallel A\parallel }{\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\sum _{j=1}^{{\overline{S}}_2}{q}^{n-j}\hfill \\ & \le {\overline{S}}_2+\frac{c}{1-\sqrt{q}}\sqrt{\frac{2D_{\omega ,X}{L}_1}{{\sigma }_{\omega }\lambda \theta \epsilon }}+\frac{2c\parallel A\parallel }{(1-q)\lambda \theta \epsilon }\sqrt{\frac{2D_{\omega ,X}{D}_{v,Y}}{{\sigma }_{\omega }{\sigma }_v}}\hfill \\ & ={\overline{S}}_2+\frac{c{\Omega }_{\omega ,X}^{\frac{1}{2}}}{1-\sqrt{q}}\sqrt{\frac{L_1}{\lambda \theta \epsilon }}+\frac{\sqrt{2}c\parallel A\parallel {\Omega }_{\omega ,X}^{\frac{1}{2}}{\Omega }_{v,Y}^{\frac{1}{2}}}{(1-q)\lambda \theta \epsilon }.\hfill \end{array}$$

(55)

This together with (54) and (55) shows that the total number of MUSL method does not exceed ${\overline{N}}_1+{\overline{N}}_2.$ □

5. Two-Stage Stochastic Programming and Numerical Experiments

In this section, the MAPL and MUSL methods are applied to solve the two-stage stochastic programming problems. Furthermore, we compare the two modified methods with the APL and USL methods. All algorithms are implemented in MATLAB (R2014a), and Mosek (8.0) is called for solving subproblems. The environment in which the program runs is Windows 7 (64 bite), Intel(R) Core(TM) i7-6700 CPU 3.40GHz and 16G memory.

Consider the following two-stage stochastic programming with recourse

$$min\{f\left(x\right)=c^{T}x+\mathbb{E}\left[Q(x,\xi )\right]:x\in X\},$$

(56)

where

$$Q(x,\xi )=min\{q^{T}y:Wy=h+Tx,y\ge 0\}.$$

(57)

Here, $x\in {\mathbb{R}}^{n_1}$ and $y\in {\mathbb{R}}^{n_2}$ are the decision variables of the first-stage problem and the second-stage problem respectively. $X\ne \varnothing$ is a compact convex set, and $\xi =(q,h,T)\in {\mathbb{R}}^{n_1+m_2+m_2+n_1}$ is a random vector with a known probability distribution and support $\Xi$. In [33], it was pointed out that by strong convexity one has

$$Q(x,\xi )=max\{{\pi }^T(h+Tx):W^T\pi \le q\}.$$

(58)

Let $\Pi \left(q\right)=\{\pi \in {\mathbb{R}}^{m_2}:W^T\le q\}$ be the feasible set of (58) and we assume that $\Pi \left(q\right)\ne \varnothing ,\forall q\in \Xi$. It is easy to know that $Q(x,\xi )$ is generally non-smooth. Therefore, for general distributions of $\xi$, (56) is a non-smooth convex programming problem. In [33] Ahmed applied Nesterov’s smoothing technique to the two-stage stochastic programming problem (56), and established a proper smooth approximation to $Q(x,\xi )$ as follows. For a given smoothing parameter $\mu$, consider the function $f_{\mu }\left(x\right)$ with the form

$$f_{\mu }\left(x\right)=c^{T}x+\mathbb{E}\left[Q_{\mu }(x,\xi )\right],$$

(59)

where

$$Q_{\mu }(x,\xi )=max\{{\pi }^T(h+Tx)-\frac{\mu }{2}{\pi }^T\pi :\pi \in \Pi \left(q\right)\}.$$

(60)

For $\xi$ with discrete distribution, it is shown in [34,35] that $f_{\mu }$ is differentiable and its gradient is Lipschitz continuous. Furthermore, when $\mu$ is sufficiently small, $f_{\mu }$ can approximate f uniformly. Based on the above analysis, we next carry out the MAPL and MUSL methods to solve the problem (59) and (60) to illustrate the effectiveness of the methods and compare them with the APL and USL methods.

We perform numerical experiments on some existing SP instances in [1,36,37] including a telecommunication design (SSN) problem and a motor freight carrier routing problem (20Term). The SSN problem studied by Sen, Doverspike, and Cosares [1] comes from the telecommunications industry: the first-stage problem allocates capacity between network links, and the second-stage problem generates demands to connections requested between point-to-point pairs. The 20Term problem studied by Mak, Morton, and Wood [37] comes from a motor freight carrier’s model: the first-stage problem determines a program of carriers, and the second-stage problem adjusts the program according a multi-commodity network. The instances of SSN, 20Term and Storm are downloaded from the link: http://pwp.gatech.edu/guanghui-lan/computer-codes/. The dimensions related to these instances are shown in Table 1. $n_i$ denotes the number of constraints in the i-stage problem, and $m_i$ denotes the number of variable in the i-stage problem. In particular, we assume that the number of possible realizations of $\xi$ is fixed, i.e. $N=50$ or 100. In this case, a total of five instances are tested. The integers in the brackets are the number of possible realizations. For given parameters $\lambda \in (0,1),\theta \in (0,1)$, tolerance $\epsilon =1.0e-6$ and stepsize ${\alpha }_k=2/(k+1)$ such that (24), we compare the number of iterations and CPU time of different methods. The results are shown in Table 2, Table 3 and Table 4.

Table 1. Data about instances.

	${\mathit{n}}_1$	${\mathit{m}}_1$	${\mathit{n}}_2$	${\mathit{m}}_2$	$\mathbf{dim}\left(\mathit{\xi }\right)$	${\mathbf{ub}}_1$	${\mathbf{\Delta }}_1$
SSN(50)	89	1	706	175	86	$2.352586\times {10}^2$	$1.008000\times {10}^3$
SSN(100)	89	1	706	175	86	$2.407279\times {10}^2$	$9.878400\times {10}^2$
20Term(50)	63	3	764	124	40	$7.718617\times {10}^5$	$1.804693\times {10}^7$
20Term(100)	63	3	764	124	40	$7.664142\times {10}^5$	$1.801832\times {10}^7$
Storm(50)	121	185	1259	528	117	$2.138447\times {10}^7$	$6.370092\times {10}^6$

Table 2. MAPL and APL methods on SP instances.

Prob	Alg	Iter	Ub	$\mathit{\delta }$	Time
SSN(50)	MAPL	135	$4.838073\times {10}^0$	$9.042063\times {10}^{-7}$	63.71
	APL	140	$4.838181\times {10}^0$	$7.252755\times {10}^{-7}$	66.39
SSN(100)	MAPL	215	$7.352607\times {10}^0$	$9.748218\times {10}^{-6}$	233.95
	APL	263	$7.352607\times {10}^0$	$9.835275\times {10}^{-7}$	292.95
20Term(50)	MAPL	281	$2.549453\times {10}^5$	$1.206150\times {10}^{-6}$	193.9
	APL	401	$2.549453\times {10}^5$	$1.168752\times {10}^{-6}$	282.83
20Term(100)	MAPL	386	$2.549453\times {10}^5$	$8.772477\times {10}^{-7}$	519.40
	APL	401	$2.532875\times {10}^5$	$1.059493\times {10}^{-4}$	536.20
Storm(50)	MAPL	119	$1.562059\times {10}^7$	$1.093373\times {10}^{-6}$	87.59
	APL	401	$1.547024\times {10}^7$	$2.211075\times {10}^{-3}$	518.62

Table 3. MUSL and USL methods on SP instances.

Prob	Alg	Iter	Ub	$\mathit{\delta }$	Time	Q
SSN(50)	MUSL	208	$4.838072\times {10}^0$	$3.003666\times {10}^{-6}$	206.81	$3.200000\times {10}^1$
	USL	322	$4.839140\times {10}^0$	$9.337998\times {10}^{-7}$	309.43	$3.200000\times {10}^1$
SSN(100)	MUSL	181	$7.352608\times {10}^0$	$1.686893\times {10}^{-6}$	351.11	$3.200000\times {10}^1$
	USL	190	$7.352699\times {10}^0$	$8.517813\times {10}^{-7}$	367.12	$3.200000\times {10}^1$
20Term(50)	MUSL	401	$2.549453\times {10}^5$	$6.026059\times {10}^{-5}$	547.93	$3.022315\times {10}^{23}$
	USL	401	$2.549453\times {10}^5$	$8.844555\times {10}^{-4}$	522.41	$2.147484\times {10}^9$
20Term(100)	MUSL	380	$2.532875\times {10}^5$	$2.171117\times {10}^{-6}$	1148.29	$1.677722\times {10}^7$
	USL	401	$2.532875\times {10}^5$	$4.172471\times {10}^{-6}$	1308.03	$1.801440\times {10}^{16}$
Storm(50)	MUSL	299	$2.138447\times {10}^7$	$1.087785\times {10}^{-6}$	437.11	$5.242880\times {10}^5$
	USL	401	$1.546949\times {10}^7$	$2.866387\times {10}^{-4}$	1084.46	$1.511157\times {10}^{23}$

Table 4. MAPL and MUSL methods on SP instances.

Prob	Alg	Iter	Ub	$\mathit{\delta }$	Time
SSN(50)	MAPL	159	$4.838072\times {10}^0$	$1.016148\times {10}^{-6}$	90.68
	MUSL	175	$4.838073\times {10}^0$	$9.421333\times {10}^{-6}$	201.82
SSN(100)	MAPL	190	$7.352608\times {10}^0$	$6.397336\times {10}^{-6}$	217.63
	MUSL	275	$7.352607\times {10}^0$	$2.900934\times {10}^{-6}$	638.96
20Term(50)	MAPL	347	$2.549453\times {10}^5$	$4.465430\times {10}^{-6}$	271.94
	MUSL	401	$2.549453\times {10}^5$	$3.039380\times {10}^{-5}$	663.75
20Term(100)	MAPL	401	$2.532875\times {10}^5$	$3.203178\times {10}^{-2}$	607.93
	MUSL	401	$2.532875\times {10}^5$	$1.512163\times {10}^{-4}$	1174.29
Storm(50)	MAPL	19	$1.866006\times {10}^7$	$1.028180\times {10}^{-6}$	18.08
	MUSL	29	$2.138447\times {10}^7$	$1.799315\times {10}^{-6}$	61.25

There are some observations on the results in Table 2, Table 3 and Table 4. When the initial gap $\Delta$ is large (up to ${10}^2$ or ${10}^7$), MAPL and APL algorithms are implemented 400 times. The results in Table 2 show that MAPL method has certain advantages over APL method in terms of CPU time and the number of iterations. The results in Table 3 show that in addition to the advantages in CPU time, MUSL algorithm can achieve higher accuracy. From the results in Table 4 we know that compared with the MUSL algorithm, the MAPL algorithm has less CPU time and fewer number of iterations. This is because that while the MAPL method solves $2N$ linear programming problems, MUSL only needs to solve N smooth quadratic programming problems in the progress of algorithm.

6. Conclusions

In this paper, we presented two modified BL-type methods, the modified accelerated prox-level (MAPL) and modified uniform smoothing level (MUSL) methods, for uniformly solving the black-box CP problems and a class of structure non-smooth problems. In addition, both MAPL and MUSL methods can achieve optimal complexities respectively. To illustrate the effectiveness of the modified methods, they were then applied to solve the two-stage stochastic programming with recourse and numerical experiments were carried out. Finally, the numerical results shown that the MAPL and MUSL methods have certain advantages in algorithm efficiency and solution time.