A Generalized Alternating Linearization Bundle Method for Structured Convex Optimization with Inexact First-Order Oracles

Abstract

In this paper, we consider a class of structured optimization problems whose objective function is the summation of two convex functions: f and h, which are not necessarily differentiable. We focus particularly on the case where the function f is general and its exact first-order information (function value and subgradient) may be difficult to obtain, while the function h is relatively simple. We propose a generalized alternating linearization bundle method for solving this class of problems, which can handle inexact first-order information of on-demand accuracy. The inexact information can be very general, which covers various oracles, such as inexact, partially inexact and asymptotically exact oracles, and so forth. At each iteration, the algorithm solves two interrelated subproblems: one aims to find the proximal point of the polyhedron model of f plus the linearization of h; the other aims to find the proximal point of the linearization of f plus h. We establish global convergence of the algorithm under different types of inexactness. Finally, some preliminary numerical results on a set of two-stage stochastic linear programming problems show that our method is very encouraging.

1. Introduction

In this paper, we consider the following structured convex optimization problem

$$F_{\ast }:=\underset{x\in {\mathbb{R}}^n}{min}\left\{F\left(x\right):=f\left(x\right)+h\left(x\right)\right\},$$

(1)

where $f:\mathrm{dom}h\to \mathbb{R}$ and $h:{\mathbb{R}}^n\to (-\infty ,\infty ]$ are closed proper convex functions, but not necessarily differentiable, and $\mathrm{dom}h:=\{x:h(x)<\infty \}$ is the effective domain of h. Problems of this type frequently arise in practice, such as compressed sensing [1], image reconstruction [2], machine learning [3], optimal control [4] and power system [5,6,7,8], and so forth. The following are three interesting examples.

Example 1.

(${\ell }_1$ minimization in compressed sensing). The signal recovery problems in compressed sensing [1] usually take the following form

$$\underset{x\in {\mathbb{R}}^n}{min}\frac{1}{2}{\parallel Ax-b\parallel }^2+\lambda {\parallel x\parallel }_1,$$

(2)

where $A\in {\mathbb{R}}^{m\times n}$, $b\in {\mathbb{R}}^m$, $\lambda >0$ and ${\parallel x\parallel }_1:={\sum }_{i=1}^n\left|x_i\right|$, which aims to get a sparse solution x of the linear system $Ax=b$. Note that by defining $f\left(x\right)=\frac{1}{2}{\parallel Ax-b\parallel }^2$ and $h\left(x\right)={\lambda \parallel x\parallel }_1$, (2) is of the form of (1).

Example 2.

(Regularized risk minimization). At the core of many machine learning problems is to minimize a regularized risk function [9,10]:

$$\underset{x\in {\mathbb{R}}^n}{min}{R}_{\mathrm{emp}}\left(x\right)+\lambda \mathsf{\Omega }\left(x\right),$$

(3)

where $R_{\mathrm{emp}}\left(x\right):=\frac{1}{m}{\displaystyle \sum _{i=1}^m}l(u_i,v_i,x)$ is the empirical risk, $\{(u_i,v_i),i=1,\cdots ,m\}$ is a training set, and l is a convex loss function measuring the gap between v and the predicted values generated by using x. In general, $R_{\mathrm{emp}}\left(x\right)$ is a nondifferentiable and computationally expensive convex function, whereas the regularization term $\mathsf{\Omega }\left(x\right)$ is a simple convex function, say $\mathsf{\Omega }\left(x\right)=\frac{1}{2}{\parallel x\parallel }_2^2$ or $\mathsf{\Omega }\left(x\right)={\parallel x\parallel }_1$. By defining $f\left(x\right)=R_{\mathrm{emp}}\left(x\right)$ and $h\left(x\right)=\lambda \mathsf{\Omega }\left(x\right)$, (3) is also of the form of (1).

Example 3.

(Unconstrained transformation of a constrained problem). Given a constrained problem

$$min\left\{f\right(x):x\in X\},$$

(4)

where f is a convex function and $X\subseteq {\mathbb{R}}^n$ is a convex set. By introducing the indicator function ${\delta }_X$ of X, that is, ${\delta }_X\left(x\right)$ equals 0 on X and infinity elsewhere, problem (4) can be written equivalently as

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right)+{\delta }_X\left(x\right).$$

(5)

Clearly, by setting $h\left(x\right)={\delta }_X\left(x\right)$, (5) becomes the form of (1). We note that such transformation could be very efficient in practice if the set X has some special structure [11,12].

The design of methods for solving problems of the form (1) has attracted the attention of many researchers. We mention here four classes of these methods—operator splitting methods [13,14,15], alternating direction methods of multipliers [5,16,17,18,19], alternating linearization methods [20,21], and alternating linearization bundle method [22]. They all fall within the well-known class of first-order black-box methods, that is, it is assumed that there is an oracle that can return the (approximate) function value and one arbitrary (approximate) subgradient at any given point. Regarding the above methods, we are concerned about the following three points:

• Smoothness of one or both functions in the objective has been assumed for many of the methods.
• Except for Reference [22], they all require the exact computation of the function values and (sub)gradients.
• The alternating linearization methods [20,21] essentially assume that both functions f and h are “simple” in the sense that minimizing the function plus a separable convex quadratic function is easy.

However, for some practical problems, the functions may be nondifferentiable (nonsmooth), not easy to handle, and computationally expensive in the sense that the exact first-order information may be impossible to calculate, or be computable but difficult to obtain. For example, if f has the form

$$f\left(x\right):=sup\{{\varphi }_u\left(x\right):u\in U\},$$

where U is an infinite set and ${\varphi }_u\left(x\right):{\mathbb{R}}^n\to \mathbb{R}$ is convex for any $u\in U$, then it is often difficult to calculate the exact function value $f\left(x\right)$. But for any tolerance $ϵ>0$, we may usually find a lower approximation $f_x^{ϵ}\approx f\left(x\right)$ in finite time such that $f_x^{ϵ}\in [f\left(x\right)-ϵ,f\left(x\right)]$ and $f_x^{ϵ}={\varphi }_{u_{ϵ}}\left(x\right)$ for some $u_{ϵ}\in U$. Then we can take a subgradient of ${\varphi }_{u_{ϵ}}$ at x as an approximate subgradient of f at x. Another example is two-stage stochastic programming (see, e.g., References [23,24]), in which the function value is generated after solving a series of linear programs (details will be given in the section of numerical experiments), therefore its accuracy is determined by the tolerance of the linear programming solver. Some other practical examples, such as Lagrangian relaxation, chance-constrained programs and convex composite functions, can be found in Reference [25].

Based on the above observation, in this paper, we focus particularly on the case where the function f is general, possibly nonsmooth and its exact function values and subgradients may be difficult to obtain, whereas the function h is assumed to be relatively simple. Our main goal is to provide an efficient method, namely, the improved alternating linearization bundle method, for such kind of structured convex optimization problems. The basic tool we used here to handle nonsmoothness and inexactness is the bundle technique, since in the nonsmooth optimization community, bundle methods [26,27,28,29] are regarded as the most robust and reliable methods, whose variants have been well studied for handling inexact oracles [23,25,30,31,32,33].

Roughly speaking, our method generalizes the alternating linearization bundle method of Kiwiel [22] from exact and inexact oracles to various oracles, including exact, inexact, partially inexact, asymptotically exact and partially asymptotically exact oracles. These oracles are covered by the so-called on-demand accuracy oracles proposed by de Oliveira and Sagastizábal [23], whose accuracy is controlled by two parameters: a descent target and an error bound. More precisely, the accuracy is bounded by an error bound whenever the function estimation reaches a certain descent target. The most advantage of oracles with on-demand accuracy is that the function and subgradient estimations can be rough without an accuracy control for some “non-critical” iterates, thus the computational effort can be saved.

At each iteration, the proposed algorithm alternately solves two interrelated subproblems: one is to find the proximal point of the polyhedron model of f plus the linearization of h; the other is to find the proximal point of the linearization of f plus h. We establish global convergence of the algorithm under different types of inexactness. Finally, some preliminary numerical results on a set of two-stage stochastic linear programming problems show that our method is very encouraging.

This paper is organized as follows. In Section 2, we recall the condition of the inexact frist-order oracles. In Section 3, we present an improved alternating linearization bundle method for structured convex optimization with inexact first-order oracles and show some properties of the algorithm. In Section 4, we establish global convergence of the algorithm under different types of inexactness. In Section 5, we provide some numerical experiments on two-stage stochastic linear programming problems. The notations are standard. The Euclidean inner product in ${\mathbb{R}}^n$ is denoted by $⟨x,y⟩:=x^{T}y$, and the associated norm by $\parallel ·\parallel$.

2. Preliminaries

For a given constant $ϵ\ge 0$, the $ϵ$-subdifferential of function f at x is defined by (see Reference [34])

$${\partial }_{ϵ}f\left(x\right):=\{g\in {\mathbb{R}}^n:f\left(y\right)\ge f\left(x\right)+⟨g,y-x⟩-ϵ,\forall y\in {\mathbb{R}}^n\},$$

with $\partial f\left(x\right):={\partial }_{0}f\left(x\right)$ being the usual subdifferential in convex analysis [35]. Each element $g\in \partial f\left(x\right)$ is called a subgradient. For simplicity, we use the following notations:

$f_x$: the approximate f value at x, that is, $f_x\approx f\left(x\right)$;

$g_x$: an approximate subgradient of f at x, that is, $g_x\approx g\left(x\right)\in \partial f\left(x\right)$;

$F_x$: the approximate F value at x, that is, $F_x:=f_x+h\left(x\right)$.

Aiming at the special structure of problem (1), we present a slight variant of the oracles with on-demand accuracy proposed in Reference [23] as follows: for a given $x\in {\mathbb{R}}^n$, a descent target ${\gamma }_x$ and an error bound ${\epsilon }_x\ge 0$, the approximate values $f_x$, $g_x$ and $F_x$ satisfy the following condition

$$\left\{\begin{array}{c}{f}_x=f\left(x\right)-\eta \left({\gamma }_x\right)\mathrm{with}\mathrm{unknown}\eta \left({\gamma }_x\right)\ge 0,\hfill \\ g_x\in \partial {}_{\eta \left({\gamma }_x\right)}f\left(x\right),\mathrm{and}\hfill \\ \mathrm{whenever}{F}_x\le {\gamma }_x\left(descenttargetreached\right),\mathrm{the}\mathrm{relation}\eta \left({\gamma }_x\right)\le {\epsilon }_x\mathrm{holds}.\hfill \end{array}\right.$$

(6)

From the relations in (6), we see that although the error $\eta \left({\gamma }_x\right)$ is unknown, it has to be restricted within the error bound ${\epsilon }_x$ whenever the descent target $F_x\le {\gamma }_x$ is reached. This ensures that the exact and inexact function values satisfy

$$f_x\in [f\left(x\right)-{\epsilon }_x,f\left(x\right)]\mathrm{and}f\left(x\right)\in [f_x,f_x+{\epsilon }_x],\mathrm{whenever}{F}_x\le {\gamma }_x.$$

(7)

The advantages of oracle (6) are that: (1) if the descent target is not reached, the calculation of oracle information can be rough without an accuracy control, which can potentially reduce the computational cost; (2) by properly choosing the parameters ${\gamma }_x$ and ${\epsilon }_x$, the oracle (6) covers various existing oracles:

• Exact (Ex) [12,21]: set ${\gamma }_x=+\infty$ and ${\epsilon }_x=0$;
• Partially Inexact (PI) [24]: set ${\gamma }_x<+\infty$ and ${\epsilon }_x=0$;
• Inexact (IE) [11,25,32,36,37]: set ${\gamma }_x=+\infty$ and ${\epsilon }_x\equiv \epsilon >0$ (possibly unknown);
• Asymptotically Exact (AE) [38,39]: set ${\gamma }_x=+\infty$ and ${\epsilon }_x\to 0$ along the iterative process;
• Partially Asymptotically Exact (PAE) [23]: set ${\gamma }_x<+\infty$ and ${\epsilon }_x\to 0$.

3. The Generalized Alternating Linearization Bundle Method

In this section, we present our generalized alternating linearization bundle method with inexact first-order oracles for solving (1).

Let k be the current iteration index, $x^j,j\in J^k\subseteq \{1,\cdots ,k\}$ be given points generated by previous iterations, and the corresponding approximate values $f_{x^j}/g_{x^j}$ be produced by the oracle (6). For notational convenience, we denote

$$f_x^j:=f_{x^j},g_x^j:=g_{x^j},F_x^j:=F_{x^j},{\epsilon }_x^j:={\epsilon }_{x^j},{\gamma }_x^j:={\gamma }_{x^j}.$$

The approximate linearizations of f at $x^j$ are given by

$$f_j(·):=f_x^j+⟨g_x^j,·-x^j⟩,j\in J^k.$$

From the second relation in (6), we have

$$f(·)\ge f\left(x^j\right)+⟨g_x^j,·-x^j⟩-\eta \left({\gamma }_x^j\right)=f_j(·),$$

which implies that $f_j$ is a lower approximation to f. Next, it is natural to define the polyhedral inexact cutting-plane model of f by

$${\check{f}}_k(·):=\underset{j\in J_k}{max}\left\{f_j(·)\right\},$$

(8)

which is obviously a lower polyhedral model for f, that is, ${\check{f}}_k(·)\le f(·)$.

Let ${\widehat{x}}^k$ (called stability center) be the “best” point obtained so far, which satisfies that ${\widehat{x}}^k=x^{k\left(l\right)}$ for some $k\left(l\right)\le k$. Frequently, it holds that $f_x^{k\left(l\right)}=\underset{j=1,\cdots ,k}{min}{f}_x^j$. Thus, from (7), we have

$$f\left({\widehat{x}}^k\right)\in [f_{\widehat{x}}^k,f_{\widehat{x}}^k+{\epsilon }_x^{k\left(l\right)}],\mathrm{whenever}{F}_{\widehat{x}}^k\le {\gamma }_{\widehat{x}}^k.$$

(9)

By applying the bundle idea to the “complex” function f, and keeping the simple function h unchanged, similar to traditional proximal bundle methods (see, e.g., Reference [28]), we may solve the following subproblem to obtain a new iterate $x^{k+1}$:

$$x^{k+1}:=argmin{\check{f}}_k(·)+h(·)+\frac{1}{2t_k}{\parallel ·-{\widehat{x}}^k\parallel }^2,$$

(10)

where $t_k>0$ is a proximal parameter. However, subproblem (10) is generally not easy to solve, so by making use of the alternating linearization idea of Kiwiel [22], we solve two easier subproblems instead of (10). These two subproblems are interrelated: one is to find the proximal point of the polyhedron model $\check{f}$ plus the linearization of h, aiming at generating an aggregate linear model of f for use in the second subproblem; the other is to find the proximal point of the aggregate linear model of f plus h, aiming at obtaining a new trial point.

Now, we are ready to present the details of our algorithm, which generalizes the work of Kiwiel [22]. We note that the choice of the model function ${\check{f}}_k$ in the algorithm may be different from the form of (8), since the subgradient aggregation strategy [40] is used to compress the bundle. The algorithm generates three sequences of iterates as follows: $\left\{y^k\right\}$, the sequence of intermediate points, at which the aggregate linear models of f are generated; $\left\{x^k\right\}$, the sequence of trial points; $\left\{{\widehat{x}}^k\right\}$, the sequence of stability centers.

We make some comments about Algorithm 1 as follows.

Algorithm 1 Generalized alternating linearization bundle method

Step 0 (Initialization). Select an initial point $x^1\in {\mathbb{R}}^n$, constants $\kappa \in (0,1)$, $t_{\min }>0$, and an initial stepsize $t_1\ge t_{\min }$. Call the oracle (6) at $x^1$ to compute the approximate values $f_x^1$ and $g_x^1$. Choose an initial error bound ${\epsilon }_x^1\ge 0$ and a descent target ${\gamma }_x^1=+\infty$. Set ${\widehat{x}}^1:=x^1$, $f_{\widehat{x}}^1:=f_x^1$, $F_{\widehat{x}}^1:=f_{\widehat{x}}^1+h\left({\widehat{x}}^1\right)$, ${\overline{f}}_0:=f_1$, and ${\overline{h}}_0(·):=h\left(x^1\right)+⟨p_h^0,·-x^1⟩$ with $p_h^0\in \partial h\left(x^1\right)$. Let $i_t^1:=0$, $l:=1$, $k\left(l\right):=1$ and $k:=1$. Step 1 (Model selection). Choose ${\check{f}}_k:{\mathbb{R}}^n\to \mathbb{R}$ closed convex and such that $$max\{{\overline{f}}_{k-1},f_k\}\le {\check{f}}_k\le f.$$ Step 2 (Solve f-subproblem). Set $$y^{k+1}:=argmin\left\{{\varphi }_f^k(·):={\check{f}}_k(·)+{\overline{h}}_{k-1}(·)+\frac{1}{2t_k}{\parallel ·-{\widehat{x}}^k\parallel }^2\right\},$$ (11) $${\overline{f}}_k(·):={\check{f}}_k\left(y^{k+1}\right)+⟨p_f^k,·-y^{k+1}⟩\mathrm{with}{p}_f^k:=\frac{1}{t_k}({\widehat{x}}^k-y^{k+1})-p_h^{k-1}.$$ (12) Step 3 (Solve h-subproblem). Set $$x^{k+1}:=argmin\left\{{\varphi }_h^k(·):={\overline{f}}_k(·)+h(·)+\frac{1}{2t_k}{\parallel ·-{\widehat{x}}^k\parallel }^2\right\},$$ (13) $${\overline{h}}_k(·):=h\left(x^{k+1}\right)+⟨p_h^k,·-x^{k+1}⟩\mathrm{with}{p}_h^k:=\frac{1}{t_k}({\widehat{x}}^k-x^{k+1})-p_f^k.$$ (14) Step 4 (Stopping criterion). Compute $$v_k:=F_{\widehat{x}}^k-\left[{\overline{f}}_k\left(x^{k+1}\right)+h\left(x^{k+1}\right)\right],p^k:=\frac{1}{t_k}({\widehat{x}}^k-x^{k+1}),{ϵ}_k:=v_k-t_k{\parallel p^k\parallel }^2.$$ (15) If $max\{\parallel p^k\parallel ,{ϵ}_k\}=0$, stop. Step 5 (Noise attenuation). If $v_k<-{ϵ}_k$, set $t_k:=10t_k$, $i_t^k:=k$, and go back to Step 2. Step 6 (Call oracle). Select a new error bound ${\epsilon }_x^{k+1}\ge 0$ and a new descent target ${\gamma }_x^{k+1}\in \mathbb{R}\cup \{+\infty \}$. Call the oracle (6) to compute $f_x^{k+1}$ and $g_x^{k+1}$. Step 7 (Descent test). If the descent condition $$F_x^{k+1}\le F_{\widehat{x}}^k-\kappa v_k$$ (16) holds, set ${\widehat{x}}^{k+1}:=x^{k+1}$, $F_{\widehat{x}}^{k+1}:=F_x^{k+1}$, $i_t^{k+1}:=0$, $k(l+1):=k+1$, and $l:=l+1$ (descent step); otherwise, set ${\widehat{x}}^{k+1}:={\widehat{x}}^k$, $F_{\widehat{x}}^{k+1}:=F_{\widehat{x}}^k$, $i_t^{k+1}:=i_t^k$, and $k(l+1)=k\left(l\right)$ (null step). Step 8 (Stepsize updating). For a descent step, select $t_{k+1}\ge t_k$. For a null step, either set $t_{k+1}:=t_k$ or choose $t_{k+1}\in [t_{min},t_k]$ if $i_t^{k+1}=0$. Step 9 (Loop). Let $k:=k+1$, and go to Step 1.

Remark 1.

(i) Theoretically speaking, the model function ${\check{f}}_k$ can be the simplest form $max\{{\overline{f}}_{k-1},f_k\}$, but in order to keep numerical stability, it may additionally consist of some active linearizations.

(ii) Alternately solving subproblems (11) and (13) can be regarded as the proximal alternating linearization method (e.g., Reference [21]) being applied to the function ${\check{f}}_k+h$.

(iii) If ${\check{f}}_k$ is a polyhedral function, then subproblem (11) is equivalent to a convex quadratic programming and thus can be solved efficiently. In addition, if h is simple, subproblem (13) can also be solved easily, or even has a closed-form solution (say $h\left(x\right)=\frac{1}{2}{\parallel x\parallel }^2$).

(iv) The role of Step 5 is to reduce the impact of inexactness. The algorithm loops between steps 2–5 by increasing the step size $t_k$ until $v_k\ge -{\epsilon }_k$.

(v) The stability center, descent target and error bound keep unchanged in the loop between Steps 2 and 5

(vi) In order to establish global convergence of the algorithm, the descent target and error bound at Step 6 should be suitably updated. Some detailed rules are presented in the next section.

The following lemma summarizes some fundamental properties of Algorithm 1, whose proof is a slight modification of that in [22], Lemma 2.2.

Lemma 1.

(i) The vectors $p_f^k$ and $p_h^k$ of (12) and (14) satisfy

$$p_f^k\in \partial {\check{f}}_k\left(y^{k+1}\right)and{p}_h^k\in \partial h\left(x^{k+1}\right).$$

(17)

The linearizations ${\overline{f}}_k$, ${\overline{h}}_k$, ${\overline{F}}_k$ satisfy the following inequalities

$${\overline{f}}_k\le {\check{f}}_k,{\overline{h}}_k\le hand{\overline{F}}_k:={\overline{f}}_k+{\overline{h}}_k\le F.$$

(18)

(ii) The aggregate subgradient $p_k$ defined in (15) and the above linearization ${\overline{F}}_k$ can be expressed as follows

$$p^k=p_f^k+p_h^k=\frac{1}{t_k}({\widehat{x}}^k-x^{k+1}),$$

(19)

$${\overline{F}}_k(·)={\overline{F}}_k\left(x^{k+1}\right)+⟨p^k,·-x^{k+1}⟩.$$

(iii) The predicted descent $v_k$ and the aggregate linearization error ${ϵ}_k$ of (15) satisfy

$$v_k=t_k{\parallel p^k\parallel }^2+{ϵ}_{k}and{ϵ}_k=F_{\widehat{x}}^k-{\overline{F}}_k\left({\widehat{x}}^k\right).$$

(20)

(iv) The aggregate linearization ${\overline{F}}_k$ is also expressed

$$F_{\widehat{x}}^k-{ϵ}_k+⟨p^k,·-{\widehat{x}}^k⟩={\overline{F}}_k(·)\le F(·).$$

(21)

(v) Denote the optimality measure by

$$V_k:=max\{\parallel p^k\parallel ,{ϵ}_k+⟨p^k,{\widehat{x}}^k⟩\},$$

(22)

which satisfies

$$V_k\le max\{\parallel p^k\parallel ,{ϵ}_k\left\}\right(1+\parallel {\widehat{x}}^k\parallel )$$

(23)

and

$$F_{\widehat{x}}^k\le F\left(x\right)+V_k(1+\parallel x\parallel ),\forall x.$$

(24)

$\left(vi\right)$ We have the relations

$$v_k\ge -{ϵ}_k\iff t_k\parallel p^k{\parallel }^2/2\ge -{ϵ}_k\iff v_k\ge t_k{\parallel p^k\parallel }^2/2,v_k\ge {ϵ}_k.$$

(25)

Moreover, if $F_{\widehat{x}}^k\le {\gamma }_{\widehat{x}}^k$, then we have $-{ϵ}_k\le {\epsilon }_x^{k\left(l\right)}$ and

$$v_k\ge max\left\{\frac{t_k{\parallel p^k\parallel }^2}{2},\left|{ϵ}_k\right|\right\}if{v}_k\ge {-ϵ}_k,$$

(26)

$$V_k\le max\left\{{\left(\frac{2v_k}{t_k}\right)}^{1/2},v_k\right\}(1+\parallel {\widehat{x}}^k\parallel )if{v}_k\ge -{ϵ}_k,$$

(27)

$$V_k<{\left(\frac{2{\epsilon }_x^{k\left(l\right)}}{t_k}\right)}^{1/2}(1+\parallel {\widehat{x}}^k\parallel )if{v}_k<-{ϵ}_k.$$

(28)

Proof. 

(i) From the optimality condition of subproblem (11), we obtain

$$\begin{array}{c}\hfill 0\in \partial {\varphi }_f^k\left(y^{k+1}\right)=\partial \check{f_k}\left(y^{k+1}\right)+p_h^{k-1}+\frac{1}{t_k}(y^{k+1}-{\widehat{x}}^k)=\partial \check{f_k}\left(y^{k+1}\right)-p_f^k,\end{array}$$

which implies $p_f^k\in \partial {\check{f}}_k\left(y^{k+1}\right)$. In addition, the fact that ${\overline{f}}_k\left(y^{k+1}\right)=\check{f_k}\left(y^{k+1}\right)$ yields ${\overline{f}}_k\le \check{f_k}$. Similarly, by the optimality condition of (14), we have

$$0\in \partial {\varphi }_h^k\left(x^{k+1}\right)=p_f^k+\partial h\left(x^{k+1}\right)+\frac{1}{t_k}(x^{k+1}-{\widehat{x}}^k)=\partial h\left(x^{k+1}\right)-p_h^k,$$

which shows $p_h^k\in \partial h\left(x^{k+1}\right)$. Further from $\overline{h}\left(x^{k+1}\right)=h\left(x^{k+1}\right)$, we obtain ${\overline{h}}_k\le h$. Finally, it follows that

$${\overline{F}}_k={\overline{f}}_k+{\overline{h}}_k\le {\check{f}}_k+h\le F.$$

(ii) By (14), we obtain

$$p_f^k+p_h^k=p_f^k+\frac{1}{t_k}({\widehat{x}}^k-x^{k+1})-p_f^k=\frac{1}{t_k}({\widehat{x}}^k-x^{k+1})=p^k.$$

Utilizing the linearity of ${\overline{F}}_k(·)$, (12) and (19), we derive

$$\begin{array}{cc}\hfill {\overline{F}}_k(·)& =\overline{f_k}(·)+{\overline{h}}_k(·)\hfill \\ \hfill & ={\check{f}}_k\left(y^{k+1}\right)+⟨p_f^k,·-y^{k+1}⟩+h\left(x^{k+1}\right)+⟨p_h^k,·-x^{k+1}⟩\hfill \\ \hfill & ={\overline{f}}_k\left(x^{k+1}\right)-⟨p_f^k,x^{k+1}-y^{k+1}⟩+⟨p_f^k,·-y^{k+1}⟩+h\left(x^{k+1}\right)+⟨p_h^k,·-x^{k+1}⟩\hfill \\ \hfill & =\overline{f_k}\left(x^{k+1}\right)+⟨p_f^k,·-x^{k+1}⟩+h\left(x^{k+1}\right)+⟨p_h^k,·-x^{k+1}⟩\hfill \\ \hfill & ={\overline{F}}_k\left(x^{k+1}\right)+⟨p^k,·-x^{k+1}⟩.\hfill \end{array}$$

(iii) We obtain directly $v_k={ϵ}_k+t_k{\parallel p^k\parallel }^2$ from (15). Combining (15) and (ii), we have

$$\begin{array}{cc}\hfill {ϵ}_k& =v_k-t_k{\parallel p^k\parallel }^2\hfill \\ \hfill & =F_{\widehat{x}}^k-[{\overline{f}}_k\left(x^{k+1}\right)+h\left(x^{k+1}\right)]-t_k{\parallel p^k\parallel }^2\hfill \\ \hfill & =F_{\widehat{x}}^k-{\overline{F}}_k\left({\widehat{x}}^k\right)+⟨p^k,{\widehat{x}}^k-x^{k+1}⟩-t_k{\parallel p^k\parallel }^2\hfill \\ \hfill & =F_{\widehat{x}}^k-{\overline{F}}_k\left({\widehat{x}}^k\right).\hfill \end{array}$$

(iv) Since ${ϵ}_k=v_k-t_k\parallel p^k{\parallel }^2=F_{\widehat{x}}^k-[{\overline{f}}_k\left(x^{k+1}\right)+h\left(x^{k+1}\right)]-t_k{\parallel p^k\parallel }^2$, the aggregate lineaization ${\overline{F}}_k(·)$ satisfies

$$F_{\widehat{x}}^k-{ϵ}_k+⟨p^k,·-{\widehat{x}}^k⟩={\overline{F}}_k\left(x^{k+1}\right)+⟨p^k,·-x^{k+1}⟩={\overline{F}}_k(·)\le F(·).$$

(v) Using the Cauchy-Schwarz inequality in the definition (22) gives

$$\begin{array}{cc}\hfill V_k& =max\{\parallel p_k\parallel ,{ϵ}_k+⟨p^k,{\widehat{x}}^k⟩\}\hfill \\ \hfill & \le max\{\parallel p^k\parallel ,{ϵ}_k+\parallel p^k\parallel \parallel {\widehat{x}}^k\parallel \}\hfill \\ \hfill & \le max\{\parallel p^k\parallel ,{ϵ}_k\}+\parallel p^k\parallel \parallel {\widehat{x}}^k\parallel \hfill \\ \hfill & \le max\{\parallel p^k\parallel ,{ϵ}_k\left\}\right(1+\parallel {\widehat{x}}^k\parallel ).\hfill \end{array}$$

From (21), we have

$$\begin{array}{cc}\hfill F_{\widehat{x}}^k& \le F\left(x\right)+{ϵ}_k-⟨p^k,x-{\widehat{x}}^k⟩\hfill \\ \hfill & =F\left(x\right)+{ϵ}_k-⟨p^k,x⟩+⟨p^k,{\widehat{x}}^k⟩\hfill \\ \hfill & \le F\left(x\right)+\parallel p^k\parallel \parallel x\parallel +{ϵ}_k+⟨p^k,{\widehat{x}}^k⟩\hfill \\ \hfill & \le F\left(x\right)+max\{\parallel p^k\parallel ,{ϵ}_k+⟨p^k,{\widehat{x}}^k⟩\left\}\right(1+\parallel x\parallel )\hfill \\ \hfill & =F\left(x\right)+V_k(1+\parallel x\parallel ),\forall x.\hfill \end{array}$$

(vi) By (iii), it is easy to get (25). Next, by (18), (20) and (9), we conclude that, if $F_{\widehat{x}}^k\le {\gamma }_{\widehat{x}}^k$,

$$-{ϵ}_k={\overline{F}}_k\left({\widehat{x}}^k\right)-F_{\widehat{x}}^k\le F\left({\widehat{x}}^k\right)-F_{\widehat{x}}^k=f\left({\widehat{x}}^k\right)-f_{\widehat{x}}^k\le {\epsilon }_x^{k\left(l\right)}.$$

Relation (26) follows from the facts that $v_k\ge {ϵ}_k$ and $v_k\ge t_k{\parallel p^k\parallel }^2/2$. Relation (27) follows from (23), $\parallel p^k\parallel \le {\left(\frac{2v_k}{t_k}\right)}^{1/2}$ and ${ϵ}_k\le v_k$. Finally, if $v_k<-{ϵ}_k$, we obtain $\parallel p^k{\parallel }^2<\frac{-2{ϵ}_k}{t_k}$, which together with $-{ϵ}_k\le {\epsilon }_x^{k\left(l\right)}$ shows that $\parallel p^k\parallel <{(\frac{2{\epsilon }_x^{k\left(l\right)}}{t_k})}^{1/2}$, and therefore (28) holds. □

Remark 2.

Relation (17) shows that $p_f^k$ is a subgradient of the model function ${\check{f}}_k$ at $y^{k+1}$ and that $p_h^k$ is a subgradient of h at $x^{k+1}$. $V_k$ defined by (22) can be viewed as an optimality measure of the iterates, which will be proved to converge to zero in the next section. Relation (24) is also a test for optimality, in that ${\widehat{x}}^k$ is an approximate optimal solution to problem (1) whenever $V_k$ is sufficiently small.

4. Global Convergence

This section aims to establish the global convergence of Algorithm 1 for various oracles. These oracles are controlled by two parameters: the error bound ${\epsilon }_x$ and the descent target ${\gamma }_x$. In Table 1, we present the choices of these two parameters for different type of instances described in Section 2, including Exact (Ex), Partially Inexact (PI), Inexact (IE), Asymptotically Exact (AE) and Partially Asymptotically Exact (PAE) oracles, where the constants are selected as $\theta$, $\kappa \in (0,1)$, and ${\kappa }_{ϵ}\in (0,\kappa )$.

Table 1. The choices for the error bound and the descent target.

Instances	${\mathit{\epsilon }}_{\mathit{x}}^{\mathit{k}+1}$	${\mathit{\gamma }}_{\mathit{x}}^{\mathit{k}+1}$
Ex	0	$+\infty$
PI	0	$F_{\widehat{x}}^k-\theta \kappa v_k$
IE	$\epsilon >0$	$+\infty$
AE	${\kappa }_{ϵ}{v}_k$	$+\infty$
PAE	${\kappa }_{ϵ}{v}_k$	$F_{\widehat{x}}^k-\theta \kappa v_k$

The following lemma is crucial to guarantee the global convergence of Algorithm 1.

Lemma 2.

The descent target is always reached at the stability centers, that is, $F_{\widehat{x}}^k\le {\gamma }_{\widehat{x}}^k$ for all $k\ge 1$.

Proof. 

For instances Ex, IE and AE, since ${\gamma }_x^k=+\infty$, the claim holds immediately.

For instances PI and PAE, we have ${\gamma }_x^{k+1}=F_{\widehat{x}}^k-\theta \kappa v_k$. Thus, for $k=1$, from Step 0 it follows that ${\widehat{x}}^1=x^1$, $f_{\widehat{x}}^1=f_x^1$ and ${\gamma }_{\widehat{x}}^1={\gamma }_x^1=+\infty$. This implies $F_{\widehat{x}}^1=f_{\widehat{x}}^1+h\left({\widehat{x}}^1\right)\le {\gamma }_{\widehat{x}}^1$. In addition, for $k\ge 2$, since $\theta \in (0,1)$, once the descent test (16) is satisfied at iteration $k-1$, we have

$$F_{\widehat{x}}^k\le F_{\widehat{x}}^{k-1}-\kappa v_{k-1}\le F_{\widehat{x}}^{k-1}-\theta \kappa v_{k-1}={\gamma }_x^k={\gamma }_{\widehat{x}}^k.$$

 □

The following lemma shows that an (approximate) optimal solution can be obtained whenever the algorithm terminates finitely or loops infinitely between Steps 2 and 5.

Lemma 3.

If either Algorithm 1 terminates at the kth iteration at Step 4, or loops between Steps 2 and 5 infinitely, then

(i) 

${\widehat{x}}^k$ is an optimal solution to problem (1) for instances Ex and PI.

(ii) 

${\widehat{x}}^k$ is ε-optimal, that is, $F\left({\widehat{x}}^k\right)\le F_{\ast }+\epsilon$, for instance IE.

(iii) 

${\widehat{x}}^k$ is ${\epsilon }_x^{k\left(l\right)}$-optimal, that is, $F\left({\widehat{x}}^k\right)\le F_{\ast }+{\epsilon }_x^{k\left(l\right)}$, for instances AE and PAE.

Proof. 

Firstly, suppose that Algorithm 1 terminates at Step 4 with iteration k. Then from (23), we have $V_k=0$. This together with (24) shows that

$$F_{\widehat{x}}^k\le inf\{F\left(x\right):x\in {\mathbb{R}}^n\}=F_{\ast }.$$

(29)

Thus, from (7), we can conclude that: $F\left({\widehat{x}}^k\right)=F_{\widehat{x}}^k\le F_{\ast }$ for instances Ex and PI; $F\left({\widehat{x}}^k\right)\le F_{\widehat{x}}^k+\epsilon \le F_{\ast }+\epsilon$ for instance IE; $F\left({\widehat{x}}^k\right)\le F_{\widehat{x}}^k+{\epsilon }_x^{k\left(l\right)}\le F_{\ast }+{\epsilon }_x^{k\left(l\right)}$ for instances AE and PAE.

Secondly, suppose that Algorithm 1 loops between Steps 2 and 5 infinitely. Then from Lemma 2 and the condition at Step 5, it follows that (28) holds and $t_k\uparrow \infty$. Thus, we obtain $V_k\to 0$. This along with (24) implies (29), and therefore the claims hold by repeating the corresponding lines in first case. □

From the above lemma, in what follows, we may assume that Algorithm 1 neither terminates finitely nor loops infinitely between Steps 2 and 5. In addition, as in Reference [22], we assume that the model subgradients $p_f^k\in \partial {\check{f}}_k\left(y^{k+1}\right)$ in (17) satisfy that $\left\{p_f^k\right\}$ is bounded if $\left\{y^k\right\}$ is bounded.

Algorithm 1 must take only one of the following two cases:

(i)

the algorithm generates finitely many descent steps;

(ii)

the algorithm generates infinitely many descent steps.

We first consider case (i), in which two subcases may occur: $t_{\infty }:={lim}_k{t}_k=\infty$ and $t_{\infty }<\infty$. The first subcase of $t_{\infty }=\infty$ is analyzed in the following lemma.

Lemma 4.

Suppose that Algorithm 1 generates finitely many descent steps, that is, there exists an index $\overline{k}$ such that only null steps occur for all $k\ge \overline{k}$, and that $t_{\infty }=\infty$. Denote $\mathcal{K}:=\{k\ge \overline{k}:t_{k+1}>t_k\}$, then $V_k\to 0$ as $k\in \mathcal{K},k\to \infty$.

Proof. 

For the last time $t_k$ increases before Step 5 for $k\in \mathcal{K}$, one has

$$V_k<{\left(\frac{2{\epsilon }_x^{\overline{k}}}{t_k}\right)}^{1/2}\left(1+∥{\widehat{x}}^{\overline{k}}∥\right),$$

which along with $t_k\to \infty$ shows the lemma. □

The following lemma analyzes the second subcase of $t_{\infty }<\infty$.

Lemma 5.

Suppose that there exists $\overline{k}$ such that ${\widehat{x}}^k={\widehat{x}}^{\overline{k}}$ and $t_{\min }\le t_{k+1}\le t_k$ for all $k\ge \overline{k}$. If the descent criterion (16) fails for all $k\ge \overline{k}$, then $V_k\to 0$.

Proof. 

In view of the facts that $t_{\min }\le t_{k+1}\le t_k$ and ${\widehat{x}}^k={\widehat{x}}^{\overline{k}}$ for all $k\ge \overline{k}$, we know that only null steps occur and $t_k$ does not increase at Step 5. By Taylor’s expansion, Cauchy-Schwarz inequality, and the properties of subproblems (11) and (13), we can conclude that $v_k\to 0$, so the conclusion holds from (27). For more details, one can refer to [11], Lemma 3.2. □

By combining Lemmas 4 and 5, we have the following lemma.

Lemma 6.

Suppose that there exists $\overline{k}$ such that only null steps occur for all $k\ge \overline{k}$. Let $\mathcal{K}:=\{k\ge \overline{k}:t_{k+1}>t_k\}$ if $t_k\to \infty$; $\mathcal{K}:=\{k:k\ge \overline{k}\}$ otherwise. Then $V_k\stackrel{\mathcal{K}}{\to }0$.

Now, we can present the main convergence result for the case where the algorithm generates finitely many descent steps.

Theorem 1.

Suppose that Algorithm 1 generates finitely many descent steps, and that ${\widehat{x}}^{\overline{k}}$ is the last stability center. Then, ${\widehat{x}}^{\overline{k}}$ is an optimal solution to problem (1) for instances Ex and PI; an ε-optimal solution for IE; and an ${\epsilon }_x^{\overline{k}\left(l\right)}$-optimal solution for AE and PAE.

Proof. 

Under the stated assumption, we know that ${\widehat{x}}^k\equiv {\widehat{x}}^{\overline{k}}$ and $f_{\widehat{x}}^k\equiv f_{\widehat{x}}^{\overline{k}}$ for all $k\ge \overline{k}$. This together with (24) and Lemma 6 shows that

$$F_{\widehat{x}}^{\overline{k}}\le inf\{F\left(x\right):x\in {\mathbb{R}}^n\}=F_{\ast }.$$

Hence, similar to the proof of Lemma 3, we obtain the results of the theorem. □

Next, we consider the second case where the algorithm generates infinitely many descent steps.

Lemma 7.

Suppose that Algorithm 1 generates infinitely many descent steps, and that $F_{\widehat{x}}^{\infty }:=\underset{k}{lim}{F}_{\widehat{x}}^k>-\infty$. Let $K:=\{k:F_{\widehat{x}}^{k+1}<F_{\widehat{x}}^k\}$. Then $v_k\stackrel{K}{\to }0$ and ${\underset{¯}{lim}}_{k\in K}{V}_k=0$. Moreover, if $\left\{{\widehat{x}}^k\right\}$ is bounded, then $V_k\stackrel{K}{\to }0$.

Proof. 

From the descent test condition (16), we may first prove that $v_k\stackrel{K}{\to }0$, and therefore ${ϵ}_k,t_k{\parallel p_k\parallel }^2\stackrel{K}{\to }0$ from (26) and $\parallel p_k\parallel \stackrel{K}{\to }0$ from the fact that $t_k\ge t_{\min }$. It can be further proved that ${ϵ}_k,\parallel p_k\parallel \stackrel{K}{\to }0$, so it follows ${\underset{¯}{lim}}_{k\in K}{V}_k=0$ from the definition of $V_k$. Moreover, under the condition that $\left\{{\widehat{x}}^k\right\}$ is bounded, we have $V_k\stackrel{K}{\to }0$ by (v) of Lemma 1. For more details, one can refer to [11], Lemma 3.4. □

Finally, we present the convergence results for the second case.

Theorem 2.

Suppose that Algorithm 1 generates infinitely many descent steps, $F_{\widehat{x}}^{\infty }>-\infty$, and that the index set K is defined in Lemma 7. Then

(i) 

$F_{\ast }\le {\underset{¯}{lim}}_{k\in K}F\left({\widehat{x}}^{k+1}\right)\le {\overline{lim}}_{k\in K}F\left({\widehat{x}}^{k+1}\right)\le F_{\widehat{x}}^{\infty }+\epsilon$ for instance IE in Table 1;

(ii) 

$F_{\ast }\le {\underset{¯}{lim}}_{k\in K}F\left({\widehat{x}}^{k+1}\right)\le {\overline{lim}}_{k\in K}F\left({\widehat{x}}^{k+1}\right)\le F_{\widehat{x}}^{\infty }$ for the remaining instances in Table 1;

(iii) 

${\underset{¯}{lim}}_{k\in K}{V}_k=0$ and $F_{\widehat{x}}^k\downarrow F_{\widehat{x}}^{\infty }\le F_{\ast }$.

Proof. 

It is obvious that

$$F_{\ast }\le \underset{k\in K}{\underset{¯}{lim}}F\left({\widehat{x}}^{k+1}\right)\le \underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right).$$

(30)

(i) For instance IE, it follows that ${\epsilon }_{\widehat{x}}^{k+1}=\epsilon$ and $F_{\widehat{x}}^{k+1}\le {\gamma }_{\widehat{x}}^{k+1}=+\infty$ for all $k\in K$. Then from (7), we have $F\left({\widehat{x}}^{k+1}\right)\le F_{\widehat{x}}^{k+1}+\epsilon$, $\forall k\in K$, which implies

$$\underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right)\le \underset{k\in K}{lim}{F}_{\widehat{x}}^{k+1}+\epsilon =F_{\widehat{x}}^{\infty }+\epsilon .$$

This along with (30) shows part (i).

(ii) Next, the other four instances in Table 1 are considered separately.

For instance Ex, we have ${\epsilon }_{\widehat{x}}^{k+1}=0$, $F_{\widehat{x}}^{k+1}\le {\gamma }_{\widehat{x}}^{k+1}=+\infty$ and $F\left({\widehat{x}}^{k+1}\right)=F_{\widehat{x}}^{k+1}$. This implies

$$\underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right)=\underset{k\in K}{lim}{F}_{\widehat{x}}^{k+1}=F_{\widehat{x}}^{\infty }.$$

For instance PI, we have ${\epsilon }_{\widehat{x}}^{k+1}=0$ and ${\gamma }_{\widehat{x}}^{k+1}=F_{\widehat{x}}^k-\theta \kappa v_k$ for all $k\in K$. Thus, we obtain

$$F_{\widehat{x}}^{k+1}\le F_{\widehat{x}}^k-\kappa v_k\le F_{\widehat{x}}^k-\theta \kappa v_k={\gamma }_{\widehat{x}}^{k+1}.$$

This implies $F\left({\widehat{x}}^{k+1}\right)=F_{\widehat{x}}^{k+1}$, and therefore

$$\underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right)=\underset{k\in K}{lim}{F}_{\widehat{x}}^{k+1}=F_{\widehat{x}}^{\infty }.$$

For instance AE, we have ${\epsilon }_{\widehat{x}}^{k+1}={\kappa }_{ϵ}{v}_k$ and $F_{\widehat{x}}^{k+1}\le {\gamma }_{\widehat{x}}^{k+1}=+\infty$ for all $k\in K$, which implies

$$F\left({\widehat{x}}^{k+1}\right)\le F_{\widehat{x}}^{k+1}+{\epsilon }_{\widehat{x}}^{k+1}\le F_{\widehat{x}}^{k+1}+{\kappa }_{ϵ}{v}_k.$$

This along with Lemma 7 ($v_k\stackrel{K}{\to }0$) shows that

$$\underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right)\le \underset{k\in K}{lim}{F}_{\widehat{x}}^{k+1}=F_{\widehat{x}}^{\infty }.$$

For instance PAE, we have ${\epsilon }_{\widehat{x}}^{k+1}={\kappa }_{ϵ}{v}_k$ and ${\gamma }_{\widehat{x}}^{k+1}=F_{\widehat{x}}^k-\theta \kappa v_k$ for all $k\in K$. Then, it follows that

$$F_{\widehat{x}}^{k+1}\le F_{\widehat{x}}^k-\kappa v_k\le F_{\widehat{x}}^k-\theta \kappa v_k={\gamma }_{\widehat{x}}^{k+1},$$

which implies

$$F\left({\widehat{x}}^{k+1}\right)\le F_{\widehat{x}}^{k+1}+{\kappa }_{ϵ}{v}_k.$$

Again from Lemma 7, we obtain

$$\underset{k\in K}{\overline{lim}}F\left({\widehat{x}}^{k+1}\right)\le \underset{k\in K}{lim}{F}_{\widehat{x}}^{k+1}=F_{\widehat{x}}^{\infty }.$$

Summarizing the above analysis and noticing (30), we complete the proof of part (ii).

(iii) From Lemma 7, we know that ${\underset{¯}{lim}}_{k\in K}{V}_k=0$. This together with (24) shows part (iii). □

5. Numerical Experiments

In this section, we aim to test the numerical efficiency of the proposed algorithm. In the fields of production and transportation, finance and insurance, power industry, and telecommunications, decision makers usually need to solve problems with uncertain information. As an effective tool to solve such problems, stochastic programming (SP) has attracted more and more attention and research on its practical instances and theories; see, for example, References [41,42]. We consider a class of two-stage SP problems with fixed recourse, whose discretization of uncertainty into N scenarios has the form (see e.g., References [23,43])

$$\begin{array}{cc}min& f\left(x\right):=⟨c,x⟩+{\sum }_{i=1}^N{p}_i{V}_i\left(x\right)\hfill \\ \mathrm{s}.\mathrm{t}.& x\in X:=\{x\in {\mathbb{R}}_{+}^{n_1}:Ax=b\},\hfill \end{array}$$

(31)

where x is the first-stage decision variable, $c\in {\mathbb{R}}^{n_1}$, $A\in {\mathbb{R}}^{m_1\times n_1}$, and $b\in {\mathbb{R}}^{m_1}$. In addition, the recourse function is

$$V_i\left(x\right):=\underset{\pi \in R_{+}^{n_2}}{min}\{⟨q,\pi ⟩:W\pi =h_i-T_{i}x\},$$

where corresponding to the ith scenario $(h_i,T_i)$, with probability $p_i>0$ for $h_i\in {\mathbb{R}}^{m_2}$ and $T_i\in {\mathbb{R}}^{m_2\times n_1}$. Here $\pi$ is the second-stage decision variable.

Clearly, by introducing the indicator function ${\delta }_X$, problem (31) can be written as the form of (5), and then becomes the form of (1) by setting $h\left(x\right)={\delta }_X\left(x\right)$.

The above recourse function can be written as its dual form:

$$V_i\left(x\right)=\underset{y\in R^{m_2}}{max}⟨h_i-T_{i}x,y⟩\mathrm{s}.\mathrm{t}.W^{T}y\le q,$$

where $q\in {\mathbb{R}}^{n_2}$ and $W\in {\mathbb{R}}^{m_2\times n_2}$. By solving these linear programming problems to return solutions with precision up to a given tolerance, one can establish an inexact oracle in the form (6), see Reference [23] for more detailed description.

The instances of SP problems are downloaded from the link: http://pwp.gatech.edu/guanghui-lan/computer-codes/.

Four instances are tested, namely, SSN(50), SSN(100), 20-term(50), 20-term(100), where the integers in the brackets mean the number of scenarios N. Here, the SSN instances come from the telecommunications and have been studied by Sen, Doverspike, and Cosares [44]. And the 20-term instances come from the motor freight carrier’s problem and have been studied by Mak, Morton, and Wood [45]. The dimensions of these instances are listed in Table 2.

Table 2. Dimensions of the SP instances.

Name	${\mathit{n}}_1$	${\mathit{m}}_1$	${\mathit{n}}_2$	${\mathit{m}}_2$
SSN	89	1	706	175
20-term	63	3	764	124

The parameters are selected as: $\kappa =0.04$, $t_{min}=0.1$ and $t_1=1.1$. The maximum bundle size is set to be 35. All the tests were performed in MATLAB (R2014a) on a PC with Intel(R) Core(TM) i7-4790 CPU 3.60GHz, 4GB RAM. The quadratic programming and linear programming subproblems were solved by the MOSEK 8 toolbox for MATLAB; see http://www.mosek.com.

We first compare our algorithm (denoted by GALBM) with the accelerated prox-level method (APL) in Reference [43], where the tolerances of the linear programming solver of MOSEK are set by default. The results are listed in Table 3, in which the number of iterations (NI), the consumed CPU time in seconds (Time), and the returned minimum values ($f_{\ast }$) are compared. Note that we use the MATLAB commands tic and toc to measure the consumed CPU time. For each instance, we run 10 times and report the average CPU time. From Table 3, we see that, when similar solution quality is achieved, GALBM can significantly outperform than APL in terms of the number of iterations and CPU time.

Table 3. The comparisons between GALBM and accelerated prox-level (APL) for the stochastic programming (SP) instances.

Name	Algorithm	NI	Time	${\mathit{f}}_{\ast }$
SSN(50)	GALBM	105	28.12	4.838238
	APL	147	72.40	4.838278
SSN(100)	GALBM	95	49.58	7.352609
	APL	155	156.53	7.352618
20-term(50)	GALBM	132	47.62	$2.549453\times {10}^5$
	APL	156	106.51	$2.549453\times {10}^5$
20-term(100)	GALBM	173	128.23	$2.532875\times {10}^5$
	APL	261	364.93	$2.532876\times {10}^5$

In what follows, we are interested in evaluating the impact of inexact oracles for GALBM. In more detail, we carry out two groups of tests. The first group adopts fixed tolerances, that is, ${\epsilon }_x^{k+1}\equiv \epsilon$, and the corresponding results are reported in Table 4, Table 5, Table 6 and Table 7. Whereas the second group adopts dynamic tolerances with a safeguard parameter $\mu >0$, that is, ${\epsilon }_x^{k+1}=min\{\mu ,{\kappa }_{ϵ}{v}_k\}$ with ${\kappa }_{ϵ}=0.7$, and the corresponding results are reported in Table 8, Table 9, Table 10 and Table 11. The symbol “-” in the following tables means that the number of iterations for the corresponding instance is greater than 500.

Table 4. Numerical results for SSN(50) with fixed tolerances.

	No.	${\mathit{\epsilon }}_{\mathit{x}}$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-4}$	206	55.21	4.838157
	2	${10}^{-5}$	202	50.37	4.838163
	3	${10}^{-6}$	190	49.64	4.838247
	4	${10}^{-7}$	132	34.35	4.838156
	5	${10}^{-8}$	105	27.81	4.838238
	6	${10}^{-9}$	97	24.49	4.838188
	7	${10}^{-10}$	99	25.62	4.838136
	8	${10}^{-11}$	84	22.98	4.838191
	9	${10}^{-12}$	97	27.47	4.838128
	10	${10}^{-13}$	84	25.53	4.838231

Table 5. Numerical results for SSN(100) with fixed tolerances.

	No.	${\mathit{\epsilon }}_{\mathit{x}}$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-4}$	-	-	-
	2	${10}^{-5}$	224	109.21	7.352932
	3	${10}^{-6}$	194	94.80	7.352854
	4	${10}^{-7}$	127	62.17	7.352937
	5	${10}^{-8}$	95	46.93	7.352758
	6	${10}^{-9}$	87	43.03	7.352750
	7	${10}^{-10}$	79	41.55	7.353074
	8	${10}^{-11}$	83	43.76	7.352939
	9	${10}^{-12}$	81	44.08	7.352748
	10	${10}^{-13}$	81	46.77	7.352734

Table 6. Numerical results for 20-term (50) with fixed tolerances.

	No.	${\mathit{\epsilon }}_{\mathit{x}}$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-2}$	250	80.86	$2.549490\times {10}^5$
	3	${10}^{-3}$	144	49.12	$2.549466\times {10}^5$
	3	${10}^{-4}$	165	57.49	$2.549463\times {10}^5$
	4	${10}^{-5}$	164	54.12	$2.549460\times {10}^5$
	5	${10}^{-6}$	211	72.42	$2.549461\times {10}^5$
	6	${10}^{-7}$	178	62.19	$2.549459\times {10}^5$
	7	${10}^{-8}$	132	44.56	$2.549457\times {10}^5$
	8	${10}^{-9}$	175	61.53	$2.549461\times {10}^5$
	9	${10}^{-10}$	132	49.53	$2.549457\times {10}^5$
	10	${10}^{-11}$	258	99.69	$2.549457\times {10}^5$
	11	${10}^{-12}$	175	67.70	$2.549461\times {10}^5$
	12	${10}^{-13}$	183	71.52	$2.549461\times {10}^5$

Table 7. Numerical results for 20-term (100) with fixed tolerances.

	No.	${\mathit{\epsilon }}_{\mathit{x}}$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-2}$	227	141.74	$2.532914\times {10}^5$
	2	${10}^{-3}$	-	-	-
	3	${10}^{-4}$	140	96.02	$2.532879\times {10}^5$
	4	${10}^{-5}$	179	117.71	$2.532877\times {10}^5$
	5	${10}^{-6}$	152	99.29	$2.532879\times {10}^5$
	6	${10}^{-7}$	139	95.07	$2.532876\times {10}^5$
	7	${10}^{-8}$	173	128.44	$2.532876\times {10}^5$
	8	${10}^{-9}$	143	103.51	$2.532878\times {10}^5$
	9	${10}^{-10}$	-	-	-
	10	${10}^{-11}$	159	110.21	$2.532877\times {10}^5$
	11	${10}^{-12}$	150	112.37	$2.532878\times {10}^5$
	12	${10}^{-13}$	132	99.46	$2.532878\times {10}^5$

Table 8. Numerical results for SSN (50) with dynamic tolerances.

	No.	$\mathit{\mu }$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-3}$	-	-	-
	2	${10}^{-4}$	201	50.54	4.838163
	3	${10}^{-5}$	202	49.07	4.838163
	4	${10}^{-6}$	190	49.21	4.838247
	5	${10}^{-7}$	132	32.26	4.838156
	6	${10}^{-8}$	105	27.10	4.838238

Table 9. Numerical results for SSN (100) with dynamic tolerances.

	No.	$\mathit{\mu }$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-3}$	-	-	-
	2	${10}^{-4}$	-	-	-
	3	${10}^{-5}$	284	141.85	7.353010
	4	${10}^{-6}$	194	97.01	7.352854
	5	${10}^{-7}$	127	63.83	7.352937
	6	${10}^{-8}$	95	47.97	7.352758

Table 10. Numerical results for 20-term (50) with dynamic tolerances.

	No.	$\mathit{\mu }$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-2}$	199	65.09	$2.549485\times {10}^5$
	2	${10}^{-3}$	140	44.73	$2.549462\times {10}^5$
	3	${10}^{-4}$	165	54.08	$2.549463\times {10}^5$
	4	${10}^{-5}$	164	54.18	$2.549460\times {10}^5$
	5	${10}^{-6}$	211	70.67	$2.549461\times {10}^5$
	6	${10}^{-7}$	178	61.90	$2.549459\times {10}^5$
	7	${10}^{-8}$	132	46.94	$2.549457\times {10}^5$

Table 11. Numerical results for 20-term (100) with dynamic tolerances.

	No.	$\mathit{\mu }$	NI	Time	${\mathit{f}}_{\ast }$
	1	${10}^{-2}$	191	119.40	$2.532901\times {10}^5$
	2	${10}^{-3}$	143	92.32	$2.532881\times {10}^5$
	3	${10}^{-4}$	140	91.68	$2.532878\times {10}^5$
	4	${10}^{-5}$	179	121.12	$2.532877\times {10}^5$
	5	${10}^{-6}$	146	104.27	$2.532878\times {10}^5$
	6	${10}^{-7}$	170	120.29	$2.532879\times {10}^5$
	7	${10}^{-8}$	173	126.57	$2.532876\times {10}^5$

6. Conclusions

In this paper, we have proposed a generalized alternating linearization bundle method for solving structured convex optimization with inexact first-order oracles. Our method can handle various inexact data by making use of the so-called on-demand accuracy oracles. At each iteration, two interrelated subproblems are solved alternately, aiming to reduce the computational cost. Global convergence of the algorithm is established under different types of inexactness. Numerical results show that the proposed algorithm is promising.