Existence of Generalized Augmented Lagrange Multipliers for Constrained Optimization Problems

Abstract

The augmented Lagrange multiplier as an important concept in duality theory for optimization problems is extended in this paper to generalized augmented Lagrange multipliers by allowing a nonlinear support for the augmented perturbation function. The existence of generalized augmented Lagrange multipliers is established by perturbation analysis. Meanwhile, the relations among generalized augmented Lagrange multipliers, saddle points, and zero duality gap property are developed.

1. Introduction

This paper is concerned with the following nonlinear programming problem:

$$\begin{array}{cc}\hfill \left(P\right)& \underset{x\in \Omega }{\min }f\left(x\right)\hfill \\ & \mathrm{s}.\mathrm{t}.g_i\left(x\right)\le 0,i=1,\cdots ,m,\hfill \\ & \hfill \hspace{1em}{h}_j\left(x\right)=0,j=1,\cdots ,l,\end{array}$$

where $\Omega$ is a nonempty and closed subset in ${\mathbb{R}}^n$, $g_i\left(x\right):{\mathbb{R}}^n\to \mathbb{R}$ for $i=1,\cdots ,m$, and $h_j\left(x\right):{\mathbb{R}}^n\to \mathbb{R}$ for $j=1,\cdots ,l$ are continuous functions. For simplification of notation, let us denote $g\left(x\right):=(g_1\left(x\right),g_2\left(x\right),\cdots ,g_m\left(x\right))$, and $h\left(x\right):=(h_1\left(x\right),h_2\left(x\right),\cdots ,h_l\left(x\right)).$ Note that the feasible region of $\left(P\right)$ can be written as $\Omega \cap \mathcal{F}$, where

$$\mathcal{F}:=\left\{x\right|g_i\left(x\right)\le 0,i=1,\cdots ,m;h_j\left(x\right)=0,j=1,\cdots ,l\}.$$

The classical Lagrangian function for the problem $\left(P\right)$ is defined as

$$L(x,\lambda ,\mu ):=f\left(x\right)+\langle \lambda ,g\left(x\right)\rangle +\langle \mu ,h\left(x\right)\rangle ,(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

A non-zero duality gap maybe arise for nonconvex optimization problems when using the above Lagrangian functions. Hence some modifications are necessary to overcome this difficulty, such as the augmented Lagrangian by introducing an augmented term, or the nonlinear Lagrangian by replacing the multiplier item and augmented term together by a nonlinear function. For example, the Hestenes–Powell–Rockafellar augmented Lagrangian [1,2,3], the cubic augmented Lagrangian [4], Mangasarian’s augmented Lagrangian [5,6], the exponential penalty function [7,8], the log-sigmoid Lagrangian [9], modified barrier functions [8,10], the p-th power augmented Lagrangian [11], and nonlinear augmented Lagrangian functions [12,13,14,15]. The other related discussion on augmented Lagrangians regarding special constrained optimization includes second-order cone programming [16,17], semidefinite programming [18,19,20], cone programming [21,22,23], semi-infinite programming [24,25], min-max programming [26], distributed optimization [27], mixed integer programming [28], stochastic mixed-integer programs [29], generalized Nash equilibrium problems [30], quasi-variational inequalities [31], composite convex programming [32], and sparse discrete problems [33].

The duality theory is closely related to the perturbation of primal problem. Precisely, for a given $(y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l$, the perturbation problem of $\left(P\right)$ is

$$\begin{array}{ll}\left(P_{(y,z)}\right)& \underset{x\in \Omega }{\min }f\left(x\right)\\ & \mathrm{s}.\mathrm{t}.g_i\left(x\right)+y_i\le 0,i=1,\cdots ,m,\\ & \hspace{1em}{h}_j\left(x\right)+z_j=0,j=1,\cdots ,l.\end{array}$$

Denote by $\mathrm{val}\left(P\right)$ and $v(y,z)(:=\mathrm{val}\left(P_{(y,z)}\right))$ the optimal values of $\left(P\right)$ and $\left(P_{(y,z)}\right)$, respectively. Clearly, $v(0,0)=\mathrm{val}\left(P\right)$. Denote by $X^{*}$ the optimal solution set of problem $\left(P\right)$, and assume throughout the paper that the optimal value $\mathrm{val}\left(P\right)$ is finite.

The augmented perturbation function is

$$v_r(y,z):=v(y,z)+r\sigma (y,z),\forall (y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l.$$

(1)

Here $\sigma$ is called an augmenting function (see Section 2 below for details). Its properties are weakened from convex to level-bounded, or valley-at-zero. For example, in Rockafellar and Wets [34], a nonnegative convex augmenting function and the corresponding augmented Lagrangian dual problem of primal problem were introduced. A sufficient condition for the zero duality gap and a necessary and sufficient condition for the existence of an exact penalty representation were obtained. It was extended in [35] by replacing the convexity condition of the augmenting function with a level-boundedness condition. Using the theory of abstract convexity, a family of augmenting functions with almost peak at zero property and a class of corresponding augmented Lagrangian dual problems were introduced in [36]. Valley-at-zero property (similar to almost peak-at-zero property) was used in [37].

A vector $(\lambda ,\mu )$ is said to be an augmented Lagrange multiplier for problem $\left(P\right)$ (cf. [22,25]), if

$$v_r(y,z)\ge v_r(0,0)+\langle \lambda ,y\rangle +\langle \mu ,z\rangle ,\forall (y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l.$$

(2)

That means that $(\lambda ,\mu )$ is a subgradient of $v_r(·,·)$ at $(y,z)=(0,0)$. The set of all subgradients $(\lambda ,\mu )$ is called the subdifferential of $v_r(y,z)$ at $(y,z)=(0,0)$ and denoted by $\partial v_r(0,0)$. Augmented Lagrange multipliers are an important concept in duality theory. Their existence is important for the global convergence analysis of primal-dual type algorithms based on the use of augmented Lagrangians [7,19,29,32,33]. In addition, augmented Lagrange multipliers are closely related to saddle points, the zero duality gap property, and exact penalty representation. Some results on the existence of augmented Lagrange multipliers are discussed for semi-infinite programming [25], cone programming [22,23], and eigenvalue composite optimization problems [38]. Moreover, CQ-free duality was proposed in the classical monograph [39] by Bonnans and Shapiro. The stronger results on CQ-free strong duality for semidefinite and general convex programming can be found in [40,41], and in more recent publications for semi-infinite, semidefinite, and copositive programming by Kostyukova and others [42,43]. Recently, Dolgopolik [44] studied the existence of augmented Lagrange multipliers for geometric constraint optimization by using the localization principle.

Recall that for convex programming, Lagrangian multiplier is a subgradient of perturbation function v at $u=0$ in the sense of convex analysis; i.e.,

$$v\left(u\right)\ge v\left(0\right)+\langle \lambda ,u\rangle ,\forall u.$$

For nonconvex programming, the Lagrangian multiplier can be used to estimate the subdifferential of the perturbation function at the origin. Precisely, for a minimization problem

$$\min f\left(x\right)+\theta \left(g\right(x\left)\right),x\in X,$$

where $f:{\mathbb{R}}^n\to \mathbb{R}$, $g:{\mathbb{R}}^n\to {\mathbb{R}}^m$, X is a closed set in ${\mathbb{R}}^n$, and $\theta :{\mathbb{R}}^m\to \overline{\mathbb{R}}:=(\infty ,+\infty ]$ is proper, lsc, and convex. This model includes the constrained optimization problems (by letting $\theta$ be a indicator function) and composite optimization problems. Denote by $S^{*}$ the solution set. For $\overline{x}\in S^{*}$, let

$$M\left(\overline{x}\right):=\left\{\lambda \right|0\in \nabla f\left(\overline{x}\right)+{\lambda }^T\nabla g\left(\overline{x}\right)+N_X\left(\overline{x}\right),\lambda \in \partial \theta \left(g\left(\overline{x}\right)\right)\}$$

and

$$M^{\infty }\left(\overline{x}\right):=\left\{\lambda \right|0\in {\lambda }^T\nabla g\left(\overline{x}\right)+N_X\left(\overline{x}\right),\lambda \in N_{\mathrm{dom}\theta }\left(g\left(\overline{x}\right)\right)\}.$$

If X is regular and $M^{\infty }\left(\overline{x}\right)=\left\{0\right\}$ for every $\overline{x}\in S^{*}$, then

$$\partial v\left(0\right)\subset \bigcup _{\overline{x}\in S^{*}}M\left(\overline{x}\right),\mathrm{lip}v\left(0\right)\le \underset{\genfrac{}{}{0pt}{}{\lambda \in M\left(\overline{x}\right)}{\overline{x}\in S^{*}}}{max}\parallel \lambda \parallel .$$

(3)

It should be pointed out that the subdifferential that appeared in (3) is the limiting/Mordukhovich subdifferential, not a subdifferential in the sense of convex analysis. Here $M^{\infty }\left(\overline{x}\right)=\left\{0\right\}$ can be regarded as constraint qualification. In particular, if $\mathrm{dom}\theta :={\mathbb{R}}_{-}^l\times {\left\{0\right\}}^l$ and $X:={\mathbb{R}}^n$, then this condition is Mangasarian–Fromovitz constraint qualification; if $\mathrm{dom}\theta$ is a convex cone with nonempty interior and $X:={\mathbb{R}}^n$, then this condition is Robinson’s constraint qualification. The result (3) indicates that the Lagrangian multiplier provides an upper bound on the subdifferential of perturbation function and gives an estimate on the Lipschitz constant of perturbed function. It is very important for the convergence analysis of numerical algorithms.

Compared with the classical Lagrangian function, the augmented Lagrangian function has been successfully applied to study nonconvex programming. Hence an interesting question is how to use the augmented Lagrangian multiplier to study the subdifferential of $v_{\tau }$, and further give an estimate on Lipschitz constant on $v_{\tau }$. On subdifferentiability in nonconvex setting, Clarke’s pioneering work on generalized gradient opened the door to the study of general nonsmooth functions. Many concepts were introduced in the past few decades. Frequently used concepts include limiting/Mordukhovich subdifferential, Ioffe’s approximate and G-subdifferential, Michel and Penot’s subdifferential, Treiman’s linear subdifferential, Sussmann’s semidifferential, etc. Compared with the abstract subdifferential (pioneered by Warga), which is defined by a set of axioms, many subdifferentials have reasonable geometric explanations. For example, a convex subdifferential means a linear support, Frech $\acute{\mathrm{e}}$t subdifferential means a smooth support, and a proximal subdifferential means a local quadratic support. The detailed discussion on other subdifferentials and their properties (particularly on calculus rules and the robust property) can be found in [34].

Clearly, the definition of an augmented Lagrangian multiplier given in (2) indicates that the augmented perturbation function is supported by a linear function at the origin. It corresponds to the subdifferential in the convex analysis. However, for a nonconvex setting, it is natural to consider whether a nonlinear support is available. Once it is done, we can establish and apply the duality theory in a more flexible environment. Define $\omega :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ such that $\omega \left(\eta \right)\to +\infty$ as $\eta \to +\infty$.

Definition 1.

A vector $(\lambda ,\mu )$ is said to be a generalized augmented Lagrange multiplier of $\left(P\right)$, if there exists $r\ge 0$ such that

$$v_r(y,z)\ge v_r(0,0)+{\varphi }_1(\lambda ,y)+{\varphi }_2(\mu ,z),\forall (y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l,$$

(4)

where ${\varphi }_i$ for $i=1,2$ possesses the following properties:

(A₁) 

${\varphi }_i$ is continuous and ${\varphi }_i(·,0)=0$;

(A₂) 

${\varphi }_i(x,y+z)\le {\varphi }_i(x,y)+{\varphi }_i(x,z)$;

(A₃) 

$\forall x\notin \mathcal{F}$, there exist a nonzero vector $(u_0,v_0)$ and $\gamma <0$ such that

$${\varphi }_1(\eta u_0,y)+{\varphi }_2(\eta v_0,z)\le \omega \left(\eta \right)\gamma ,$$

whenever $(y,z)$ satisfies $y+g\left(x\right)\le 0,z+h\left(x\right)=0,$ and $\eta >0$ is sufficiently large.

Since ${\varphi }_i$ includes the inner product as special cases, (4) is an essential extension of (2) from linear support to nonlinear support.

As mentioned above, the augmented Lagrange multiplier is a subgradient (in the sense of convex analysis) of an augmented perturbation function at the origin. That means the augmented perturbation function has a linear support. The augmented Lagrange multiplier is extended in this paper to a new concept called the generalized augmented Lagrangian multiplier, in which a nonlinear support is allowed. The main aim of this paper is to study the existence of generalized augmented Lagrange multipliers. It helps us to better understand properties of an augmented perturbation function at the origin. Based on this nonlinear support, we need to re-investigate the corresponding duality theory, particularly be discussing the relations among generalized augmented Lagrange multipliers, saddle points, and the zero duality gap property. The existence of generalized augmented Lagrange multipliers is established by perturbation analysis of the primal problem.

We organize our paper as follows. Section 2 introduces the preliminaries. In Section 3, we present the duality theory based on generalized augmented Lagrangians. Section 4 discusses the existence of generalized augmented Lagrange multipliers by perturbation analysis.

2. Preliminaries

In this section we clarify the notation, recall some background materials we need from duality theory, and develop some preliminary results.

Recall that

$$v_r(y,z):=v(y,z)+r\sigma (y,z),\forall (y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l.$$

where $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}:=[0,+\infty )$ satisfies the following valley-at-zero property:

(i)

$\sigma$ is continuous at 0 with $\sigma (0,0)=0$;

(ii)

$\inf \left\{\sigma (y,z)\right|\parallel (y,z)\parallel \ge \eta ,y\in {\mathbb{R}}^m,z\in {\mathbb{R}}^l\}>0$ for all $\eta >0$.

The definition of the growth condition defined below was introduced in [23], as an extension of the one given in [3], where the augmenting function is restricted to be a quadratic function.

Definition 2.

A function $v(y,z)$ is said to satisfy the growth condition with σ, if for any $\tau >0$, there exist $a,c\in \mathbb{R}$ such that

$$v(y,z)\ge c-a\sigma (y,z),\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}},$$

where ${\mathbb{B}}_{{\mathbb{R}}^{m+l}}$ denotes the closed unit ball in ${\mathbb{R}}^{m+l}$.

The dualizing parametrization function of the primal problem is defined as

$$\begin{array}{c}\hfill F(x,y,z):=\left\{\begin{array}{cc}f\left(x\right),\hfill & \mathrm{if}x\in \Omega \mathrm{and}y+g\left(x\right)\le 0,z+h\left(x\right)=0,\hfill \\ +\infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(5)

For $(x,\lambda ,\mu )\in {\mathbb{R}}^n\times {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$, the corresponding generalized augmented Lagrangian is

$$L(x,\lambda ,\mu ,r):=\inf \left\{F(x,y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r\sigma (y,z)|(y,z)\in {\mathbb{R}}^{m+l}\right\}.$$

(6)

The generalized Lagrangian function is defined as

$$L_0(x,\lambda ,\mu ):=f\left(x\right)-{\varphi }_1\left(\lambda ,-g\left(x\right)\right)-{\varphi }_2\left(\mu ,-h\left(x\right)\right),$$

which reduces to the classical Lagrangian of (P) when ${\varphi }_1(\lambda ,y)=\langle \lambda ,y\rangle$ and ${\varphi }_2(\mu ,z)=\langle \mu ,z\rangle$.

If in particular $x\in \Omega$, the generalized augmented Lagrangian can be rewritten as

$$\begin{array}{lll}L(x,\lambda ,\mu ,r)& =& \underset{z+h\left(x\right)=0}{\underset{y+g\left(x\right)\le 0}{\inf }}\left\{f\left(x\right)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r\sigma (y,z)\right\}\\ & \ge & \underset{{\xi }_1\le 0,{\xi }_2=0}{\inf }\{f\left(x\right)-{\varphi }_1\left(\lambda ,-g\left(x\right)\right)-{\varphi }_2\left(\mu ,-h\left(x\right)\right)-{\varphi }_1(\lambda ,{\xi }_1)\\ & & \hspace{1em}\hspace{1em}-{\varphi }_2(\mu ,{\xi }_2)+r\sigma \left({\xi }_1-g\left(x\right),{\xi }_2-h\left(x\right)\right)\}\end{array}$$

(7)

$$\begin{array}{rrr}& =& \underset{{\xi }_1\le 0,{\xi }_2=0}{\inf }\left\{L_0(x,\lambda ,\mu )-{\varphi }_1(\lambda ,{\xi }_1)+r\sigma \left({\xi }_1-g\left(x\right),{\xi }_2-h\left(x\right)\right)\right\},\end{array}$$

(8)

where the inequality comes from $\left(A_2\right)$.

Definition 3.

A solution $(x^{*},{\lambda }^{*},{\mu }^{*})\in \Omega \times {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$ is said to be a global saddle point of the generalized augmented Lagrangian L for $r\ge 0$, if

$$L(x^{*},\lambda ,\mu ,r)\le L(x^{*},{\lambda }^{*},{\mu }^{*},r)\le L(x,{\lambda }^{*},{\mu }^{*},r),\forall x\in \Omega ,(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(9)

If the above inequalities hold for all $x\in {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )\cap \Omega$, where ${\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )$ denotes the ball with center $x^{*}$ and radius $\delta >0$, then $(x^{*},{\lambda }^{*},{\mu }^{*})$ is said to be a local saddle point of L.

The generalized augmented Lagrangian dual problem of $\left(P\right)$ is defined as

$$\begin{array}{cc}\hfill \left(D\right)& \underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r),\hfill \end{array}$$

where $\theta (\lambda ,\mu ,r)$ is the generalized augmented Lagrangian dual function given as

$$\begin{array}{c}\hfill \theta (\lambda ,\mu ,r):=\inf \left\{L\right(x,\lambda ,\mu ,r\left)\right|x\in \Omega \}.\end{array}$$

(10)

Taking into account of (7) and (10), we have

$$\begin{array}{cc}& \underset{(y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l}{\inf }\left\{v_r(y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)\right\}\hfill \\ =& \underset{(y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l}{\inf }\underset{x\in \Omega }{\inf }\left\{f\left(x\right)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r\sigma (y,z)|g\left(x\right)+y\le 0,h\left(x\right)+z=0\right\}\hfill \\ =& \underset{x\in \Omega }{\inf }\underset{y+g\left(x\right)\le 0,z+h\left(x\right)=0}{\inf }\left\{f\left(x\right)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r\sigma (y,z)\right\}\hfill \\ =& \underset{x\in \Omega }{\inf }L(x,\lambda ,\mu ,r)\hfill \\ =& \theta (\lambda ,\mu ,r).\hfill \end{array}$$

(11)

In addition, it also follows from (5) that

$$\begin{array}{c}\hfill v(y,z)=\underset{\genfrac{}{}{0pt}{}{x\in \Omega }{g\left(x\right)+y\le 0,h\left(x\right)+z=0}}{\inf }f\left(x\right)=\underset{x\in \Omega }{\inf }F(x,y,z).\end{array}$$

It is well known that a zero duality gap between the problem $\left(P\right)$ and its generalized augmented Lagrangian dual problem $\left(D\right)$ holds if

$$\mathrm{val}\left(P\right)=\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r).$$

For $r\ge 0$, consider the following r-dual problem of $\left(P\right)$, denoted by $\left(D_r\right)$,

$$\begin{array}{cc}\hfill \left(D_r\right)& \underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }\theta (\lambda ,\mu ,r)=\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }\underset{x\in \Omega }{\inf }L(x,\lambda ,\mu ,r).\hfill \end{array}$$

Similarly, if for some fixed $r\ge 0$ such that

$$\mathrm{val}\left(P\right)=\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }\theta (\lambda ,\mu ,r),$$

then the zero duality gap property holds for the pair of problems $\left(P\right)$ and $\left(D_r\right)$.

Define the optimal values of problems $\left(D\right)$ and $\left(D_r\right)$ by $\mathrm{val}\left(D\right)$ and $\mathrm{val}\left(D_r\right)$, respectively. It is clear that $\mathrm{val}\left(D\right)={\sup }_{r\in {\mathbb{R}}_{+}}\mathrm{val}\left(D_r\right).$

3. Duality Theory Based on Generalized Augmented Lagrangian Functions

In this section, we study the relationships among generalized augmented Lagrange multipliers, global saddle points, and the zero duality gap property between the primal problem and its generalized augmented Lagrangian dual problem. The related conclusions are given in Theorem 3 and Theorem 4.

Firstly, the weak duality theorem is given below, which shows that the dual problem provides a lower bound for $\left(P\right)$.

Proposition 1.

Let x be a feasible point of $\left(P\right)$ and $(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}$. Then

$$\theta (\lambda ,\mu ,r)\le val\left(P\right)\le f\left(x\right).$$

Proof. 

Since x is feasible, i.e., $x\in \Omega$ and $g\left(x\right)\le 0,h\left(x\right)=0$, then $-g\left(x\right)\ge 0,-h\left(x\right)=0$. So

$$L(x,\lambda ,\mu ,r)=\underset{y\le -g\left(x\right),z=-h\left(x\right)}{\inf }\left\{f\left(x\right)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r\sigma (y,z)\right\}\le f\left(x\right),$$

(12)

where the inequality follows by letting $y=0,z=0$ and $\varphi (·,0)=0.$ Hence

$$\theta (\lambda ,\mu ,r)=\underset{x\in \Omega }{\inf }L(x,\lambda ,\mu ,r)\le f\left(x\right).$$

The arbitrariness of x ensures

$$\theta (\lambda ,\mu ,r)\le \underset{\genfrac{}{}{0pt}{}{x\in \Omega }{g\left(x\right)\le 0,h\left(x\right)=0}}{\inf }f\left(x\right)=val\left(P\right).$$

(13)

□

Theorem 1.

Let $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ and $({\lambda }^{*},{\mu }^{*},r^{*})\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}$. Then $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$ with $r^{*}$ if and only if $({\lambda }^{*},{\mu }^{*},r^{*})$ is an optimal solution of $\left(D\right)$ and the zero duality gap property holds for problems $\left(P\right)$ and $\left(D\right)$.

Proof. 

(Necessity). If $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$ with $r^{*}$, then

$$v_{r^{*}}(0,0)=\underset{(y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l}{\inf }\left\{v_{r^{*}}(y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\right\},$$

where the above equation is due to Definition 1. According to (11), we have

$$val\left(P\right)=v_{r^{*}}(0,0)=\theta ({\lambda }^{*},{\mu }^{*},r^{*}).$$

This implies

$$val\left(P\right)=\theta ({\lambda }^{*},{\mu }^{*},r^{*})\le \underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }\theta (\lambda ,\mu ,r^{*})\le val\left(D\right)=\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)\le val\left(P\right),$$

where the third inequality is due to (13). Hence, $({\lambda }^{*},{\mu }^{*},r^{*})$ is an optimal solution of $\left(D\right)$ and

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)=val\left(P\right).$$

(Sufficiency). Suppose $({\lambda }^{*},{\mu }^{*},r^{*})$ is an optimal solution of $\left(D\right)$ and the zero duality gap property between $\left(P\right)$ and $\left(D\right)$ holds. Then

$$val\left(P\right)=\theta ({\lambda }^{*},{\mu }^{*},r^{*})\le val\left(D_{r^{*}}\right)\le val\left(D\right)\le val\left(P\right).$$

Hence

$$\theta ({\lambda }^{*},{\mu }^{*},r^{*})=val\left(D_{r^{*}}\right)=val\left(P\right)=v(0,0)=v_{r^{*}}(0,0),$$

which together with (11) implies

$$v_{r^{*}}(0,0)=\theta ({\lambda }^{*},{\mu }^{*},r^{*})=\underset{(y,z)\in {\mathbb{R}}^m\times {\mathbb{R}}^l}{\inf }\left\{v_{r^{*}}(y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\right\}.$$

Therefore, $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$ with $r^{*}$. □

From the proof of Theorem 1, we can see that $({\lambda }^{*},{\mu }^{*})$ is an optimal solution of $\left(D_{r^{*}}\right)$ and the zero duality gap property holds between $\left(P\right)$ and $\left(D_{r^{*}}\right)$. It should be emphasized that the existence of generalized augmented Lagrange multipliers does not require that the primal problem $\left(P\right)$ must be solvable. Indeed, in general, the optimal solution of a primal problem cannot be known in advance. The relation between the zero duality gap property and global saddle points is given below.

Theorem 2.

Let $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ and $r^{*}\ge 0$. Then $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a global saddle point of $L(x,\lambda ,\mu ,r^{*})$ if and only if $\mathrm{val}\left(P\right)=\mathrm{val}\left(D_{r^{*}}\right)$, and $x^{*}\in \Omega ,$$({\lambda }^{*},{\mu }^{*})\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$ are optimal solutions of $\left(P\right)$ and $\left(D_{r^{*}}\right)$, respectively.

Proof. 

We first claim that

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})=\left\{\begin{array}{cc}f\left(x\right),\hfill & x\in \Omega \mathrm{and}g\left(x\right)\le 0,h\left(x\right)=0,\hfill \\ +\infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(14)

Consider the following two cases:

Case 1.x is infeasible. Then either $x\notin \Omega$ or $x\in \Omega$ while $x\notin \mathcal{F}$. If $x\notin \Omega$, from (5) and (6) we get

$$L(x,\lambda ,\mu ,r^{*})=+\infty ,\forall (\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(15)

If $x\in \Omega$, but $x\notin \mathcal{F}$, it follows from the property $\left(A_3\right)$ that there exist nonzero $({\lambda }_0,{\mu }_0)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$ and $\gamma <0$ such that

$$\begin{array}{c}\hfill {\varphi }_1(\eta {\lambda }_0,y)+{\varphi }_2(\eta {\mu }_0,z)\le \omega \left(\eta \right)\gamma ,\end{array}$$

(16)

whenever $(y,z)$ satisfies $y+g\left(x\right)\le 0,z+h\left(x\right)=0$, and $\eta >0$ sufficiently large. Hence

$$\begin{array}{lll}L(x,\eta {\lambda }_0,\eta \mu ,r^{*})& =& \underset{y+g\left(x\right)\le 0,z+h\left(x\right)=0}{\inf }\left\{f\left(x\right)-{\varphi }_1(\eta {\lambda }_0,y)-{\varphi }_2(\eta \mu ,z)+r^{*}\sigma (y,z)\right\}\hfill \\ & \ge & \underset{y+g\left(x\right)\le 0,z+h\left(x\right)=0}{\inf }\left\{f\left(x\right)-{\varphi }_1(\eta {\lambda }_0,y)-{\varphi }_2(\eta {\mu }_0,z)\right\}\\ & \ge & f\left(x\right)-\omega \left(\eta \right)\gamma ,\end{array}$$

where the first inequality comes from the nonnegativity of $\sigma$, and the second inequality is due to (16). This together with $\gamma <0$ further implies that

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})\ge L(x,\eta {\lambda }_0,\eta {\mu }_0,r^{*})\ge f\left(x\right)-\omega \left(\eta \right)\gamma \to +\infty ,as\eta \to +\infty ;$$

i.e.,

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})=+\infty .$$

(17)

Therefore, either $x\notin \Omega$ or $x\in \Omega ,x\notin \mathcal{F},$ so it follows from (15) and (17) that

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})=+\infty .$$

(18)

Case 2.x is feasible i.e., $x\in \Omega$ and $g\left(x\right)\le 0,h\left(x\right)=0$. In this case, it follows from (12) that for any $(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l,$

$$L(x,\lambda ,\mu ,r^{*})=\underset{y\le -g\left(x\right),z=-h\left(x\right)}{\inf }\{f\left(x\right)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)+r^{*}\sigma (y,z)\}\le f\left(x\right).$$

(19)

According to the nonnegativity of $\sigma$, we also have

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})\ge L(x,0,0,r^{*})\ge f\left(x\right),$$

which together with (19) means that

$$\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*})=f\left(x\right).$$

(20)

Putting (18) and (20) together yields the desired formula (14). Hence

$$val\left(P\right)=\underset{x\in \Omega }{\inf }\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x,\lambda ,\mu ,r^{*}).$$

On the other hand, note that the dual problem can be rewritten as

$$val\left(P\right)=\underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }\underset{x\in \Omega }{\inf }L(x,\lambda ,\mu ,r^{*}).$$

The desired result follows by applying the minimax relations theorem (Theorem 11.50 [34]). □

Indeed, Theorem 2 shows that $\mathrm{val}\left(P\right)=\mathrm{val}\left(D\right)$, and $x^{*}$, $({\lambda }^{*},{\mu }^{*},r^{*})$ are optimal solutions of $\left(P\right)$ and $\left(D\right)$ respectively, provided that $\mathrm{val}\left(D\right)=\mathrm{val}\left(D_r\right)$, i.e., $\mathrm{val}\left(D\right)={\sup }_{r\in {\mathbb{R}}_{+}}\mathrm{val}\left(D_r\right)$ by Proposition 1, and the maximum can be attained at some r. The converse statement obviously holds true. As just mentioned above, compared with the existence of augmented Lagrange multipliers, global saddle points require that the primal problem is solvable.

Theorem 3.

Suppose that $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ has a valley at zero, v satisfies the growth condition with σ, and

$$\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)<+\infty .$$

(21)

The following statements hold:

(i)

$$\underset{(0,0,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (0,0,r)=\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z);$$

(ii)

v is lower semi-continuous at the origin if and only if the zero duality gap property holds for problems $\left(P\right)$ and $\left(D\right)$.

Proof .

(i). First, according to the condition (21) we show that

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)=\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z).$$

(22)

Assume that $\left\{(y^{\left(s\right)},z^{\left(s\right)})\right\}$ is the sequence such that the liminf in (21) is attained; i.e.,

$$\underset{s\to \infty }{lim}(y^{\left(s\right)},z^{\left(s\right)})=0,\underset{s\to \infty }{lim}v\left((y^{\left(s\right)},z^{\left(s\right)})\right)=\underset{s\to 0}{lim\; inf}v(y,z).$$

(23)

Consider the following two cases:

Case 1.$\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)=-\infty$. For $(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}$, it follows from (11) that

$$\begin{array}{lll}\theta (\lambda ,\mu ,r)& =& \underset{(y,z)\in {\mathbb{R}}^{m+l}}{\inf }\left\{v_r(y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)\right\}\\ & \le & v(y^{\left(s\right)},z^{\left(s\right)})+r\sigma (y^{\left(s\right)},z^{\left(s\right)})-{\varphi }_1(\lambda ,y^{\left(s\right)})-{\varphi }_2(\mu ,z^{\left(s\right)}),\end{array}$$

(24)

where the inequality comes from (1). Passing to limit (24), together with (23), we get

$$\theta (\lambda ,\mu ,r)\le \underset{s\to \infty }{lim}\left\{v(y^{\left(s\right)},z^{\left(s\right)})+r\sigma (y^{\left(s\right)},z^{\left(s\right)})-{\varphi }_1(\lambda ,y^{\left(s\right)})-{\varphi }_2(\mu ,z^{\left(s\right)})\right\}=\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)=-\infty ,$$

where the first equality comes from the continuity of $\sigma$ and $\varphi$ by $\left(A_1\right)$. Hence

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)=\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z).$$

Case 2.$\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)>-\infty$. Noting that $\theta (\lambda ,\mu ,r)\le \underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)$, then

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)\le \underset{(y,z)\to (0,0)}{lim\; inf}v(y,z).$$

(25)

Conversely, take k satisfying $\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)>k$. Then there exists $\tau >0$ such that

$$v(y,z)+r\sigma (y,z)\ge v(y,z)\ge k,\forall (y,z)\in \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}},r\ge 0,$$

(26)

where the first inequality follows from the nonnegativity of $\sigma$. Since $\sigma$ has a valley at zero, there exists $\epsilon >0$ such that

$$\sigma (y,z)\ge \epsilon ,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(27)

Using the growth condition of v with $\sigma$, for the above $\tau >0$ there exist $a,c\in \mathbb{R}$ such that

$$v(y,z)\ge c-a\sigma (y,z),\forall \tau >0,(y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

This together with (27) yields

$$v(y,z)+r\sigma (y,z)\ge c-a\sigma (y,z)+r\sigma (y,z)\ge c+(r-a)\epsilon \ge k,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}},r\ge r_0,$$

(28)

where $r_0:=max\{a+(k-c)/\epsilon ,0\}+1$. From (26) and (28) we get

$$\theta (0,0,r)=\underset{(y,z)\in {\mathbb{R}}^{m+l}}{\inf }\left\{v(y,z)+r\sigma (y,z)\right\}\ge k,\forall r\ge r_0,$$

and

$$\underset{r\in {\mathbb{R}}_{+}}{\sup }\theta (0,0,r)\ge k.$$

Since $k<\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)$ is arbitrary, then

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)\ge \underset{(y,z)\to (0,0)}{lim\; inf}v(y,z).$$

Taking into account (25), we get in the last inequality

$$\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)=\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z).$$

The statement (i) follows by letting $(\lambda ,\mu )=(0,0)$ in (24) and (25) and by a similar argumentation as above.

(ii). If v is lower semi-continuous at origin, then

$$\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)\ge v(0,0)=val\left(P\right),$$

which together with (22) yields

$$val\left(P\right)\le \underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)=\underset{(\lambda ,\mu ,r)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l\times {\mathbb{R}}_{+}}{\sup }\theta (\lambda ,\mu ,r)\le val\left(P\right).$$

Therefore, the zero duality gap property holds for $\left(P\right)$ and $\left(D\right)$.

Conversely, according to (22), it is easy to see that the lower semi-continuity of v at the origin can be obtained if the zero duality gap property holds for problems (P) and (D). □

Corollary 1.

Suppose that $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ has a valley at zero, v satisfies the growth condition with σ, and

$$\underset{(y,z)\to (0,0)}{lim\; inf}v(y,z)<+\infty .$$

If v is lower semi-continuous at origin and $r^{*}\in arg{\sup }_{r>0}\theta (0,0,r)$, then the following statements hold:

(i)

$(0,0)$ is a generalized augmented Lagrange multiplier;

(ii)

If the primal problem (P) has the optimal solution $x^{*}$, then $(x^{*},0,0)$ and $(x^{*},0,0,r^{*})$ are saddle points of $D_{r^{*}}$ and D, respectively.

Proof. 

The results follow immediately from Theorem 3. □

Theorem 3 shows that the zero duality gap property is closely related with the lower semi-continuity of the perturbation function. In the definition of generalized augmented Lagrange multipliers, the inequality involved in (4) is required to be satisfied for all $(y,z)\in {\mathbb{R}}^{m+l}$, but Theorem 4 shows that this restriction can be weakened by just checking all $(y,z)$ in some neighborhood of the origin once some additional assumptions are imposed on augmented functions. In the following, we further require the $\varphi$ satisfying the following property:

$\left(A_4\right)$ For any $(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$, there exist $\rho >0,\tau >0$ such that

$$\rho \sigma (y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)\ge 0,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

Theorem 4.

Suppose that σ has a valley at zero and v satisfies the growth condition with σ. Then (P) has a generalized augmented Lagrange multiplier $({\lambda }^{*},{\mu }^{*})\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l$ if and only if there exists $r^{*}\in {\mathbb{R}}_{+}$ such that

$$v_{r^{*}}(y,z)\ge v_{r^{*}}(0,0)+{\varphi }_1({\lambda }^{*},y)+{\varphi }_2({\mu }^{*},z),\forall (y,z)\in \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(29)

Proof. 

(Necessity). The necessity is clear by the definition of generalized augmented Lagrange multiplier.

(Sufficiency). Since v satisfies the growth condition with $\sigma$, then for any $\tau >0$ there exist $a,c\in \mathbb{R}$ such that

$$v(y,z)\ge c-a\sigma (y,z),\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(30)

Since $\sigma$ has a valley at zero, there exists $d>0$ such that

$$\sigma (y,z)\ge d,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(31)

Combining the property $\left(A_4\right)$ with (31) means that for any $(y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}$ we have

$$\begin{array}{ll}{v}_r(y,z)& -v_r(0,0)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\\ \ge & c-a\sigma (y,z)-v_r(0,0)+r\sigma (y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\hfill \\ =& c-\mathrm{val}\left(P\right)+(r-a-\rho )\sigma (y,z)+\left[\rho \sigma (y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\right]\hfill \\ \ge & c-\mathrm{val}\left(P\right)+(r-a-\rho )\sigma (y,z)\hfill \\ \ge & c-\mathrm{val}\left(P\right)+(r-a-\rho )d.\hfill \end{array}$$

Pick $r>max\left\{r^{*},a+\rho ,a+\rho -\left[\left(c-\mathrm{val}\left(P\right)\right)/d\right]\right\}$. Then

$$v_r(y,z)-v_r(0,0)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\ge 0,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(32)

It follows from (29) and (32) that

$$v_r(y,z)\ge v_r(0,0)+{\varphi }_1({\lambda }^{*},y)+{\varphi }_2({\mu }^{*},z),\forall (y,z)\in {\mathbb{R}}^{m+l}.$$

Hence $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$. □

Here we list two classes of nonlinear functions satisfying the above assumptions $\left(A_1\right)$–$\left(A_4\right)$.

(1) Let $\theta :\mathbb{R}\to \mathbb{R}$ be sublinear, continuous, and increasing with $\theta \left(0\right)=0$.

Let

$$\varphi (x,y):=\parallel x\parallel \theta \left(x^{T}y\right).$$

(1-1) $\varphi (x,0)=\parallel x\parallel \theta \left(0\right)=0$, $\forall x$.

(1-2) For any $x,y,z$,

$$\begin{array}{lll}\varphi (x,y+z)& =& \parallel x\parallel \theta \left(x^T(y+z)\right)=\parallel x\parallel \theta (x^{T}y+x^{T}z)\\ & \le & \parallel x\parallel \theta \left(x^{T}y\right)+\parallel x\parallel \theta \left(x^{T}z\right)=\varphi (x,y)+\varphi (x,z).\hfill \end{array}$$

(1-3) For any $x\notin \mathcal{F}$, then $(g\left(x\right),h\left(x\right))\notin {\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l$, i.e., $(0,0)\notin {\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l-(g\left(x\right),h\left(x\right))$. If $0\notin {\mathbb{R}}_{-}^m-g\left(x\right)$, then according to convex set sperate theorem, there exist a nonzero vector $u_0$ and $\xi <0$ such that $u_0^{T}y<\xi$ whenever $y+g\left(x\right)\le 0$. Hence taking $v_0:=0$ and $\gamma :=\parallel u_0\parallel \theta \left(\xi \right)$, we have

$$\begin{array}{lll}{\varphi }_1(\eta u_0,y)+{\varphi }_2(\eta v_0,z)& =& {\varphi }_1(\eta u_0,y)=\eta \parallel u_0\parallel \theta \left(\eta u_0^{T}y\right)\\ & \le & \eta \parallel u_0\parallel \theta \left(\eta \xi \right)\le \eta \parallel u_0\parallel \theta \left(\xi \right)=\omega \left(\eta \right)\gamma ,\end{array}$$

where $\omega \left(\eta \right):=\eta$. Similarly, if $0\notin {\left\{0\right\}}^l-h\left(x\right)$, there exists a nonzero vector $v_0$ and $\xi <0$ such that $v_0^{T}z\le \xi$ for $z=-h\left(x\right)$. Hence taking $u_0:=0$ and $\gamma :=\parallel v_0\parallel \theta \left(\xi \right)$, we have

$${\varphi }_1(\eta u_0,y)+{\varphi }_2(\eta v_0,z)={\varphi }_2(\eta v_0,z)=\eta \parallel v_0\parallel \theta \left(\eta v_0^{T}z\right)\le \eta \parallel v_0\parallel \theta \left(\xi \right)=\omega \left(\eta \right)\gamma .$$

(1-4) Let $\beta :{\mathbb{R}}_{+}\to \mathbb{R}$ satisfy $\beta \left(t\right)>0$ as $t>0$. Assume that there exist $\alpha >0$, $\tau >0$ such that for all $t>\alpha$ and u with $\parallel u\parallel \ge \tau$, we have $\theta (t\parallel u\parallel )\le \beta \left(t\right)\sigma \left(u\right)$.

For any $(\lambda ,\mu )$, letting $\varrho :=\parallel \lambda \parallel +\parallel \mu \parallel +\alpha$ we have

$$\begin{array}{lll}{\varphi }_1(\lambda ,y)+{\varphi }_2(\mu ,z)& =& \parallel \lambda \parallel \theta \left({\lambda }^{T}y\right)+\parallel \mu \parallel \theta \left({\mu }^{T}z\right)\le \parallel \lambda \parallel \theta (\parallel \lambda \parallel \parallel y\parallel )+\parallel \mu \parallel \theta (\parallel \mu \parallel \parallel z\parallel )\\ & \le & \varrho \theta (\varrho \parallel y\parallel )+\varrho \theta (\varrho \parallel z\parallel )\le 2\varrho \theta (\varrho \parallel (y,z)\parallel ).\end{array}$$

As $\parallel (y,z)\parallel \ge \tau$, we have $2\varrho \theta (\varrho \parallel (y,z)\parallel )\le 2\varrho \beta \left(\varrho \right)\sigma (y,z)$. Hence for all $\parallel (y,z)\parallel \ge \tau$ and $\rho :=2\varrho \beta \left(\varrho \right),$

$$\rho \sigma (y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)\ge 0.$$

In particular, we can take $\theta$ as a piecewise linear function or a support function over a bounded closed interval, $w\left(t\right):=t$, $\beta \left(t\right):=t^2$, and $\sigma \left(u\right):=\parallel u\parallel$.

(2) Let $\theta :{\mathbb{R}}_{+}\to \mathbb{R}$ satisfy $\theta \left(t\right)>0$ if $t\ne 0$ and $\theta \left(t\right)\ge t^q$ as $t>0$ are sufficiently large, where q is positive integer.

Define

$$\varphi (x,y):=\theta (\parallel x\parallel )\left(x^{T}Ay\right),$$

where A is a symmetric and invertible matrix.

(2-1) $\varphi (x,0)=0$, $\forall x$.

(2-2) For any $x,y,z$,

$$\varphi (x,y+z)=\theta (\parallel x\parallel )\left(x^{T}A(y+z)\right)=\theta (\parallel x\parallel )(x^{T}Ay+x^{T}Az)=\varphi (x,y)+\varphi (x,z).$$

(2-3) Similar to the argument given in (1-3), if $0\notin {\mathbb{R}}_{-}^m-g\left(x\right)$, then according to convex set sperate theorem, there exists a nonzero vector ${\tilde{u}}_0$ and $\xi <0$ such that ${\tilde{u}}_0^{T}y<\xi$ whenever $y+g\left(x\right)\le 0$. Hence taking $u_0:=A^{-1}{\tilde{u}}_0$, $v_0:=0$, and $\gamma :=\theta (\parallel u_0\parallel )\xi$, we have

$$\begin{array}{lll}{\varphi }_1(\eta u_0,y)+{\varphi }_2(\eta v_0,z)& =& {\varphi }_1(\eta u_0,y)=\theta (\eta \parallel u_0\parallel )\eta u_0^{T}Ay\\ & \le & {\eta }^{q+1}\parallel u_0{\parallel }^q\xi \le {\eta }^q\theta (\parallel u_0\parallel )\xi =\omega \left(\eta \right)\gamma ,\end{array}$$

whenever $\eta \ge \theta (\parallel u_0\parallel )/\parallel u_0{\parallel }^q$ and $\omega \left(\eta \right):={\eta }^q$.

If $0\notin {\left\{0\right\}}^l-h\left(x\right)$, there exists a nonzero vector ${\tilde{v}}_0$ and $\xi <0$ such that ${\tilde{v}}_0^{T}z\le \xi$ for $z=-h\left(x\right)$. Hence taking $u_0:=0$, $v_0:=A^{-1}{\tilde{v}}_0$, and $\gamma :=\theta (\parallel v_0\parallel )\xi$, we have

$$\begin{array}{lll}{\varphi }_1(\eta u_0,y)+{\varphi }_2(\eta v_0,z)& =& {\varphi }_2(\eta v_0,z)=\theta (\eta \parallel v_0\parallel )\eta v_0^{T}Az\\ & \le & {\eta }^{q+1}\parallel v_0{\parallel }^q\xi \le {\eta }^q\theta (\parallel v_0\parallel )\xi \le \omega \left(\eta \right)\gamma ,\end{array}$$

whenever $\eta \ge \theta (\parallel v_0\parallel )/\parallel v_0{\parallel }^q$.

(2-4) Assume that there exists $\tau >0$ such that $\sigma \left(u\right)\ge \parallel u\parallel$ as u with $\parallel u\parallel \ge \tau$. For any $(\lambda ,\mu )$, we have

$$\begin{array}{lll}{\varphi }_1(\lambda ,y)+{\varphi }_2(\mu ,z)& =& \theta (\parallel \lambda \parallel ){\lambda }^{T}Ay+\theta (\parallel \mu \parallel ){\mu }^{T}Az\hfill \\ & \le & \theta (\parallel \lambda \parallel )(\parallel A\lambda \parallel \parallel y\parallel )+\theta (\parallel \mu \parallel )(\parallel A\mu \parallel \parallel z\parallel )\\ & \le & \left(\theta (\parallel \lambda \parallel )\parallel A\lambda \parallel +\theta (\parallel \mu \parallel )\parallel A\mu \parallel \right)\parallel (y,z)\parallel \\ & \le & \left(\theta (\parallel \lambda \parallel )\parallel A\lambda \parallel +\theta (\parallel \mu \parallel )\parallel A\mu \parallel \right)\sigma (y,z).\end{array}$$

Let $\rho :=\left(\theta (\parallel \lambda \parallel )\parallel A\lambda \parallel +\theta (\parallel \mu \parallel )\parallel A\mu \parallel \right)$. Then for any $(y,z)$ with $\parallel (y,z)\parallel \ge \tau$,

$$\rho \sigma (y,z)-{\varphi }_1(\lambda ,y)-{\varphi }_2(\mu ,z)\ge 0.$$

In particular, we can take $q:=2$, $\omega \left(t\right):=t^2$, $\theta \left(t\right):=t^3$, and $\sigma \left(u\right):={\parallel u\parallel }^2$.

4. Existence of Generalized Augmented Lagrange Multipliers

In this section, we develop some sufficient conditions for the existence of generalized augmented Lagrange multipliers. Given $\epsilon \ge 0$, define

$$W_1\left(\epsilon \right):=\left\{x\in \Omega |\mathrm{dist}\left(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)\le \epsilon \right\},$$

and

$$W_2\left(\epsilon \right):=\left\{x\in \Omega \left|f\right(x)-v(0,0)\le \epsilon \right\}.$$

Lemma 1.

Suppose that $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ has a valley at zero and

$$\underset{x\in \Omega }{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}>-\infty .$$

(33)

Then for any $\epsilon >0$, we have

$$\underset{r\to +\infty }{lim}\underset{x\in \Omega /W_1\left(\epsilon \right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r)=+\infty ,$$

(34)

and

$$\left\{x\in \Omega |L(x,{\lambda }^{*},{\mu }^{*},r)\le v(0,0)\right\}\subseteq W_1\left(\epsilon \right)\cap W_2\left(\epsilon \right),$$

(35)

whenever $r>0$ is sufficiently large.

Proof. 

The proofs of (34) and (35) are given in parts (a) and (b), respectively.

(a) For any fixed $x\in \Omega /W_1\left(\epsilon \right)$, it follows from the definition of $W_1\left(\epsilon \right)$ that

$$\mathrm{dist}\left(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)>\epsilon ,$$

which implies that for any $({\xi }_1,{\xi }_2)$ with ${\xi }_1\le 0,{\xi }_2=0$ we have

$$\underset{x\in \Omega /W_1\left(\epsilon \right)}{\inf }\parallel ({\xi }_1,{\xi }_2)-\left(g\left(x\right),h\left(x\right)\right)\parallel \ge \underset{x\in \Omega /W_1\left(\epsilon \right)}{\inf }\mathrm{dist}\left(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)\ge \epsilon .$$

According to the valley-at-zero property of $\sigma$, for any $x\in \Omega /W_1\left(\epsilon \right)$, there exists $\zeta >0$ such that

$$\sigma \left({\xi }_1-g\left(x\right),{\xi }_2-h\left(x\right)\right)\ge \zeta .$$

(36)

It follows from (8) that

$$\begin{array}{lll}L(x,{\lambda }^{*},{\mu }^{*},r)& \ge & \underset{{\xi }_1\le 0,{\xi }_2=0}{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-{\varphi }_1({\lambda }^{*},{\xi }_1)+r\sigma \left({\xi }_1-g\left(x\right),{\xi }_2-h\left(x\right)\right)\right\}\\ & \ge & \underset{{\xi }_1\le 0}{\inf }\{L_0(x,{\lambda }^{*},{\mu }^{*})-{\varphi }_1({\lambda }^{*},{\xi }_1)+r\zeta \}\\ & =& L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)+r\zeta ,\end{array}$$

where the second inequality comes from (36). This implies that

$$\underset{x\in \Omega /W_1\left(\epsilon \right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r)\ge \underset{x\in \Omega }{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}+r\zeta .$$

Passing to limit in the above inequality, we get

$$\underset{r\to +\infty }{lim}\underset{x\in \Omega /W_1\left(\epsilon \right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r)\ge \underset{x\in \Omega }{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}+\underset{r\to +\infty }{lim}r\zeta =+\infty ,$$

where the equality comes from the fact that ${\inf }_{x\in \Omega }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-{\sup }_{{\xi }_1\le 0}{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}$ is finite by (33). Hence, (34) is true.

(b) First prove that

$$\{x\in \Omega |L(x,{\lambda }^{*},{\mu }^{*},r)\le v(0,0)\}\subseteq W_1\left(\epsilon \right).$$

We argue it by contradiction. If there exist ${\epsilon }_0>0$, $r_k\to \infty$, and $x_k\in \Omega$ such that

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\le v(0,0),x_k\notin W_1\left({\epsilon }_0\right),$$

then

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge \underset{x\in \Omega /W_1\left({\epsilon }_0\right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r_k).$$

Passing to limit in the above inequality, we get

$$\underset{k\to \infty }{lim\; inf}L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge \underset{k\to \infty }{lim}\underset{x\in \Omega /W_1\left({\epsilon }_0\right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r_k)=+\infty ,$$

where the equality comes from part $\left(a\right)$. Clearly, this contradicts the finiteness of $v(0,0)$.

Next, we claim that

$$\{x\in \Omega |L(x,{\lambda }^{*},{\mu }^{*},r)\le v(0,0)\}\subseteq W_2\left(\epsilon \right).$$

Suppose, on the contrary, that there exist ${\epsilon }_0>0$, $r_k\to \infty$ and $x_k\in \Omega$ such that

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\le v(0,0),x_k\notin W_2\left({\epsilon }_0\right).$$

(37)

From (7) and (37), we conclude that there exist $y_k+g\left(x_k\right)\le 0,z_k+h\left(x_k\right)=0$ such that

$$\begin{array}{lll}v(0,0)+\frac{{\epsilon }_0}{2}& \ge & L(x_k,{\lambda }^{*},{\mu }^{*},r_k)+\frac{{\epsilon }_0}{2}\\ & \ge & f\left(x_k\right)-{\varphi }_1({\lambda }^{*},y_k)-{\varphi }_2({\mu }^{*},z_k)+r_k\sigma (y_k,z_k).\end{array}$$

(38)

By the property $\left(A_4\right)$, for above $({\lambda }^{*},{\mu }^{*})$, there exist $\rho >0,\tau >0$ such that

$$\rho \sigma (y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\ge 0,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(39)

Further using valley-at-zero property of $\sigma$, for above $\tau >0$ there exists $d_1>0$ such that

$$\sigma (y,z)\ge d_1,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

(40)

Now, let us prove that $(y_k,z_k)\to 0$. Let us consider the following cases:

Case 1. There exists an infinite subset $N_1\subseteq N$ such that $\parallel (y_k,z_k)\parallel \ge \tau$ for all $k\in N_1$. Note that

$$\begin{array}{lll}v(0,0)+{\epsilon }_0& \ge & f\left(x_k\right)-{\varphi }_1({\lambda }^{*},y_k)-{\varphi }_2({\mu }^{*},z_k)+\rho \sigma (y_k,z_k)+(r_k-\rho )\sigma (y_k,z_k)\\ & \ge & f\left(x_k\right)+(r_k-\rho )\sigma (y_k,z_k)\\ & \ge & f\left(x_k\right)+(r_k-\rho )d_1,\end{array}$$

(41)

where the second inequality comes from the assumption (39), and the third step is due to (40). The right side in (41) can be arbitrary large as $N_1\ni k\to \infty$, which contradicts the finiteness of $v(0,0)$.

Case 2.$\parallel (y_k,z_k)\parallel \le \tau$ as k sufficiently large. Since ${\varphi }_1$ and ${\varphi }_2$ are continuous by the property $A_1$, for above $({\lambda }^{*},{\mu }^{*})$ and $\tau >0$, we can find $d_2\in \mathbb{R}$ such that

$${\varphi }_1({\lambda }^{*},y)+{\varphi }_2({\mu }^{*},z)\le d_2,\forall (y,z)\in {\mathbb{R}}^{m+l}\backslash \tau {\mathbb{B}}_{{\mathbb{R}}^{m+l}}.$$

Then

$$\begin{array}{lll}v(0,0)+{\epsilon }_0& \ge & f\left(x_k\right)-{\varphi }_1({\lambda }^{*},y_k)-{\varphi }_2({\mu }^{*},z_k)+r_k\sigma (y_k,z_k)\\ & \ge & f\left(x_k\right)-d_2+r_k\sigma (y_k,z_k)\\ & \ge & -d_2+r_k\sigma (y_k,z_k).\end{array}$$

Due to the boundedness of $(y_k,z_k),$ we have

$$\sigma (y_k,z_k)\le \frac{v(0,0)+{\epsilon }_0+d_2}{r_k}\to 0,ask\to \infty ,$$

which in turn implies that $(y_k,z_k)\to 0$ by the valley-at-zero property of $\sigma$.

Applying $(y_k,z_k)\to 0$ into (38) yields $v(0,0)+\frac{{\epsilon }_0}{2}\ge f\left(x_k\right).$ It contradicts $x_k\notin W_2\left({\epsilon }_0\right)$ by the definition of $W_2\left(\epsilon \right)$. Therefore, (35) holds. □

Remark 1.

Note that ${\varphi }_2({\mu }^{*},{\xi }_2)$ is not used in the assumption (33). The reason is ${\varphi }_2({\mu }^{*},{\xi }_2)=0$ as ${\xi }_2=0$, since the perturbation for equality constraint is restricted to the subspace ${\left\{0\right\}}^l$.

Theorem 5.

Suppose that $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ has a valley at zero and

$$\underset{x\in \Omega }{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}>-\infty .$$

For any $x^{*}\in X^{*}$, $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a local saddle point of $L(x,{\lambda }^{*},{\mu }^{*},r)$ for some $r^{*}>0$ and there exist a bounded subset $\mathsf{\Lambda }\subset {\mathbb{R}}^n$ and ${\epsilon }_0>0$ such that

$$\left\{x\in \Omega |\mathrm{dist}\left(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)\le {\epsilon }_0,f\left(x\right)-v(0,0)\le {\epsilon }_0\right\}\subset \mathsf{\Lambda }.$$

(42)

Then $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$.

Proof. 

According to the relationship among the generalized augmented Lagrange multiplier, the zero duality gap property, and global saddle points established in Theorems 1 and 2, we only need to justify that $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a global saddle point of $L(x,\lambda ,\mu ,r)$.

According to the definition of local saddle points, there exists $\delta >0$ such that

$$L(x^{*},\lambda ,\mu ,r^{*})\le L(x^{*},{\lambda }^{*},{\mu }^{*},r^{*})\le L(x,{\lambda }^{*},{\mu }^{*},r^{*}),\forall x\in {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )\cap \Omega ,(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(43)

It follows by invoking (14) and the first inequality in (43) that

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r^{*})=f\left(x^{*}\right).$$

(44)

By the monotonicity of $L(x^{*},{\lambda }^{*},{\mu }^{*},r)$ in r, we also have

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r^{*})\le L(x^{*},{\lambda }^{*},{\mu }^{*},r)\le f\left(x^{*}\right),\forall r\ge r^{*},$$

(45)

where the second inequality comes from (12). Combining (44) and (45) implies

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r)=f\left(x^{*}\right),\forall r\ge r^{*},$$

(46)

which together with (12) again yields

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r)=f\left(x^{*}\right)\ge L(x^{*},\lambda ,\mu ,r),\forall r\ge r^{*},(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(47)

Now, we establish the first inequality in (9). To complete the proof, it remains to show that

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r)\le L(x,{\lambda }^{*},{\mu }^{*},r),\forall x\in \Omega \backslash {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta ),$$

(48)

whenever r is sufficiently large. Suppose on the contrary that we can find $r_k\to +\infty$ and $x_k\in \Omega \backslash {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )$ such that

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)<L(x^{*},{\lambda }^{*},{\mu }^{*},r_k).$$

(49)

Hence, applying (46) into (49) and together with the fact $x^{*}\in X^{*}$ yields

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)<f\left(x^{*}\right)=v(0,0),$$

(50)

which means that $x_k$ belongs to the set $\{x\in \Omega |L(x,{\lambda }^{*},{\mu }^{*},r_k)\le v(0,0)\}.$ Taking into account of (35) in Lemma 1, we obtain that for any $\epsilon \in (0,{\epsilon }_0)$, $x_k\in W_1\left(\epsilon \right)\bigcap W_2\left(\epsilon \right)$, which further implies that $x_k\in \mathsf{\Lambda }$ by (42). We can assume without loss of generality that $x_k$ converges to $\overline{x}$. According to the continuity of $f\left(x\right)$, $g\left(x\right)$ and $h\left(x\right)$, together with the closedness of $W_1\left(\epsilon \right)$ and $W_2\left(\epsilon \right)$, we obtain that $\overline{x}\in W_1\left(\epsilon \right)\bigcap W_2\left(\epsilon \right)$. Therefore, $\overline{x}\in W_1\left(0\right)\bigcap W_2\left(0\right)$ by the arbitrariness of $\epsilon >0$, which further implies that $\overline{x}\in X^{*}$. By assumption, $(\overline{x},{\lambda }^{*},{\mu }^{*})$ is also a local saddle point of $L(x,\lambda ,\mu ,r)$ for some $\overline{r}>0$; i.e., there exists $\overline{\delta }>0$ such that

$$L(\overline{x},\lambda ,\mu ,\overline{r})\le L(\overline{x},{\lambda }^{*},{\mu }^{*},\overline{r})\le L(x,{\lambda }^{*},{\mu }^{*},\overline{r}),\forall x\in {\mathbb{B}}_{{\mathbb{R}}^n}(\overline{x},\overline{\delta })\cap \Omega ,(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(51)

Similar to the above argument, it follows from (44) that

$$L(\overline{x},{\lambda }^{*},{\mu }^{*},\overline{r})=f\left(\overline{x}\right)=v(0,0)=val\left(P\right).$$

(52)

Since $x_k\in {\mathbb{B}}_{{\mathbb{R}}^n}(\overline{x},\overline{\delta })$ and $r_k\ge \overline{r}$ for k large enough, from (51) and (52), it follows

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge L(x_k,{\lambda }^{*},{\mu }^{*},\overline{r})\ge L(\overline{x},{\lambda }^{*},{\mu }^{*},\overline{r})=f\left(\overline{x}\right)=f\left(x^{*}\right)=v(0,0),$$

which contradicts (50). This justifies (48).

By the fact (43), (47) and (48), we conclude that $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a global saddle point of $L(x,\lambda ,\mu ,r)$ for r large enough. Therefore, $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$. □

Theorem 6.

Suppose that $\sigma :{\mathbb{R}}^{m+l}\to {\mathbb{R}}_{+}$ has a valley at zero and

$$\underset{x\in \Omega }{\inf }\left\{L_0(x,{\lambda }^{*},{\mu }^{*})-\underset{{\xi }_1\le 0}{\sup }{\varphi }_1({\lambda }^{*},{\xi }_1)\right\}>-\infty .$$

Let $x^{*}$ be the unique global optimal solution of $\left(P\right)$. If $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a local saddle point of $L(x,\lambda ,\mu ,r)$ for some $r\ge 0$, and there exists ${\epsilon }_0>0$ such that

$$\{x\in \Omega |\mathrm{dist}(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l)\le {\epsilon }_0\}\subset \mathsf{\Lambda },$$

(53)

where Λ is a bounded subset in ${\mathbb{R}}^n$, then $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$.

Proof. 

For ${\epsilon }_0>0$, it follows from (34) in Lemma 1 that there exists $r_1>0$ such that

$$\underset{x\in \Omega /W_1\left({\epsilon }_0\right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r_1)\ge f\left(x^{*}\right).$$

That is to say, for any $x\in \Omega \backslash W_1\left({\epsilon }_0\right)$, we have

$$L(x,{\lambda }^{*},{\mu }^{*},r_1)\ge f\left(x^{*}\right),\forall x\in \Omega \backslash W_1\left({\epsilon }_0\right).$$

(54)

To complete the proof, we next need to show that there exists $r_2>0$ such that

$$L(x,{\lambda }^{*},{\mu }^{*},r_2)\ge f\left(x^{*}\right),\forall x\in W_1\left({\epsilon }_0\right).$$

(55)

Suppose on the contrary that there exist $r_k\to \infty$ and $\left\{x_k\right\}\subset W_1\left({\epsilon }_0\right)$ such that

$$f\left(x^{*}\right)>L(x_k,{\lambda }^{*},{\mu }^{*},r_k).$$

(56)

According to (53) and $\mathsf{\Lambda }$ being bounded, $W_1\left({\epsilon }_0\right)$ is bounded, which further implies that $\left\{x_k\right\}$ has at least a cluster point $\overline{x}$. We assume without loss of generality that $x_k$ converges $\overline{x}$.

We now claim that $\overline{x}$ is a feasible point of $\left(P\right)$. If $\overline{x}$ is not feasible, then $\mathrm{dist}\left(g\left(x\right),h\left(x\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)>2m_0$ for some $m_0>0$. Therefore, $\mathrm{dist}\left(g\left(x_k\right),h\left(x_k\right);{\mathbb{R}}_{-}^m\times {\left\{0\right\}}^l\right)>m_0$ as k sufficiently large. It in turn implies

$$L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge \underset{x\in \Omega /W_1\left(m_0\right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r_k).$$

Taking the limits on both sides yields

$$\underset{k\to \infty }{lim\; inf}L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge \underset{k\to \infty }{lim}\underset{x\in \Omega /W_1\left(m_0\right)}{\inf }L(x,{\lambda }^{*},{\mu }^{*},r_k)=+\infty ,$$

(57)

where the equality comes from Lemma 1. Combining (56) with (57) together yields a contradiction to the finiteness of $f\left(x^{*}\right)$. This justifies the feasibility of $\overline{x}$ for $\left(P\right)$.

By hypothesis, $(x^{*},{\lambda }^{*},{\mu }^{*})$ is a local saddle point of $L(x,\lambda ,\mu ,r)$ for some $r\ge 0$; then there exists a neighborhood ${\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )$ such that

$$L(x^{*},\lambda ,\mu ,r)\le L(x^{*},{\lambda }^{*},{\mu }^{*},r)\le L(x,{\lambda }^{*},{\mu }^{*},r),\forall x\in {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )\cap \Omega ,(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l.$$

(58)

Putting (14), (58), and the monotonicity of $L(x^{*},{\lambda }^{*},{\mu }^{*},r)$ with respect to r together means that for any $r^{\prime }\ge r$,

$$\begin{array}{lll}f\left(x^{*}\right)& =& \underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x^{*},\lambda ,\mu ,r)=L(x^{*},{\lambda }^{*},{\mu }^{*},r)\le L(x^{*},{\lambda }^{*},{\mu }^{*},r^{\prime })\\ & \le & \underset{(\lambda ,\mu )\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}^l}{\sup }L(x^{*},\lambda ,\mu ,r^{\prime })=f\left(x^{*}\right).\end{array}$$

That is,

$$f\left(x^{*}\right)=L(x^{*},{\lambda }^{*},{\mu }^{*},r^{\prime }),\forall r^{\prime }\ge r.$$

(59)

Taking into account of (56), we obtain that

$$L(x^{*},{\lambda }^{*},{\mu }^{*},r)=f\left(x^{*}\right)>L(x_k,{\lambda }^{*},{\mu }^{*},r_k)\ge L(x_k,{\lambda }^{*},{\mu }^{*},r),$$

where the last step is due to the monotonicity of $L(x,{\lambda }^{*},{\mu }^{*},r)$ with respect to r. Thus, it follows from (58) that $x_k\notin {\mathbb{B}}_{{\mathbb{R}}^n}(x^{*},\delta )$ whenever k is sufficiently large; i.e., $\overline{x}\ne x^{*}$. Since $x^{*}$ is the unique global optimal solution of (P) and invoking that $\overline{x}$ is feasible, we have

$$f\left(\overline{x}\right)-f\left(x^{*}\right)>0.$$

Define $\varrho :=\left(f\left(\overline{x}\right)-f\left(x^{*}\right)\right)/2>0$. Using (7) and (56), there exist $y_k\le -g\left(x_k\right),z_k=-h\left(x_k\right)$ such that

$$\begin{array}{lll}f\left(x^{*}\right)+\varrho & >& L(x_k,{\lambda }^{*},{\mu }^{*},r_k)+\varrho \\ & \ge & f\left(x_k\right)-{\varphi }_1({\lambda }^{*},y_k)-{\varphi }_2({\mu }^{*},z_k)+r_k\sigma (y_k,z_k).\end{array}$$

Similarly to the argument given in Lemma 1, we conclude from the property $\left(A_4\right)$ that $(y_k,z_k)\to 0$. Hence $f\left(x^{*}\right)+\varrho \ge f\left(x_k\right).$ Passing to limit, we get $f\left(x^{*}\right)+\varrho \ge f\left(\overline{x}\right),$ which together with the definition of $\varrho$ implies that $f\left(x^{*}\right)\ge f\left(\overline{x}\right)$. It is clearly the case that $\overline{x}$ is also an optimal solution. Hence $\overline{x}=x^{*}$, since the optimal solution is unique. This justifies (55).

Let $r^{*}:=max\{r_1,r_2\}$. Taking into account of (54) and (55), we obtain that

$$L(x,{\lambda }^{*},{\mu }^{*},r^{*})\ge f\left(x^{*}\right)=L(x^{*},{\lambda }^{*},{\mu }^{*},r^{*}),\forall x\in \Omega ,$$

(60)

where the equation comes from (59). Hence, according to (11) and (60),

$$\begin{array}{l}\underset{(y,z)\in {\mathbb{R}}^{m+l}}{\inf }\left\{v_{r^{*}}(y,z)-{\varphi }_1({\lambda }^{*},y)-{\varphi }_2({\mu }^{*},z)\right\}=\underset{x\in \Omega }{\inf }L(x,{\lambda }^{*},{\mu }^{*},r^{*})=f\left(x^{*}\right)=v(0,0),\end{array}$$

where the last step comes from that $x^{*}$ is the optimal solution of (P). This further implies

$$v_{r^{*}}(y,z)\ge v_{r^{*}}(0,0)+{\varphi }_1({\lambda }^{*},y)+{\varphi }_2({\mu }^{*},z),\forall (y,z)\in {\mathbb{R}}^{m+l}.$$

Hence $({\lambda }^{*},{\mu }^{*})$ is a generalized augmented Lagrange multiplier of $\left(P\right)$. □

The existence of generalized augmented Lagrange multipliers is established in two different scenarios: one is applicable to the case of unique solution while another is applicable to the case of multiple optimal solutions.

5. Conclusions

In this paper, we studied the generalized augmented Lagrangian multiplier, which is an extension of the augmented Lagrangian multiplier from linear support to nonlinear support for an augmented perturbation function. Some sufficient conditions for the existence of generalized augmented Lagrangian multipliers were developed. In particular, the relationships among global saddle points, generalized augmented Lagrangian multipliers, and the zero duality gap property between the primal problem and its generalized augmented Lagrangian dual problem were established. Several interesting topics are left for further investigation. For example, one is developing some necessary and sufficient conditions for the existence of generalized augmented Lagrangian multipliers by using the localization principle; another is studying the generalized differentiation of support functions from the subdifferential view.