The Alternating Direction Search Pattern Method for Solving Constrained Nonlinear Optimization Problems

Abstract

We adopt the alternating direction search pattern method to solve the equality and inequality constrained nonlinear optimization problems. Firstly, a new augmented Lagrangian function with a nonlinear complementarity function is proposed to transform the original constrained problem into a new unconstrained problem. Under appropriate conditions, it has been proven that there is a 1-1 correspondence between the local and global optimal solutions of the new unconstrained problem and the original constrained problem. In this way, the optimal solution of the original problem can be obtained by solving the new unconstrained optimization problem. Furthermore, based on the characteristics of the new problem, the alternating direction pattern search method was designed and its convergence was demonstrated. Numerical experiments were implemented to illustrate the availability of the new augmented Lagrangian function and the algorithm.

1. Introduction

With the big data era coming, the demand for optimization in resource allocation, transportation, production management, marketing planning, engineering design, and other aspects is constantly increasing. As one of the most basic models, the constrained nonlinear optimization problems are widely used in many fields, such as finance [1], power grids [2], dynamics [3,4], logistics management [5], portfolio [6], etc. There are many ways to solve the constrained nonlinear optimization problems, including the filter method [7], penalty function method [8,9], and so on. For example, the alternating direction method has been continuously developed in recent years [10,11]. Ref. [10] proposes a new acceleration approximation linearization ADMM algorithm.

The augmented Lagrangian function is to add the constraints as penalty terms to the objective function to form a new optimization problem, which was presented by Di Pillo and Grippo in the 1980s, see [12,13,14]. Furthermore, PU [15,16] proposed a new class of augmented Lagrangian functions and methods with piecewise nonlinear complementarity function (denote NCP) for the constrained nonlinear programming. A class of new augmented Lagrangian functions with Fischer–Burmeister NCP functions and methods was presented in [17]. A generalized Newtonian method of global convergence to solve non-smooth equations with non-differentiable functions but continuous Lipschitz was presented in [18]. A class of new Lagrangian multiplier methods with NCP functions was presented for solving the constrained nonlinear programming satisfying equation constraints and inequality constraints, see [19]. The reference [20] developed and analyzed several new constructors for constructing nonlinear complementary functions. The reference [21] provided a new class of complementary functions for regularization problems, which use smooth complementary functions to transform the complementarity problem into a nonlinear system of equations. In [22,23], a class of new augmented Lagrangian functions with NCP functions is presented for the constrained nonlinear programming.

We can use the Lagrange multiplier method to solve the constrained nonlinear optimization problems. The following the constrained nonlinear optimization problems(P) is studied in this paper:

$$\begin{array}{c}minf\left(x\right)\hfill \\ s.t.c\left(x\right)=0,c:R^n\to R^p\hfill \\ s\left(x\right)\le 0,s:R^n\to R^m\hfill \end{array}$$

let $c\left(x\right)={(c_1\left(x\right),c_2\left(x\right),\cdots ,c_p\left(x\right))}^T$, $s\left(x\right)={(s_1\left(x\right),s_2\left(x\right),\cdots ,s_m\left(x\right))}^T$, which are twice the continuously differentiable functions.

The Lagrangian function of the problem (P) is equivalent to the following function:

$$L(x,\omega ,\lambda )=f\left(x\right)+{\omega }^{T}c\left(x\right)+{\lambda }^{T}s\left(x\right),$$

(1)

where $\omega ={({\omega }_1,{\omega }_2,\cdots ,{\omega }_p)}^T$$\in R^p$, $\lambda ={({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_m)}^T$$\in R^m$, $\omega$ and $\lambda$ are the multiplier vectors. Conveniently, $(x,\omega ,\lambda )$ is denoted as the column vector ${(x^T,{\omega }^T,{\lambda }^T)}^T$.

A Karush–Kuhn–Tucker (KKT) point $(\overline{x},\overline{\omega },\overline{\lambda })\in R^n\times R^p\times R^m$ holds the following conditions:

$${\nabla }_{x}L(\overline{x},\overline{\omega },\overline{\lambda })=0,c\left(\overline{x}\right)=0,s\left(\overline{x}\right)\le 0,\overline{\lambda }\ge 0,{\overline{\lambda }}_j^T{s}_j\left(\overline{x}\right)=0,1\le j\le m.$$

(2)

This condition is called the first-order optimal necessary condition for problem (P).

We also define a class of new augmented Lagrangian functions with NCP functions for the problem (P). The difference between this new function and the previous one is that the parameters are different. The previous study used the same parameters in the penalty terms of equation and inequality constraints, and now we consider adding two penalty terms with different parameters, which has the advantage of improving the algorithm efficiency. We then prove a 1-1 correspondence between the local and global optimal solutions of the new unconstrained problem and the original constrained problem. In particular, considering the characteristics of the new problem, an alternating direction search pattern method was designed and its convergence was demonstrated, while experiment simulations are performed according to the designed new augmented Lagrangian function and algorithm.

2. Constructing New Augmented Lagrangian Function

In order to construct the new augmented Lagrangian function, some necessary preparations and assumptions are given in this section.

The NCP function is a function that satisfies the following conditions:

$$\psi (a,b)=0,\iff a\ge 0,b\ge 0,ab=0.$$

The point $(x,\omega ,\lambda )$ subject to the following conditions:

• The first-order KKT condition is satisfied;
• For any $d\in P\left(x\right)=\left\{d\right|d^T\nabla c_i\left(x\right)=0,i=1,2,\cdots ,p,d^T\nabla s_j\left(x\right)\le 0,j\in J\left(x\right)\},$ with $d^T{\nabla }_{xx}^{2}L(x,\omega ,\lambda )d>0$. Here, $d\ne 0$ and $j\in J\left(x\right),J\left(x\right)=\left\{j\right|j=1,2,\cdots ,m,s_j\left(x\right)=0\}$.

Then, $(x,\omega ,\lambda )$ satisfies the strong second-order sufficiency condition [20].

In this paper, NCP function was used $\psi (a,b)=(a+b)\sqrt{a^2+b^2}-a^2-b^2$, it is continuously differentiable except at the origin, and strongly semismooth at the origin, see [19,20,21].

Denote

$${\psi }_j(x,{\lambda }_j,\gamma )=\psi ((-\gamma s_j\left(x\right)),{\lambda }_j),1\le j\le m.$$

where $\gamma >0$ is a parameter.

${\psi }_j(x,\lambda ,\gamma )=0$ iff $s_j\left(x\right)\le 0$, ${\lambda }_j\ge 0$ and ${\lambda }_j{s}_j\left(x\right)=0$ for any $\gamma >0$. Denote $\Psi (x,\lambda ,\gamma )={\left({\psi }_1(x,{\lambda }_1,\gamma ),\cdots ,{\psi }_m(x,{\lambda }_m,\gamma )\right)}^T$.

For problem (P), we defined the new augmented Lagrangian function F as follows:

$$\begin{array}{c}F(x,\omega ,\lambda ,\gamma ,\rho )\hfill \\ \stackrel{def}{=}f\left(x\right)+{\omega }^{T}c\left(x\right)+\rho c\left(x\right){}^2/2+\left[{∥\Psi (x,\lambda ,\gamma )+{\lambda }^2∥}^2-{∥{\lambda }^2∥}^2\right]+{\nabla }_{x}L(x,\omega ,\lambda ){}^2/2\hfill \end{array}$$

(3)

where $\rho >0$ is the other parameter.

Obviously, the conditions of KKT point convert to the following conditions:

$$\psi (x,\lambda ,\gamma )=0,c\left(x\right)=0,{\nabla }_{x}L(x,\omega ,\lambda )=0.$$

(4)

Some assumptions are as follows:

A1. $f\left(x\right)$, $c_i\left(x\right)$, and $s_j\left(x\right)$ are two times Lipschitz continuously differentiable. For example, for $f\left(x\right)$, L is a positive number, if all points x in the field of point $x_0$ have,

$$\left|f\left(x\right)-f\left(x_0\right)\right|\le L{\left|x-x_0\right|}^2,$$

then it is said that $f\left(x\right)$ holds two Lipschitz conditions at point $x_0$. The same for $c_i\left(x\right)$ and $s_j\left(x\right)$.

A2. At the point with $\nabla F(x,\omega ,\lambda ,\gamma ,\rho )=0$ for any $\gamma >0$ and $\rho >0$, $\{\nabla c_i\left(x\right),\nabla s_j\left(x\right)|i=1,2,\cdots ,p,j=1,2,\cdots ,m\}$ are linear independent.

A3. The strict complementarity condition holds at any KKT point of problem (P).

Suppose $(x,\omega ,\lambda )$ is a KKT point. ${\lambda }_j{s}_j\left(x\right)=0,\forall j=1,\cdots ,m$ is called the complementary condition. If there is also ${{\lambda }_j}^2+{\left(s_j\left(x\right)\right)}^2>0$, then the strict complementarity condition holds.

A4. The strict second-order sufficiency condition holds at any KKT point of the problem (P).

3. The Equivalence of Local Optimal Solution

To prove the equivalence of locally optimal solutions, we give the following two lemmas.

Lemma 1.

For sufficiently large γ, $\rho >0$. $\nabla F(x,\omega ,\lambda ,\gamma ,\rho )=0$ holds iff $(x,\omega ,\lambda )$ is a KKT point.

Proof. 

The index sets is denoted

$$I_0(x,\lambda )=\left\{j\right|(s_j\left(x\right),{\lambda }_j)=(0,0)\},I_1(x,\lambda )=\left\{j\right|(s_j\left(x\right),{\lambda }_j)\ne (0,0)\}.$$

By the ${\psi }_j(x,{\lambda }_j,\gamma )$ definition, for any $j\in I_0$, ${\psi }_j(x,{\lambda }_j,\gamma )=0$, ${\psi }_j(x,{\lambda }_j,\gamma )+{\lambda }_j^2=0$, and $\parallel \psi (x,{\lambda }_j,\gamma )+{\lambda }_j^2\parallel =0$.

For any $j\in I_1$, the derivative of F is:

$$\begin{array}{c}{\nabla }_{x}F(x,\omega ,\lambda ,\gamma ,\rho )=\nabla f\left(x\right)+\nabla c\left(x\right)\omega +\rho \nabla c\left(x\right)c\left(x\right)\hfill \\ +\frac{1}{2{\gamma }^4}[{\nabla }_x{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }_{}^2\right)}^2]+{\nabla }_{xx}^{2}L(x,\omega ,\lambda ){\nabla }_{x}L(x,\omega ,\lambda )\hfill \\ ={\nabla }_{x}L(x,\omega ,\lambda )+\rho \nabla c\left(x\right)c\left(x\right)+\nabla s\left(x\right)\Psi (x,\lambda ,\gamma )\hfill \\ +{\nabla }_{xx}^{2}L(x,\omega ,\lambda ){\nabla }_{x}L(x,\omega ,\lambda )+{\displaystyle \sum _{j\in I_1}}{\tau }_j\hfill \end{array}$$

(5)

$${\nabla }_{\omega }F(x,\omega ,\lambda ,\gamma ,\rho )=c\left(x\right)+{\left(\nabla c\left(x\right)\right)}^T{\nabla }_{x}L(x,\omega ,\lambda )$$

(6)

$$\begin{array}{c}{\nabla }_{\lambda }F(x,\omega ,\lambda ,\gamma ,\rho )\hfill \\ =\frac{1}{2{\gamma }^4}{\nabla }_{\lambda }\left[{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }^2\right)}^2\right]+{(\nabla s\left(x\right))}^T{\nabla }_{x}L(x,\omega ,\lambda )\hfill \\ ={\displaystyle \sum _{j\in I_1}}{\xi }_j+{(\nabla s\left(x\right))}^T{\nabla }_{x}L(x,\omega ,\lambda )\hfill \end{array}$$

(7)

where ${\tau }_j,{\xi }_j$ denotes the following:

$$\begin{array}{c}{\tau }_j=-{\psi }_j(x,{\lambda }_j,\gamma )\nabla s_j\left(x\right)-{\lambda }_j\nabla s_j\left(x\right)+\frac{1}{{\gamma }^3}({\psi }_j(x,{\lambda }_j,\gamma )+{\lambda }_j^2)\hfill \\ \left[\frac{\gamma s_j\left(x\right)({\lambda }_j-\gamma s_j\left(x\right))}{\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}}-\sqrt{{{\lambda }_j}^2+{\left(\gamma s_j\left(x\right)\right)}^2}+2\gamma s_j\left(x\right)\right]\nabla s_j\left(x\right)\hfill \end{array}$$

$${\xi }_j=\frac{1}{2{\gamma }^4}\{({\psi }_j(x,{\lambda }_j,\gamma )+{\lambda }_j^2)\left[\frac{{\lambda }_j({\lambda }_j-\gamma s_j\left(x\right))}{\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}}+\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}\right]-4{\lambda }_j^3\}$$

Suppose $(x,\omega ,\lambda )$ is a KKT point, then for any $\gamma >0,$

$${\nabla }_{x}L(x,\omega ,\lambda )=0,c\left(x\right)=0,s\left(x\right)\le 0,\lambda \ge 0,{\lambda }_i^T{s}_i\left(x\right)=0.$$

The combined (5) can be obtained from this,

$$\begin{array}{c}{\nabla }_{x}F(x,\omega ,\lambda ,\gamma ,\rho )=\nabla f\left(x\right)+\nabla c\left(x\right)\omega +\rho \nabla c\left(x\right)c\left(x\right)\hfill \\ +\frac{1}{2{\gamma }^4}[{\nabla }_x{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }_{}^2\right)}^2]+{\nabla }_{xx}^{2}L(x,\omega ,\lambda ){\nabla }_{x}L(x,\omega ,\lambda )\hfill \\ ={\nabla }_{x}L(x,\omega ,\lambda )+\rho \nabla c\left(x\right)c\left(x\right)+\nabla s\left(x\right)\Psi (x,\lambda ,\gamma )\hfill \\ +{\nabla }_{xx}^{2}L(x,\omega ,\lambda ){\nabla }_{x}L(x,\omega ,\lambda )+\frac{1}{2{\gamma }^4}[{\nabla }_x{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }_{}^2\right)}^2]\hfill \\ =\frac{1}{2{\gamma }^4}[{\nabla }_x{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }_{}^2\right)}^2]\hfill \end{array}$$

put (4) into the above equation in order to obtain

$$\frac{1}{2{\gamma }^4}[{\nabla }_x{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }_{}^2\right)}^2]=0.$$

therefore, ${\nabla }_{x}F(x,\omega ,\lambda ,\gamma ,\rho )=0$.

It can be obtained according to (6),

$${\nabla }_{\omega }F(x,\omega ,\lambda ,\gamma ,\rho )=c\left(x\right)+{\left(\nabla c\left(x\right)\right)}^T{\nabla }_{x}L(x,\omega ,\lambda )=0.$$

By (7),

$$\begin{array}{c}{\nabla }_{\lambda }F(x,\omega ,\lambda ,\gamma ,\rho )\hfill \\ =\frac{1}{2{\gamma }^4}{\nabla }_{\lambda }\left[{\left(\Psi (x,\lambda ,\gamma )+{\lambda }^2\right)}^2-{\left({\lambda }^2\right)}^2\right]+{(\nabla s\left(x\right))}^T{\nabla }_{x}L(x,\omega ,\lambda )\hfill \\ =0.\hfill \end{array}$$

As a result, $\nabla F(x,\omega ,\lambda ,\gamma ,\rho )=0$.

Conversely, suppose $\nabla F(x,\omega ,\lambda ,\gamma ,\rho )=0$ for some $\gamma >0,{\psi }_j(x,{\lambda }_j,\gamma )=0,$${\nabla }_{x}L(x,\omega ,\lambda )=0$.

Then, we have

$$s\left(x\right)\le 0,\lambda \ge 0,{\lambda }_i^T{s}_i\left(x\right)=0,j=1,2,\cdots ,m.$$

by (6) it is easy to know

$${\nabla }_{\omega }F(x,\omega ,\lambda ,\gamma ,\rho )=c\left(x\right)+{\left(\nabla c\left(x\right)\right)}^T{\nabla }_{x}L(x,\omega ,\lambda )=0.$$

as ${\nabla }_{x}L(x,\omega ,\lambda )=0$, we have $c\left(x\right)=0$.

Thus, $(x,\omega ,\lambda )$ makes the KKT conditions of problem (P) hold.

Furthermore, we assume $\nabla F(x,\omega ,\lambda ,\gamma ,\rho )=0$ for a sufficiently large $\gamma >0$.

By A1, $g_j\left(x\right)$ is bounded and we have

$$\begin{array}{c}{\xi }_j=\frac{1}{{\gamma }^4}\{2{\lambda }_j{\left(\gamma s_j\left(x\right)\right)}^2-3{\left({\lambda }_j\right)}^2\left(\gamma s_j\left(x\right)\right)-{\left(\gamma s_j\left(x\right)\right)}^3\hfill \\ -{\left(\gamma s_j\left(x\right)\right)}^2[\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}+\frac{{\lambda }_j({\lambda }_j-\gamma s_j\left(x\right))}{\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}}]\}\hfill \end{array}$$

where $\gamma \to \infty ,{\xi }_j\to 0$.

By (7), we obtain

$${(\nabla s\left(x\right))}^T{\nabla }_{x}L(x,\omega ,\lambda )=0$$

(8)

By A2, $s_j\left(x\right)$ are linear independent,

$${\nabla }_{x}L(x,\omega ,\lambda )=0,$$

(9)

there is

$${\lambda }_j{s}_j\left(x\right)=0.$$

(10)

Otherwise, let ${\lambda }_j{s}_j\left(x\right)>0$ for any $\gamma >0,{\tau }_j>0$ or ${\xi }_j<0$, which is a contradiction to $F(x,\omega ,\lambda ,\gamma ,\rho )=0.$

By (6) and (9), we have

$$c\left(x\right)=0.$$

(11)

Put (9)–(11) into (5), then for $j\in I_1$, we obtain

$$\begin{array}{c}{\nabla }_{x}F(x,\omega ,\lambda ,\gamma ,\rho )=\frac{1}{{\gamma }^3}{\displaystyle \sum _{j\in I_1}}({\psi }_j(x,{\lambda }_j,\gamma )+{\lambda }_j^2)\hfill \\ \left[\frac{\gamma s_j\left(x\right)({\lambda }_j-\gamma s_j\left(x\right))}{\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}}-\sqrt{{{\lambda }_j}^2+{\left(\gamma s_j\left(x\right)\right)}^2}-2\gamma s_j\left(x\right)\right]\nabla s_j\left(x\right).\hfill \end{array}$$

(12)

By A2, $\nabla s_j\left(x\right)$ are linear independent and then we obtain

$$\frac{1}{{\gamma }^3}({\psi }_j(x,{\lambda }_j,\gamma )+{\lambda }_j^2)\left[\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}+\frac{\gamma s_j\left(x\right)(\gamma s_j\left(x\right)-{\lambda }_j)}{\sqrt{{\left({\lambda }_j\right)}^2+{\left(\gamma s_j\left(x\right)\right)}^2}}+2\gamma s_j\left(x\right)\right]=0.$$

(13)

If $s_j\left(x\right)=0$, then ${\psi }_j(x,{\lambda }_j,\gamma )=0$.

If $s_j\left(x\right)\ne 0$, then ${\lambda }_j=0$.

For a sufficiently large $\gamma$,

$$\left[{∥s_j\left(x\right)∥}^2+{\left(s_j\left(x\right)\right)}^2\right]/∥s_j\left(x\right)∥+2s_j\left(x\right)={\left[∥s_j\left(x\right)∥+s_j\left(x\right)\right]}^2=0.$$

then, $s_j\left(x\right)<0$.

So, it follows from (13) that ${\psi }_j(x,{\lambda }_j,\gamma )=0$; that means $s\left(x\right)\le 0,{\lambda }_j\ge 0,{\lambda }_i^T{s}_i\left(x\right)=0$, in other words $(x,\omega ,\lambda )$ is KKT point of problem (P).    □

Lemma 2.

For $\rho >0$ and a sufficiently large γ, suppose $(x^{*},{\omega }^{*},{\lambda }^{*})$ is the local minima of $F(x,\omega ,\lambda ,\gamma ,\rho )$ and $x^{*}$ is the local minima of the problem (P).

The proof of this lemma can be found in [17].

4. The Equivalence of Globe Optimal Solution

Lemma 3.

For a sufficiently large γ, suppose $(\overline{x},\overline{\omega },\overline{\lambda })$ is the KKT point of problem (P), and $F(x,\omega ,\lambda ,\gamma ,\rho )$ is strong convex at $(\overline{x},\overline{\omega },\overline{\lambda })$ when A1–A3 hold.

Proof. 

Suppose $(\overline{x},\overline{\omega },\overline{\lambda })$ is a KKT point of problem (P), thus

$${\nabla }_{x}L(\overline{x},\overline{\omega },\overline{\lambda })=0,c_i\left(\overline{x}\right)=0,{\psi }_j(\overline{x},{\overline{\lambda }}_j,\gamma )=0.$$

In other words,

$$({\overline{\lambda }}_j-\gamma s_j\left(\overline{x}\right))\sqrt{{\overline{\lambda }}_j^2+{\left(\gamma s_j\left(\overline{x}\right)\right)}^2}-{\left(\gamma s_j\left(\overline{x}\right)\right)}^2={\overline{\lambda }}_j^2\ge 0.$$

The supposition A1 connotes ${\overline{\lambda }}_j^2+{\left(\gamma s_j\left(\overline{x}\right)\right)}^2\ne 0$. For any $j\in I_1$, the Hessian matrix of $F(x,\omega ,\lambda ,\gamma ,\rho )$ at a KKT point $(\overline{x},\overline{\omega },\overline{\lambda })$ is:

$$\begin{array}{c}{\nabla }_{xx}^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )\hfill \\ ={\nabla }_{xx}L(\overline{x},\overline{\omega },\overline{\lambda })+\rho \nabla c\left(\overline{x}\right){(\nabla c\left(\overline{x}\right))}^T+\gamma {\nabla }_{xx}^{2}L(\overline{x},\overline{\omega },\overline{\lambda },\gamma ){\nabla }_{xx}^{2}L(\overline{x},\overline{\omega },\overline{\lambda })\hfill \\ +{\displaystyle {\displaystyle \sum _{j=1}^m}}\left[{\lambda }_j^2+16{\left(\gamma s_j\left(\overline{x}\right)\right)}^2+{\lambda }_j^4\right]\nabla s_j\left(\overline{x}\right){(\nabla s_j\left(\overline{x}\right))}^T\hfill \end{array},$$

$${\nabla }_{x\omega }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )=\nabla c\left(\overline{x}\right)+{\nabla }_{xx}^{2}L(\overline{x},\overline{\omega },\overline{\lambda })\nabla \left(c\left(\overline{x}\right)\right),$$

$${\nabla }_{x\lambda }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )=\nabla s\left(\overline{x}\right))diag\left(\zeta \right)+{\nabla }_{xx}^{2}L(\overline{x},\overline{\omega },\overline{\lambda },\gamma ),$$

$${\nabla }_{\lambda \lambda }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )=\nabla s\left(\overline{x}\right)diag\left(\eta \right)+{\nabla }_{xx}^{2}L(\overline{x},\overline{\omega },\overline{\lambda })\nabla s\left(\overline{x}\right),$$

$${\nabla }_{\omega \omega }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )=\nabla c\left(\overline{x}\right)(\nabla {\left(c\left(\overline{x}\right)\right)}^T,$$

$${\nabla }_{\lambda \omega }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )={(\nabla s\left(\overline{x}\right))}^T\nabla c\left(\overline{x}\right),$$

let $\zeta ,\eta$ represent the column vector,

$$\begin{array}{c}{\zeta }_j=\frac{{\lambda }_j^3}{\sqrt{{\lambda }_j^2+{\left(\gamma s_j\left(\overline{x}\right)\right)}^2}},\hfill \\ {\eta }_j=6{\lambda }_j^2+\frac{{\left(\gamma s_j\left(\overline{x}\right)\right)}^4}{{\lambda }_j^2+{\left(\gamma s_j\left(\overline{x}\right)\right)}^2}).\hfill \end{array}$$

Therefore,

$$\begin{array}{c}{(x_i,{\omega }_i,{\lambda }_i)}^T{\nabla }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )(x_i,{\omega }_i,{\lambda }_i)\hfill \\ =x^T{\nabla }_{xx}L(\overline{x},\overline{\omega },\overline{\lambda })x+{\displaystyle \sum _{i=1}^p}{\rho }_i{(x^T\nabla c\left(x\right))}^2\hfill \\ +{\displaystyle \sum _{j=1}^m}[{\lambda }_j^2+16{\left(\gamma s_j\left(\overline{x}\right)\right)}^2+{\lambda }_j^4]{(x^T\nabla s_j\left(x\right))}^2\hfill \end{array}.$$

Moreover,

$${(x^T\nabla c\left(x\right))}^2\ge 0,{(x^T\nabla s\left(x\right))}^2\ge 0,{\lambda }_j^2+16{\left(\gamma s_j\left(\overline{x}\right)\right)}^2+{\lambda }_j^4\ge 0.$$

So,

$${(x_i,{\omega }_i,{\lambda }_i)}^T{\nabla }^{2}F(\overline{x},\overline{\omega },\overline{\lambda },\gamma ,\rho )(x_i,{\omega }_i,{\lambda }_i)\ge 0.$$

The lemma is proved.    □

Lemma 4.

Let $(\overline{x},\overline{\omega },\overline{\lambda })\in X\times P\times M$ is the KKT point of problem (P) in the compact set. Suppose $\overline{x}$ is the unique global minima of problem (P) on the X. $(\overline{x},\overline{\omega },\overline{\lambda })$ is the unique global minima of $F(x,\omega ,\lambda ,\gamma ,\rho )$ when A2–A4 hold.

From the above several lemmas, it is easy to obtain Lemma 4, which is proved but ignored in the remainder of this paper.

5. Solution Algorithms

Based on the above four lemmas, we obtain the solution of the problem (P) by solving the unconstrained nonlinear optimization problems. Considering the nature of the problem (P), we design the algorithm for solving the nonlinear constrained problem using the alternating direction search pattern method. The specific algorithm is as shown below.

Algorithm 1: Alternating direction search pattern method

Input: Parameter initialization. ${\gamma }^0>0,{\rho }^0>0,0\le \eta <<1$ and $0<{\theta }_1<1<{\theta }_2$. Give a starting point $x_0\in R^n$, ${\omega }^0=({\omega }_1^0,{\omega }_2^0,\cdots ,{\omega }_p^0)\in R^p$ and ${\lambda }^0=({\lambda }_1^0,{\lambda }_2^0,\cdots ,{\lambda }_m^0)\in R^m$. Let $k=0$. Output: Stop until a certain stopping criterion is met. Step 1: Solve the optimal solution. Use Algorithm 2 to solve the problem $minF(x,{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)$ to obtain $x^{k+1}.$ If $\parallel c\left(x^{k+1}\right)\parallel \le \eta$ and $\parallel \Psi (x^{k+1},{\lambda }^k,{\gamma }^k){\parallel }_{\infty }\le \eta ,$ then stop. Step 2: If $|c_i\left(x^{k+1}\right)|\le {\theta }_1\left|c_i\left(x^k\right)\right|,$ for any i, then ${\rho }_i^{k+1}={\rho }_i^k$. Otherwise ${\rho }_i^{k+1}={\theta }_2{\rho }_i^k.$ If $|{\psi }_j(x^{k+1},{\lambda }_j^k,{\gamma }^k)|\le {\theta }_1\left|{\psi }_j(x^k,{\lambda }_j^k,{\gamma }^k)\right|$, for any j, then ${\gamma }^{k+1}={\gamma }^k$, otherwise ${\gamma }^{k+1}={\theta }_2{\gamma }^k$. Step 3: Compute ${\omega }^{k+1}$ and ${\lambda }^{k+1}$: $${\omega }_i^{k+1}={\omega }_i^k+{\rho }_i^k{c}_i\left(x^{k+1}\right),$$ (14) $${\left({\lambda }_j^{k+1}\right)}^2=(-{\gamma }^k{s}_j\left(x^{k+1}\right)+{\lambda }_j^k)\sqrt{{\left({\gamma }^k{s}_j\left(x^{k+1}\right)\right)}^2+{\left({\lambda }_j^k\right)}^2}-({\gamma }^k{s}_j{\left(x^{k+1}\right)}^2.$$ (15) Step 4: Set $k=k+1$, go to Step 1.

Algorithm 2 is a sub-algorithm of Algorithm 1.

Algorithm 2: The search pattern method

Input: Select initial value $x^0\in R^n,\epsilon >0,k=1.$ Output: Stop until a certain stopping criterion is met. Step 1: Let $x_1^k=x^k$; for $i=1,2,···,n,$ computer ${\alpha }_i^k=argminF(x_i^k+\alpha e_i,{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)),$ $x_{i+1}^k=x_i^k+{\alpha }_i^k{e}_i.$ Step 2: Set $d_k=x^{k+1}-x_k$, if $\left|\right|d^k\left|\right|\le \epsilon$, stop; Otherwise, let ${\alpha }_i^k=argminF(x_i^k+\alpha d_k,{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)),$ $x_{i+1}^k=x_i^k+{\alpha }_i^k{d}_k.$ Let $k:=k+1$, go to step 1.

Note: $e_i$ is the i-th direction of coordinate axis. The Algorithm 2 is also called the search pattern method; its main idea is to find a new point in each n coordinate rotation search in order to obtain a direction and then conduct a search.

6. Convergence

In this section, let ${\mu }_i^k={\omega }_i^k/\sqrt{{\rho }_i^k},{\nu }_j^k={\left({\lambda }_j^k\right)}^2/{\gamma }^k,{\mu }^k={({\mu }_1^k,{\mu }_2^k,\dots ,{\mu }_p^k)}^T,{\nu }^k={({\nu }_1^k,{\nu }_2^k,\dots ,{\nu }_m^k)}^T.$

Lemma 5.

Let the non-empty set X is the feasible solutions set of the problem (P), for $x\in X$. $\exists \tau$, supposing that $f\left(x\right)$ has a lower bound, then $\parallel {\mu }^k{\parallel }^2+{\parallel {\nu }^k\parallel }^2\le \tau k$.

Proof. 

For any k, to prove ${\parallel {\mu }^k\parallel }^2+{\parallel {\nu }^k\parallel }^2\le \tau k$ is equal in order to prove

$${\displaystyle \sum _{i=1}^p}\frac{{\left({\omega }_i^k\right)}^2}{{\rho }_i^k}+{\displaystyle \sum _{j=1}^m}\frac{{\left({\lambda }_j^k\right)}^4}{2{\left({\gamma }^k\right)}^4}\le \tau k.$$

Because

$$\begin{array}{c}F(x^{k+1},{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)-f\left(x^{k+1}\right)-{\nabla }_{x}L(x^{k+1},{\omega }^k,{\lambda }^k){}^2/2\hfill \\ ={\displaystyle \sum _{i=1}^p}\frac{{\left({\omega }_i^{k+1}\right)}^2-{\left({\omega }_i^k\right)}^2}{2{\rho }_i^k}+{\displaystyle \sum _{j=1}^m}\frac{{\left({\lambda }_j^{k+1}\right)}^4-{\left({\lambda }_j^k\right)}^4}{2{\left({\gamma }^k\right)}^4}\hfill \\ ={\displaystyle \sum _{i=1}^p}\frac{{\left({\omega }_i^{k+1}\right)}^2}{2{\rho }_i^k}+{\displaystyle \sum _{j=1}^m}\frac{{\left({\lambda }_j^{k+1}\right)}^2}{2{\left({\gamma }^k\right)}^4}-{\displaystyle \sum _{i=1}^p}\frac{{\left({\omega }_i^k\right)}^2}{2{\rho }_i^k}-{\displaystyle \sum _{j=1}^m}\frac{{\left({\lambda }_j^k\right)}^2}{2{\left({\gamma }^k\right)}^4}\hfill \\ =\frac{1}{2}({\mu }^{k+1}{}^2+{\nu }^{k+1}{}^2)-\frac{1}{2}({\mu }^k{}^2+{\nu }^k{}^2).\hfill \end{array}$$

(16)

Therefore,

$$\begin{array}{c}\frac{1}{2}({\mu }^{k+1}{}^2+{\nu }^{k+1}{}^2)=\frac{1}{2}({\mu }^k{}^2+{\nu }^k{}^2)\hfill \\ +F(x^{k+1},{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)-f\left(x^{k+1}\right)-{\nabla }_{x}L(x^{k+1},\omega ,\lambda ){}^2/2.\hfill \end{array}$$

For any $\overline{x}\in X$, in other words

$$c\left(\overline{x}\right)=0,s\left(\overline{x}\right)\le 0,\overline{\lambda }\ge 0,{\overline{\lambda }}_j^T{s}_j\left(\overline{x}\right)=0,{\varphi }_j(\overline{x},{\lambda }_j^k,{\gamma }^k)+{\left({\lambda }_j\right)}^2\le {\left({\lambda }_j^k\right)}^2,$$

so,

$$\begin{array}{c}F(x^{k+1},{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)-f\left(x^{k+1}\right)-{\nabla }_{x}L(x^{k+1},{\omega }^k,{\lambda }^k){}^2\hfill \\ ={\displaystyle \sum _{j=1}^m}\left[{\psi }_j(\overline{x},{\lambda }_i^k,{\gamma }^k)+{\left({\lambda }_i^2\right)}^2-{\lambda }_i^4\right]/2{\left({\gamma }^k\right)}^4\le 0,\hfill \end{array}$$

(17)

$${\mu }^{k+1}{}^2+{\nu }^{k+1}{}^2\le {\mu }^k{}^2+{\nu }^k{}^2+2(f\left(\overline{x}\right)-f\left(x^{k+1}\right)).$$

(18)

Now that $f\left(x\right)$ has a lower bound, there is a $\tau$ such that ${\parallel {\mu }^k\parallel }^2+{\parallel {\nu }^k\parallel }^2\le \tau k.$ □

The lemma is proved.

Theorem 1.

Let the non-empty set X be the feasible solutions set of the problem (P); when Algorithm 1 reaches the stopping criterion or $\underset{k\to \infty }{liminf}f\left(x^k\right)=-\infty$, then end the experiment.

Proof. 

Suppose the contrary; let the algorithm be implemented and if after the finite number of iterations of the algorithm does not stop or $\underset{k\to \infty }{liminf}f\left(x^k\right)=-\infty$, then let

$$\begin{array}{c}{J}_i=\{i|\underset{k\to \infty }{liminf}{c}_i\left(x^k\right)\ne 0\},{\overline{J}}_i=\{i|\underset{k\to \infty }{liminf}{\rho }_i^k=-\infty \},\hfill \\ J_j=\{j|\underset{k\to \infty }{liminf}{\psi }_j\left(x^k\right)\ne 0\},{\overline{J}}_j=\{j|\underset{k\to \infty }{liminf}{\gamma }^k=-\infty \}.\hfill \end{array}$$

By assumption of this theorem: $J_i\cup J_j,{\overline{J}}_i\cup {\overline{J}}_j$ is not empty. For any $x\in X$,

$$\begin{array}{c}f\left(\overline{x}\right)+{\displaystyle \sum _{j=1}^m}\left[{\left({\psi }_j(\overline{x},{\lambda }_j^k,{\gamma }^k)+{\lambda }_j^k\right)}^2-{\left({\lambda }_j^k\right)}^2\right]/2{\gamma }^k\hfill \\ \ge f\left(x^{k+1}\right)+{\left({\omega }^k\right)}^{\mathrm{T}}c\left(x^{k+1}\right)+{\displaystyle \sum _{i=1}^m}{\rho }_{i}c\left(x^{k+1}\right)/2\hfill \\ +{\displaystyle \sum _{j=1}^m}\left[{\left({\psi }_j(x^{k+1},{\lambda }_j^k,{\gamma }^k)+{\lambda }_j^k\right)}^2-{\left({\lambda }_j^k\right)}^2\right]/2{\gamma }^k.\hfill \end{array}$$

Because ${\gamma }^{k+1}\ge {\gamma }^k$ and ${\rho }_i^{k+1}\ge {\rho }_i^k$, therefore

$$\begin{array}{c}f\left(\overline{x}\right)-f\left(x^{k+1}\right)\ge O\left(k\right)+{\displaystyle \sum _{i\in {\overline{J}}_i}^p}{\rho }_i^k\left[{(c_i\left(x^{k+1}\right)}^2+{{\omega }_i^k/{\rho }_i^k)}^2-{({\omega }_i^k/{\rho }_i^k)}^2/2\right]\hfill \\ +{\displaystyle \sum _{j\in {\overline{J}}_j}^m}{\gamma }^k\left[{({\varphi }_j(x^{k+1},{\lambda }_j^k,{\gamma }^k)/{\gamma }^k+{\lambda }_j^k/{\gamma }^k)}^2-{({\lambda }_j^k/{\gamma }^k)}^2/2\right].\hfill \end{array}$$

(19)

where $O\left(k\right)$ is a higher order infinitesimal of k.

Since Algorithm 1 does not stop, for any $\overline{k}$, there exists $k>\overline{k}$, either for a $i\in {\overline{J}}_i$ and ${\tau }_1>0,{\rho }_i^{k+1}\ge {\rho }_i^k$, and $c_i\left(x^{k+1}\right)\ge {\tau }_1$, either ${\rho }_i^k\ge {\theta }_2^{(k-1)/p}\ge k^2$, such that,

$${\rho }_i^k{(h_i{\left(x^{k+1}\right)}^2+{\omega }_i^k/{\rho }_i^k)}^2/2>{\rho }_i^k{\tau }_1^2/2>k^2{\tau }_1^2$$

or for a $j\in {\overline{J}}_j$, ${\gamma }_j^k\ge {\theta }_2^{(k-1)/m}\ge k^2$,

$${\gamma }_j^k{(s_j{\left(x^{k+1}\right)}^2+{\lambda }_j^k/{\gamma }_j^k)}^2/2>{\gamma }_j^k{\tau }_1^2/2>k^2{\tau }_1^2/2$$

The above formula shows,

$$f\left(\overline{x}\right)-f\left(x^{k+1}\right)\ge {\left(k{\tau }_1\right)}^2/4.$$

This is a contradiction. That is the end of the proof. □

Theorem 2.

Let the non-empty set X is the feasible solutions set of the problem (P); ${\omega }^k$ and ${\lambda }^k,$ are bounded. Let $\eta =0$ in Algorithm 1; at the k-th iteration Algorithm 1 satisfies the stopping criterion, then $x^k$ is solution to problem(P), or let $x^{*}$ be the accumulation point of $\left\{x^k\right\}$, then $x^{*}$ is solution to problem (P).

Proof. 

As Algorithm 1 is stopped at the k-th iteration,

$${\psi }_j(x^k,{\lambda }_j^k,{\gamma }^k)=0,s\left(x^k\right)\le 0,{\lambda }_j^k\ge 0,{\lambda }_j^T{s}_j\left(x^k\right)=0.$$

(20)

It is clear for any $\gamma >0$,

$${\lambda }_j^{k+1}=\left(\frac{{\gamma }^k{s}_j\left(x^k\right)}{\sqrt{{\left({\lambda }_j^k\right)}^2+{\left({\gamma }^k{s}_j\left(x^k\right)\right)}^2}}+1\right)\left(\sqrt{{\left({\lambda }_j^k\right)}^2+{\left({\gamma }^k{s}_j\left(x\right)\right)}^2}+{\gamma }^k{s}_j\left(x^k\right)\right)={\lambda }_j^k$$

(21)

Since $\parallel {\Psi }_j(x^k,{\lambda }_j^k,{\gamma }^k)\parallel =0$, $\parallel c\left(x^k\right)\parallel =0$, by Equations (16) and (21),

$$\nabla F(x^k,{\omega }^k,{\lambda }^k,{\gamma }^k,{\rho }^k)=\nabla L(x^k,{\omega }^k,{\lambda }^{k+1})=0.$$

(22)

so, $x^k$ is the solution of $\left(P\right)$.

In other words, at the k-th iteration Algorithm 1 does not satisfy the stopping criterion, for any minima $(x^{*},{\omega }^{*},{\lambda }^{*})$ of sequence $(x^k,{\omega }^k,{\lambda }^k),$ where $\gamma >0$ or for any $\overline{x}\in X$, $∥\psi (x^{*},{\lambda }_j^{*},\gamma )∥=0,∥c\left(x^k\right)∥=0,$

$$\begin{array}{c}f\left(\overline{x}\right)\ge f\left(x^{k+1}\right)+O\left(k\right)\hfill \end{array}.$$

(23)

As $k\to \infty$

$$f\left(\overline{x}\right)\ge f\left(x^{*}\right).$$

It is clear that $x^{*}$ is local minima of the primal constrained problem (P).

That is the end of the proof. □

7. Numerical Experiment

To verify the reliability of the algorithm, some numerical experiments were conducted. The results are shown in Table 1. The experimental examples in Table 1 are constrained nonlinear programming from [24], where the problem number is the same as the number of the problem in [24].

Table 1. Computational results of Algorithm 1.

Problem No.	Initial Point	NIT	NF	NG	Initial Point	NIT	NF	NG
Problem 215	0.6, 0.6	11	15	19	1.8, 1.8	18	25	33
Problem 227	0.8, 0.8	7	14	25	1.5, 1.2	18	23	34
Problem 232	4, 3	12	16	23	6, 6	9	19	22
Problem 250	8, 6, 9	15	18	39	−6, −7, −8	14	19	27
Problem 264	1, 0.8, 1, 0.8	18	24	21	1.2, 1.2, 1.2, 1.2	24	36	35

In the experiments, the parameters are set to $\eta =0.00001$, ${\theta }_1=0.6$, ${\theta }_2=1.6$, and $\parallel \psi \parallel \le {10}^{-5}$ as the termination condition. By selecting different initial points, “NIT, NF and NG” are set as the evaluation criteria. NIT denotes the number of iterations; NF is the number of times the functions are evaluated—the number of NF will only increase by one when all functions are estimated once; NG is the number of times the $\Psi$ (or gradient) is evaluated times.

It is clear from the above table that the “NIT, NF and NG” are small for different problems with different initial points, which means that the new augmented Lagrange methods are effective as shown by these numerical results.

In conclusion, a new augmented Lagrangian function with nonlinear complementarity functions is proposed to solve the constrained nonlinear optimization problems. Under the given conditions, the solution of the new unconstrained programming is equivalent to the solution of the original constrained programming. Furthermore, the alternating direction based search pattern method was designed, and its convergence was demonstrated. At the same time, experiments were conducted and the results of the experiments demonstrated the effectiveness of the presented new augmented Lagrangian function and the algorithms.