An Efficient Penalty Method without a Line Search for Nonlinear Optimization

Abstract

In this work, we integrate some new approximate functions using the logarithmic penalty method to solve nonlinear optimization problems. Firstly, we determine the direction by Newton’s method. Then, we establish an efficient algorithm to compute the displacement step according to the direction. Finally, we illustrate the superior performance of our new approximate function with respect to the line search one through a numerical experiment on numerous collections of test problems.

1. Introduction

The nonlinear optimization is a fundamental subject in the modern optimization literature. It focuses on the problem of optimizing an objective function in the presence of inequality and/or equality constraints. Furthermore, the optimization problem is obviously linear if all the functions are linear, otherwise it is called a nonlinear optimization problem.

This research field is motivated by the fact that it arises in various problems encountered in practice, such as business administration, economics, agriculture, mathematics, engineering, and physical sciences.

In our knowledge, Frank and Wolfe are the deans in nonlinear optimization problems. They established a powerful algorithm in [1] to solve them. Later, they used another method in [2] based on the application of the Simplex method on the nonlinear problem after converting it to a linear one.

This pioneer work inspired many authors to propose and develop several methods and techniques to solve this class of problems. We refer to [3,4] for interior point methods to find the solution of nonlinear optimization problems with a high dimension.

In order to make this theory applicable in practice, other methods are designed on the linear optimization history, among robust algorithms with polynomial complexity. In this perception, Khachian succeeded in 1979 to introduce a new ellipsoid method from approaches applied originally to nonlinear optimization.

Interior point methods outperform the Simplex ones, and they have recently been the subject of several monographs including Bonnans and Gilbert [5], Evtushenko and Zhadan [6], Nesterov and Emirovski [7], and Wright [8] and Ye [9].

Interior point methods can be classified into three different groups as follows: projective methods and their alternatives as in Powell [10] and Rosen [11,12], central trajectory methods (see Ouriemchi [13] and Forsgren et al. [14]), and barrier/penalty methods, where majorant functions were originally proposed by Crouzeix and Merikhi [15] to solve a semidefinite optimization problem. Inspired by this work, Menniche and Benterki [16] and Bachir Cherif and Merikhi [17] applied this idea to linear and nonlinear optimizations, respectively.

A majorant function for the penalty method in convex quadratic optimization was proposed by Chaghoub and Benterki [18]. On the other hand, A. Leulmi et al. [19,20] used new minorant functions for semidefinite optimization, and this idea was extended to linear programming by A. Leulmi and S. Leulmi in [21].

As far as we know, our new approximate function has not been studied in the nonlinear optimization literature. These approximate functions are more convenient and efficient than the line search method for rapidly computing the displacement step.

Therefore, in our work, we aim to optimize a nonlinear problem based on prior efforts. Thus, we propose a straightforward and effective barrier penalty method using new minorant functions.

More precisely, we first introduce the position of the problem and its perturbed problem with the results of convergence in Section 2 and Section 3 of our paper. Then, in Section 4, we establish the solution of the perturbed problem by finding new minorant functions. Section 5 is devoted to presenting a concise description of the algorithm and to illustrating the outperformance of our new approach by carrying out a simulation study. Finally, we summarize our work in the conclusion.

Throughout this paper, the following notations are adopted. Let $⟨.,.⟩$ and $∥.∥$ denote the scalar product and the Euclidean norm, respectively, given by the following:

$$⟨x,y⟩=x^{T}y=\sum _{i=1}^n{x}_i{y}_i,x,y\in {\mathbb{R}}^n$$

and

$$∥x∥=\sqrt{⟨x,x⟩}=\sqrt{\sum _{i=1}^n{x}_i^2}$$

2. The Problem

We aim to present an algorithm for solving the following optimization problem:

$$\left\{\begin{array}{c}\min f\left(x\right)\hfill \\ Ax=b\hfill \\ x\ge 0,\hfill \end{array}\right.$$

(P)

where $b\in {\mathbb{R}}^m$ and A$\in {\mathbb{R}}^{m\times n}$ is a full-rank matrix with $m<n.$

For this purpose, we need the following hypothesis:

Hypothesis 1.

f is nonlinear, twice continuously differentiable, and convex on $\mathcal{L}$, where $\mathcal{L}=\left\{x\in {\mathbb{R}}^n:Ax=b;x\ge 0\right\}$ is the set of realizable solutions of (P).

Hypothesis 2.

(P) satisfies the condition of interior point (IPC), i.e., there exists $x_0>0$ such that $A{x}_0=b.$

Hypothesis 3.

The set of optimal solutions of (P) is nonempty and bounded.

Notice that these conditions are standard in this context. We refer to [17,20].

If $x^{*}$ is an optimal solution, there exist two Lagrange multipliers $p^{*}$$\in {\mathbb{R}}^m$ and $q^{*}$$\in {\mathbb{R}}^n,$ such that

$$\left\{\begin{array}{c}▿f\left(x^{*}\right)+A^T{p}^{*}=q^{*}\ge 0,\hfill \\ A{x}^{*}=b,\hfill \\ ⟨q^{*},x^{*}⟩=0.\hfill \end{array}\right.$$

(1)

3. Formulation of the Perturbed Problem of (P)

Let us first consider the function $\psi$ defined on $\mathbb{R}\times {\mathbb{R}}^n$ by the following:

$$\psi (\eta ,x)=\left\{\begin{array}{ccc}f\left(x\right)+{\displaystyle \sum _{i=1}^n}\xi (\eta ,x_i)\hfill & \mathrm{if}\hfill & x\ge 0,Ax=b\hfill \\ +\infty \hfill & \mathrm{if}\mathrm{not},\hfill \end{array}\right.$$

where $\xi :{\mathbb{R}}^2⟶\left(-\infty ,+\infty \right]$ is a convex, lower semicontinuous and proper function given by the following:

$$\xi (\eta ,\alpha )=\left\{\begin{array}{ccc}\eta \ln \left(\eta \right)-\eta \ln \left(\alpha \right)\hfill & \mathrm{if}\hfill & \alpha >0\mathrm{and}\eta >0,\hfill \\ 0\hfill & \mathrm{if}\hfill & \alpha \ge 0\mathrm{and}\eta =0,\hfill \\ +\infty \hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

Thus, $\psi$ is a proper, convex, and lower semicontinuous function.

Furthermore, the function g defined by

$$g\left(\eta \right)=\underset{x\in {\mathbb{R}}^n}{\inf }\left[{\psi }_{\eta }\left(x\right)=f\left(x\right)+\stackrel{n}{\sum _{i=1}}\xi (\eta ,x_i)\right]$$

(Pη)

is convex. Notice that for $\eta =0,$ the perturbed problem $\left(P\eta \right)$ coincides with the initial problem (P); then, $f^{*}=g\left(0\right).$

3.1. Existence and Uniqueness of Optimal Solution

To show that the perturbed problem $\left(P\eta \right)$ has a unique optimal solution, it is sufficient to demonstrate that the recession cone of ${\psi }_{\eta }$ is reduced to zero.

Proof. 

For a fixed $\eta ,$ the function ${\psi }_{\eta }$ is proper, convex, and lower semicontinuous. The asymptotic function of ${\psi }_{\eta }$ is defined by the following:

$${\left({\psi }_{\eta }\right)}_{\infty }\left(d\right)=\underset{\alpha \to +\infty }{\lim }\frac{{\psi }_{\eta }\left(x_0+\alpha d\right)-{\psi }_{\eta }\left(x_0\right)}{\alpha },$$

thus, the asymptotic functions of f and ${\psi }_{\eta }$ satisfy the relation:

$${\left({\psi }_{\eta }\right)}_{\infty }\left(d\right)=\left\{\begin{array}{ccc}{\left(f\right)}_{\infty }\left(x\right)\hfill & \mathrm{if}\hfill & d\ge 0,Ad=0,\hfill \\ +\infty \hfill & \mathrm{if}\mathrm{not}.\hfill \end{array}\right.$$

Moreover, hypothesis H3 is equivalent to

$$\left\{d\in {\mathbb{R}}^n:{\left(f\right)}_{\infty }\left(x\right)\le 0,d\ge 0,Ad=0\right\}=\left\{0\right\}.$$

Then,

$$\left\{d\in {\mathbb{R}}^n:{\left({\psi }_{\eta }\right)}_{\infty }\left(d\right)\le 0\right\}=\left\{0\right\}$$

and from [17], for each non-negative real number $\eta ,$ the strictly convex problem $\left(P\eta \right)$ admits a unique optimal solution noted by $x_{\eta }^{*}.$ The solution of the problem (P) is the limit of the solutions sequence of the perturbed problem $\left(P\eta \right)$ when $\eta$ tends to 0. □

3.2. Convergence of the Solution

Now, we are in a position to state the convergence result of $\left(P\eta \right)$ to (P), which is proved in Lemma 1 on [18].

Let $\eta >0,$ for all $x\in \mathcal{L};$ we define $\psi (x,\eta )=f_{\eta }\left(x\right).$

Lemma 1

([18]). We consider $\eta >0.$ If the perturbed problem $\left(P\eta \right)$ admits an optimal solution $x_{\eta },$ such that ${\displaystyle \underset{\eta \to 0}{\lim }}{x}_{\eta }=x^{*}$, then the problem (P) admits an optimal solution $x^{*}.$

We use the classical prototype of penalty methods. We begin our process with $(x_0,{\eta }_0)\in \tilde{\mathcal{L}}\times \left(0,\infty \right),$ where

$$\tilde{\mathcal{L}}=\left\{x\in {\mathbb{R}}^n:x>0,Ax=b\right\}$$

(2)

and the iteration scheme is divided into the following steps:

1. Select ${\eta }_{k+1}\in \left(0,{\eta }_k\right).$

2. Establish an approximate solution $x_{k+1}$ for $\left(P{\eta }_k\right).$ It is obvious that $\psi ({\eta }_k,x_{k+1})<\psi ({\eta }_k,x_k).$

Remark 1.

If the values of the objective functions of the problem (P) and the perturbed problem $\left(P\eta \right)$ are equal and finite, then (P) will have an optimal solution if and only if $\left(P\eta \right)$ has an optimal solution.

The iterative process stops when we obtain an acceptable approximation of $g\left(0\right).$

4. Computational Resolution of the Perturbed Problem

Our approach to the numerical solution of the perturbed problem $\left(P\eta \right),$ consists of two stages. In the first one, we calculate the descent direction using the Newton approach, and in the second one, we propose an efficient new-minorant-functions approach to compute the displacement step easily and quickly relative to the line search method.

4.1. The Descent Direction

As $\left(P\eta \right)$ is strictly convex, the necessary and sufficient optimality conditions state that $x_{\eta }$ is an optimal solution of $\left(P\eta \right)$ if and only if it satisfies the nonlinear system:

$$\nabla {\psi }_{\eta }\left(x_{\eta }\right)=0.$$

Using the Newton approach, a penalty method is provided to solve the above system, where the vector $x_{k+1}$ in each is given by $x_{k+1}=x_k+{\alpha }_k{d}_k.$

The solution of the following quadratic convex optimization problem is necessary to obtain the Newton descent direction d:

$$\underset{d}{\min }\left[⟨▿{\psi }_{\eta }\left(x\right),d⟩+\frac{1}{2}⟨{▿}^2{\psi }_{\eta }\left(x\right)d,d⟩:Ad=0\right]=\underset{d}{\min }\left[H\left(\eta ,x\right):Ad=0\right],$$

where $x\in \tilde{\mathcal{L}}$ and

$$\begin{array}{ccc}\hfill {\psi }_{\eta }\left(x\right)& =& f\left(x_{\eta }\right)+n\eta \ln \eta -\eta \sum _{i=1}^n\ln \left(x_i\right)\hfill \\ \hfill \nabla {\psi }_{\eta }\left(x\right)& =& \nabla f\left(x\right)-\eta X^{-1}e\hfill \\ \hfill {\nabla }^2{\psi }_{\eta }\left(x\right)& =& {\nabla }^{2}f\left(x\right)+\eta X^{-2}e\hfill \\ \hfill H\left(\eta ,x\right)& =& ⟨▿{\psi }_{\eta }\left(x\right),d⟩+\frac{1}{2}⟨{▿}^2{\psi }_{\eta }\left(x\right)d,d⟩,\hfill \end{array}$$

with the diagonal matrix $X=\mathbf{diag}{\left(x_i\right)}_{i=\overline{1,n}}.$

The Lagrangian is given by the following:

$$L\left(x,s\right)=⟨\nabla f\left(x\right)-X^{-1}\eta ,d⟩+\frac{1}{2}⟨\left({\nabla }^{2}f\left(x\right)+X^{-2}\eta \right)d,d⟩+⟨Ad,s⟩,$$

where $s\in {\mathbb{R}}^m$ is the Lagrange multiplier. It is sufficient for solving the linear system equations with $n+m:$

$$\left\{\begin{array}{cc}\nabla f\left(x\right)-\eta X^{-1}e+⟨\left({\nabla }^{2}f\left(x\right)+X^{-2}\eta \right)d,d⟩+A^{t}s\hfill & =0\hfill \\ Ad\hfill & =0,\hfill \end{array}\right.$$

then,

$$\left\{\begin{array}{cc}\left({▿}^{2}f\left(x\right)-\eta X^{-2}\right)d+A^{T}s\hfill & =\eta X^{-1}e-▿f\left(x\right)\hfill \\ Ad\hfill & =0.\hfill \end{array}\right.$$

(3)

It is simple to prove that system (3) is non-singular. We obtain

$$\left\{\begin{array}{cc}{d}^T\left({▿}^{2}f\left(x\right)-\eta X^{-2}\right)d+d^T{A}^{T}s\hfill & =d^T\eta X^{-1}e-d^T▿f\left(x\right)\hfill \\ d^{T}Ad\hfill & =0,\hfill \end{array}\right.$$

As $d^T{A}^{T}s={\left(Ad\right)}^{T}s=0$ and $Ad=0,$ we obtain

$$⟨{▿}^{2}f\left(x\right)d,d⟩+⟨▿f\left(x\right),d⟩=\eta \left[⟨X^{-1}d,e⟩-{∥X^{-1}d∥}^2\right].$$

(4)

The system can also be written as follows:

$$\left\{\begin{array}{cc}\left(X{▿}^{2}f\left(x\right)X\right)\left(X^{-1}d\right)+\eta I\left(X^{-1}d\right)+X{A}^{T}s\hfill & =\eta X{X}^{-1}e-X▿f\left(x\right)\hfill \\ AX\left(X^{-1}d\right)\hfill & =0,\hfill \end{array}\right..$$

(5)

Thus, the Newton descent direction is obtained.

Throughout this paper, we take x instead of $x_{\eta }.$

4.2. Computation of the Displacement Step

This section deals with the numerical solution of the displacement step. We give a brief highlight of the line search methods used in nonlinear optimization problems. Then, we collect some important results of approximate function approaches applied to both semidefinite and linear programming problems. Finally, we propose our new approximate function method for the nonlinear optimization problem (P).

4.2.1. Line Search Methods

The line search methods consists of determining a displacement step ${\alpha }_k$, which ensures the sufficient decrease in the objective at each iteration $x_{k+1}=x_k+{\alpha }_k{d}_k,$ where ${\alpha }_k>0,$ along the descent direction $d_k;$ in other words, it involves solving the following one-dimensional problem:

$$\phi \left(\alpha \right)=\underset{\alpha >0}{\min }{\psi }_{\eta }\left(x_k+\alpha d_k\right).$$

The disadvantage of this method is that the solution $\alpha$ is not necessarily optimal, which make the feasibility of $x_{k+1}$ not guaranteed.

The line search techniques of Wolfe, Goldstein-Armijo, and Fibonacci are the most widely used ones. However, generally, their computational volume is costly. This is what made us search for another alternative.

4.2.2. Approximate Functions Techniques

These methods are based on sophisticated techniques introduced by J.P. Crouzeix et al. [15] and A. Leulmi et al. [20] to obtain the solution of a semidefinite optimization problem.

The aim of these techniques is to give a minimized approximation of one real-variable function $\phi \left(\alpha \right)$ defined by

$$\begin{array}{ccc}\hfill \phi \left(\alpha \right)& =& \frac{1}{\eta }[{\psi }_{\eta }(x+\alpha d)-{\psi }_{\eta }\left(x\right)]\hfill \\ & =& \frac{1}{\eta }[f(x+\alpha d)-f\left(x\right)]-\stackrel{n}{\sum _{i=1}}\ln (1+\alpha t_i),t=X^{-1}d.\hfill \end{array}$$

The function $\phi$ is convex, and we obtain the following:

$$\begin{array}{ccc}\hfill {\phi }^{\prime }\left(\alpha \right)& =& \frac{1}{\eta }⟨▿f(x+\alpha d),d⟩-\stackrel{n}{\sum _{i=1}}\frac{t_i}{1+\alpha t_i},\hfill \\ \hfill {\phi }^{″}\left(\alpha \right)& =& \frac{1}{\eta }⟨{▿}^{2}f(x+\alpha d),d⟩+\stackrel{n}{\sum _{i=1}}\frac{t_i^2}{{\left(1+\alpha t_i\right)}^2}.\hfill \end{array}$$

We find that ${\phi }^{\prime }\left(0\right)+{\phi }^{″}\left(0\right)=0,$ deduced from (4), which is expected since d is the direction of Newton’s descent direction.

We aim to avoid the disadvantages of line search methods and accelerate the convergence of the algorithm. For this reason, we have to identify an $\overline{\alpha }$ that yields a significant decrease in the function $\phi \left(\alpha \right).$ This is the same as solving a polynomial equation of degree $n+1,$ where f is a linear function.

Now, we include a few helpful inequalities below, which are used throughout the paper.

H. Wolkowicz et al. [22] see also Crouzeix and Seeger [23] presented the following inequalities:

$$\begin{array}{ccc}\hfill \overline{z}-{\sigma }_z\sqrt{n-1}& \le & \underset{i}{\min }{z}_i\le \overline{z}-\frac{{\sigma }_z}{\sqrt{n-1}}\hfill \\ \hfill \overline{z}+\frac{{\sigma }_x}{\sqrt{n-1}}& \le & \underset{i}{\max }{z}_i\le \overline{z}+{\sigma }_z\sqrt{n-1},\hfill \end{array}$$

where $\overline{z}$ and ${\sigma }_z$ represent the mean and the standard deviation, respectively, of a statistical real numbers series $\{z_1,z_2,...,z_n\}$. The later quantities are defined as follows:

$$\overline{z}=\frac{1}{n}\sum _{i=1}^n{z}_i\mathrm{and}{\sigma }_z^2=\frac{1}{n}\sum _{i=1}^n{z}_i^2-{\overline{z}}^2=\frac{1}{n}\sum _{i=1}^n{(z_i-\overline{z})}^2.$$

Theorem 1

([15]). Let $z_i>0,$ for $i=1,2,...,n.$ We have the following:

$$\sum _{i=1}^n\ln \left(z_i\right)\le \ln \left(\overline{z}+{\sigma }_z\sqrt{n-1}\right)+(n-1)\ln \left(\overline{z}-\frac{{\sigma }_z}{\sqrt{n-1}}\right).$$

(6)

where $z_i=1+\alpha t_i,$ $z=1+\alpha t$, and ${\sigma }_z=\alpha {\sigma }_t.$

We will proceed to present the paper’s principal result.

4.2.3. New Approximate Functions Approach

Let

$$\phi \left(\alpha \right)=\frac{1}{\eta }[f(x+\alpha d)-f\left(x\right)]-\stackrel{n}{\sum _{i=1}}\ln (1+\alpha t_i),$$

be defined on $\tilde{\alpha }=\underset{i\in I_{-}}{\min }\left\{\frac{-1}{t_i}\right\}$ such that $I_{-}=\{i:t_i<0\}.$

To find the displacement step, it is necessary to solve ${\phi }^{\prime }\left(\alpha \right)=0.$ Considering the difficulty of solving a non-algebraic equation, approximate functions are recommended alternatives.

Two novel approximation functions of $\phi$ are introduced in the following lemma.

Lemma 2.

For all $\alpha \in \left[0,{\alpha }_1^{*}\right[$ with ${\alpha }_1^{*}=\min \left(\widehat{\alpha },{\widehat{\alpha }}_1\right),$ we have

$$\phi \left(\alpha \right)\ge {\widehat{\phi }}_1\left(\alpha \right),$$

and for all $\alpha \in \left[0,{\alpha }_2^{*}\right[$ with ${\alpha }_2^{*}=\min \left(\widehat{\alpha },{\widehat{\alpha }}_2\right),$ we obtain

$${\widehat{\phi }}_2\left(\alpha \right)\le \phi \left(\alpha \right),$$

where

$${\widehat{\phi }}_1\left(\alpha \right)=\frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\ln \left(1+\delta \alpha \right)-(n-1)\ln \left(1+\beta \alpha \right),$$

and

$${\widehat{\phi }}_2\left(\alpha \right)=\frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\tau \ln (1+{\beta }_1\alpha ),$$

with

$$\left\{\begin{array}{c}\beta =\overline{t}-\frac{{\sigma }_t}{\sqrt{n-1}}\hfill \\ \delta =\overline{t}+{\sigma }_t\sqrt{n-1}\hfill \\ {\beta }_1=\frac{{∥t∥}^2}{n\overline{t}}.\hfill \end{array}\right.$$

(7)

Furthermore, we have

$${\widehat{\phi }}_2\left(\alpha \right)\le {\widehat{\phi }}_1\left(\alpha \right)\le \phi \left(\alpha \right),$$

Proof. 

We start by proving that

$$\phi \left(\alpha \right)\ge {\widehat{\phi }}_1\left(\alpha \right),$$

Theorem 1 gives

$$\sum _{i=1}^n\ln \left(z_i\right)\le \ln \left(\overline{z}+{\sigma }_z\sqrt{n-1}\right)+(n-1)\ln \left(\overline{z}-\frac{{\sigma }_z}{\sqrt{n-1}}\right),$$

then,

$$\stackrel{n}{\sum _{i=1}}\ln (1+\alpha t_i)\le \ln \left(1+\alpha \delta \right)+(n-1)\ln \left(1+\alpha \beta \right),$$

and

$$-\stackrel{n}{\sum _{i=1}}\ln (1+\alpha t_i)\ge -\ln \left(1+\alpha \delta \right)-(n-1)\ln \left(1+\alpha \beta \right).$$

Hence,

$$\begin{array}{c}\frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-{\displaystyle \sum _{i=1}^n}\ln (1+\alpha t_i)\ge \frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\ln \left(1+\alpha \delta \right)\hfill \\ -(n-1)\ln \left(1+\alpha \beta \right).\hfill \end{array}$$

Therefore,

$$\begin{array}{ccc}\hfill \phi \left(\alpha \right)& \ge & {\widehat{\phi }}_1\left(\alpha \right)=\frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\ln \left(1+\alpha \delta \right)-(n-1)\ln \left(1+\alpha \beta \right),\hfill \\ \hfill {\widehat{\phi }}_1^{\prime }\left(\alpha \right)& =& \frac{1}{\eta }⟨▿f(x+\alpha d)-d⟩-\frac{\delta }{1+\alpha \delta }-(n-1)\frac{\beta }{1+\alpha \beta }.\hfill \end{array}$$

Let us consider the following:

$$g\left(\alpha \right)=\phi \left(\alpha \right)-{\widehat{\phi }}_2\left(\alpha \right).$$

We have

$$g^{″}\left(\alpha \right)=\sum _{i=1}^n\frac{t_i^2}{{(1+\alpha t_i)}^2}-\tau \frac{{\beta }_1}{{\left(1+\alpha {\beta }_1\right)}^2}.$$

Because of the fact that $|t_i|\le \parallel t\parallel$ and $n\overline{t}\le \parallel t\parallel ,$ it is easy to see that $\forall \alpha \ge 0,g^{″}\left(\alpha \right)\ge 0.$

Therefore,

$$\frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\sum _{i=1}^n\ln (1+\alpha t_i)\ge \frac{1}{\eta }\left(f(x+\alpha d)-f\left(x\right)\right)-\tau \ln \left(1+{\beta }_1\alpha \right),$$

then, $\phi \left(\alpha \right)\le {\widehat{\phi }}_2\left(\alpha \right).$ □

Hence, the domain of ${\left({\widehat{\phi }}_i\right)}_{i=1,2}$ is included in the domain of $\phi ,$ which is $(0,\tilde{\alpha }),$ where

$$\tilde{\alpha }=\max \left[\alpha :1+\alpha \delta >0,1+\alpha \beta >0\right].$$

Let us remark that

$$0={\widehat{\phi }}_i\left(0\right)=\phi \left(0\right),-{\phi }^{\prime }\left(0\right)=-{\widehat{\phi }}_i^{\prime }\left(0\right)={\phi }_i^{″}\left(0\right)={\widehat{\phi }}_i^{″}\left(0\right)>0,i=1,2.$$

Thus, $\phi$ is well approximated by ${\widehat{\phi }}_i$ in a neighborhood of $0.$ Since ${\widehat{\phi }}_i$ is strictly convex, it attains its minimum at one unique point $\overline{\alpha },$ which is the unique root of the equation ${\widehat{\phi }}_i^{\prime }\left(\alpha \right)=0$. This point belongs to the domain of ${\widehat{\phi }}_i\left(i=1,2\right).$ Therefore, $\phi$ is bounded from below by ${\widehat{\phi }}_1:$

$${\widehat{\phi }}_1\left(\overline{\alpha }\right)\le \phi \left(\overline{\alpha }\right)<0$$

And it is also bounded from below by ${\widehat{\phi }}_2:$

$${\widehat{\phi }}_2\left(\overline{\alpha }\right)\le \phi \left(\overline{\alpha }\right)<0.$$

Then, $\overline{\alpha }$ gives an apparent decrease in the function $\phi .$

4.3. Minimize an Auxiliary Function

We now consider the minimization of the function

$${\phi }_1\left(\alpha \right)=n\gamma \alpha -\ln \left(1+\delta \alpha \right)-(n-1)\ln \left(1+\beta \alpha \right),$$

and we also have the following approximate function:

$${\phi }_2\left(\alpha \right)=n\gamma \alpha -\tau \ln (1+{\beta }_1\alpha ),$$

where ${\beta }_1$ is defined in (7). Then, we have the following:

$$\begin{array}{ccc}\hfill {\phi }_1^{\prime }\left(\alpha \right)& =& n\gamma -\frac{\delta }{1+\delta \alpha }-(n-1)\frac{\beta }{1+\beta \alpha },\hfill \\ \hfill {\phi }_1^{″}\left(\alpha \right)& =& \frac{{\delta }^2}{{\left(1+\delta \alpha \right)}^2}+(n-1)\frac{{\beta }^2}{{\left(1+\beta \alpha \right)}^2}.\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill {\phi }_2^{\prime }\left(\alpha \right)& =& n\gamma -\tau \frac{{\beta }_1}{(1+{\beta }_1\alpha )},\hfill \\ \hfill {\phi }_2^{″}\left(\alpha \right)& =& \tau \frac{{\beta }_1^2}{{\left(1+{\beta }_1\alpha \right)}^2}.\hfill \end{array}$$

We remark that for $i=1,2:$

$${\phi }_i\left(0\right)=0,{\phi }_i^{\prime }\left(0\right)=n\left(\gamma -\overline{t}\right),{\phi }_i^{″}\left(0\right)={∥t∥}^2.$$

We present the conditions ${\phi }_i^{\prime }\left(0\right)<0$ and ${\phi }_i^{″}\left(0\right)>0.$ The function $\phi$ is strictly convex. It attains its minimum at one unique point $\alpha$ such that ${\phi }_i^{\prime }\left(\alpha \right)=0,$ which is one of the roots of the equations

$$\gamma \delta \beta {\alpha }^2+\alpha \left(\gamma \delta +\gamma \beta -\delta \beta \right)+\gamma -\overline{t}=0,$$

(8)

and

$$n\gamma \beta (1+\beta \alpha )-{∥t∥}^2=0.$$

(9)

For Equation (8), the roots are explicitly calculated, and we distinguish the following cases:

• If $\delta =0,$ we obtain ${\overline{\alpha }}_1=\frac{\overline{t}-\gamma }{\gamma \beta }.$
• If $\beta =0,$ we obtain ${\overline{\alpha }}_1=\frac{\overline{t}-\gamma }{\gamma \delta }.$
• If $\gamma =0,$ we have ${\overline{\alpha }}_1=\frac{-\overline{t}}{\delta \beta }.$
• If $\gamma \delta \beta \ne 0,$ $\overline{\alpha }$ is the only root of the second-degree equation that belongs to the domain of definition of $\phi .$ We obtain $▵=\frac{1}{{\gamma }^2}+\frac{1}{{\beta }^2}+\frac{1}{{\delta }^2}-\frac{2}{\delta \beta }+\left(\frac{2n-4}{n}\right)\left[\frac{1}{\beta \gamma }-\frac{1}{\gamma \delta }\right].$ Both roots are

  $${\overline{\alpha }}_{1.1}=\frac{1}{2}\left(\frac{1}{\gamma }-\frac{1}{\beta }-\frac{1}{\delta }+\sqrt{▵}\right),$$

  and

  $${\overline{\alpha }}_{1.2}=\frac{1}{2}\left(\frac{1}{\gamma }-\frac{1}{\beta }-\frac{1}{\delta }+\sqrt{▵}\right).$$

Then, the root of Equation (9) is explicitly calculated, and we have

$${\overline{\alpha }}_2=\left(\frac{{∥t∥}^2}{\beta (1-\beta )}-\frac{1}{\beta }\right).$$

Consequently, we compute the two values ${\overline{\alpha }}_i,i=1,2$, explicitly. Then, we take ${\overline{\alpha }}_1,{\overline{\alpha }}_2\in \left[0,\tilde{\alpha }-\epsilon \right[$, where $\epsilon >0$ is a fixed precision and ${\phi }^{\prime }\left({\overline{\alpha }}_i\right)>0,i=1,2.$

Remark 2.

The computation of ${\overline{\alpha }}_i,i=1,2$ is performed through a dichotomous procedure in the cases where ${\overline{\alpha }}_i\notin (0,\tilde{\alpha }-\epsilon ),$ and ${\phi }^{\prime }\left({\overline{\alpha }}_i\right)>0$, as follows:

Put $a=0,b=\tilde{\alpha }-\epsilon$.

While $\left|b-a\right|>\epsilon$ do

If ${\phi }^{\prime }\left(\frac{a+b}{2}\right)<0$ then, $b=\frac{a+b}{2},$

else $a=\frac{a+b}{2},$ so ${\overline{\alpha }}_i=b.$

This computation guarantees a better approximation of the minimum of ${\phi }^{\prime }\left(\alpha \right)$ while remaining in the domain of $\phi .$

4.4. The Objective Function f Is

4.4.1. Linear

For all $x,$ there exists $c\in {\mathbb{R}}^n$ such that $f\left(x\right)=$$⟨c,x⟩.$

The minimum of ${\widehat{\phi }}_i$ is reached at the unique root $\overline{\alpha }$ of the equation ${\phi }^{\prime }\left(\alpha \right)=0.$ Then,

$${\widehat{\phi }}_i\left(\overline{\alpha }\right)\le \phi \left(\overline{\alpha }\right)<{\widehat{\phi }}_i\left(0\right)=\phi \left(0\right)=0.$$

Take $\gamma =n^{-1}⟨c,d⟩$ in the auxiliary function $\phi .$ The two functions $\phi$ and ${\left({\widehat{\phi }}_i\right)}_{i=1,2}$ coincide.

$\overline{\alpha }$ yields a significant decrease in the function ${\psi }_{\eta }$ along the descent direction $d.$ It is interesting to note that the condition ${\widehat{\phi }}_i^{\prime }\left(0\right)+{\widehat{\phi }}_i^{″}\left(0\right)=0$$(i=1,2)$ implies the following:

$$-{\widehat{\phi }}_i^{\prime }\left(0\right)=n\left(\overline{t}-\gamma \right)={∥t∥}^2=n\left({\overline{t}}^2-{\sigma }_t^2\right)={\widehat{\phi }}_i^{\prime \prime }\left(0\right)>0.$$

4.4.2. Convex

$▿f(x+\alpha d)$ is no longer constant, and the equation ${\widehat{\phi }}_i^{\prime }\left(\alpha \right)=0$ is not reduced to one equation of a second degree for $i=1,2.$

We consider another function $\tilde{\phi }$ less than $\phi .$ Given $\widehat{\alpha }\in \left(0,\tilde{\alpha }\right),$ we have, for all $\alpha \in (0,\widehat{\alpha }]$, the following:

$$\frac{f\left(x+\alpha d\right)-f\left(x\right)}{\eta }\le \frac{f\left(x+\widehat{\alpha }d\right)-f\left(x\right)}{\eta \widehat{\alpha }}\alpha ,$$

(10)

then,

$$\phi \left(\alpha \right)\ge {\tilde{\phi }}_1\left(\alpha \right)=\frac{f\left(x+\widehat{\alpha }d\right)-f\left(x\right)}{\eta \widehat{\alpha }}\alpha -\ln \left(1+\alpha \delta \right)-(n-1)\ln \left(1+\alpha \beta \right)$$

and

$$\phi \left(\alpha \right)\ge {\tilde{\phi }}_2\left(\alpha \right)=\frac{f\left(x+\widehat{\alpha }d\right)-f\left(x\right)}{\eta \widehat{\alpha }}\alpha -\tau \ln (1+{\beta }_1\alpha ),\tau \in \left]0,1\right[.$$

We choose $\gamma =\frac{f\left(x+\widehat{\alpha }d\right)-f\left(x\right)}{n\eta \widehat{\alpha }}$ in the auxiliary function $\phi$, and we compute the root $\overline{\alpha }$ of the equation ${\phi }_i^{\prime }\left(\alpha \right)=0$ with $i=1,2.$

Therefore, we have two cases:

• Where $\overline{\alpha }\le \widehat{\alpha }:$ We have the following:

  $$\phi \left(\overline{\alpha }\right)\ge {\widehat{\phi }}_i\left(\overline{\alpha }\right)\ge {\tilde{\phi }}_i\left(\overline{\alpha }\right),\mathrm{for}i=1,2$$

  and, thus, along the direction $d,$ we obtain a significant decrease in the function ${\psi }_{\eta }.$ The approximation accuracy of $\phi$ by ${\tilde{\phi }}_i$ being better for small values of $\widehat{\alpha }\left(\mathrm{for}i=1,2\right),$ it is recommended to use a new value of $\widehat{\alpha },$ situated between $\tilde{\alpha }$ and the former $\widehat{\alpha },$ for the next iteration. Moreover, the cost of the supplementary computation is small since it is the cost of one evaluation of f and the resolution of a second-order equation.
• Where $\overline{\alpha }>\widehat{\alpha }:$ The computation of $\widehat{\alpha }$ is performed through a dichotomous procedure (see Remark 3).

5. Description of the Algorithm and Numerical Simulations

5.1. Description of the Algorithm

This section is devoted to introducing our algorithm for obtaining an optimal solution $\overline{x}$ of (P).

Begin

Initialization

$\epsilon >0$ is a given precision. $\widehat{\eta }>0$ and $\sigma \in \left[0,1\right]$ are given.

$x_0$ is a strictly realizable solution from $\tilde{\mathcal{L}},$ $d_0\in {\mathbb{R}}^m.$

Iteration

• Start with $\eta >\widehat{\eta }.$
• Calculate d and $t=X^{-1}d.$
• If $∥t∥>\epsilon ,$ calculate $\overline{t},\gamma ,\delta ,\beta$, and ${\beta }_1.$
• Determine $\overline{\alpha }$ following (8), (10), or (9) depending on the linear or nonlinear case.
• Take the new iterate $x=x+\overline{\alpha }d=X\left(e+\overline{\alpha }t\right)$ and go back to step 2.
• If $∥t∥\le \epsilon ,$ a well approximation of $g\left(\eta \right)$ has been obtained.

  (a) 

  If $\eta \ge$ $\widehat{\eta }$ and $\eta =\sigma \eta ,$ return to step 2.

  (b) 

  If $\eta <$ $\widehat{\eta },$ STOP: a well approximate solution of $\left(P\right)$ has been obtained.

End algorithm.

The aim of this method is to reduce the number of iterations and the time consumption. In the next section, we provide some examples.

5.2. Numerical Simulations

To assess the superior performance and accuracy of our algorithm, based on our minorant functions, numerical tests are conducted to make comparisons between our new approach and the classical line search method.

For this purpose, in this section, we present comparative numerical tests on different examples taken from the literature [5,24].

We report the results obtained by implementing the algorithm in MATLAB on an Intel Core i7-7700HQ (2.80 GHz) machine with 16.00 Go RAM.

5.2.1. Examples with a Fixed Size

Nonlinear Convex Objective

Example 1.

Let us take the following problem:

$$\left\{\begin{array}{c}\min 2x_1^2+2x_2^2-2x_1{x}_2-4x_1-6x_2\hfill \\ x_1+x_2+x_3=2\hfill \\ x_1+5x_2+x_4=5\hfill \\ x_1,x_2,x_3,x_4\ge 0.\hfill \end{array}\right.$$

The optimal value is $-7.1613,$ and the optimal solution is $x^{*}={\left(\begin{array}{cccc}1.1290\hfill & 0.7742\hfill & 0.0968\hfill & 0\hfill \end{array}\right)}^t.$

Example 2.

Let us take the following problem:

$$\left\{\begin{array}{c}\min x_1^3+x_2^3\hfill \\ x_1-x_2+x_3+x_4=3\hfill \\ 2x_1+x_2-x_3+x_4=2.0086\hfill \\ x_1+x_3+2x_4=4.9957\hfill \\ x_1,x_2,x_3,x_4\ge 0.\hfill \end{array}\right.$$

The optimal value is $0.0390,$ and the optimal solution is $x^{*}={\left(\begin{array}{cccc}0.3391\hfill & 0\hfill & 0.6652\hfill & 1.9957\hfill \end{array}\right)}^t.$

Example 3.

Let us consider the following problem:

$$\left\{\begin{array}{c}\min x_1^3+x_2^3+x_1{x}_2\hfill \\ 2x_1-x_2+x_3=8\hfill \\ x_1+2x_2+x_4=6\hfill \\ x_1,x_2,x_3,x_4\ge 0.\hfill \end{array}\right.$$

The optimal value is $1.6157,$ and the optimal solution is $x^{*}={\left(\begin{array}{cccc}1.1734\hfill & 0\hfill & 5.6532\hfill & 4.8265\hfill \end{array}\right)}^t.$

This table presents the results of the previous examples:

Example		st1			st2			LS
		iter	Time (s)		iter	Time (s)		iter	Time (s)
1		12	0.0006		19	0.0015		6	0.0091
2		5	0.0004		9	0.0009		44	0.099
3		3	0.0001		5	0.0006		65	0.89

5.2.2. Example with a Variable Size

The Objective Function f Is

1-Linear: Let us consider the linear programming problem:

$$\zeta =\min [c^{T}x:x\ge 0,Ax=b],$$

where A is an $m\times 2m$ matrix given by the following:

$$\begin{array}{ccc}\hfill A[i,j]& =& \left\{\begin{array}{ccc}1\hfill & \mathrm{if}i=j\hfill & \mathrm{or}j=i+m\hfill \\ 0\hfill & \mathrm{if}\mathrm{not},\hfill \end{array}\right.\hfill \\ \hfill c\left[i\right]& =& -1,c[i+m]=0\mathrm{and}b\left[i\right]=2,\forall i=1,...m,\hfill \end{array}$$

where $c,b\in {\mathbb{R}}^{2m}.$

The results are presented in the table below.

Size		st1			st2		LS
		iter	Time (s)		iter	Time (s)	iter	Time (s)
$5\times 10$		1	0.0021		2	0.0039	9	0.0512
$20\times 40$		1	0.0031		3	0.0045	13	0.0821
$50\times 100$		2	0.0049		3	0.0032	17	0.3219
$100\times 200$		2	0.0053		4	0.0088	19	0.5383
$200\times 400$		2	0.0088		4	0.0098	22	0.9220
$250\times 500$		3	0.0096		5	0.0125	26	9.2647

2-Nonlinear:

Example 4

(Quadratic case [13]). Let the quadratic problem be as follows:

$$\zeta =\min [f\left(x\right):x\ge 0,Ax\ge b],$$

with $f\left(x\right)=\frac{1}{2}⟨x,Qx⟩,$ Q is the matrix defined for $n=2m$ by the following:

$$\begin{array}{ccc}\hfill Q[i,j]& =& \left\{\begin{array}{cc}2j-1\hfill & \mathit{if}i>j\hfill \\ 2i-1\hfill & \mathit{if}i<j\hfill \\ i\left(i+1\right)-1\hfill & \mathit{if}i=j,i,j=1,..,n\hfill \end{array}\right.\hfill \\ \hfill A[i,j]& =& \left\{\begin{array}{cc}1\hfill & \mathit{if}i=j\mathit{or}j=i+m,i=1,..,m\mathit{and}j=1,..,n\hfill \\ 0\hfill & \mathit{if}\mathit{not}\hfill \end{array}\right.\hfill \\ \hfill c\left[i\right]& =& -1,c[i+m]=0\mathit{and}b\left[i\right]=2,\forall i=1,..,m.\hfill \end{array}$$

This example is tested for many values of $n.$

The obtained results are given by the following table:

ex($\mathit{m},\mathit{n}$)		st1		st2		LS
		iter	Time (s)	iter	Time (s)	iter	Time (s)
$300\times 600$		5	0.9968	4	0.9699	26	19.5241
$400\times 800$		7	18.1448	5	9.6012	35	86.1259
$600\times 1200$		12	36.3259	5	19.0099	23	98.2354
$1000\times 2000$		21	56.9912	17	41.1012	33	109.2553
$1500\times 3000$		28	140.1325	23	95.6903	40	1599.1596

Example 5

(The problem of Erikson [25]). Let the following be the quadratic problem:

$$\zeta =\min \left[f\left(x\right)=\stackrel{n}{\sum _{i=1}}{x}_i\ln \left(\frac{x_i}{a_i}\right):x_i+x_{i+m}=b,x\ge 0\right],$$

where $n=2m,a_i>0$ and $b\in {\mathbb{R}}^m$ are fixed.

This example is tested for different values of $n,a_i$, and $b_i.$

The following table resumes the obtained results in the case $(a_i=2,\forall i=1,...,n,$ $b_i=4,\forall i=1,...,m):$

ex($\mathit{m},\mathit{n}$)		st1		st2		LS
		iter	Time (s)	iter	Time (s)	iter	Time (s)
$10\times 20$		1	0.0001	2	0.0012	4	0.0236
$40\times 100$		2	0.0021	3	0.0033	5	0.7996
$100\times 200$		2	0.0043	3	0.0201	5	1.5289
$500\times 1000$		2	3.0901	4	5.9619	12	22.1254

In the above tables, we take $\epsilon$ = 1.0 × ${10}^{-4}$.

We also denote the following:

- (iter) is the number of iterations.

- (time) is the computational time in seconds (s).

- (sti)_i=1,2 represents the strategy of approximate functions introduced in this paper.

- (LS) represents the classical line search method.

Commentary: The numerical tests carried out show, without doubt, that our approach leads to a very significant reduction in the cost of calculation and an improvement in the result. When comparing the approximate functions to the line search approach, the number of iterations and computing time are significantly reduced.

6. Conclusions

The contribution of this paper is particular focused on the study of nonlinear optimization problems by using the logarithmic penalty method based on some new approximate functions. We first formulate the problems $\left(P\right)$ and $\left(P\eta \right)$ with the results of the convergence. Then, we find their solutions by using new approximate functions.

Finally, to lend further support to our theoretical results, a simulation study is conducted to illustrate the good accuracy of the studied approach. More precisely, our new approximate functions approach outperforms the line search one as it significantly reduces the cost and computing time.