An Efficient Penalty Method without a Line Search for Nonlinear Optimization

: In this work, we integrate some new approximate functions using the logarithmic penalty method to solve nonlinear optimization problems. Firstly, we determine the direction by Newton’s method. Then, we establish an efficient algorithm to compute the displacement step according to the direction. Finally, we illustrate the superior performance of our new approximate function with respect to the line search one through a numerical experiment on numerous collections of test problems.


Introduction
The nonlinear optimization is a fundamental subject in the modern optimization literature.It focuses on the problem of optimizing an objective function in the presence of inequality and/or equality constraints.Furthermore, the optimization problem is obviously linear if all the functions are linear, otherwise it is called a nonlinear optimization problem.
This research field is motivated by the fact that it arises in various problems encountered in practice, such as business administration, economics, agriculture, mathematics, engineering, and physical sciences.
In our knowledge, Frank and Wolfe are the deans in nonlinear optimization problems.They established a powerful algorithm in [1] to solve them.Later, they used another method in [2] based on the application of the Simplex method on the nonlinear problem after converting it to a linear one.
This pioneer work inspired many authors to propose and develop several methods and techniques to solve this class of problems.We refer to [3,4] for interior point methods to find the solution of nonlinear optimization problems with a high dimension.
In order to make this theory applicable in practice, other methods are designed on the linear optimization history, among robust algorithms with polynomial complexity.In this perception, Khachian succeeded in 1979 to introduce a new ellipsoid method from approaches applied originally to nonlinear optimization.
Interior point methods can be classified into three different groups as follows: projective methods and their alternatives as in Powell [10] and Rosen [11,12], central trajectory methods (see Ouriemchi [13] and Forsgren et al. [14]), and barrier/penalty methods, where majorant functions were originally proposed by Crouzeix and Merikhi [15] to solve a semidefinite optimization problem.Inspired by this work, Menniche and Benterki [16] and Bachir Cherif and Merikhi [17] applied this idea to linear and nonlinear optimizations, respectively.
A majorant function for the penalty method in convex quadratic optimization was proposed by Chaghoub and Benterki [18].On the other hand, A. Leulmi et al. [19,20] used new minorant functions for semidefinite optimization, and this idea was extended to linear programming by A. Leulmi and S. Leulmi in [21].
As far as we know, our new approximate function has not been studied in the nonlinear optimization literature.These approximate functions are more convenient and efficient than the line search method for rapidly computing the displacement step.
Therefore, in our work, we aim to optimize a nonlinear problem based on prior efforts.Thus, we propose a straightforward and effective barrier penalty method using new minorant functions.
More precisely, we first introduce the position of the problem and its perturbed problem with the results of convergence in Sections 2 and 3 of our paper.Then, in Section 4, we establish the solution of the perturbed problem by finding new minorant functions.Section 5 is devoted to presenting a concise description of the algorithm and to illustrating the outperformance of our new approach by carrying out a simulation study.Finally, we summarize our work in the conclusion.
Throughout this paper, the following notations are adopted.Let ⟨., .⟩and ∥.∥ denote the scalar product and the Euclidean norm, respectively, given by the following:

The Problem
We aim to present an algorithm for solving the following optimization problem: where b ∈ R m and A ∈ R m×n is a full-rank matrix with m < n.
For this purpose, we need the following hypothesis: Hypothesis 1. f is nonlinear, twice continuously differentiable, and convex on L, where L = {x ∈ R n : Ax = b; x ≥ 0} is the set of realizable solutions of (P).
Hypothesis 2. (P) satisfies the condition of interior point (IPC), i.e., there exists x 0 > 0 such that Ax 0 = b.Hypothesis 3. The set of optimal solutions of (P) is nonempty and bounded.
Notice that these conditions are standard in this context.We refer to [17,20].If x * is an optimal solution, there exist two Lagrange multipliers p * ∈ R m and q * ∈ R n , such that (1)

Formulation of the Perturbed Problem of (P)
Let us first consider the function ψ defined on R × R n by the following: where ξ : R 2 −→ (−∞, +∞] is a convex, lower semicontinuous and proper function given by the following: Thus, ψ is a proper, convex, and lower semicontinuous function.Furthermore, the function g defined by is convex.Notice that for η = 0, the perturbed problem (Pη) coincides with the initial problem (P); then, f * = g(0).

Existence and Uniqueness of Optimal Solution
To show that the perturbed problem (Pη) has a unique optimal solution, it is sufficient to demonstrate that the recession cone of ψ η is reduced to zero.
Proof.For a fixed η, the function ψ η is proper, convex, and lower semicontinuous.The asymptotic function of ψ η is defined by the following: thus, the asymptotic functions of f and ψ η satisfy the relation: Moreover, hypothesis H3 is equivalent to and from [17], for each non-negative real number η, the strictly convex problem (Pη) admits a unique optimal solution noted by x * η .The solution of the problem (P) is the limit of the solutions sequence of the perturbed problem (Pη) when η tends to 0.

Convergence of the Solution
Now, we are in a position to state the convergence result of (Pη) to (P), which is proved in Lemma 1 on [18].
We use the classical prototype of penalty methods.We begin our process with and the iteration scheme is divided into the following steps: If the values of the objective functions of the problem (P) and the perturbed problem (Pη) are equal and finite, then (P) will have an optimal solution if and only if (Pη) has an optimal solution.
The iterative process stops when we obtain an acceptable approximation of g(0).

Computational Resolution of the Perturbed Problem
Our approach to the numerical solution of the perturbed problem (Pη), consists of two stages.In the first one, we calculate the descent direction using the Newton approach, and in the second one, we propose an efficient new-minorant-functions approach to compute the displacement step easily and quickly relative to the line search method.

The Descent Direction
As (Pη) is strictly convex, the necessary and sufficient optimality conditions state that x η is an optimal solution of (Pη) if and only if it satisfies the nonlinear system: Using the Newton approach, a penalty method is provided to solve the above system, where the vector x k+1 in each is given by The solution of the following quadratic convex optimization problem is necessary to obtain the Newton descent direction d : where x ∈ L and The Lagrangian is given by the following: where s ∈ R m is the Lagrange multiplier.It is sufficient for solving the linear system equations with n + m : It is simple to prove that system (3) is non-singular.We obtain As d T A T s = (Ad) T s = 0 and Ad = 0, we obtain The system can also be written as follows: Thus, the Newton descent direction is obtained.Throughout this paper, we take x instead of x η .

Computation of the Displacement Step
This section deals with the numerical solution of the displacement step.We give a brief highlight of the line search methods used in nonlinear optimization problems.Then, we collect some important results of approximate function approaches applied to both semidefinite and linear programming problems.Finally, we propose our new approximate function method for the nonlinear optimization problem (P).

Line Search Methods
The line search methods consists of determining a displacement step α k , which ensures the sufficient decrease in the objective at each iteration x k+1 = x k + α k d k , where α k > 0, along the descent direction d k ; in other words, it involves solving the following onedimensional problem: The disadvantage of this method is that the solution α is not necessarily optimal, which make the feasibility of x k+1 not guaranteed.
The line search techniques of Wolfe, Goldstein-Armijo, and Fibonacci are the most widely used ones.However, generally, their computational volume is costly.This is what made us search for another alternative.

Approximate Functions Techniques
These methods are based on sophisticated techniques introduced by J.P. Crouzeix et al. [15] and A. Leulmi et al. [20] to obtain the solution of a semidefinite optimization problem.
The aim of these techniques is to give a minimized approximation of one real-variable function φ(α) defined by The function φ is convex, and we obtain the following: We find that φ ′ (0) + φ ′′ (0) = 0, deduced from (4), which is expected since d is the direction of Newton's descent direction.
We aim to avoid the disadvantages of line search methods and accelerate the convergence of the algorithm.For this reason, we have to identify an ᾱ that yields a significant decrease in the function φ(α).This is the same as solving a polynomial equation of degree n + 1, where f is a linear function.Now, we include a few helpful inequalities below, which are used throughout the paper.H. Wolkowicz et al. [22] see also Crouzeix and Seeger [23] presented the following inequalities: where z and σ z represent the mean and the standard deviation, respectively, of a statistical real numbers series {z 1 , z 2 , . . ., z n }.The later quantities are defined as follows: Theorem 1 ([15]).Let z i > 0, for i = 1, 2, . . ., n.We have the following: where z i = 1 + αt i , z = 1 + αt, and σ z = ασ t .
We will proceed to present the paper's principal result.

New Approximate Functions Approach
Let To find the displacement step, it is necessary to solve φ ′ (α) = 0. Considering the difficulty of solving a non-algebraic equation, approximate functions are recommended alternatives.
Two novel approximation functions of φ are introduced in the following lemma.
Lemma 2. For all α ∈ 0, α * 1 with α * 1 = min( α, α 1 ), we have and for all α ∈ [0, α * 2 [ with α * 2 = min( α, α 2 ), we obtain where and Furthermore, we have Proof.We start by proving that φ(α) ≥ φ 1 (α), and Hence, Let us consider the following: We have Because of the fact that |t i | ≤ ∥t∥ and nt ≤ ∥t∥, it is easy to see that ∀α ≥ 0, g ′′ (α) ≥ 0. Therefore, Hence, the domain of ( φ i ) i=1,2 is included in the domain of φ, which is (0, α), where Thus, φ is well approximated by φ i in a neighborhood of 0. Since φ i is strictly convex, it attains its minimum at one unique point ᾱ, which is the unique root of the equation φ ′ i (α) = 0.This point belongs to the domain of φ i (i = 1, 2).Therefore, φ is bounded from below by φ 1 : And it is also bounded from below by φ 2 : Then, ᾱ gives an apparent decrease in the function φ.

Minimize an Auxiliary Function
We now consider the minimization of the function and we also have the following approximate function: where β 1 is defined in (7).Then, we have the following: We remark that for i = 1, 2 : We present the conditions φ ′ i (0) < 0 and φ ′′ i (0) > 0. The function φ is strictly convex.It attains its minimum at one unique point α such that φ ′ i (α) = 0, which is one of the roots of the equations and For Equation ( 8), the roots are explicitly calculated, and we distinguish the following cases:
Remark 2. The computation of ᾱi , i = 1, 2 is performed through a dichotomous procedure in the cases where ᾱi / ∈ (0, α − ε), and φ ′ (ᾱ i ) > 0, as follows: 2 , so ᾱi = b.This computation guarantees a better approximation of the minimum of φ ′ (α) while remaining in the domain of φ.For all x, there exists c ∈ R n such that f (x) = ⟨c, x⟩.The minimum of φ i is reached at the unique root ᾱ of the equation φ ′ (α) = 0.Then, Take γ = n −1 ⟨c, d⟩ in the auxiliary function φ.The two functions φ and ( φ i ) i=1,2 coincide.
ᾱ yields a significant decrease in the function ψ η along the descent direction d.It is interesting to note that the condition φ ′ i (0) + φ ′′ i (0) = 0 (i = 1, 2) implies the following:

Convex
▽ f (x + αd) is no longer constant, and the equation φ ′ i (α) = 0 is not reduced to one equation of a second degree for i = 1, 2.
Where ᾱ ≤ α : We have the following: and, thus, along the direction d, we obtain a significant decrease in the function ψ η .
The approximation accuracy of φ by φ i being better for small values of α (for i = 1, 2), it is recommended to use a new value of α, situated between α and the former α, for the next iteration.Moreover, the cost of the supplementary computation is small since it is the cost of one evaluation of f and the resolution of a second-order equation.

Description of the Algorithm
This section is devoted to introducing our algorithm for obtaining an optimal solution x of (P).
x 0 is a strictly realizable solution from L, d 0 ∈ R m .Iteration
If ∥t∥ ≤ ε, a well approximation of g(η) has been obtained.

(b)
If η < η, STOP: a well approximate solution of (P) has been obtained.

End algorithm.
The aim of this method is to reduce the number of iterations and the time consumption.In the next section, we provide some examples.

Numerical Simulations
To assess the superior performance and accuracy of our algorithm, based on our minorant functions, numerical tests are conducted to make comparisons between our new approach and the classical line search method.
For this purpose, in this section, we present comparative numerical tests on different examples taken from the literature [5,24].
We report the results obtained by implementing the algorithm in MATLAB on an Intel Core i7-7700HQ (2.80 GHz) machine with 16.00 Go RAM.

Examples with a Fixed Size Nonlinear Convex Objective
Example 1.Let us take the following problem: The optimal value is −7.1613, and the optimal solution is x * = 1.1290 0.7742 0.0968 0 t .
Example 2. Let us take the following problem: The optimal value is 0.0390, and the optimal solution is x * = 0.3391 0 0.6652 1.9957 t .
Example 3. Let us consider the following problem: The optimal value is 1.6157, and the optimal solution is x * = where A is an m × 2m matrix given by the following: with f (x) = 1 2 ⟨x, Qx⟩, Q is the matrix defined for n = 2m by the following: This example is tested for many values of n.
The obtained results are given by the following In the above tables, we take ε = 1.0 × 10 −4 .We also denote the following: -(iter) is the number of iterations.
-(time) is the computational time in seconds (s).
-(sti) i=1,2 represents the strategy of approximate functions introduced in this paper.
-(LS) represents the classical line search method.
Commentary: The numerical tests carried out show, without doubt, that our approach leads to a very significant reduction in the cost of calculation and an improvement in the result.When comparing the approximate functions to the line search approach, the number of iterations and computing time are significantly reduced.

Conclusions
The contribution of this paper is particular focused on the study of nonlinear optimization problems by using the logarithmic penalty method based on some new approximate functions.We first formulate the problems (P) and (Pη) with the results of the convergence.Then, we find their solutions by using new approximate functions.
Finally, to lend further support to our theoretical results, a simulation study is conducted to illustrate the good accuracy of the studied approach.More precisely, our new approximate functions approach outperforms the line search one as it significantly reduces the cost and computing time.

4. 4 .
The Objective Function f Is 4.4.1.Linear where c, b ∈ R 2m .The results are presented in the table below.

table :
[25]ple 5 (The problem of Erikson[25]).Let the following be the quadratic problem:ζ = min f (x) = + x i+m = b, x ≥ 0 ,where n = 2m, a i > 0 and b ∈ R m are fixed.This example is tested for different values of n, a i , and b i .The following table resumes the obtained results in the case (a i = 2, ∀i = 1, . . ., n, b i = 4, ∀i = 1, . . ., m) :