Improved Algorithms Based on Trust Region Framework for Solving Unconstrained Derivative Free Optimization Problems

Abstract

This paper is devoted to developing new derivative-free optimization (DFO) methods for solving optimization problems where derivative information is not available or cannot be calculated numerically. To overcome the computational difficulties arising from time-consuming estimations of the objective function, we propose two algorithms. One algorithm is a variant of the surrogate model-based DFO algorithm under the trust region method, where the surrogate model is formulated through sparse low-rank interpolation and quadratic polynomial interpolation. This algorithm serves as a comparative baseline. The second algorithm leverages the characteristics of the sparse regression model, which can handle sparsity and noise issues, to construct the surrogate model. The coefficients of the sparse surrogate model are then estimated using the alternating direction method of multipliers and refined through a correction strategy based on the R-square. Finally, numerical results, evaluated in terms of performance and data profiles, demonstrate the effectiveness and competitiveness of the proposed algorithms.

1. Introduction

Derivative-free optimization (DFO) is a common topic in nonlinear optimization literature, which aims to solve problems where the derivatives of the objective function (or constraints) are difficult to obtain or even unavailable. The growth in computing power for scientific, engineering, and other applications has made DFO methods widely used in many real-world scenarios. For example, the objective function (or constraints) might be formulated through data analytic tools [1,2] or simulation methods [3,4]. In such cases, once the inputs are provided, the outputs can be computed. However, the disadvantage of these approaches is that the algebraic expression of the models can be complex or unknown, making the corresponding derivatives hard to compute or unavailable. Thus, there is a need for the study of DFO methods to address such problems.

Based on whether the algorithm has a foundation for convergence analysis, DFO methods can be broadly classified into heuristic methods and non-heuristic methods. Common heuristic-based methods include particle swarm optimization [5], genetic algorithms [6], and estimation of distribution algorithms [7], etc. In this paper, we focus on studying the algorithms that belong to non-heuristic methods. The development of non-heuristic methods DFO methods dates back to the works of Spendley et al. [8], and Powell [9]. A considerable amount of research has been conducted from the theoretical perspective, for example, in [10,11,12]. These approaches can be classified into local search methods and global search methods [13]. Methods such as mesh adaptive direct search (MADS) and trust region-based methods are classical examples of local search methods. The former are categorized as direct search DFO methods, while the latter are model-based DFO methods [10]. Conversely, global search methods can be further divided into three categories: global deterministic search, model-based search, and stochastic search methods. More details about DFO methods can be found in [10,14]. Beside, many research have been done from the practical perspective. Besides, much research has been done from the practical perspective. For example, Gao et al. compared three non-heuristic DFO algorithms and applied them to the design of polymer microstructures and the control of polymerization processes [15].

For example, Gao et al. compared three non-heuristic DFO algorithms and applied them to the design of polymer microstructures and the control of polymerization processes [15]. Lucidi et al. utilized DFO algorithms to facilitate the planning and management of hospital services [16]. Boukouvala et al. applied the DFO algorithm proposed in [17] to optimize the pressure swing adsorption process [18]. Liu et al. developed a model-based DFO algorithm and applied it to solve the operation optimization problem in the basic oxygen furnace (BOF) steelmaking process [19].

For many optimization problems abstracted from practical issues, the objective function may be smooth or locally smooth. It is valuable to make full use of known information and approximate the objective function using a surrogate model [14]. Furthermore, as mentioned above, trust region-based DFO methods are a type of model-based approach that has been shown to perform well and has been studied by many researchers [13]. Therefore, this paper is devoted to employing surrogate model-based DFO algorithms within the framework of the trust region method for solving unconstrained optimization problems extracted from real-world applications. The problem is formulated as follows.

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right),$$

(1)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ for which derivatives are difficult to obtain or even absent. $x\in {\mathbb{R}}^n$. In practical problems, such as those abstracted from the BOF steelmaking process [19], x corresponds to the amounts of raw materials added, and $f\left(x\right)$ represents the corresponding output, which measures the performance of the products. Furthermore, the goal of the optimization problem (1) is to find an optimal value $x^{*}$ that meets practical production requirements. Therefore, it is crucial to study methods for solving problem (1).

As with other algorithm research, performance is a key criterion for deploying and measuring model-based DFO methods. Powell’s work promoted the development of these methods [14]. Furthermore, to enhance algorithm performance, he adopted various interpolation error bounds to ensure convergence, which was subsequently verified through numerical experiments. Additionally, he discovered that quadratic interpolation is more effective than linear interpolation [20]. Conn and Toint [21] employed a DFO algorithm utilizing a trust region framework for unconstrained optimization problems, and they utilized quadratic interpolation for constructing the surrogate model. Numerical experiments were conducted to assess the performance of the proposed algorithm. Conn et al. [22] derived another surrogate model-based trust region method for problem-solving, adopting the interpolation error bound from [23]. In this work, first-order convergence was established. Moreover, the geometry of sample sets, which influences the quality of the interpolation model, is another crucial aspect for DFO algorithms. For instance, works in [22,24,25,26] all examined the convergence of their proposed algorithms from the perspective of the geometric properties of interpolation sets. It is important to note that the geometry of the interpolation set impacts the interpolation error and, consequently, the algorithm’s convergence.

In addition, the number of function evaluations is also a crucial factor to consider when designing DFO algorithms, as it directly influences the execution time. This factor is partly determined by the size of the interpolated sample set. When a quadratic model is used to approximate the objective function in problem (1), ${\displaystyle \frac{1}{2}}(n+1)(n+2)$ independent parameters need to be evaluated, which can be computationally challenging for large n. To address this issue, Powell developed the NEWUOA software, which requires only m samples (where $m(n+2\le m\le {\displaystyle \frac{1}{2}}(n+1)(n+2))$ to build the quadratic interpolation model [27]. Inspired by the work in [26,28], and considering the sparsity of the Hessian matrix, Bandeira et al. [29] proposed a derivative-free algorithm within the framework of a sparse low-degree interpolation trust region. In this algorithm, only $n+1<m\le {\displaystyle \frac{1}{2}}(n+1)(n+2)$ samples are needed at each iteration. Additionally, $l_1$-norm optimization and $l_2$-norm optimization are employed to determine the coefficients of the interpolation model when $m<{\displaystyle \frac{1}{2}}(n+1)(n+2)$. The numerical results presented in [29] showed that the proposed algorithms outperform NEWUOA, and using the minimum $l_1$-norm model was superior to using the minimum $l_2$-norm model. Furthermore, besides the interpolation method, when the cardinality of the sample set is more than ${\displaystyle \frac{1}{2}}(n+1)(n+2)$ the regression method can also be used to establish the surrogate model [30]. This suggests that both interpolation and regression are suitable for formulating the surrogate model when the number of samples is adequate. This encourages us to explore the application of sparse modeling to formulate the surrogate model when the number of samples is insufficient.

This paper targets to propose novel efficient algorithms for solving the unconstrained DFO problems. One of the algorithms described in this paper is based on the work found on [10], p. 249. The objective of this algorithm is to validate the efficacy of incorporating the alternating direction method of multipliers (ADMM) into a DFO method for solving $l_1$-norm optimization problem. Furthermore, in the literature, there are few works that verify the effectiveness of sparse regression modeling when used to determine the surrogate model. Considering that function evaluation is time-consuming for some practical problems, another algorithm proposed in this paper employs the sparse regression method, LASSO, to find the surrogate model. Moreover, numerical experiments demonstrate that the proposed algorithm is comparable and competitive. In addition to the aforementioned work, the following work are also accomplished.

(1)

To enhance the goodness of fit and ensure the sparsity of the surrogate model, a correction strategy is implemented to guarantee the feasibility of the proposed algorithm when applying LASSO to find the surrogate model.

(2)

ADMM is utilized to determine the coefficients of the sparse interpolation model and for LASSO modeling.

The remainder of this paper is organized as follows. Section 2 presents the framework of the model-based trust region method. Section 3 describes the sparse regression modeling, as well as the proposed DFO algorithms. In Section 4, the numerical experiments are performed. Conclusions of our work are given in the last section.

2. The Framework of Algorithms

In this paper, a model-based trust region framework is adopted to develop new approaches for solving DFO problems. The trust region method is an iterative algorithm for solving nonlinear programming problems [10]. The basic algorithm flowchart is shown in Figure 1. The procedure starts from a given initial solution and, through gradual iteration and continuous updating, the algorithm terminated until a satisfactory approximate optimal solution is reached. Indeed, the basic idea of the traditional trust region method is to transform the optimization problem into a series of local sub-problems. In this process, we first

The key step of trust region methods is to determine a trial step d at each iteration, where d satisfies

$$\mathcal{B}=\{d\in {\mathbb{R}}^n|\parallel d{\parallel }_2\le \Delta \},$$

(2)

where $\mathcal{B}$ represents the trust region at the k-th iteration, $\Delta$ is the corresponding trust region radius. ${\parallel ·\parallel }_2$ denotes the Euclidean $l_2$-norm, i.e., ${\parallel d\parallel }_2=\sqrt{{\sum }_{i=1}^n{d}_i^2}$. At each iteration, to obtain d, solution to the following trust region sub-optimization problem (3) should be sought.

$$\underset{d\in \mathcal{B}}{min}q(x+d)=f\left(x\right)+\langle g,d\rangle +{\displaystyle \frac{1}{2}}\langle d,Hd\rangle ,$$

(3)

where $g\in {\mathbb{R}}^n$ and $H\in S_n$ are the gradient and Hessian matrix information of $f\left(x\right)$, respectively. $\langle ·,·\rangle$ represents the inner product, for example, $\langle x,y\rangle ={\sum }_i^n{x}_i{y}_i,x,y\in {\mathbb{R}}^n$. And $S_n$ denotes the set of $n\times n$ symmetric matrices.

In trust region method, the ratio r is defined to measure the agreement between $q\left(x\right)$ and $f\left(x\right)$, which affects the update of x and $\Delta$ at each iteration. As shown in Figure 1, when $r\ge {\tilde{h}}_2$ holds, there will be $x^{k+1}=x^k+d^k$ and ${\Delta }^{k+1}={\tilde{\lambda }}_2{\Delta }^k$. Following gives the definition of ratio r.

$$r={\displaystyle \frac{Ared}{Pred}},$$

(4)

where $Ared=f\left(x\right)-f(x+d)$ and $Pred=q\left(x\right)-q(x+d)$.

Apart from the aforementioned steps, as shown in Figure 1, we need to select an initial point $x_0\in {\mathbb{R}}^n$ during the initialization process, define constants ${\tilde{h}}_1$ and ${\tilde{h}}_2$ to determine how to update the parameters based on the comparison of the ratio r with ${\tilde{h}}_1$ and ${\tilde{h}}_2$. Additionally, two constants ${\tilde{\lambda }}_1$ and ${\tilde{\lambda }}_2$ need to be set for shrinking and expanding the trust region radius $\Delta$. To determine whether the algorithm should terminate, we also need to specify constants ${\epsilon }_g$ and $\delta$ to assess termination based on the norm values of the gradient and Hessian matrix.

It is evident that g and H are essential for solving the sub-problem (3) in the general trust region method. Therefore, when the derivatives of $f\left(x\right)$ are difficult to obtain or unavailable, it becomes crucial to find a surrogate model for $q\left(x\right)$ to approximate g and H. Consequently, finding a surrogate model for the objective function is another key step in the model-based trust region method. In this paper, sparse regression modeling methods and quadratic polynomial interpolation are employed to determine this surrogate model.

3. Description of the Proposed Approach

In the study of DFO algorithm, the number of function evaluation is an important factor which effects the algorithm execution time. In fact, computing the objective function value is expensive for many applications. However, saving computing time is an important issue to resolve urgently for most of the practicality problems. Thus, methods proposed in this paper devote to time-saving.

In this section, the proposed DFO algorithms will be presented in detail, and the implementation of the surrogate model will be described. Thereinto, one of the proposed algorithms is a variant of the algorithm employed in [29], so that only specific differences are described. When the number of samples is sufficient, polynomial interpolation provide an alternative which is presented in Section 3.1. In Section 3.2, sparse regression modeling is taken into consideration when the number of samples is inadequate, where ADMM connecting with the proposed correction strategy help to determine the model coefficient. Finally, two DFO methods are described in Section 3.3.

3.1. Interpolation Based Surrogate Model Formulation

In model-based DFO, fully quadratic models of $f\left(x\right)$ are often obtained from the class of quadratic polynomials by interpolating $f\left(x\right)$ on some sample points. A comprehensive description of quadratic polynomial interpolation can be found in [10]. In this section, we will provide a concise overview of the fundamental concepts and the essential notation.

Define ${\mathcal{P}}_n^2$ as the space of polynomials in ${\mathbb{R}}^n$ of degree less than or equal to 2. Obviously, the dimension of this space is $q={\displaystyle \frac{1}{2}}(n+1)(n+2)$. Suppose $\phi \left(x\right)=\{{\phi }_0\left(x\right),\cdots ,{\phi }_{q-1}\left(x\right)\}$ be a basis in ${\mathcal{P}}_n^2$ and $m\left(x\right)\in {\mathcal{P}}_n^2$ be the surrogate model of the objective function of optimization problem (1), then $m\left(x\right)$ can be written as (5).

$$m\left(x\right)=\sum _{i=0}^{q-1}{\beta }_i{\phi }_i\left(x\right).$$

(5)

where ${\beta }_i(i=1,\cdots ,q-1)$ is the interpolation coefficients. Assume the interpolation set at k-th iteration is $Y=\{y_1,y_2,\cdots ,y_m\}\in {\mathbb{R}}^n$, and m is the number of interpolation samples. Then, the coefficients ${\beta }_i(i=1,\cdots ,q-1)$ can be determined from the interpolation conditions (6).

$$m\left(y_j\right)=\sum _{i=0}^{q-1}{\beta }_i{\phi }_i\left(y_j\right)=f\left(y_j\right),i=1,\cdots ,m$$

(6)

Conditions (6) form a linear system which can be written in matrix form as (7).

$$M(\phi ,Y)\beta =f\left(Y\right),$$

(7)

where $f\left(Y\right)={(f\left(y_1\right),\cdots ,f\left(y_m\right))}^T$ and $M(\phi ,Y)\in {\mathbb{R}}^{m\times q}$ is defined as

$$M(\phi ,Y)=\left(\begin{array}{cccc}{\phi }_0\left(y_1\right)& {\phi }_1\left(y_1\right)& \cdots & {\phi }_{q-1}\left(y_1\right)\\ {\phi }_0\left(y_2\right)& {\phi }_1\left(y_2\right)& \cdots & {\phi }_{q-1}\left(y_2\right)\\ ⋮& ⋮& \cdots & ⋮\\ {\phi }_0\left(y_m\right)& {\phi }_1\left(y_m\right)& \cdots & {\phi }_{q-1}\left(y_m\right)\end{array}\right),$$

(8)

Generally, to build a fully quadratic class of model $m\left(x\right)$, it is necessary that the following Theorem 1 holds ([10], Chapter 6).

Theorem 1. 

Assume that $f\left(x\right)$ is twice differentiable with a Lipschitz continuous Hessian. There exists a $m\left(x\right)$ satisfies conditions including a Lipschitz continuous Hessian with a corresponding Lipschitz constant bounded by a positive constant $v_2^m$. where $m\left(x\right)$ is the quadratic approximation surrogate model for $f\left(x\right)$ determined by the quadratic polynomial interpolation method. And sample set Y is suitable, namely Y is poised for polynomial interpolation in ${\mathbb{R}}^n$ provided that the corresponding matrix $M(\varphi ,Y)$ is non-singular for the basis ϕ ([10], Page 37). For any $y\in Y$, the following holds.

(1) $m\left(y\right)\stackrel{def}{=}f\left(y\right)$((indicating that $m\left(x\right)$ and $f\left(x\right)$ are equal at the sample point y));

(2) $\parallel {\nabla }_{y}m\left(y\right)-{\nabla }_y{f\left(y\right)\parallel }_2\le {\kappa }_{eg}{(\Delta )}^2$;

(3) $\parallel {\nabla }_{yy}m\left(y\right)-{\nabla }_{yy}{f\left(y\right)\parallel }_2\le {\kappa }_{e_H}\Delta$;

where ${\kappa }_{eg}$ and ${\kappa }_{e_H}$are positive constants, $\Delta =\Delta \left(Y\right)$ states the smallest radios of a ball $B\left(Y\right)$ containing Y.

Theorem 1 implies that, in order to formulate fully quadratic surrogate model $m\left(x\right)$ for the objective function of optimization problem (1), in the context of quadratic interpolation, a total of ${\displaystyle \frac{1}{2}}(n+1)(n+2)$ samples are required. Thus, constructing a fully quadratic surrogate model can be computationally expensive, especially as the dimension n increases, due to the cost of obtaining function values and maintaining sample points. While the cost is negligible for small n, it becomes significant for larger values. Furthermore, in numerous practical problems, evaluating the function values is a time-consuming process. To overcome this issue, Powell [27], Bandeira et al. [29] and other researchers tried to use a small number of interpolation points to formulate the interpolation model. In this way, the number of interpolation points $m<{\displaystyle \frac{1}{2}}(n+1)(n+2)$, which makes the linear system (7) an underdetermined one.

3.2. Sparse Surrogate Model Formulation

As mentioned above, when the number of sample points $m<{\displaystyle \frac{1}{2}}(n+1)(n+2)$, the linear system (7) becomes underdetermined. Assuming that the Hessian of the objective function is sparse, Bandeira et al. have incorporated sparse low-degree quadratic polynomials for interpolation to overcome this issue [29]. Unfortunately, in many optimization problems, the structure of the Hessian is unknown or the sparse structure of the model is unclear. Moreover, Theorem 1 assumes that $f\left(x\right)$ is twice differentiable with a Lipschitz continuous Hessian, but this condition may not be satisfied in many optimization problems. To address these challenge, when $m<q$, this paper employs the sparse regression modeling method to identify an appropriate surrogate model.

3.2.1. Formulation of Sparse Model

In the case where $m<{\displaystyle \frac{1}{2}}(n+1)(n+2)$, to construct the quadratic surrogate model, the following $l_0$-norm minimization optimization problem is formulated to determine model coefficients $\beta$.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{\beta }{argmin}{\parallel \beta \parallel }_0\hfill \\ \hfill & s.t.M(\phi ,Y)\beta =f\left(Y\right).\hfill \end{array}\end{array}$$

(9)

Here, ${\parallel ·\parallel }_0$ denotes Euclidean $l_0$-norm, which counts the number of non-zero elements in a vector. The problem (9) is known to be NP-hard [31]. A common strategy to address this challenge is to relax it to the $l_1$ minimization optimization problem (10), where ${\parallel \beta \parallel }_1$ is the sum of the absolute values of all elements in vector $\beta$. Additionally, if the restricted isometry property (RIP) [31] holds, the optimization problems (9) and (10) have the same solution.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{\beta }{argmin}{\parallel \beta \parallel }_1\hfill \\ \hfill & s.t.M(\phi ,Y)\beta =f\left(Y\right).\hfill \end{array}\end{array}$$

(10)

However, when using regression methods to establish the surrogate model, there will be regression errors at the sample data points, or noise present in practical problems. Consequently, problem (10) is often relaxed to the following optimization problem (11) with inequality constraints.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{\beta }{argmin}{\parallel \beta \parallel }_1\hfill \\ \hfill & {s.t.\parallel M(\phi ,Y)\beta -f\left(Y\right)\parallel }_2^2\le \epsilon ,\hfill \end{array}\end{array}$$

(11)

where $\epsilon \in \mathbb{R}$ is a constant. (11) is a quadratically constrained convex optimization problem. With Lagrangian relaxation, (11) can be reformulated as:

$$\underset{\beta ,{\widehat{\lambda }}_L}{min}{\displaystyle \frac{1}{2}}{\parallel M(\phi ,Y)\beta -f\left(Y\right)\parallel }_2^2+{\widehat{\lambda }}_L{\parallel \beta \parallel }_1.$$

(12)

where ${\widehat{\lambda }}_L>0$ is a Lagrange multiplier. When Lagrange multiplier ${\widehat{\lambda }}_L$ specified, problem (12) becomes an $l_1$ regularization regression problem, also known as the LASSO model [32]. In this problem, ${\widehat{\lambda }}_L$ controls the sparsity of the coefficient vector $\beta$. This formulation allows for a more flexible and computationally efficient solution compared to directly solving the quadratically constrained problem. Furthermore, the method for determining ${\widehat{\lambda }}_L$ will be introduced later in the text.

3.2.2. ADMM Algorithm for Estimating Model Coefficient

According to algorithm comparison experiment in literature [33], it found that ADMM is one of the most effective approach for solving optimization problem with the form $minf\left(x\right)+{\parallel x\parallel }_1$, where $f\left(x\right)$ is convex and easy solvable. Thus, ADMM is adopted to estimate coefficient $\beta$ of problem (12).

Define

$$h\left(\beta \right)={\displaystyle \frac{1}{2}}{\parallel M(\phi ,Y)\beta -f\left(Y\right)\parallel }_2^2,$$

and

$$g\left(\xi \right)={\widehat{\lambda }}_L{\parallel \xi \parallel }_1,$$

where $h:{\mathbb{R}}^q\to \mathbb{R}$ and $g:{\mathbb{R}}^q\to \mathbb{R}$ are both convex. Then problem (12) is equal to the following optimization problem (13).

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{\beta ,\xi }{min}{\displaystyle \frac{1}{2}}{\parallel M(\phi ,Y)\beta -f\left(Y\right)\parallel }_2^2+{\widehat{\lambda }}_L{\parallel \xi \parallel }_1\hfill \\ \hfill & s.t.\beta -\xi =0.\hfill \end{array}\end{array}$$

(13)

Thus, the augmented Lagrangian function of problem (13) is

$$L(\beta ,\xi ,\lambda )=h\left(\beta \right)+g\left(\xi \right)+\langle \lambda ,\beta -\xi \rangle +{\displaystyle \frac{\rho }{2}}{\parallel \beta -\xi \parallel }_2^2,$$

(14)

where $\lambda \in {\mathbb{R}}^q$ is a Lagrangian multiplier vector and constant $\rho >0$ is a penalty parameter. For convenience of code programming process, introduce a new variable $u=\lambda /\rho$, (14) can be convert to (15).

$$L(\beta ,\xi ,u)=h\left(\beta \right)+g\left(\xi \right)+{\displaystyle \frac{\rho }{2}}{\parallel \beta -\xi +u\parallel }_2^2-{\displaystyle \frac{\rho }{2}}{\parallel u\parallel }_2^2.$$

(15)

And $L(\beta ,\xi ,u)$ is named as scaled Lagrangian function. Then, the iteration scheme of ADMM for solving (13) is as follows.

$$\begin{array}{cc}\hfill & \beta \mathrm{step}:{\beta }^{k+1}=\underset{\beta }{argmin}\{h\left(\beta \right)+{\displaystyle \frac{\rho }{2}}{\parallel \beta -{\xi }^k+u^k\parallel }_2^2\},\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & \xi \mathrm{step}:{\xi }^{k+1}=\underset{\xi }{argmin}\{g\left(\xi \right)+{\displaystyle \frac{\rho }{2}}{\parallel {\beta }^{k+1}-\xi +u^k\parallel }_2^2\},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & u\mathrm{step}:u^{k+1}=u^k+{\beta }^{k+1}-{\xi }^{k+1}.\hfill \end{array}$$

(18)

This scheme follows the Gaussian-Seidel form, it decomposes problem (13) into three sub-problems at each iteration and makes treat function $h\left(\beta \right)$ and $g\left(\xi \right)$ individually. To solve sub-problems (16) and (17), according to the first-order condition, we have

$${M(\phi ,Y)}^T(M(\phi ,Y)\beta -f\left(Y\right))+\rho (\beta -{\xi }^k+u^k)=0,$$

(19)

and

$$0\in \partial \left({\widehat{\lambda }}_L{\parallel \xi \parallel }_1\right)+\rho (\xi -({\beta }^{k+1}+u^k)),$$

(20)

where ∂ denotes the subdifferential operator. Obviously, in (19), ${M(\phi ,Y)}^{T}M(\phi ,Y)+\rho I$ is non-singular, thus, we have

$${\beta }^{k+1}={({M(\phi ,Y)}^{T}M(\phi ,Y)+\rho I)}^{-1}({M(\phi ,Y)}^{T}f\left(Y\right)+\rho ({\xi }^k-u^k)).$$

(21)

As for (20), according to the definition of subdifferential, we have

$${\xi }^{k+1}=S_{{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }}}({\beta }^{k+1}+u^k),$$

(22)

where

$$S_{{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }}}({\beta }^{k+1}+u^k)=\left\{\begin{array}{ccc}{\beta }^{k+1}+u^k-{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }},\hfill & & {\beta }^{k+1}+u^k>{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }}\hfill \\ 0,\hfill & & otherwise\hfill \\ {\beta }^{k+1}+u^k+{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }},\hfill & & {\beta }^{k+1}+u^k<-{\displaystyle \frac{{\widehat{\lambda }}_L}{\rho }}\hfill \end{array}\right.$$

The following Algorithm 1 illustrates the ADMM algorithm scheme for solving problem (13). With ADMM algorithm, a sequence of ${\beta }^k$, ${\xi }^k$ and $u^k$ are produced. The algorithm stopped until the primary residuals and dual residuals satisfy the termination criterion.

Algorithm 1 ADMM algorithm to solve problem (13)

Initialization: Determine the initial point ${\beta }^0$, ${\xi }^0$, $u^0$ 1: Compute ${\beta }^{k+1}={({M(\phi ,Y)}^{T}M(\phi ,Y)+\rho I)}^{-1}{M(\phi ,Y)}^{T}f\left(Y\right)+\rho ({\xi }^k-u^k)$, 2: Compute ${\xi }^{k+1}=S_{{\widehat{\lambda }}_L/\rho }({\beta }^{k+1}+u^k)$, 3: Compute $u^{k+1}=u^k+{\beta }^{k+1}-{\xi }^{k+1}$.

Notably, in Algorithm 1, there is a parameter ${\widehat{\lambda }}_L>0$ that controls the sparsity of coefficient vector $\beta$ in problem (12). The value of ${\widehat{\lambda }}_L>0$ affects effectiveness and feasibility of our algorithm (see test details in Section 4). To balance the trade-off between model fit and model sparsity, we propose the following correction strategy for ${\widehat{\lambda }}_L$.

(1)

Initialization: Given the constant ${\mathtt{R}}_1>{\mathtt{R}}_2>0$, $1<{\chi }_2<{\chi }_1$. And set the initial value ${\widehat{\lambda }}_{L_0}$ and ${\widehat{\lambda }}_{L_{max}}={\parallel M(\phi ,Y)f\left(Y\right)\parallel }_{\infty }$, where ${\parallel ·\parallel }_{\infty }$ defined for a vector $x\in {\mathbb{R}}^n$ with ${\parallel x\parallel }_{\infty }={max}_i^n\left|x_i\right|$;

(2)

Use Algorithm 1 to find value of $\beta$;

(3)

R-square statistic [34] is used to test for goodness of fit, and $\mathrm{sum}\left(\right|\beta |>eps)$ is used to judge the sparsity of the coefficient, where $\mathrm{sum}\left(\right|\beta |>eps)$ denotes the number of elements in $\beta$ satisfying $|{\beta }_i|>eps(i=1,\cdots ,q)$. Define $R^2=S_R^2/S_T^2$, where $S_R^2={\parallel M(\phi ,Y)\beta -\overline{f}\left(Y\right)I\parallel }_2^2$, $S_T^2={\parallel f\left(Y\right)-\overline{f}\left(Y\right)I\parallel }_2^2$. $\overline{f}\left(Y\right)={\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=1}^m}f\left(y_i\right)$ is the mean value. And $I\in {\mathbb{R}}^m$ is a vector of all elements with 1.

(4)

Determine shrink or expansion of factor ${\widehat{\lambda }}_L$:

If $R^2\ge {\mathtt{R}}_1$, $\mathrm{sum}\left(\right|\beta |>eps)>2\ast n+1$ and ${\widehat{\lambda }}_L\le \widehat{\lambda }{L}_{max}$, then ${\widehat{\lambda }}_L={\chi }_1{\widehat{\lambda }}_L$.

If $R^2<{\mathtt{R}}_2$, then ${\widehat{\lambda }}_L={\chi }_2{\widehat{\lambda }}_L$.

3.2.3. Ensure the Geometry of the Interpolation Set

As mentioned previously, it is necessary to ensure the geometric properties of the sample (or interpolation) sets. In this paper, to recover a sparse quadratic model and maintain the geometry of the interpolation set, it is crucial to select an appropriate basis for solving (10). The basis $\Phi$ used for constructing $M(\phi ,Y)$ needs to satisfy the orthogonality property described below, which is cited from [35].

Definition 1 

(K-bounded orthogonal basis). For a measure μ, if a set of monomial functions $\Phi =\{{\phi }_1,\cdots ,{\phi }_N\}$ satisfy

$${\int }_D{\phi }_i\left(x\right){\overline{\phi }}_j\left(x\right)d\mu \left(x\right)={\delta }_{i,j}=\left\{\begin{array}{cc}0,\hfill & ifi\ne j\\ 1,\hfill & ifi=j\end{array}\right.$$

(23)

where $\left|\right|{\phi }_j\left(x\right){\left|\right|}_{L^{\infty }\left(D\right)}\le K$, for all $j\in 1,\cdots ,N$. Φ is an orthogonal basis of a certain function space.

Let $\mu$ be the uniform probability measure on ${\mathcal{B}}_{\infty }(0,\Delta )$, considering the sparse quadratic polynomial contains the linear terms, quadratic terms, and cross terms, to recover a sparse quadratic model of function f that satisfy the bounded orthogonal property, Bandeira et al. [29] provided a set of basis of the n-dimensional polynomial space ${\mathcal{P}}_n^2$ that is described as Definition 2. The surrogate model formulated on this basis helps to ensure the performance of the proposed algorithm.

Definition 2. 

The basis Φ satisfies the 3-bounded orthogonal property, if 

$$\left\{\begin{array}{c}{\phi }_0\left(x\right)=1\hfill \\ {\phi }_{1,i}\left(x\right)=\frac{\sqrt{3}}{\Delta }{x}_i\hfill \\ {\phi }_{2,ij}\left(x\right)=\frac{3}{{\Delta }^2}{x}_i{x}_j\hfill \\ {\phi }_{2,i}\left(x\right)=\frac{3\sqrt{5}}{2{\Delta }^2}{x}_i^2-\frac{\sqrt{5}}{2}\hfill \end{array}\right.$$

where $i,j=1,\cdots ,n$, and $i\ne j$.

3.3. Derivative Free Algorithms

In this subsection, the proposed algorithms in this paper for solving problem (1) will be detailed. We will first introduce the extension algorithm of [29] named as DFO-ADMM-TR ${\mathrm{l}}_1$ based on ADMM, and then present the proposed algorithm DFO-LASSOADMM-TR based on the method introduced in Section 3.2.

3.3.1. DFO-ADMM-TR ${\mathrm{l}}_1$

In the case of the number of interpolation samples being less than ${\displaystyle \frac{(n+1)(n+2)}{2}}$, literature [29] deployed a sparse low degree interpolating polynomials method to formulate the surrogate model when solving the optimization problem (1). And the basis $\varphi \left(x\right)$ is split into linear components ${\varphi }_L\left(x\right)$ and quadratic components ${\varphi }_Q\left(x\right)$. Accordingly, the interpolation model is built as follows:

$$m\left(x\right)={\beta }_L^T{\varphi }_L\left(x\right)+{\beta }_Q^T{\varphi }_Q\left(x\right),$$

(24)

where ${\beta }_Q\in {\mathbb{R}}^{{\displaystyle \frac{n(n+1)}{2}}}$ is supposed to be a sparse coefficient vector and ${\beta }_L\in {\mathbb{R}}^{n+1}$. Then to obtain ${\beta }_L$ and ${\beta }_Q$, an optimization problem is formulated as follows.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & min{\displaystyle \frac{1}{t}}{\parallel {\beta }_Q\parallel }_t^t\hfill \\ \hfill & s.t.A{\beta }_Q+B{\beta }_L=f\left(Y\right),\hfill \end{array}\end{array}$$

(25)

where $A\in {\mathbb{R}}^{m\times {\displaystyle \frac{n(n+1)}{2}}}$, and $B\in {\mathbb{R}}^{m\times (n+1)}$. When $t=1$, it was transformed into linear programming and solved by the function linprog.m from Matlab toolbox. The numerical experiments verified that DFO-TR $l_1$ outperforms DFO-TR Frob ($t=2$) and NEWUOA [27].

With a guess of ADMM may reduce the computing time when solving optimization problem (25), unlike the work in [29], in the first proposed algorithm DFO-ADMM-TR ${\mathrm{l}}_1$, ADMM is adopted to solve problem (25). And the scaled augmented Lagrangian function for the problem (25) is as follows.

$$L({\beta }_Q,{\beta }_L,u)=h\left({\beta }_Q\right)+g\left({\beta }_L\right)+{\displaystyle \frac{\rho }{2}}{\parallel A{\beta }_Q+B{\beta }_L-c+u\parallel }_2^2-{\displaystyle \frac{\rho }{2}}{\parallel u\parallel }_2^2,$$

where $h\left({\beta }_Q\right)={\parallel {\beta }_Q\parallel }_1$ and $g\left({\beta }_L\right)=0$. The scaled ADMM iteration scheme for solving (25) is shown as (26).

$$\begin{array}{cc}\hfill & {\beta }_L^{k+1}=\underset{{\beta }_L}{argmin}\left\{{\displaystyle \frac{\rho }{2}}{\parallel A{\beta }_Q^k+B{\beta }_L-f\left(Y\right)+u^k\parallel }_2^2\right\},\hfill \\ \hfill & {\beta }_Q^{k+1}=\underset{{\beta }_Q}{argmin}\{\parallel {\beta }_Q{\parallel }_1+{\displaystyle \frac{\rho }{2}}{\parallel A{\beta }_Q+B{\beta }_L^{k+1}-c+u^k\parallel }_2^2\},\hfill \\ \hfill & u^{k+1}=u^k+A{\beta }_Q^{k+1}+B{\beta }_L^{k+1}-c.\hfill \end{array}$$

(26)

3.3.2. DFO-LASSOADMM-TR

In DFO-LASSOADMM-TR for sparse modeling, the LASSO (Least Absolute Shrinkage and Selection Operator) method is employed to formulate the surrogate model for solving the optimization problem (1) when the number of sample points m is less than q. The ADMM algorithm, outlined in Algorithm 1, is then utilized to solve the optimization problem for estimating the parameters of the surrogate model. Furthermore, a correction strategy is introduced to guarantee the efficacy of the surrogate model. In the case where $m={\displaystyle \frac{1}{2}}(n+1)(n+2)$, a linear system (7) is derived, which can be efficiently solved using the routine svd.m from the Matlab toolbox. The procedure for constructing the surrogate model has been elaborated upon previously. Therefore, in this subsection, we will present the DFO-LASSOADMM-TR algorithm itself. This algorithm comprises five primary steps: initialization, model construction, trust region step calculation, iteration point and trust radius updating, and sample set updating.

(0)

Initialization. Parameters and sample of interpolation set are both initialized. As for parameter initialization, it can be divided into the following three steps.

(a)

Parameter in ADMM

• ${\beta }^0=\mathbf{0},{\xi }^0=\mathbf{0},{\beta }_Q^0=\mathbf{0},{\beta }_L^0=\mathbf{0}$: the original variables in the optimization problem;
• $u^0=\mathbf{0}$: the scaled variable;
• $\rho$: a constant satisfies $\rho ={\displaystyle \frac{\sqrt{5}+1}{2}}>0$;
• ${ϵ}^{abs}={10}^{-2},{ϵ}^{rel}={10}^{-3}$: tolerances for the primal and dual feasibility conditions, which used to be define the stopping criteria.

(b)

Parameter in correction strategy

• ${\widehat{\lambda }}_L=0.001$: controls the sparsity of coefficient vector $\beta$ in problem (12);
• ${\mathtt{R}}_1=0.8,{\mathtt{R}}_2=0.5$: determine shrink or expansion of factor ${\widehat{\lambda }}_L$.

(c)

General parameter in DFO.

• ${\Delta }^0=1$, ${\Delta }^{max}=100$, and ${\Delta }^{min}={10}^{-10}$: denote the initial, maximum, and minimum trust region radius, respectively.
• ${\gamma }_1=0.8$ and ${\gamma }_2=1.5$: denote the radius shrink and expansion factor respectively;
• ${\epsilon }_g={10}^{-15}$: the smallest norm-square of gradient ${\parallel g\parallel }_2^2$ to stop;
• ${\eta }_1=eps$: ${\displaystyle \frac{Pred}{Ared}}$ level to judge whether the current iteration is successful;
• ${\eta }_2=0.75$: ${\displaystyle \frac{Pred}{Ared}}$ level to expand trust region radius;
• $\mathrm{maxiter}=1000$: the maximum number of iteration of the DFO algorithms;
• $f_{\tau }={10}^{-c}$, $c=-8\mathrm{or}c=-6$: the minimum change value of function $f\left(x\right)$;

To initialize the sample set for interpolation, the initial point $x^0$ is supposed to be given. The initial interpolation sets are selected as $Y^0=\{x^0,x^0+{\Delta }^0{e}_i,x^0-{\Delta }^0{e}_i:i=1,\cdots ,n\}$, where n is the dimension of the optimization problem and $e_i$ is the i-th column of the identity matrix of order n. Then, the corresponding function values are calculated, and the interpolation sets are sorted according to the order of these function values (from smallest to largest).

(1)

Model building and trust region step calculation. At the first iteration, the initial interpolation set $Y^0$ is used to create the quadratic surrogate model either by the quadratic interpolation method or the LASSO method, depending on the number of samples. Without loss of generality, at the k-th ($k\ge 0$) iteration, the surrogate model $q^k\left(x\right)$ is formulated based on the updated sample set. Simultaneously, $d^k$ is computed by solving the sub-optimization problem (3). Then, $f_{change}=|\langle g^k,d^k\rangle +{\displaystyle \frac{1}{2}}\langle d^k,H^k{d}^k\rangle |$ is calculated. The algorithm’s termination is determined based on the values of $f_{change}$ and k.

(2)

Iteration point and trust region radius updating. Compute

$$r^k={\displaystyle \frac{f\left(x^k\right)-f(x^k+d^k)}{f_{change}}},$$

(27)

then updating $x^k$ and ${\Delta }^k$ according to $r^k$. If $r^k>{\eta }_1$, $sucess=1$.

(3)

Sample set updating. Calculate the square of the Euclidean distance $D(y_i^k,x^{k+1})$ between all points $y_i^k$ in the sample set $Y^k$ and the next trial point $x^{k+1}$, where $D(y_i^k,x^{k+1})={\parallel y_i^k-x^{k+1}\parallel }_2^2$ for $i=1,\dots ,m$. Denote $D^{max}=argmax{\parallel y_i^k-x^{k+1}\parallel }_2^2$. Then, update the sample set $Y^k$ according to the $sucess$ step (lines 28–43). In Algorithm 2, $Card\left(Y^k\right)$ denotes the number of samples in the set $Y^k$.

Algorithm 2 details the pseudocode of DFO-LASSOADMM-TR that based on LASSO and interpolation based trust region method.

Algorithm 2 DFO-LASSOADMM-TR: a novel derivative free algorithm

Initialization: Give the initial algorithm parameters, and construct the initial sample sets as well as compute the corresponding objective values. 1: Set the initial iteration $k=0$, $k^{max}=1000$ 2: while condition do 3:     $success=0$ 4:     if $m<{\displaystyle \frac{(n+1)(n+2)}{2}}$ then 5:         Creating quadratic model $q^k(x+d)$ use LASSO method 6:         if $R^2>{\mathtt{R}}_1$ && $sum\left(\right|\widehat{\beta }|>eps)>2n+1$ && ${\widehat{\lambda }}_L\le \widehat{\lambda }{L}_{max}$ then 7:            ${\widehat{\lambda }}_L={\chi }_1{\widehat{\lambda }}_L$, and solving problem (13) again 8:         if $R^2<{\mathtt{R}}_2$ then 9:            ${\widehat{\lambda }}_L={\chi }_2{\widehat{\lambda }}_L$, and solving problem (13) again 10:     else 11:         Creating quadratic model $q^k(x+d)$ by interpolation 12:     if ${\parallel g^k\parallel }_2^2\le {\epsilon }_g$∥${\Delta }^k\le {\Delta }^{min}$ then 13:         break 14:     if $f_{change}<f_{\tau }$∥$k>maxiter$ then 15:         break 16:     Evaluate the function value $f(x^k+d^k)$, and then compute $r^k$ through the Equation (27) 17:     if $r^k\ge {\eta }_1$ then 18:         $success=1$ 19:         $x^{k+1}=x^k+d^k$ 20:         if $r^k\ge {\eta }_2$ then 21:            ${\Delta }^{k+1}=min\{{\gamma }_2{\Delta }^k,{\Delta }^{max}\}$ 22:         else 23:            ${\Delta }^{k+1}={\Delta }^k$ 24:     else 25:         $x^{k+1}=x^k$ 26:         if $Card\left(Y^k\right)\ge n+1$ then 27:            ${\Delta }^{k+1}={\gamma }_1{\Delta }^k$ 28:     Compute $D(y_i^k,x^{k+1})$, $D^{max}$ 29:     if $success$ then 30:         $reachmax=1$ 31:         if $Card\left(Y^k\right)<{\displaystyle \frac{(n+1)(n+2)}{2}}\mathbf{then}$ 32:            $reachmax=0$ 33:         if $reachmax$ then 34:            $y_{out}=D^{max}$ 35:            $Y^{k+1}=Y^k\cup \left\{x^{k+1}\right\}\setminus \left\{y_{out}\right\}$ 36:         else 37:            $Y^{k+1}=Y^k\cup \left\{x^{k+1}\right\}$ 38:     else 39:         if $Card\left(Y^k\right)={\displaystyle \frac{(n+1)(n+2)}{2}}$ then 40:            if ${\parallel (x^k+d^k)-x^k\parallel }_2^2\le {\parallel y_{out}-x^k\parallel }_2^2$ then 41:                $Y^{k+1}=Y^k\cup \{x^k+d^k\}\setminus \left\{y_{out}\right\}$ 42:         else 43:            $Y^{k+1}=Y^k\cup \left\{x^{k+1}\right\}$ 44:     $k=k+1$ 45: return result

4. Numerical Experiments

In this section, to evaluate the performance of the proposed algorithms, we conduct numerical experiments on both smooth optimization problems (i.e., problems without noise) and noisy optimization problems. We will compare our algorithms with the one presented in [29]. All numerical experiments are executed using Matlab 2013b on a personal computer equipped with an Intel 3.40 GHz CPU, 4 GB of RAM, and the Windows 7 operating system.

4.1. Benchmark Functions and Experimental Design

The performance of the proposed algorithms is measured by solving 24 benchmark smooth optimization problems and 20 noisy ones. Within this context, the smooth optimization problems are selected from the CUTEr/st collection [36], as shown in

Table 1. Details of smooth problems.

Problem	Sourse or Formula Form	n
ARGLINB	[37], Problem 33	10
ARGLINC	[37], Problem 34	8
ARWHEAD	[38], Problem 55	15
BDQRTIC	[38], Problem 61	10
BROYDN3DLS	[37], Problem 30	10
DIXMAANC	[39], Page 49	12
DIXMAANG	[39], Page 49	12
DIXMAANI	[39], Page 49	12
DIXMAANK	[39], Page 49	12
DIXON3DQ	[39], Page 51	12
DQDRTIC	[40], Problem 22	10
FLETCHCR	[41], Problem 2	10
FREUROTH	[37], Problem 2	10
GENHUMPS	$f_1\left(x\right)$	10
HIMMELBH	[39], Page 60	2
MOREBVNE	[39], Page 75	10
NONDIA	[39], Page 76	10
NONDQUAR	$f_2\left(x\right)$	10
POWELLSG	[37], Problem 13	4
POWER	[39], Page 83	18
ROSENBR	[37], Problem 1	2
TRIDIA	[40], Problem 8	10
VARDIM	[37], Problem 25	10
WOODS	[37], Problem 14	4

. Specifically, for the test functions without identified sources, we have provided their mathematical expressions based on the CUTEr/st collection. In the Table 1, the mathematical expressions for $f_1\left(x\right)$ and $f_2\left(x\right)$ are as formula (28) and (29).

$$f_1\left(x\right)=\sum _{i=1}^{n-1}\left({sin}^2\left(x_i\right)·{sin}^2\left(2x_{i+1}\right)+0.05(x_i^2+x_{i+1}^2)\right)$$

(28)

$$f_2\left(x\right)={(x_1-x_2)}^2+\sum _{i=1}^{n-2}{\left(x_i+x_{i+1}+x_n\right)}^4+{(x_{n-1}+x_n)}^2$$

(29)

Besides, n denotes the number of variables for each problem, and structure represents the feature of the functions.

Moreover, the algorithms are tested on noise functions constructed based on the Sphere function and the Rosenbrock function [42]. The formulas for these base functions are as follows:

$$\mathrm{Rosenbrock}\mathrm{function}:f\left(x\right)=\sum _{i=1}^{n-1}[b{(x_{i+1}-x_i^2)}^2+{(a-x_i)}^2]$$

(30)

$$\mathrm{Sphere}\mathrm{function}:f\left(x\right)=\sum _{i=1}^n{x}_i^2$$

(31)

where the noise functions are constructed by incorporating log-normal noise or uniform noise into these two base functions. For details on constructing the noise function, please refer to [42]. Noticeably, in cases with moderate noise, problem sizes of 2, 4, and 10 variables are considered. However, for instances with severe noise, the sizes are limited to 2 and 4 variables. The characteristics of these noisy problems are summarized in

Table 2. Details of noisy problems.

Problem	Noise	n
Sphere	moderate gaussian noise	2, 4, 10
	moderate uniform noise
	severe gaussian noise	2, 4
	severe uniform noise
Rosenbrock	moderate gaussian noise	2, 4, 10
	moderate uniform noise
	severe gaussian noise	2, 4
	severe uniform noise

. Additionally, many of these problems have a sparse structure. Furthermore, although some benchmark functions may indeed have boundary constraints, we did not explicitly take into account these bounds as our focus has been on unconstrained optimization. Finally, when conducting the numerical experiments, the initial points and the optimal solutions for each benchmark problem refer to references [36,37,38,39,40,41,43]. It is worth noting that the initial point chosen for numerical experiments is consistent for the same benchmark functions.

4.2. Parameter Correction Strategy Tests

Cross-validation is a crucial step in data analytics modeling. In our proposed algorithm, we employ sparse regression modeling, specifically LASSO, to identify the surrogate model. Additionally, we adopt a correction strategy to ensure the accurate determination of the surrogate model. Specifically, a correction strategy for ${\widehat{\lambda }}_L$ has been introduced in Section 3.

In this section, the necessity of the correction strategy involving the parameter ${\widehat{\lambda }}_L>0$ is validated through numerical experiments. Experiments were conducted using ten different problems, each with varying values of ${\widehat{\lambda }}_L>0$. The results are presented in

Table 3. Run time(s) of the algorithm with different ${\widehat{\lambda }}_L$.

${\widehat{\mathit{\lambda }}}_{\mathit{L}}$	S	0.0001	0.0005	0.001	0.005	0.01	0.05	0.1	0.5	1	5	10
ARGLINB	0.72	0.58	0.56	0.59	0.54	0.55	0.56	0.53	0.54	0.61	0.54	0.54
ARGLINC	0.53	0.48	0.47	0.49	0.47	0.52	0.48	0.46	0.51	∼	0.46	0.49
BDQRTIC	1.57	1.42	∼	1.57	1.26	∼	1.46	1.20	1.32	1.31	1.64	1.33
DIXMAANC	2.07	1.32	1.36	∼	1.19	1.36	1.46	∼	∼	∼	∼	∼
DIXMAANI	2.26	∼	∼	∼	0.92	0.82	0.97	∼	∼	∼	∼	∼
DIXON3DQ	0.84	0.64	0.67	0.70	0.50	0.96	0.94	∼	0.76	0.77	∼	∼
DQDRTIC	0.42	∼	∼	∼	∼	0.86	∼	0.84	0.93	0.87	∼	∼
FREUROTH	1.81	∼	∼	1.80	1.57	1.44	∼	∼	1.48	1.43	1.36	∼
POWER	0.83	0.80	0.84	0.82	0.83	0.75	0.79	0.75	0.80	∼	0.88	∼
TRIDIA	0.63	0.73	0.71	0.70	0.57	0.67	∼	∼	∼	∼	0.75	∼

, where S denotes the algorithm incorporating the strategy with ${\widehat{\lambda }}_L>0$, and the values $0.0001,0.0005,\cdots ,10$ represent different settings for ${\widehat{\lambda }}_L>0$. The symbol ∼ indicates that the DFO algorithm was unable to reach the optimal solution before meeting the termination condition. For instance, an entry such as 0.72 in the table signifies the CPU time required to solve the ARGLINB problem using the DFO algorithm with the strategy S.

Table 3 illustrates that the parameter ${\widehat{\lambda }}_L$ not only governs the sparsity and precision of the model but also impacts the viability of the proposed LASSO and interpolation-based DFO algorithm. Furthermore, in terms of successfully solving the optimization problem, the algorithm employing strategy S outperforms the other configurations. Therefore, incorporating the correction strategy is crucial to ensure the effectiveness of the proposed algorithm.

4.3. Experimental Results

In the experiments, only algorithms that belong to local model-based search algorithm, especially trust region methods based on the interpolation, are selected as comparison. In the work of Bandeira et al., numerical experiments have shown that their propose DFO-TR ${\mathrm{l}}_1$ performs better than DFO-TR Frob and software NEWUOA [29], where NEWUOA is the classical algorithm for solving derivative DFO problems. Thus, only comparisons among DFO-TR ${\mathrm{l}}_1$, DFO-ADMM-TR ${\mathrm{l}}_1$ and DFO-LASSOADMM-TR are made in our numerical experiments.

To evaluate the performance of algorithms in this paper, performance profile [44] and data profile [45] are adopted. These two criteria measure the algorithms from different perspectives. Performance profile provides a view of the relative performance of solvers in terms of the amount of computing time. And data profile provides valuable information in terms of the number of evaluations. In other words, the former is informative for a user with a computation time limitation, while the latter is informative for a user whose objective functions are expensive to compute.

In order to better understand them, a brief description of the performance profile and data profile will be presented first. In addition, let $\mathrm{S}$ be the set of solvers and $\mathrm{P}$ be the set of test problems.

(1)

Performance profile. Let $t_{p,s}$ be the computing time required to solve problem $p\in \mathrm{P}$ by solver $s\in \mathrm{S}$. Let ${\mathrm{R}}_{p,s}$ denotes the performance ratio, it is defined as follows

$${\mathrm{R}}_{p,s}=\left\{\begin{array}{ccc}{\displaystyle \frac{t_{p,s}}{min\{t_{p,s}:s\in \mathrm{S}\}}}\hfill & \hfill & \mathrm{if}\mathrm{solver}s\mathrm{solves}p\mathrm{properly},\hfill \\ {\mathrm{R}}_M\hfill & \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(32)

where ${\mathrm{R}}_M\ge max\{{\mathrm{R}}_{p,s}:p\in \mathrm{P},s\in \mathrm{S}\}$, and ‘properly’ means that solver s solves p with the certain accuracy. Define

$${\rho }_s\left(\tau \right)={\displaystyle \frac{\mathrm{size}\{p\in \mathrm{P}:{\mathrm{R}}_{p,s}\le \tau \}}{n_p}},$$

where ${\rho }_s\left(\tau \right)$ is the cumulative probability of ${\mathrm{R}}_{p,s}$, it is named as the performance profile. And $n_p$ is the number of problems in the test sets $\mathrm{P}$.

(2)

Data profile. For a given tolerance ${\tau }_f$, let $n_{p,s}$ be the number of function evaluations satisfy (33) when solver $s\in \mathrm{S}$ is adopted to solve problem $p\in \mathrm{P}$.

$$f\left(x\right)\le f_L+{\tau }_f(f\left(x_0\right)-f_L),$$

(33)

where ${\tau }_f>0$ is a tolerance, $x_0$ is the initial point for solving problem, here, $f_L$ is an accurate estimate of $f\left(x\right)$ at a global minimum point. Thus, the data profile of a solver $s\in \mathrm{S}$ is defined as

$$d_s\left(\alpha \right)={\displaystyle \frac{1}{n_p}}\mathrm{size}\{p\in \mathrm{P}:{\displaystyle \frac{n_{p,s}}{p_n+1}}\le \alpha \},$$

(34)

where $p_n$ is the number of variables in problem p.

4.3.1. Numerical Experiments for Smooth Case

When using the performance profile as a metric, accuracies of ${10}^{-6}$ and ${10}^{-8}$ are set in advance. Besides, the maximum number of iterations is set to 1000; that is, the algorithms stop when $k>1000$. When using the data profile as the metric, tolerances ${\tau }_f$ are defined as ${\tau }_f={10}^{-6}$ and ${\tau }_f={10}^{-8}$, respectively.

As for the performance profiles, in the numerical results, both in the cases of $\mathrm{acc}=6$ and $\mathrm{acc}=8$, all functions except DIXMAANC reached the given accuracy. And Figure 2 illustrates the comparison results. It is worth noting that the ${\rho }_s\left(\tau \right)$ values are plotted on a log-scale. In this context, ${\rho }_s\left(0\right)$ denotes the probability that a given algorithm is the best among all algorithms tested. Moreover, according to the definition of ${\rho }_s\left(\tau \right)$, in a certain sense, the smaller the $\tau$ (on a log-scale), the larger the ${\rho }_s\left(\tau \right)$, indicating that the algorithm can solve more problems.

From Figure 2, it can be easily observed that, for both $\mathrm{acc}=6$ and $\mathrm{acc}=8$, DFO-ADMM-TR${\mathrm{l}}_1$ and DFO-LASSOADMM-TR outperform DFO-TR${\mathrm{l}}_1$. Figure 3 illustrates the comparison of the data profiles, and it can be reported that DFO-ADMM-TR ${\mathrm{l}}_1$ performs better than the other two algorithms.

To be more specific, the following comprehensive conclusions can be deduced from Figure 2 and Figure 3: DFO-ADMM-TR${\mathrm{l}}_1$ demonstrates superiority over DFO-TR${\mathrm{l}}_1$ in both performance and data profiles, indicating the effectiveness of integrating ADMM into DFO-TR${\mathrm{l}}_1$. In terms of performance profiles, the two proposed algorithms, DFO-ADMM-TR${\mathrm{l}}_1$ and DFO-LASSOADMM-TR, consistently outperform DFO-TR${\mathrm{l}}_1$ for both $\mathrm{acc}=6$ and $\mathrm{acc}=8$, regardless of the value of $\tau$. Furthermore, when examining data profiles, DFO-ADMM-TR${\mathrm{l}}_1$ is superior to the other two algorithms. It is understandable that DFO-LASSOADMM-TR might be inferior to DFO-TR${\mathrm{l}}_1$ because the accuracy of the surrogate model obtained by interpolation might be higher than that obtained by LASSO. However, adopting LASSO reduces the execution time of the algorithm, albeit at the cost of a slight increase in the number of function evaluations. Nevertheless, upon comparing Figure 2 and Figure 3, it is evident that DFO-ADMM-TR${\mathrm{l}}_1$ not only maintains its superiority but also significantly reduces the algorithm’s runtime, while the increase in the number of function evaluations is negligible.

4.3.2. Numerical Experiments for Noisy Case

For the noisy optimization problems, we conduct numerical experiments in the same way as in the smooth case. However, for the performance profile, accuracy is set to $\mathrm{acc}=2$ and $\mathrm{acc}=4$. Correspondingly, tolerance ${\tau }_f$ is defined as ${\tau }_f={10}^{-2}$ and ${\tau }_f={10}^{-4}$, respectively. Figure 4 and Figure 5 demonstrate that the application of the LASSO method enables the proposed DFO algorithm to attain enhanced performance, both in terms of performance profiles and data profiles.

As regards the size of the noise, taking the case $\mathrm{acc}=2$ as an example,

Table 4. The percentage of problems solved by the algorithm.

	DFO-TR ${\mathbf{l}}_1$	DFO-TR ${\mathbf{l}}_1$	DFO-LASSOADMM-TR
moderate noise	58.33%	58.33%	75.00%
severe noise	50.00%	50.00%	62.50%

shows that adopting LASSO improves the algorithm’s ability to solve problems with noise. Besides, for a given accuracy, the proportion of problems that the algorithm can solve decreases as the size of the noise increases.

Overall, the proposed two algorithms are easy to adopt for solving problems with noise, particularly when LASSO is used to find the surrogate model. It is notable that the other proposed algorithm also benefits from the DFO-TR${\mathrm{l}}_1$ framework. The reason is that ADMM only converges to a modest accuracy, which is sufficient for many applications [33], making DFO-ADMM-TR${\mathrm{l}}_1$ more suitable for solving noisy problems. In conclusion, the numerical results provide a perspective for users solving optimization problems originating from practical applications. In other words, when the user needs to obtain the optimal solution as soon as possible, DFO-LASSOADMM-TR is a good choice. Moreover, DFO-LASSOADMM-TR is an alternative when there is noise in the problems.

5. Conclusions

In many practical industrial settings, one often encounters a type of optimization problem where the derivatives are hard to compute or unavailable. To address this issue, DFO methods have attracted the attention of researchers. Algorithms based on the surrogate model trust region framework constitute an important class of DFO methods. In this paper, we present two DFO algorithms that consider sparsity and noise, aiming to reduce computing time and decrease the number of function evaluations. One of the algorithms is an extension of the method proposed in [29]. Building on this, another algorithm is developed by integrating sparse modeling methods and interpolation techniques for constructing the surrogate model. Furthermore, in both algorithms, ADMM is applied to solve the optimization problem formulated for determining the coefficients of the surrogate models. Finally, the numerical experiments reported in this paper suggest that the proposed algorithms are both comparable and competitive.