About the Performance of a Calculus-Based Approach to Building Model Functions in a Derivative-Free Trust-Region Algorithm

Abstract

This paper examines a calculus-based approach to building model functions in a derivative-free algorithm. This calculus-based approach can be used when the objective function considered is defined via more than one blackbox. Two versions of a derivative-free trust-region method are implemented. The first version builds model functions by using a calculus-based approach, and the second version builds model functions by directly considering the objective function. The numerical experiments demonstrate that the calculus-based approach provides better results in most situations and significantly better results in specific situations.

1. Introduction

Trust-region methods are a popular class of algorithms for finding the solutions of nonlinear minimization optimization problems [1,2]. Trust-region algorithms build a model of the objective function in a neighborhood of the incumbent solution. The region in which the model function behaves similarly to the objective function is called the trust region and is defined through a trust-region radius. The optimization algorithm then finds a point in the trust region at which the model sufficiently decreases. This step is known as the trust-region sub-problem, and the point that provides a sufficient decrease is called the trial point. The value of the objective function is then computed at the trial point. If the ratio of the achieved reduction versus the reduction in the model is sufficient, then the incumbent solution is updated and set to be equal to the trial point. If the ratio is not sufficient, then the trust-region radius is decreased. This method iterates until a stopping condition implies that a local minimizer has been located.

Extensive research has been done on this topic since 1944, when Levenberg published what is known to be the first paper related to trust-region methods [3]. Early work on trust-region methods includes the work of Dennis and Mei [4], Dennis and Schnabel [5], Fletcher [6], Goldfeldt, Quandt, and Trotter [7], Hebden [8], Madsen [9], Moré [10,11], Moré and Sorensen [12], Osborne [13], Powell [14,15,16,17,18], Sorensen [19,20], Steihaug [21], Toint [22,23,24,25], and Winfield [26,27], to name a few. The name trust region seems to have been used for the first time in 1978 by Dennis in [28].

In the works mentioned above, trust-region methods were designed and analyzed under the assumption that first-order information about the objective function is available and that second-order information (i.e., Hessians) may, or may not, be available. In the case in which both first-order and second-order information is not available, or it is hard to directly obtain, derivative-free trust-region (DFTR) methods can be used. This type of method has become more popular in the last two decades due to the rise of blackbox optimization problems. Early works on DFTR methods include those of Conn, Scheinberg, and Toint [29], Marazzi and Nocedal [30], Powell [31,32], and Colson and Toint [33], to name a few.

In 2009, the convergence properties of general DFTR algorithms for unconstrained optimization problems were rigorously investigated [34]. The pseudo-code of an algorithm that converges to a first-order critical point and the pseudo-code of an algorithm that converges to a second-order critical point were provided. A complete review of the DFTR methods for unconstrained optimization problems is available in [35] (Chapters 10 and 11) and [36] (Chapter 11). There now exist several DFTR algorithms for solving unconstrained optimization problems, such as Advanced DFO-TRNS [37], BOOSTERS [38], CSV2 [39], DFO [29,40], UOBYQA [31], and WEDGE [30]. In recent years, DFTR algorithms have also been developed for constrained optimization problems. When an optimization problem is bound-constrained, some of the algorithms available in the literature are BC-DFO [41], BOBYQA [42], ORBIT [43,44], SNOBFIT [45], and TRB-POWELL [46]. Other DFTR algorithms dealing with more general constrained optimization problems include CONDOR [47], CONORBIT [48], DEFT-FUNNEL [49], LCOBYQA [50], LINCOA [51], and S [52].

We define a blackbox as any process that returns an output whenever an input is provided, but where the inner mechanism of the process is not analytically available to the optimizer [36]. In this paper, we consider a situation in which the objective function, $F,$ is obtained by manipulating several blackboxes. For instance, F could be the product of two functions, say, $f_1$ and $f_2$, where the function values for $f_1$ are obtained through one blackbox and the function values for $f_2$ are obtained through a different blackbox. An objective function that is defined by manipulating more than one function is called a composite objective function.

Composite objective functions have inspired a particular direction of research under the assumption that the functions involved do not have the same computational costs. For instance, Khan et al. [53] developed an algorithm for minimizing $F=\varphi +h\circ f$, where $\varphi$ is smooth with known derivatives, h is a known nonsmooth piecewise linear function, and f is smooth but expensive to evaluate. In [54], Larson et al. investigated the minimization problem $F=h\circ f$, where h is nonsmooth and inexpensive to compute and f is smooth, but its Jacobian is not available. Recently, Larson and Menickelley developed algorithms for bound-constrained nonsmooth composite minimization problems in which the objective function F has the form $F=h\left(f\right(x\left)\right)$, where h is cheap to evaluate, and f requires considerable time to evaluate.

These ideas have led to research on more general calculus-based approaches to approximating gradients or Hessians of composite functions. In [55,56,57], the authors provided calculus rules (integer power, product, quotient, and chain) for (generalized) simplex gradients. These results were advanced to the (generalized) centered simplex gradient in [58] and to the (generalized) simplex Hessian in [59]. In [60], a unified framework that provides general error bounds for gradient and Hessian approximation techniques by using calculus rules was presented.

Previous research has shown that a calculus-based approach to approximating gradients or Hessians can be substantially more accurate than a non-calculus-based approach in several situations [55,58,59,60]. However, it is still unclear if these theoretical results translate to a significant improvement in a derivative-free algorithm. The main goal of this paper is to compare two versions of a DFTR algorithm designed to solve a box-constrained blackbox optimization problem in which the objective function is a composite function: one that employs a calculus-based approach and one that does not employ a calculus-based approach. It is worth emphasizing that it is not our intention to show that the DFTR algorithm developed in this paper is better than state-of-the-art derivative-free algorithms. The main goal is to analyze any benefits resulting from using a calculus-based approach in a DFTR algorithm designed to minimize a composite objective function.

This paper is organized as follows. In Section 2, the context is established and fundamental notions related to the topic of this paper are presented. In Section 3, the pseudo-code of the DFTR algorithm is introduced, and details related to the algorithm are discussed. In Section 4, numerical experiments are conducted to compare the calculus-based approach with the non-calculus-based approach. In Section 5, the results of the numerical experiments are scrutinized. Lastly, the main results of this paper, the limitations of this paper, and future research directions are briefly discussed in Section 6.

2. Background

Unless otherwise stated, we use the standard notation found in [36]. We work in the finite-dimensional space ${{ℝ}}^n$ with the inner product $x^{\top }y={\sum }_{i=1}^n{x}_i{y}_i$. The norm of a vector is denoted as $\parallel x\parallel$ and is taken to be the ${\ell }_2$ norm. Given a matrix $A\in {{ℝ}}^{n\times m},$ the ${\ell }_2$ induced matrix norm is used.

We denote by $B(x^0;\Delta )$ the closed ball centered at $x^0$ with radius $\Delta .$ The identity matrix in ${{ℝ}}^{n\times n}$ is denoted by ${Id}_n$, and the vector of all ones in ${{ℝ}}^n$ is denoted by ${\mathbf{1}}_n.$ The Minkowski sum of two sets of vectors A and B is denoted by $A\oplus B.$ That is,

$$\begin{array}{c}\hfill A\oplus B=\{a+b:a\in A,b\in B\}.\end{array}$$

In the next sections, we will refer to a calculus-based approach and a non-calculus approach to approximate gradients and Hessians. Let us clarify the meanings of these two approaches.

We begin with the non-calculus-based approach. Let $F:{{ℝ}}^n\to {ℝ}$ and let $Q_F\left(x^k\right)$ be a quadratic interpolation of F at $x^k$ using $(n+1)(n+2)/2$ distinct sample points poised for quadratic interpolation. An approximation of the gradient of F at $x^k$, denoted by $g^k,$ is obtained by computing $\nabla Q_F\left(x^k\right),$ and an approximation of the Hessian of F at $x^k$, denoted by $H^k,$ is obtained by computing ${\nabla }^2{Q}_F\left(x^k\right).$

We now explain the calculus-based approach. Let $F:{{ℝ}}^n\to {ℝ}$ be constructed using $f_1:{{ℝ}}^n\to {ℝ}$ and $f_2:{{ℝ}}^n\to {ℝ}$. Let $Q_{f_1}$ and $Q_{f_2}$ be quadratic interpolation functions of $f_1$ and $f_2$, respectively. When a calculus-based approach is employed and the composite objective function F has the form $F=f_1·f_2,$ then

$$\begin{array}{cc}\hfill g^k& =\nabla \left(Q_{f_1}·Q_{f_2}\right)\left(x^k\right)=f_1\left(x^k\right)\nabla Q_{f_2}\left(x^k\right)+f_2\left(x^k\right)\nabla Q_{f_1}\left(x^k\right),\hfill \end{array}$$

(1)

and

$$\begin{array}{cc}\hfill H^k& ={\nabla }^2\left(Q_{f_1}·Q_{f_2}\right)\left(x^k\right)=f_2\left(x^k\right){\nabla }^2{Q}_{f_1}\left(x^k\right)+\nabla Q_{f_1}\left(x^k\right){\left(\nabla Q_{f_2}\left(x^k\right)\right)}^{\top }+\nabla Q_{f_2}\left(x^k\right){\left(\nabla Q_{f_1}\left(x^k\right)\right)}^{\top }+f_1\left(x^k\right){\nabla }^2{Q}_{f_2}\left(x^k\right).\hfill \end{array}$$

(2)

Similarly, when the composite objective function F has the form $F=\frac{f_1}{f_2},$ then (assuming $f_2\left(x^k\right)\ne 0$)

$$\begin{array}{cc}\hfill g^k& =\nabla \left(\frac{Q_{f_1}}{Q_{f_2}}\right)\left(x^k\right)=\frac{f_2\left(x^k\right)\nabla Q_{f_1}\left(x^k\right)-f_1\left(x^k\right)\nabla Q_{f_2}\left(x^k\right)}{{\left[f_2\left(x^k\right)\right]}^2}.\hfill \end{array}$$

(3)

and

$$\begin{array}{cc}{H}^k& ={\nabla }^2\left(\frac{Q_{f_1}}{Q_{f_2}}\right)\left(x^k\right)=\frac{1}{{\left[f_2\left(x^k\right)\right]}^3}\left[{\left[f_2\left(x^k\right)\right]}^2{\nabla }^2{Q}_{f_1}\left(x^k\right)-f_1\left(x^k\right)f_2\left(x^k\right){\nabla }^2{Q}_{f_2}\left(x^k\right)+2f_1\left(x^k\right)\nabla Q_{f_2}\left(x^k\right)\nabla Q_{f_2}{\left(x^k\right)}^{\top }\right.\\ \hfill & \left.-f_2\left(x^k\right)\left(\nabla Q_{f_1}\left(x^k\right)\nabla Q_{f_2}{\left(x^k\right)}^{\top }+\nabla Q_{f_2}\left(x^k\right)\nabla Q_{f_1}{\left(x^k\right)}^{\top }\right)\right],\hfill \end{array}$$

(4)

To compute $H^k,$ the technique called the (generalized) simplex Hessian is utilized [59]. When the two matrices involved in the computation of the simplex Hessian are square matrices that have been properly chosen, the simplex Hessian is equal to the Hessian of the quadratic interpolation function. The formula for computing simplex Hessians is based on simple matrix algebra. Hence, this technique is straightforward to implement in an algorithm. The formula for computing the simplex Hessian requires the computation of $n+1$ simplex gradients.

Definition 1

([55]). (Simplex gradient). Let $f:domf\subseteq {{ℝ}}^n\to {ℝ}$. Let $x^0\in domf$ be the point of interest. Let $T=\left[\begin{array}{cccc}{t}^1& t^2& \cdots & t^n\end{array}\right]\in {{ℝ}}^{n\times n}$ with $x^0\oplus T\in domf.$ The simplex gradient of f at $x^0$ over T is denoted by ${\nabla }_{s}f(x^0;T)$ and defined by

$${\nabla }_{s}f(x^0;T)=T^{-\top }\delta f(x^0;T)$$

where

$$\delta f(x^0;T)=\left[\begin{array}{c}f(x^0+t^1)-f\left(x^0\right)\\ f(x^0+t^2)-f\left(x^0\right)\\ ⋮\\ f(x^0+t^n)-f\left(x^0\right)\end{array}\right]\in {{ℝ}}^n.$$

Definition 2

([59]). (Simplex Hessian). Let $f:domf\subseteq {{ℝ}}^n\to {ℝ}$ and let $x^0\in domf$ be the point of interest. Let $S=\left[\begin{array}{cccc}{s}^1& s^2& \cdots & s^n\end{array}\right]\in {{ℝ}}^{n\times n}$ and $T\in {{ℝ}}^{n\times n}$ with $x^0\oplus S,x^0\oplus T,x^0\oplus S\oplus T\in domf.$ The simplex Hessian of f at $x^0$ over S and T is denoted by ${\nabla }_s^{2}f(x^0;S;T)$ and defined by

$${\nabla }_s^{2}f(x^0;S;T)=S^{-\top }\delta {\nabla }_{s}f(x^0;S;T),$$

where

$$\delta {\nabla }_{s}f(x^0;S;T)=\left[\begin{array}{c}{({\nabla }_{s}f(x^0+s^1;T)-{\nabla }_{s}f(x^0;T))}^{\top }\\ {({\nabla }_{s}f(x^0+s^2;T)-{\nabla }_{s}f(x^0;T))}^{\top }\\ ⋮\\ {({\nabla }_{s}f(x^0+s^n;T)-{\nabla }_{s}f(x^0;T))}^{\top }\end{array}\right]\in {{ℝ}}^{n\times n}.$$

The simplex gradient and simplex Hessian do not necessarily require the matrices S and T to be square matrices with a full rank. These two definitions may be generalized by using the Moore–Penrose pseudoinverse of a matrix [55,59].

Last, we recall the definition of a fully linear class of models.

Definition 3

([36]). (Class of fully linear models). Given $f\in {\mathcal{C}}^1,x\in {{ℝ}}^n$ and $\overline{\Delta }>0,$ we say that $\{{\tilde{f}}_{\Delta }:\Delta \in (0,\overline{\Delta }]\}$ is a class of fully linear models of f at x parametrized by Δ if there exists a pair of scalars ${\kappa }_f\ge 0$ and ${\kappa }_g\ge 0$ such that, given any $\Delta \in (0,\overline{\Delta }],$ the model ${\tilde{f}}_{\Delta }$ satisfies

• $|f(x+s)-{\tilde{f}}_{\Delta }(x+s)|\le {\kappa }_f\left(x\right){\Delta }^2\mathrm{for}\mathrm{all}s\in B(0;\Delta ),$
• $\parallel \nabla f(x+s)-\nabla {\tilde{f}}_{\Delta }(x+s)\le {\kappa }_g\left(x\right)\Delta \mathrm{for}\mathrm{all}s\in B(0;\Delta ).$

Note that a fully linear model is equivalent to a model that is order-1 gradient accurate and order-2 function accurate based on the definitions introduced in [56].

We are now ready to introduce the pseudo-code of the DFTR algorithm that will be used to compare the calculus-based approach with the non-calculus-based approach.

3. Materials and Methods

In this section, the algorithm designed to minimize a box-constrained blackbox optimization problem involving a composite objective function is described. The algorithm will be used to perform our comparison between the calculus-based approach and the non-calculus-based approach in Section 4.

Let ${\Delta }_{max}>0$ be the maximum trust-region radius allowed in the DFTR algorithm. The minimization problem considered is

$$\begin{array}{cc}\hfill & \underset{\ell \le x\le u}{min}F\left(x\right)\hfill \end{array}$$

(5)

where $F:domF\subseteq {{ℝ}}^n\to {ℝ}$ is twice continuously differentiable on

$$\begin{array}{cc}\hfill & \bigcup _{\ell \le x\le u}B(x;{\Delta }_{max})\hfill \end{array}$$

(6)

where $\ell ,u\in {{ℝ}}^n$, and the inequalities $\ell \le x\le u$ are taken component-wise. It is assumed that F is a composite function obtained from two blackboxes with similar computational costs.

Before introducing the pseudo-code of the algorithm, let us clarify some details about the model function and the trust-region sub-problem.

The model function built at an iteration k will be denoted by $m^k.$ The model is a quadratic function that can be written as

$$\begin{array}{cc}\hfill m^k(x^k+s^k)& =F\left(x^k\right)+{\left(g^k\right)}^{\top }{s}^k+\frac{1}{2}{\left(s^k\right)}^{\top }{H}^k{s}^k,\hfill \end{array}$$

(7)

where $g^k$ denotes an approximation of the gradient $\nabla F\left(x^k\right),$$H^k$ is a symmetric approximation of the Hessian ${\nabla }^{2}F\left(x^k\right)$, and $s^k\in {{ℝ}}^n$ is the independent variable.

When a calculus-based approach is used, both $g^k$ and $H^k$ are built by using the calculus-based approach presented earlier in Section 2. Similarly, if the non-calculus-based approach is used, then both $g^k$ and $H^k$ are built with the non-calculus approach. We define the sampling radius, ${\Delta }_s^k$, as the maximum distance between the incumbent solution $x^k$ and a sampling point used to build the approximations $g^k$ and $H^k.$

The trust-region subproblem solved in the DFTR algorithm (Step 2 of Algorithm 1) at an iteration k is

$$\begin{array}{cc}\hfill & \underset{s\in {{ℝ}}^n}{minimize}{m}^k(x^k+s)=F\left(x^k\right)+{\left(g^k\right)}^{\top }s+\frac{1}{2}{\left(s\right)}^{\top }{H}^{k}s\hfill \\ \hfill & subject\; to\parallel s\parallel \le {\Delta }^k,\mathrm{and}\ell \le x^k+s\le u.\hfill \end{array}$$

(8)

where ${\Delta }^k>0$ is the trust-region radius at an iteration $k.$

Recall that the first-order necessary condition for solving (5) is that the projected gradient is equal to zero. The projected gradient of the model function $m^k$ at $s=0$ onto the feasible box, which is delimited by ℓ and u where $\ell <u$, is defined by

$$\begin{array}{cc}\hfill & {\pi }_m^k=x^k-{Proj}_{[\ell ,u]}(x^k-\nabla m^k\left(x^k\right))=x^k-{Proj}_{[\ell ,u]}(x^k-g^k)\hfill \end{array}$$

where the projection operator, ${Proj}_{[\ell ,u]}\left(y\right),$ is defined component-wise by

$${\left[{Proj}_{[\ell ,u]}\left(y\right)\right]}_i=\left\{\begin{array}{ccc}{\ell }_i\hfill & \hfill & \mathrm{if}{y}_i<{\ell }_i,\hfill \\ u_i\hfill & \hfill & \mathrm{if}{y}_i>u_i,\hfill \\ y_i\hfill & \hfill & \mathrm{if}\mathrm{otherwise}.\hfill \end{array}\right.$$

(9)

Similarly, we define the projected gradient of the objective function at $s=0$ onto the box by

$$\begin{array}{c}\hfill {\pi }_F^k=x^k-{Proj}_{Box}(x^k-\nabla F\left(x^k\right)).\end{array}$$

Next, the pseudo-code of our DFTR algorithm is presented. This is essentially the pseudo-code in [34] (Algorithm 4.1), but adapted for a box-constrained problem (see also Algorithm 10.1 in [35] and Algorithm 2.1 in [61]). The algorithm is a first-order method in the sense that it converges to a first-order critical point.

Several details about the pseudo-code require some clarification. The procedure employed in our algorithm to build the model function at an arbitrary iteration k guarantees that the model $m^k$ is fully linear on $B(x^k;{\Delta }^k).$ To see this, first note that when a model $m^k$ is built in Algorithm 1, we always have that the sampling radius ${\Delta }_s^k\le {\Delta }^k.$ It is known that the gradient of a quadratic interpolation function built with $(n+1)(n+2)/2$ sample points poised for quadratic interpolation is $\mathcal{O}\left({\Delta }_s^2\right).$ The Hessian of this quadratic interpolation is $\mathcal{O}\left({\Delta }_s\right).$ It was shown in [60] that the calculus-based approach to approximating gradients described in (1) and (3) is $\mathcal{O}\left({\Delta }_s^2\right).$ It was also shown in [59,60] that the calculus-based approach to approximating the Hessians described in (2) and (4) was $\mathcal{O}\left({\Delta }_s\right).$ Hence, $g^k$ and $H^k$ are $\mathcal{O}\left({\Delta }^k\right)$-accurate. The next proposition shows that if $g^k$ and $H^k$ are both $\mathcal{O}\left({\Delta }^k\right)$-accurate approximations of the gradient and Hessian at $x^k$, respectively, then the model function $m^k$ is fully linear on $B(x^k;{\Delta }^k).$

Proposition 1.

Let $F\in {\mathcal{C}}^2,$ $x^k\in domF,s^k\in {{ℝ}}^n$ and ${\Delta }^k>0.$ Assume that $x^k+s^k\in B(x^k;{\Delta }^k)$, where k is any iteration of Algorithm 1. Let the model function $m^k$ be defined as in Equation (7). If $g^k$ is an $\mathcal{O}\left({\Delta }^k\right)$-accurate approximation of the gradient of F at $x^k$ and $H^k$ is an $\mathcal{O}\left({\Delta }^k\right)$-accurate approximation of the Hessian of F at $x^k,$ then the model $m^k$ is fully linear on $B(x^k;{\Delta }^k).$

Proof. 

For any $x^k+s^k\in B(x^k;{\Delta }^k),$ we have

$$\begin{array}{cc}\hfill \parallel \nabla F(x^k+s^k)-\nabla m^k(x^k+s^k)\parallel & =\parallel \nabla F(x^k+s^k)-g^k-{\left(s^k\right)}^{\top }{H}^k\parallel \hfill \\ \hfill & \le \parallel \nabla F(x^k+s^k)-\nabla F\left(x^k\right)+\nabla F\left(x^k\right)-g^k\parallel +{\Delta }^k\parallel H^k\parallel \hfill \\ \hfill & \le L_g{\Delta }^k+C_1{\Delta }^k+\parallel H^k\parallel {\Delta }^k\hfill \end{array}$$

where $L_g\ge 0$ is the Lipschitz constant of $\nabla F$ on $B(x^k;{\Delta }^k),$ and $C_1\ge 0.$ Note that $\parallel H^k\parallel$ is bounded above, since

$$\begin{array}{cc}\hfill \parallel H^k\parallel & =\parallel H^k-{\nabla }^{2}F\left(x^k\right)+{\nabla }^{2}F\left(x^k\right)\parallel \hfill \\ \hfill & \le C_2{\Delta }^k+\parallel {\nabla }^{2}F\left(x^k\right)\parallel .\hfill \end{array}$$

Since F is twice continuously differentiable on the box constraint, we have $\parallel {\nabla }^{2}F\left(x^k\right)\parallel \le M$ for some nonnegative scalar M independently of $k.$ Therefore, $\parallel H^k\parallel \le C_2{\Delta }_{max}+M.$ Letting ${\kappa }_g=(L_g+C_1+C_2{\Delta }_{max}+M),$ we obtain

$$\begin{array}{cc}\hfill \parallel \nabla F(x^k+s^k)-\nabla m^k(x^k+s^k)\parallel & ={\kappa }_g{\Delta }^k.\hfill \end{array}$$

Hence, the first property in Definition 3 is verified. The second property can be obtained by using a similar process to that used in [62] (Proposition 19.1). Therefore, the model $m^k$ is fully linear on $B(x^k;{\Delta }^k).$    □

Algorithm 1 DFTR pseudo-code.

Step 0: Initialization.   Choose a feasible initial point $x^0,$ an initial trust-region radius ${\Delta }^0>0,$ an initial sampling radius $0<{\Delta }_s^0\le {\Delta }^0,$ and a maximal trust-region radius ${\Delta }_{max}>0.$ Build an initial fully linear model $m^0(x^0+s)$ on $B(x^0;{\Delta }^0)$. Denote by $g^0$ and $H^0$ the gradient and Hessian of the initial model at $s=0.$ Choose the constants $0\le {\eta }_1\le {\eta }_2<1$ (with ${\eta }_2\ne 0)$, $0<\gamma <1<{\gamma }_{inc},{ϵ}_{stop}>0,\mu >0.$ Set $k=0.$ for$\mathbf{do}k=0,1,2,\cdots$    Step 1: Criticality step.      If $\parallel {\pi }_m^k\parallel \le {ϵ}_{stop}$ and ${\Delta }^k\le \mu \parallel {\pi }_m^k\parallel$, stop. Return $x^k.$      If $\parallel {\pi }_m^k\parallel \le {ϵ}_{stop}$ and ${\Delta }^k>\mu \parallel {\pi }_m^k\parallel$, set ${\Delta }^k\leftarrow min\{\mu \parallel {\pi }_m^k\parallel ,{\Delta }^k\}.$ If ${\Delta }_s^k>{\Delta }^k$,      set ${\Delta }_s^k\leftarrow {\Delta }^k$ update the model $m^k$ to make it fully linear on $B(x^k;{\Delta }^k).$    Step 2: Trust-region sub-problem.      Find an approximate solution $s^k$ to the trust-region sub-problem (8).    Step 3: Acceptance of the trial point.      Compute $${\rho }^k=\frac{F\left(x^k\right)-F(x^k+s^k)}{m^k\left(x^k\right)-m^k(x^k+s^k)}.$$ If ${\rho }^k\ge {\eta }_1$,      set $x^{k+1}\leftarrow x^k+s^k$ and build a fully linear model $m^{k+1}.$      Otherwise (${\rho }_k<{\eta }_1$),      set $m^{k+1}\leftarrow m^k$ and $x^{k+1}\leftarrow x^k.$      Step 4: Trust-region radius update. $${\Delta }^{k+1}\in \{\begin{array}{ll}[{\Delta }^k,min\{{\gamma }_{inc}{\Delta }^k,{\Delta }_{max}\}],& \mathrm{if}{\rho }^k\ge {\eta }_2,\\ \left\{\gamma {\Delta }^k\right\},& \mathrm{if}{\rho }^k<{\eta }_2.\end{array}$$      If ${\Delta }_s^k>{\Delta }^k$,      set ${\Delta }_s^{k+1}\leftarrow {\Delta }^k.$      If ${\rho }^k<{\eta }_1$,      attempt to improve the accuracy of the model.      Increment k by one and go to Step 1. end for

It follows from Proposition 1 that Algorithm 1 always builds a fully linear model on $B(x^k;{\Delta }^k).$

In [61], Hough and Roberts analyzed the convergence properties of a DFTR algorithm for an optimization problem of the form

$$\begin{array}{c}\hfill \underset{x\in C}{min}F\left(x\right)\end{array}$$

where $C\subseteq {{ℝ}}^n$ is closed and convex with a nonempty interior and $F:{{ℝ}}^n\to {ℝ}.$ The main algorithm developed in their paper, Algorithm 2.1, follows the same steps as those of our algorithm. Since a box constraint is a convex closed set with a nonempty interior, the convergence results that were proven in [61] may be applied to Algorithm 1. For completeness, we recall the three assumptions and convergence results of [61].

Assumption 1.

The objective function F is bounded below and continuously differentiable. Furthermore, the gradient $\nabla F$ is Lipschitz continuous with the constant $L_{\nabla F}\ge 0$ in ${\cup }_{k}B(x^k;{\Delta }_{max}).$

Assumption 2.

There exists ${\kappa }_H\ge 0$ such that $\parallel H^k\parallel \le {\kappa }_H$ for all k.

Assumption 3.

There exists a constant $c_1\in (0,1)$ such that the computed step $s^k$ satisfies $x^k+s^k\in C,\parallel s^k\parallel \le {\Delta }^k$, and the generalized Cauchy decrease condition is

$$\begin{array}{cc}\hfill m^k\left(x^k\right)-m^k(x^k+s^k)& \ge c_1{\pi }_m^{k}min\left(\frac{{\pi }_m^k}{1+\parallel H^k\parallel },{\Delta }^k,1\right).\hfill \end{array}$$

Theorem 1.

Suppose that Algorithm 1 is applied to the minimization problem (5). Suppose that Assumptions 1–3 hold. Then,

$$\begin{array}{cc}\hfill \underset{k\to \infty }{lim}{\pi }_F^k& =\underset{k\to \infty }{lim}{x}^k-{Proj}_{[\ell ,u]}(x^k-\nabla F\left(x^k\right))=0.\hfill \end{array}$$

To conclude this section, we provide details on the different choices made while implementing Algorithm 1.

3.1. Implementing the Algorithm

Let us begin by specifying the value used for all of the parameters involved in the algorithm. Our choices were influenced by some preliminary numerical results and the values proposed in the literature, such as [1] (Chapter 6).

In our implementation, the parameters are set to:

$$\begin{array}{cccc}\hfill {\Delta }^0& =1\hfill & \hfill & (\mathrm{initial}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}),\hfill \\ \hfill {\Delta }_s^0& =0.5\hfill & \hfill & (\mathrm{initial}\mathrm{sampling}\mathrm{radius}),\hfill \\ \hfill {\Delta }_{max}& ={10}^3\hfill & \hfill & (\mathrm{maximal}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}),\hfill \\ \hfill {\eta }_1& =0.1\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{accepting}\mathrm{the}\mathrm{trial}\mathrm{point}),\hfill \\ \hfill {\eta }_2& =0.9\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{the}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}\mathrm{update}),\hfill \\ \hfill \gamma & =0.5\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{decreasing}\mathrm{the}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}),\hfill \\ \hfill {\gamma }_{inc}& =2\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{increasing}\mathrm{the}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}),\hfill \\ \hfill {ϵ}_{stop}& ={10}^{-5}\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{verifying}\mathrm{optimality}),\hfill \\ \hfill \mu & =1\hfill & \hfill & (\mathrm{parameter}\mathrm{for}\mathrm{verifying}\mathrm{the}\mathrm{size}\mathrm{of}\mathrm{the}\mathrm{trust}\text{-}\mathrm{region}\mathrm{radius}).\hfill \end{array}$$

To build an approximation of the Hessian $H^k,$ we compute a simplex Hessian, as defined in Definition 2. Two matrices of directions S and T must be chosen; at every iteration k, the matrices $S^k$ and $T^k$ are set to

$$\begin{array}{cc}\hfill S^k& =\frac{{\Delta }_s^k}{2}{Id}_n,\hfill \\ \hfill T^k& =\frac{{\Delta }_s^k}{2}{Id}_n.\hfill \end{array}$$

Multiplying the identity matrix by $\frac{{\Delta }_s}{2}$ to form $S^k$ and $T^k$ guarantees that the sampling radius for building an approximation of the Hessian is equal to ${\Delta }_s.$ Setting $S^k$ and $T^k$ in this fashion creates $(n+1)(n+2)/2$ distinct sample points poised for quadratic interpolation. This implies that the simplex Hessian is equal to the Hessian of the quadratic interpolation function [59]. Clearly, other matrices $S^k$ and $T^k$ could be used, and $S^k$ does not necessarily need to be equal to $T^k$, nor to always be a multiple of the identity matrix for every iteration $k.$ More details on how to choose S and T so that the resulting set of sample points is poised for quadratic interpolation can be found in [59].

To build an approximation of the gradient $g^k,$ the gradient of the quadratic interpolation function built using the same $(n+1)(n+2)/2$ sample points used to obtain $H^k$ is simply computed. Therefore, building a model $m^k$ requires $(n+1)(n+2)/2$ function evaluations.

Note that the sample points utilized to build $g^k$ and $H^k$ may be outside of the box constraint. This is allowed in our implementation.

To solve the trust-region sub-problem, the Matlab command quadprog with the algorithm trust-region-reflective is used. In theory, this method satisfies Assumption 3.

Based on the preliminary numerical results, the version of Algorithm 1 that is utilized to conduct the numerical experiments in this section does not attempt to improve the accuracy of the model in Step 4. The results obtained suggest that rebuilding a new model function when ${\rho }^k<{\eta }_1$ decreases the efficiency of our algorithm.

Reducing Numerical Errors

We next discuss two strategies that decrease the risk of numerical errors. When the sampling radius ${\Delta }_s$ is sufficiently small, numerical errors occur while computing $g^k$ and $H^k$, and this can cause $g^k$ and $H^k$ to be very bad approximations of the gradient and Hessian at $x^k.$ To avoid this situation, a minimal sampling radius ${\Delta }_{smin}$ is defined. Every time the sampling radius ${\Delta }_s^k$ is updated in Algorithm 1 (this may happen in Step 1 or Step 5), the rule implemented is the following:

$$\begin{array}{cc}\hfill {\Delta }_s^k& \leftarrow max\{{\Delta }_{smin},{\Delta }_s^k\}.\hfill \end{array}$$

(10)

In our implementation, ${\Delta }_{smin}={10}^{-4}.$ Numerical errors can occur with relatively large values of the sampling radius ${\Delta }_s.$ The following example illustrates this situation. This motivates our choice of setting ${\Delta }_{smin}={10}^{-4}.$

Example 1.

Let

$$F\left(x\right)=f_1\left(x\right)·f_1\left(x\right)={\left(0.5x^{\top }\left[\begin{array}{cc}10& 9\\ 9& 10\end{array}\right]x+\left[\begin{array}{cc}10& 9\end{array}\right]x\right)}^2$$

Let $x^0={\left[\begin{array}{cc}5& 5\end{array}\right]}^{\top }.$ Set $S=T=\frac{h}{2}{Id}_2,$ where $h>0.$ Note that the sampling radius is ${\Delta }_s=h.$ 

Table 1. Relative error of the simplex Hessian ${\nabla }_s^{2}F(x^0;S,T)$ for different values of h.

${\Delta }_{\mathit{s}}$	$RE\left({\mathit{\nabla }}_{\mathit{s}}^2\mathit{F}({\mathit{x}}^0;\mathit{S},\mathit{T})\right)$
$5\times {10}^{-1}$	$4.7\times {10}^{-2}$
$1\times {10}^{-1}$	$9.3\times {10}^{-3}$
$1\times {10}^{-2}$	$9.2\times {10}^{-4}$
$1\times {10}^{-3}$	$9.2\times {10}^{-5}$
$1\times {10}^{-4}$	$8.8\times {10}^{-6}$
$1\times {10}^{-5}$	$4.5\times {10}^{-5}$

presents the relative error of the simplex Hessian ${\nabla }_s^{2}F(x^0;S,T),$ which is denoted by $RE\left({\nabla }_s^{2}F(x^0;S,T)\right).$

We see that numerical errors occur at a value of ${\Delta }_s$ between ${10}^{-4}$ and ${10}^{-5}$.

A maximal sampling radius ${\Delta }_{smax}$ is also defined to ensure that the sampling radius ${\Delta }_s$ does not become excessively large when the trust-region radius is large. The parameter is set to ${\Delta }_{smax}=0.5.$ Therefore, after checking (10), the following update on ${\Delta }_s^k$ is performed:

$$\begin{array}{cc}\hfill {\Delta }_s^k& \leftarrow min\{{\Delta }_{smax},{\Delta }_s^k\}.\hfill \end{array}$$

In Step 3, in the computation of ${\rho }^k,$ the denominator satisfies

$$\begin{array}{cc}{m}^k\left(x^k\right)-m^k(x^k+s^k)& =F\left(x^k\right)-\left(F\left(x^k\right)+{\left(g^k\right)}^{\top }{s}^k+\frac{1}{2}{\left(s^k\right)}^{\top }{H}^k{s}^k\right)\hfill \\ \hfill & ={\left(g^k\right)}^{\top }{s}^k+\frac{1}{2}{\left(s^k\right)}^{\top }{H}^k{s}^k.\hfill \end{array}$$

(11)

As mentioned in [1] (Section 17.4), to reduce the numerical errors, we use (11) to compute this value.

To compute the ratio ${\rho }^k,$ we again follow the advice given in [1] (Section 17.4.2) and proceed in the following way. Let $ϵ={10}^4·{ϵ}_M$, where ${ϵ}_M$ is the relative machine precision. Let ${\delta }^k=ϵmax(1,|F\left(x^k\right)\left|\right).$ Define

$$\begin{array}{cc}\hfill \delta F^k& =F(x^k+s^k)-F\left(x^k\right)-{\delta }^k,\hfill \\ \hfill \delta m^k& =m^k(x^k+s^k)-m^k\left(x^k\right)-{\delta }^k=F\left(x^k\right)+{\left(g^k\right)}^{\top }{s}^k+\frac{1}{2}{\left(s^k\right)}^{\top }{H}^k{s}^k-{\delta }^k\hfill \end{array}$$

Then,

$${\rho }^k\in \left\{\begin{array}{cccc}\hfill & 1,\hfill & \hfill & \mathrm{if}|\delta F^k|<ϵ\mathrm{and}|F\left(x^k\right)|>ϵ,\hfill \\ \hfill & \frac{\delta F^k}{\delta m^k},\hfill & \hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

3.2. Data Profiles

To perform the comparisons in the next section, data profiles were built [63]. The convergence test for the data profiles is

$$f\left(x\right)\le f_L+\tau \left(f\left(x^0\right)-f_L\right)$$

(12)

where $\tau >0$ is a tolerance parameter and $f_L$ is the best known minimum value for each problem p in $\mathcal{P}.$ Let $t_{p,s}$ be the number of function evaluations required to satisfy (12) for a problem $p\in \mathcal{P}$ by using a solver $s\in \mathcal{S}$ given a maximum number of function evaluations ${\mu }_f.$ In this paper, the parameter is ${\mu }_f=1000\left(n_p\right)$, where $n_p$ is the dimension of the problem $p\in \mathcal{P}.$ Note that ${\mu }_f$ is set to ∞ if (12) is not satisfied after ${\mu }_f$ function evaluations. The data profile of a solver $s\in \mathcal{S}$ is defined by

$$d_s\left(\alpha \right)=\frac{1}{\left|\mathcal{P}\right|}\mathrm{size}\left\{p\in \mathcal{P}:\frac{t_{p,s}}{n_p+1}\le \alpha \right\}$$

Three different values of $\tau$ will be used to build the data profiles: ${10}^{-1},{10}^{-3},{10}^{05}$.

3.3. Experimental Setup

Algorithm 1 was implemented in Matlab 2021a while using both the calculus-based and non-calculus-based approach to approximating gradients and Hessians. The computer that was utilized had an Intel Core i7-9700 CPU at 3.00 GHz and 16.0 GB of DDR4 RAM at 2666 MHz on a 64-bit version of Microsoft Windows 11 Home. These implementations were tested on a suite of test problems, which are detailed below. The Matlab function fmincon was also tested on each problem for all of the experiments mentioned below. This was done as a validation step to demonstrate that Algorithm 1 was correctly implemented (and reasonably competitive with respect to the currently used methods).

To conclude this section, we present the details of the different situations tested for the product rule and the quotient rule.

3.3.1. Numerical Experiment with the Product Rule

In these numerical experiments, the composite objective function F had the form $F=f_1·f_2,$ where $f_i:{{ℝ}}^n\to {ℝ},i\in \{1,2\}.$ Three different situations were considered:

• $f_1$ and $f_2$ were linear functions;
• $f_1$ was a linear function and $f_2$ was a quadratic function;
• $f_1$ and $f_2$ were quadratic functions.

Recall that a linear function $L:{{ℝ}}^n\to {ℝ}$ has the form

$$L\left(x\right)=b^{\top }x+c$$

where b $\in {{ℝ}}^n$ and $c\in {ℝ}.$ A quadratic function $Q:{{ℝ}}^n\to {ℝ}$ has the form

$$Q\left(x\right)=\frac{1}{2}{x}^{\top }Ax+b^{\top }x+c,$$

where $A\in {{ℝ}}^{n\times n}$ is a symmetric matrix. At the beginning of each experiment, the seed of the random number generator was fixed to 54321 by using the command rng. The dimension of the space was between 1 and 30, and it was generated with the command randi. All integers were generated with randi. In each experiment, the coefficients involved in $f_1$ and $f_2$ were integers between −10 and 10 inclusively. Each component of the starting point $x^0$ was an integer between −5 and 5 inclusively. The box constraint was built in the following way:

$$\begin{array}{cc}\hfill {\ell }_i& =x_i^0-1\mathrm{for}\mathrm{all}i\in \{1,\cdots ,n\},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill u_i& =x_i^0+1\mathrm{for}\mathrm{all}i\in \{1,\cdots ,n\}.\hfill \end{array}$$

(14)

This created a randomly generated test problem of the form of (5). This problem was solved via Algorithm 1 by using the starting point $x^0$ and by using both versions—the calculus-based approach and the non-calculus-based approach. Each of the three situations listed above was repeated 100 times. Each time, all of the parameters were generated again: the dimension n, the starting point $x^0,$ the box constraint $[\ell ,u]$, and the functions $f_1,f_2.$

3.3.2. Numerical Experiment with the Quotient Rule: Easy Case

In these experiments, the composite objective function took the form $F=\frac{f_1}{f_2},$ where $f_i$ was either a linear function or a quadratic function for $i\in \{1,2\}$. We began by building the function $f_2$ such that its real roots, if any, were relatively far from the box constraint. To do so, each component of the starting point $x^0$ was taken to be an integer between 1 and 100 inclusively. The coefficients in $f_2$ are taken to be integers between 1 and 10 inclusively. When $f_2$ was a quadratic function, note that having all positive entries in the matrix A did not necessary imply that A was positive semi-definite. The box constraint was built in the same fashion as in the previous experiment. Thus, the bounds were

$$\begin{array}{cc}\hfill {\ell }_i& =x_i^0-1\ge 0\mathrm{for}\mathrm{all}i\in \{1,\cdots ,n\},\hfill \\ \hfill u_i& =x_i^0+1\mathrm{for}\mathrm{all}i\in \{1,\cdots ,n\}.\hfill \end{array}$$

Note that if $r\in {{ℝ}}^n$ is a root of $f_2,$ then r must have at least one negative component. Since $f_2(\ell )>0$ and $f_2$ is an increasing function, there is no root of $f_2$ in the box constraint. Hence, F is twice continuously differentiable on the box constraint. The following four situations were tested:

• $f_1$ and $f_2$ were linear functions;
• $f_1$ was a linear function and $f_2$ was a quadratic function;
• $f_1$ was a quadratic function and $f_2$ was a linear function;
• $f_1$ and $f_2$ were quadratic functions.

Each situation was repeated 100 times.

3.3.3. Numerical Experiment with the Quotient Rule: Hard Case

Last, the quotient rule was tested again, but this time, the function $f_2$ was built so that there was a root of $f_2$ near (but not within) the box constraint. The differences from the previous quotient experiments were the following. Each component of the starting point was taken as an integer between −5 and 5 inclusively. Let $f_2=\widetilde{f_2}+c$, where $c\in {ℝ}.$ The coefficients in $\widetilde{f_2}$ were taken to be between −10 and 10 inclusively. Before generating the constant c in $f_2,$ the minimum value on the box constraint of $\widetilde{f_2}$, say, ${\widetilde{f_2}}^{*},$ was found. Then, c was set to $c=0.001-{\widetilde{f_2}}^{*}.$ Hence, $f_2$ was built such that the minimum value of $f_2$ on the box constraint was 0.001 and $f_2\left(x\right)\ge 0.001$ for all x in the box constraint.

4. Results

In this section, the data profiles for each of the three experiments are presented. We begin by presenting the data profiles for the product rule. The following nine data profiles (three data profiles per situation) were obtained (Figure 1, Figure 2 and Figure 3). In the following figures, the x-axis represents the independent variable $\alpha$, as defined in Section 3.2. The y-axis represents the percentage of problems solved. For a given tolerance parameter $\tau$, the robustness of an algorithm was obtained by looking at the value of its curve at the greatest value of $\alpha$ in the figures. For example, fmincon solved 100% of the problems for all three values of $\tau$ in Figure 1. For a given percentage of problems solved, the curve with the smallest value of $\alpha$ represented the most efficient algorithm, as it used fewer function evaluations to solve a certain percentage of the problems. For example, when $\tau =1$ in Figure 1, the fmincon algorithm was the most efficient and the most robust of all three algorithms. Roughly speaking, if a curve is on top of all other curves in a data profile, then it was the most efficient and the most robust algorithm on this set of problems for a given value of $\tau .$ Testing different values of $\tau$ provides information about the level of accuracy that an algorithm can achieve. Small values of $\tau$ provide information about the capabilities of an algorithm for finding accurate minimizers, and large values of $\tau$ provide information about the capabilities of an algorithm for finding rough approximations of the minimizers.

Next, we present the data profiles for the quotient rule as described in Section 3.3.2. The following 12 data profiles were obtained (Figure 4, Figure 5, Figure 6 and Figure 7).

Last, we present the data profiles for the quotient rule as described in Section 3.3.3. The following 12 data profiles were obtained (Figure 8, Figure 9, Figure 10 and Figure 11).

5. Discussion

In this section, the data profiles presented in Section 4 are analyzed. First, the data profiles related to the product rule are scrutinized.

5.1. Numerical Experiment with the Product Rule

Figure 1 agrees with the theory: The calculus-based approach provided the exact same results as the non-calculus-based approach. Indeed, when both $f_1$ and $f_2$ are linear, the theory states that both approaches will build models with an exact gradient and an exact Hessian. As such, both methods should behave identically.

Figure 2 and Figure 3 show that the calculus-based approach was slightly more efficient and robust. Note that the calculus-based approach built the model functions $m^k$ with an exact gradient $g^k$ and an exact Hessian $H^k.$ This was not the case with the non-calculus-based approach when F was a cubic function or quartic function. However, the accuracy of the approximate gradients and Hessians in the models $m^k$ was sufficiently good that a significant difference from the exact models built with the calculus-based approach was not made. We also observe that Algorithm 1 was more robust than fmincon for these two situations, which supported that our algorithm is competitive with currently used solvers.

Although there were no drastic differences between the calculus-based approach and the non-calculus-based approach, we conclude that the calculus-based approach was better or at least as good as the non-calculus-based approach in these three situations.

5.2. Numerical Experiment with the Quotient Rule: The Easy Case

We now analyze Figure 4, Figure 5, Figure 6 and Figure 7. The performance of both approaches was almost identical. The calculus-based approach provided slightly better results. Compared to fmincon, Algorithm 1 was competitive and even outperformed fmincon when the numerator of F was a linear function (Figure 4). We note that having no roots of $f_2$ near the box constraint implies that the value of the composite objective function F does not drastically change on the box constraint. In other words, the Lipschitz constant of F on the box constraint is not a huge number. This helped obtain an accurate gradient $g^k$ when using the non-calculus-based approach. As such, in these four situations, the accuracy of the approximate gradients and Hessians computed with the non-calculus-based approach was similar to this computed with the calculus-based approach. Hence, the performance of the non-calculus-based approach was almost as good as that of the calculus-based approach.

5.3. Numerical Experiment with the Quotient Rule: The Hard Case

The most interesting results were obtained when the composite objective function F had the form $F=\frac{f_1}{f_2},$ and $f_2$ had a real root near the box constraint $[\ell ,u]$. Figure 8, Figure 9, Figure 10 and Figure 11 clearly show that the calculus-based approach was significantly more efficient and robust than the non-calculus-based approach for these four situations. Note that the composite objective function F was twice continuously differentiable on the box constraint, but F was not twice continuously-differentiable on the expanded box constraint (6). The composite objective function F was also not twice continuously differentiable on $\ell +{\Delta }_{smax}{\mathbf{1}}_n\le xu+{\Delta }_{smax}{\mathbf{1}}_n.$ Using the non-calculus-based approach, it could be the case that $g^k$ or $H^k$ was undefined if one of the sample points landed exactly on a real root of $f_2.$ This was very unlikely and it did not happen during the experiment. When using a calculus-based approach, this issue cannot occur, as $f_1$ and $f_2$ are twice continuously differentiable everywhere. This is clearly an advantage of the calculus-based approach. If a sample point is near a root of $f_2,$ the value of F at this point can be a very large number. This could make the accuracy of $g^k$ and $H^k$ very poor when using the non-calculus-based approach.

Note that the Lipschitz constants of $\nabla F$ and ${\nabla }^{2}F$ on the box constraint were very large numbers. Hence, the error bounds associated with the approximate gradient $g^k$ and the approximate Hessian $H^k$ were very large numbers (see [59]). This tells us that it is possible to obtain very bad approximations for $g^k$ and $H^k$ when using the non-calculus-based approach.

To see how the non-calculus-based approach can fail to provide accurate approximations of the gradient and Hessian, we present Example 2.

Example 2.

Let

$$F=\frac{f_1}{f_2}=\frac{10x+10}{-10x^2+10x+20.0001}.$$

Suppose that $x^k=-1.$ The derivative of F at $x^k$ is $F^{\prime }\left(x^k\right)={10}^5$, and the second-order derivative of F at $x^k=-1$ is $F^{″}\left(x\right)=-6\times {10}^{10}$.

Table 2. An example where the non-calculus-based approach provides inaccurate approximations.

h	${\mathit{g}}^{\mathit{k}}$	RE(${\mathit{g}}^{\mathit{k}}$)	${\mathit{H}}^{\mathit{k}}$	RE(${\mathit{H}}^{\mathit{k}}$)
$5\times {10}^{-1}$	$1.1\times {10}^0$	$9.9\times {10}^{-1}$	$-1.2\times {10}^0$	$1.0\times {10}^0$
$1\times {10}^{-1}$	$5.1\times {10}^0$	$9.9\times {10}^{-1}$	$-3.3\times {10}^1$	$1.0\times {10}^0$
$1\times {10}^{-2}$	$5.0\times {10}^1$	$9.9\times {10}^{-1}$	$-3.3\times {10}^3$	$1.0\times {10}^0$
$1\times {10}^{-3}$	$4.9\times {10}^2$	$9.9\times {10}^{-1}$	$-3.3\times {10}^5$	$1.0\times {10}^0$
$1\times {10}^{-4}$	$4.8\times {10}^3$	$9.5\times {10}^{-1}$	$-3.1\times {10}^7$	$9.9\times {10}^{-1}$
$1\times {10}^{-5}$	$3.5\times {10}^4$	$6.4\times {10}^{-1}$	$-2.1\times {10}^9$	$9.6\times {10}^{-1}$
$1\times {10}^{-6}$	$9.1\times {10}^4$	$8.6\times {10}^{-2}$	$-2.8\times {10}^{10}$	$5.1\times {10}^{-1}$
$1\times {10}^{-7}$	$9.9\times {10}^4$	$1.6\times {10}^{-3}$	$-5.4\times {10}^{10}$	$8.4\times {10}^{-2}$
$1\times {10}^{-8}$	$9.9\times {10}^4$	$1.7\times {10}^{-5}$	$-5.9\times {10}^{10}$	$8.9\times {10}^{-3}$
$1\times {10}^{-9}$	$1.0\times {10}^5$	$3.3\times {10}^{-7}$	$-5.9\times {10}^{10}$	$1.1\times {10}^{-3}$
$1\times {10}^{-10}$	$1.0\times {10}^5$	$1.8\times {10}^{-9}$	$-5.9\times {10}^{10}$	$8.9\times {10}^{-5}$
$1\times {10}^{-11}$	$1.0\times {10}^5$	0	$-6.0\times {10}^{10}$	$2.9\times {10}^{-6}$
$1\times {10}^{-12}$	$-1.0\times {10}^5$	$2.0\times {10}^0$	$1.7\times {10}^{13}$	$2.9\times {10}^2$

provides the value and the relative error associated with the approximations $g^k$ and $H^k$ obtained by using the non-calculus-based approach and $S=T=h$ for different values of $h.$ The relative error of the approximation technique is denoted by $RE\left(·\right)$ in the following table.

Several modifications to Algorithm 1 were tested to see if it was possible to improve the performance of the non-calculus-based approach. None of those modifications resulted in a significant increase in performance for all four situations, which made the non-calculus-based approach competitive with the calculus-based approach.

We observed that numerical errors occurred at $h={10}^{-11}$ for the approximate gradient $g^k$ and at $h={10}^{-12}$ for the approximate Hessian $H^k.$ Note that in this experiment, the sampling radius was ${\Delta }_s=2h.$ Assuming that numerical errors do not occur, the theory guarantees that the relative error will be of order ${\Delta }_s^2$ for $g^k$ and of order ${\Delta }_s$ for $H^k$ when ${\Delta }_s$ is sufficiently small. However, in this experiment, numerical errors occurred before attaining the expected accuracy for $g^k$ and $H^k.$ Therefore, in this experiment, $g^k$ and $H^k$ were extremely inaccurate.

6. Conclusions

When dealing with a composite objective function, the numerical results obtained in Section 4 clearly suggest that a calculus-based approach should be used. In particular, in all cases tested, the calculus-based approach was better or as good as the non-calculus-based approach. Since a calculus-based approach is not more difficult to implement than a non-calculus-based approach, the former approach seems to be the best approach to implement and use whenever a composite objective function is optimized.

When the composite objective function was a quotient ($F=\frac{f_1}{f_2}$) and there was a real root of $f_2$ near the box constraint, the calculus-based approach greatly outperformed the other methods. In this case, the sampling radius needed to be very small to obtain a relatively accurate gradient and Hessian when using the non-calculus-based approach. In some situations, numerical errors may occur before obtaining the accuracy required for convergence. In general, based on the results obtained, it seems reasonable to think that a calculus-based approach will outperform a non-calculus-based approach when the Lipschitz constant of the gradient and∖or Hessian of F on the box constraint are large numbers. By increasing the range of numbers allowed to generate the functions $f_1$ and $f_2$ when using the product rule in the experiment described in Section 3.3.1, it is relatively easy to create a test set of problems in which the differences in performance between the calculus-based approach and the non-calculus-based approach are more pronounced than the results presented in this paper (Figure 1, Figure 2 and Figure 3), which are in favor of the calculus-based approach. However, the gain in performance is not as drastic as the results obtained with the quotient rule in our third experiment (Figure 8, Figure 9, Figure 10 and Figure 11). Further investigation is needed to determine situations in which the calculus-based approach significantly outperforms the non-calculus-based approach when using the product rule.

We remark that our implementation of Algorithm 1 is simple. This is intentional, as the goal is to compare the calculus-based approach with the non-calculus-based approach. Nonetheless, we remark that, recently, Nocedal et al. found that a DFTR algorithm that used quadratic models built from a forward-finite-difference gradient and a forward-finite-difference Hessian can be surprisingly competitive with state-of-the-art derivative-free algorithms [64]. From our perspective, now that the calculus-based approach has been established as the superior method within a model-based derivative-free algorithm, the next logical step is to use this knowledge to improve current state-of-the-art software (e.g., [65]). Applying a calculus-based approach to real-world optimization problems is, then, the obvious direction of future research.

Another direction to explore is inspired by model-based methods for high-dimensional blackbox optimization [66]. In [66], the authors used subspace decomposition to reduce a high-dimensional blackbox optimization problem into a sequence of low-dimensional blackbox optimization problems. At each iteration, the subspace was rotated, requiring new models to be frequently constructed. An examination of whether calculus rules could be adapted to help in this situation would be an interesting direction for future research.

This further links to a direction of future research based on underdetermined gradient/Hessian approximations. An underdetermined approximation does not contain accurate information about all entries of a gradient/Hessian, but uses fewer function evaluations than the determined case. It would be valuable to explore if calculus-based approaches could be merged with underdetermined approximations to create more accuracy with a minimal increase in function evaluations.