Automatic Convexity Deduction for Efﬁcient Function’s Range Bounding

: Reliable bounding of a function’s range is essential for deterministic global optimization, approximation, locating roots of nonlinear equations, and several other computational mathematics areas. Despite years of extensive research in this direction, there is still room for improvement. The traditional and compelling approach to this problem is interval analysis. We show that accounting convexity/concavity can signiﬁcantly tighten the bounds computed by interval analysis. To make our approach applicable to a broad range of functions, we also develop the techniques for handling nondifferentiable composite functions. Traditional ways to ensure the convexity fail in such cases. Experimental evaluation showed the remarkable potential of the proposed methods.


Introduction
Reliable bounding of univariate functions is one of the primary techniques in global optimization, i.e., finding the solution for the following problem: The problem (1) has many practical applications [1][2][3][4][5][6]. Besides solving problems of one variable, univariate search serves as an auxiliary method in multivariate global optimization. A promising optimization technique known as space-filling curves reduces an optimization [7,8] or approximation [9] problem of multiple variables to a sequence of univariate problems. Univariate optimization techniques are widely used in separable programming [10], where an objective and constraints are sums of functions of one variable. Many univariate optimization methods are directly extended to the multivariate case [11,12].
Univariate global optimization has been intensively studied last decades. The first results date back to the early 1970s. Seminal works in this area [13][14][15][16] relied on the Lipschitzian property of a function: In [14,15] the "saw-tooth cover" lower and upper bounding functions for Lipschitzian objectives were proposed. The lower (upper) bounding functions were defined as max i∈1,...,n f (x i ) − L|x − x i | (min i∈1,...,n f (x i ) + L|x − x i |), where L is a Lipschitz constant and {x 1 , . . . , x n } is a set of function evaluation points. Since the functions are piecewise linear, their range can be easily computed. This makes such estimates attractive for bounding an objective from below and/or above. Other approaches exploiting the property (2) were studied in numerous papers [17][18][19]. In papers [20][21][22], the Lipschitzian first derivatives were used to facilitate the global search. Good surveys on Lipschitzian optimization can be found in [7,[23][24][25].
Interval analysis [26,27] is another powerful technique for global optimization. The goal of interval analysis is to find the tightest enclosing interval for a range of a function. The left end of the enclosing interval provides a lower bound for a function over an interval that can be used to reduce the search space in global optimization methods. Most promising approaches are based on interval arithmetic, and more advanced techniques based on interval Taylor expansions [26,28]. Promising approaches based on combining Lipschitzian optimization and interval analysis ideas were proposed in [29]. Efficient optimization algorithms based on piecewise linear [30,31], piecewise convex [32], slopes techniques [33], and DC-decomposition [34,35] should also be mentioned.
The approaches outlined above apply various methods to obtain bounds on a range of a function. However, they do not analyze the convexity of the objective function. Meanwhile, the convexity plays an essential role in global optimization. If the objective is proved to be convex, then efficient local search techniques can be applied to locate its minimum. For example, the univariate convexification technique developed in [36] even sacrifices the dimensionality of the problem for convexity.
The convexity test [26] helps to reduce the search region by pruning areas where a function is proved to be nonconvex. Usually, the convexity is checked by analyzing the range of the second derivative. If this range lies above (below) zero, the function is convex (concave). This approach works only for functions with continuous second derivatives.
Checking convexity is, in general, an NP-hard problem (see [37]) and references therein. Approaches based on the symbolical proof and the numerical disproof of convexity are described in [38]. In the context of convexity checking, it is necessary to mention the disciplined convex programming [39,40], which also relies on a set of rules for proving the convexity of the problem under consideration. However, authors limit their techniques to proving the convexity of the entire mathematical programming problem for a subsequent use of convex programming methods. As we show below, monotonicity, convexity and concavity properties can also remarkably improve the accuracy of interval bounds when applied to subexpression of the function's algebraic representation.
The main contribution of our paper is the novel techniques for bounding the function's range by accounting monotonicity, convexity or concavity of subexpressions of its algebraic expression. This approach efficiently restricts the objective function's range even if the latter is not convex neither concave. We proved experimentally that the introduced techniques can significantly reduce the bounds on the function's range and remarkably enhance the conventional interval global search procedures. A set of rules for deducing monotonicity, concavity and convexity properties of a univariate function from its algebraic expression is clearly and concisely formulated and proved. These rules complement the traditional ways of establishing the properties of the objective function based on evaluating its derivatives' ranges. Notation: R -the set of real numbers; Z -the set of integers; N -the set of positive integers (natural numbers); IR -the set of all intervals in R;

Type Smooth Non-Smooth
One variable x n , n √ x, 1/x, ln(x), e x |x| sin(x), arcsin(x), arctan(x) Two variables x + y, x · y max(x, y), max(x, y) The paper is organized as follows. Section 2 describes the deduction techniques to evaluate the convexity/concavity of a function automatically. Then, in Section 3 it is shown how this technique is used to bound the range of a function. Section 4 contains the experimental results demonstrating the the proposed approach's efficiency. Section 5 concludes the paper and discusses possible future research directions.

Deducing Monotonicity
The monotonicity significantly helps in global optimization. If a function f (x) is monotonically nondecreasing on a segment [a, b], then min x∈ [a,b] and the segment [a, b] can be eliminated from further consideration after updating the record (best known solution so far). A similar statement is valid for a nonincreasing function. This techniques is known as the monotonicity test [26,41,42]. Moreover, as it is shown below, the monotonicity is crucial for evaluating the convexity/concavity of a composite function.
The usual way to ensure the monotonicity of a differentiable univariate function f (x) on an interval [a, b] is to compute an interval extension for its derivative [c, d] = f ([a, b]). If c ≥ 0, then the function is nondecreasing monotonic on [a, b]. Similarly, if d ≤ 0, then the function is nonincreasing monotonic on [a, b].
If a function is not differentiable, its monotonicity can still be evaluated using the rules described below. The Proposition 1 lists rules for evaluating an expression's monotonicity composed with the simple arithmetic operations. Proposition 1. The following rules hold: The proof of Proposition 1 is obvious. The rules for evaluating the monotonicity of the composition of functions are summarized in Proposition 2. The proof is intuitive and not presented here.
Proposition 2. Let f (x) be a composition of univariate functions g(x) and h(x): f (x) = g(h(x)). Then, the following four statements hold.
The monotonicity of elementary univariate functions on a given interval can easily be established as these functions' behavior is well-known (Table 2).

Function
Increase Decrease The monotonicity of a composite function defined by an arbitrary complex algebraic expression can be evaluated automatically using Propositions 1 and 2 and the data from the Tables 2. Let us consider an example.
It is worth noting that the rules outlined above help to prove the monotonicity of nondifferentiable functions. However, for differentiable functions, the analysis of the the range of the first derivative is a better way to establish monotonicity. For example, a function f (x) = e x + sin(x) is monotonic on an interval [0, 2π]. Indeed, the range [0, e 2π + 1] of its first derivative f (x) = e x + cos(x) computed by the natural interval expansion is non-negative. However, its monotonicty cannot be established by the outlined rules since sin(x) is not monotonic on [0, 2π]. The general recommendation is to compute the first derivative's range when the function is smooth and use Propositions 1 and 2 otherwise.
Monotonicity itself plays a vital role in optimization. The following obviously valid Proposition shows how the interval bounds can be computed for a monotonic function.

Deducing Convexity
First, we recall some well-known mathematical notions used in the rest of the paper.
for any Convexity plays an important role in optimization due to the following two observations. If a function is convex on some interval, then a minimum point of f (x) can be efficiently found by well elaborated local search techniques [43,44].
If the function is two times differentiable, the convexity can be deduced from the second derivative. If one can prove that f (x) ≥ 0(≤ 0) on a segment [a, b], then f (x) is convex (concave) on this segment. However, if the function is nonsmooth, the convexity property should be computed in some other way. Even if f (x) is smooth, the accurate bounding of its second derivative can be a complicated task, and the convexity test becomes difficult.
The conical combination and the maximum of two functions are known to preserve convexity. The proof can be found in seminal books on convex analysis, e.g., [43]. For the sake of completeness, we reproduce these rules in the following Proposition 4.
Proposition 4. Let f (x) and g(x) be convex functions on an interval [a, b]. Then, the following statements hold: The product of two convex functions is not always a convex function. For example, (x − 1)(x 2 − 4) is not convex while both x − 1 and x 2 − 4 are convex functions on R. In [45], it is proved that if f and g are two positive convex functions defined on an interval [a, b], then their product is convex provided that they are synchronous in the sense that for all x, y ∈ I. However checking this general property automatically is difficult. Instead, we propose the following sufficient condition that can be effectively evaluated.
Propositions 4 and 5 can be readily reformulated for concave functions. The following Proposition gives rules for evaluating the convexity of a composite function.
Then, the following holds: The proof of the Proposition 6 can be found in numerous books for convex analysis, e.g., [43].
Many elementary functions are convex/concave on a whole domain of the definition, e.g., e x , ln x, x n for even natural n. For other functions, the intervals of concavity/convexity can be efficiently established as these function's behavior is well-known (Table 3).

Function
Convex Concave . According to the Proposition 6 function, 2 −x is convex. Since 2 x is also convex, we conclude (Proposition 4) that It is worth noting that the convexity can be proved by computing the interval bounds for the second derivative in the considered example. Indeed, f (x) = ln 2 (2)×2 x + ln 2 (2) 2 x is obviously positive on [−1, 1]. Since there are plenty of tools for automatic differentiation and interval computations, the convexity can be proved automatically.
However, a convex function does not necessarily have derivatives in all points. Moreover, even if it is piecewise differentiable, locating the points where the function is not continuously differentiable can be difficult. Fortunately, the theory outlined above efficiently copes with such situations.

Example 3. Consider the following function
The convexity of e −x follows from the convexity of the linear function −x and the Proposition 6. From the convexity of x, 2 − sin(x), e −x and Proposition 4 we derive that f (x) is convex.
Notice that automatic symbolic differentiation techniques cannot compute the derivative of f (x) because it involves computing the intersection points of x and 2 − sin(x) functions, which is a rather complex problem.

Application to Bounding the Function's Range
An obvious application of the proposed techniques is the convexity/concavity test [26] that helps to eliminate the interval from the further consideration and reduce the number of steps of branch-and-bound algorithms. Consider the following problem: If the objective f (x) is concave on [a, b], then the global minimum can be easily computed as follows: f (x * ) = min( f (a), f (b)). If the concavity does not hold for the entire search region, the test can be used in branch-and-bound, interval Newton or other global minimization methods by applying it to subintervals of [a, b] processed by the algorithm.
However, if the objective is convex on [a, b], then any local minimum in [a, b] is a global minimum and can be easily found by a local search procedure. Since any continuously differentiable function is convex or concave on a sufficiently small interval, the convexity/concavity test can tremendously reduce the number of algorithm's steps by preventing excessive branching.
Another situation commonly encountered in practice occurs when a subexpression represents a convex/concave function while the entire function is not convex neither concave. For example, the function 0.5 − cos(x) is convex on interval [−π/2, π/2] while (0.5 − cos(x)) 3 is not. In such cases, the interval cannot be discarded by the convexity/concavity test. Nevertheless, the convexity and concavity can be used to compute tight upper and lower bounds for the sub-expression yielding better bounds for the entire function.
For computing upper and lower bounds, recall that a convex function graph always lies above any of its tangents. This property and the convexity definition yield the Proposition 7. where Proof. First, prove that f (x) is an underestimator for f (x). For a function f (x) convex on an interval [a, b] and a point t ∈ [a, b], the following inequality holds [43]: Substituting t with a and b in (7), we get the following system of valid inequalities: From (8), it directly follows that The right part is f (x) from (6). (3) we obtain: The rightmost part is f (x) from (6). This completes the proof.
Proposition 7 is illustrated in Figure 1. The figure shows the original function f (x) (blue curve), its overestimator consisting of one green line segment and the underestimator consisting of two connected line segments AC and CB marked with red. The estimators are constructed by following (6). The similar proposition holds for concave functions. where Proof. This statement is a straightforward corollary of Proposition 7. Indeed, if f (x) is concave then − f (x) is convex and one can apply Formula (6) to obtain its estimators. After changing the sign and reversing the inequalities we get (10).
Fortunately, the minimum and maximum of estimators f (x) and f (x) can be found analytically as stated by the following propositions.  (14) otherwise. (11) is obviously valid. Denote α = f (a), β = f (b). Equation (12) follows from the fact the function f (x) lies above its tangent f (a) + α(x − a), coincides with it at x = a and the tangent is a monotonically nondecreasing function. In the same way, Equation (13) is proved. For the remaining case α < 0 < β the minimum of the underestimator is achieved at the intersection point of lines defined by f (a) Figure 1). This point is the solution of the following equation:

Proof. Equation
Simple transformations yield: Since α = β the minimum of the underestimator is achieved at the point Substituting this value to f (a) + α(x − a) we obtain: This concludes the proof.
Similarly, the validity of the following Proposition giving bounds for a concave function is justified.  (18) otherwise.
Proof. This statement is a straightforward corollary of Proposition 9. Indeed, if f (x) is concave then − f (x) is convex and one can apply Formulas (11)- (14) to obtain its estimators. After changing the sign and reversing the inequalities, we get Formulas (15)- (18).
mode automatic differentiation [28], which is merely the application of differentiation rules at a point. Table 4 compares bounds computed with the interval analysis techniques and the bounds computed by the proposed method for five convex functions. The convexity of these functions can be easily deduced by the introduced convexity evaluation rules. For an interval [a, b], three bounds are presented in the respective columns: Natural-a bound computed by the natural interval expansion techniques, Taylor-a bound computed by the 1st order Taylor expansion, Convex-a bound computed according to Propositions 9 and 10. For all functions except No. 4, the bound produced by the proposed techniques contain both intervals produced by interval techniques and significantly more tight. For function number 4, the interval computed by the Convex method is narrower than the natural interval expansion but does not enclose it. However, as neither of these intervals contains each other, they can be intersected to obtain a better enclosing interval [1.65, 90.02]. The 5th function is non-differentiable in x = 0. Thus the symbolic differentiation does not give a meaningful result, and the Taylor expansion cannot be applied in this case. For that reason, the respective cell is marked with "−".

Impact on the Performance of Global Search
In Section 4.1, we observed that accounting convexity can significantly improve the interval bounds. As expected, the application of these bounds entails reducing the number of steps of the global search algorithm.
We implemented a standard branch-and-bound algorithm that uses the lower-bound test to discard subintervals from the further search. The description of this algorithm can be found elsewhere [26,41]. For completeness, we outline it here ( Figure 2). We consider three variants of computing the interval extension: 1. Natural-the natural interval expansion techniques, 2. Taylor-the 1-st order Taylor expansion, 3. Convex-the range is computed according to Propositions 3, 9 and 10.
The described methods can be applied in combination, when the intervals computed by several methods are intersected to obtain the resulting range. We considered four different combinations of the range bounding techniques to compute the enclosing interval of the objective function: 1. Natural-pure natural interval expansion; 2. Natural + Convex-the natural interval expansion combined with the proposed techniques; 3. Natural + Taylor-the natural interval expansion combined with the first-order Taylor expansion; 4. Natural + Taylor + Convex-the natural interval expansion combined with the firstorder Taylor expansion and the proposed techniques.
The convexity and the monotonicity are detected by analyzing the ranges of the first/second derivatives in differentiable cases or by using the introduced evaluation rules for the non-differentiable expressions. Table 5 lists the set of test problems used in the experiments. For each problem, the objective function ( f (x)), the interval ([a, b]), and the global optimum value ( f (x * )) are presented. The first ten problems are taken from [29]. The objective functions in these problems have both first and second derivatives.
To demonstrate the applicability of the proposed automatic convexity deduction techniques, we also have added four nondifferentiable problems. Test cases 11 and 12 were proposed by us, and 13 and 14 were taken from [33].  | The results of numerical experiments are summarized in Table 6. The cells contain the number of steps performed by the branch-and-bound method. Columns correspond to different ways for computing the range of the objective functions, and rows correspond to test problems. The Taylor expansion cannot be applied to nondifferentiable problems 11-14, and the respective cells are blank. No Natural Natural + Convex Natural + Taylor Natural + Taylor + Convex   1  35  15  29  15  2  135,043  199  267  81  3  98,995  107  269  79  4  72,953  151  311  91  5  443  39  83  39  6  187  19  47  19  7  183  39  69  39  8  189  49  91  49  9  857  31  75  31  10  51  19  27  19  11  55  5  --12  579  23  --13  35  27  --14  125 125 --Experimental results demonstrate that the proposed techniques tremendously improve the standard branch-and-bound algorithm's performance that uses the natural interval expansion for the majority of the test problems. The combination of the natural interval expansion and the proposed method always outperform the combination of the natural and the first-order Taylor interval expansion. The comparison of the last two columns of Table 6 indicates that the Taylor expansion version of branch-and-bound can be further improved when combined with the proposed techniques. However, for problems 1 and 5-10, the proposed method does not benefit from the Taylor expansion.

Discussion
The standard way to ensure the convexity is to bound the range of the function's second derivative. However, this approach is only applicable to smooth functions. We defined a set of rules that can efficiently handle nonsmooth functions. The algebraic representation for the function, however, should be available.
It is worth noting that the proposed approach can be efficiently coded in modern programming languages supporting the operator's overloading techniques. To run the experiments presented in Tables 4 and 6 we have implemented our approach in Python. The elementary functions and operators were overloaded to support a particular data type that carries monotonicity and convexity information and the range of the function. The overloaded methods work according to the rules described in Section 2 and interval arithmetic.
As we have shown above, evaluating convexity can improve interval bounds on the function's range and accelerate the global optimization algorithms. Moreover, the over-and underestimators defined by the Propositions 7 and 8 enable efficient reduction techniques. The reduction techniques are widely used to accelerate the search for a global minimum of a function or a root of an equation.
We believe that the proposed approach has great potential as it can be extended to various generalized notions of convexity, e.g., quasiconvexity [46]. Quasiconvex functions possess the unimodality property, and thus recognizing the quasiconvexity (quasiconcavity) can tremendously enhance global optimization algorithms.