COMPARATIVE STUDY OF ALGORITHMS FOR RESPONSE SURFACE OPTIMIZATION

-Response Surface Methodology (RSM) is a method that uses a combination of statistical techniques and experimental design for modelling and optimization problems. Many researchers have studied the integration of heuristic methods and RSM in recent years. The purpose of this study is to compare two popular heuristic methods, namely Genetic Algorithms (GA) and Simulated Annealing (SA), with two commonly used gradient-based methods, namely Sequential Quadratic Programming (SQP) and Generalized Reduced Gradient (GRG), to obtain optimal conditions. Moreoever, real quadratic and cubic response surface models are selected from literature and used in this study. The comparison results indicate that the heuristic methods outperform the traditional methods on majority of the problems.


INTRODUCTION
Response Surface Methodology (RSM) seeks to relate a response variable to some input variables by conducting experiments and applying regression analysis.Box and Wilson [1] first introduced RSM.It was initially used in chemical industry but it is now used in physical, engineering, biologicial, clinical and social sciences.The main advantage of RSM is the reduced number of experimental trials needed to evaluate multiple parameters and their interactions [2,3].RSM usually contains three stages i.Design of experiment, ii.Response surface modeling through regression, iii.Optimization [4].
Optimization methods used in the third phase can generally be classified into three broad groups: gradient-based techniques (e.g., the model trust region algorithm, the quasi-Newton method, etc.), direct search techniques (e.g., the method of Hooke and Jeeves, the method of Rosenbrock, etc.) and meta-heuristic techniques (e.g., genetic algorithms, simulated annealing, particle swarm optimization, ant colony optimization etc.).Among the optimization techniques, the steepest ascent (or descent) is commonly used (see, [5]), but the method is relatively inefficient and capable of finding only local optima.Recently, many researchers have tried to find a way of integrating RSM with modern heuristic methods.
Khoo and Chen [6] showed how to integrate RSM with GAs for the determination of near optimal values in response surface design.They presented a framework of the prototype system.A pseudo-objective function, which can be used to deal with one response and two response problems, was derived.The prototype system was validated quality variables.The techniques were performed and results were compared.Results indicated that both techniques were capable of locating good conditions.
Bashiri et al. [19] proposed a new meta-heuristic including SA and PSO to optimize all responses in multi response surface problems.The proposed method was applied to two problems from literature.Results of the study showed that the proposed method outperforms the other approaches and can find better solutions.
Luangpaiboon and Sermpattarachai [20] compared steepest ascent, SA and ant colony optimization (ACO) algorithms on eight benchmark problems which have 2-5 variables.Results of the experiment showed that combined algorithm of SA and ACO enabled to search optimal response faster.
In this article, real quadratic and cubic response surface models are selected from literature and two popular heuristic methods viz.GA and SA are compared with commonly used gradient-based methods, viz.SQP and GRG to obtain optimal conditions.Although there are many works in RSM literature, this is the first study for the performance assestment of optimization methods on real response surface models collected from literature.
The rest of the paper is organized as follows.Section 2 describes the basic concepts of RSM.Section 3 is devoted to detailed description of GA and SA.In Section 4, we give a detailed description of SQP and GRG.The results of computational experiments are presented in Section 5. Section 6 concludes and summarizes the main results obtained in this paper.

RESPONSE SURFACE METHODOLOGY
RSM is a collection of statistical and mathematical techniques useful for developing, improving, and optimizing processes in which a response of interest is influenced by several variables and the objective is to optimize this response.Applications of RSM can be found in a wide range of fields such as material science, the chemical industry, mechanicals, the food industry, manufacturing, etc.
In most RSM problems, a form of the relationship between the response and the independent variables is unknown.Thus, the first step in RSM is to find a suitable approximation for the true functional relationship between y and the set of its independent variables denoted by x 1 ,x 2 ,…,x n .Usually, a low-order polynomial in some region of the independent variables is employed.If the response is well modeled by a linear function of the independent variables, the approximating function is the first order model as Since the real world problems are usually very complicated, linear estimation may not perform well in providing a good representation of the objective function.If a curvature appears in the system, then quadratic model will be or cubic model will be (3) In Eqs. ( 1)-(3), y is the response variable, β 0 is the constant term, β i is the coefficient of the linear term, β ii is the coefficient of quadratic single term, β ij is the coefficient of the quadratic cross product term, β iii is the coefficient of the cubic single term, β iij is the coefficient of the cubic two cross product terms, β ijk is the coefficient of the cubic three cross product terms.The x i , x j , x k terms represent the independent variables.
Optimization theory consists of a body of numerical methods for finding and identifying the best candidate from a collection of alternatives without having to explicitly evaluate all possible alternatives [21].Complex response surfaces may have many local optima, and more advanced optimization techniques may be needed.

Genetic algorithms
GAs, which were proposed by Holland in the 1960s, are search algorithms that emulate the adaptive processes of natural biological systems [22].Based on the survival and reproduction of the fittest, they continually search for new and better solutions without any pre-assumptions such as continuity and unimodality.
GAs have been applied in many complex optimization and search problems and also outperforming traditional optimisation and search methods [6,23] The solution of the problem that GAs attempt to solve is coded into a string of binary numbers known as a chromosome.Each chromosome contains the information of a set of possible process parameters such as the temperature and pressure settings.Initially, a population of chromosomes is formed randomly.The fitness of each chromosome is then evaluated using an objective function after the chromosome has been decoded.Upon completion of the evaluation, a biased roulette wheel is used to select randomly pairs of chromosomes to undergo genetic operations such as crossover and mutation to produce offspring for fitness evaluation.This process continues until a near optimal solution is found [24].

Simulated annealing
In this section, the fundamental intuition of the SA and how it processes are given briefly.The SA was proposed by Kirkpatrick et al. [25] to deal with complex nonlinear combinatorial optimization problems.SA is motivated by an analogy to annealing in solids.The idea of SA comes from a paper published by Metropolis et al. [26].It is inspired from thermodynamic to simulate the physical process of annealing of molten metals [27,28].It obtains the minimum value of energy by simulating annealing which is a process employed to obtain a perfect crystal by gradual cooling of molten metals in order to keep the system of melt in a thermodynamic equilibrium at given temperature [29].Thus, it exploits an analogy between the way in which a metal cools and freezes into a minimum energy crystalline structure.At high temperature, the atoms in the molten metal can move freely with respect to each other as the cooling proceeds, the atoms of metal become more ordered and the system naturally converges towards a state of minimal energy.This formation of crystal mostly depends on the cooling rate.If the metal is cooled at very fast rate, the atoms will form an irregular structure and the crystalline state may not be achieved.The Metropolis algorithm provides an efficient simulation of a collection of atoms in equilibrium at given temperature.The SA makes use of the Metropolis algorithm which provides an efficient simulation according to a probabilistic criterion stated as: where if 0 E , the probability, P, is one and the change -the new point-is accepted as () / E Tk e  . Each set of points of all atoms of a system is scaled by its Boltzmann probability factor.E  is the change in the energy value from one point to the next, k is the Boltzmann's constant, T is the current temperature as a control parameter.Even at a low temperature, there is a chance for the system being in a high-energy state.
Thus, if 0 E , the probability, P, is one and the change -the new point -is accepted.Otherwise, the modification is accepted at some finite probability.Each set of points of all atoms of a system is scaled by its Boltzmann probability factor ( / )   E Tk e  , where '' E  " is the change in the energy value from one point to the next, ''k " is the Boltzmann's constant and ''T" is the current temperature as a control parameter.Even at a low temperature, there is a chance for the system being in a high-energy state.Thus, there is a corresponding chance for getting out of a local energy minimum in favor of a better solution, a global one.The following is the general procedure for employing the SA: Step 1: Start with a random initial solution, X, and an initial temperature, T, which should be high enough to allow all candidate solutions to be accepted and evaluate the objective function.The initial temperature is problem specific and depends on the scaling of the objective function. Step

Sequential quadratic programing (SQP)
The basic concept of SQP is to model a nonlinear programming problem by using an iterative algorithm in which, at a current iterate x k , the step to the next iterate is obtained through the information generated by solving a quadratic sub-problem.This sub-problem is assumed to reflect, in some way, the local properties of the original problem.The main idea is the formulation of a sub-problem based on a quadratic approximation of the Lagrangian function [30].The nonlinear programming problem to be solved is min ( ); subject to: in which F c (x) is the objective function; m is the number of the equality constraints h(x); p is the number of inequality constraints g(x) and x is the vector containing the design parameters a 1 , a 2 , a 3 , d 2 , d 3 ,α 1 , and α 2 .
To take nonlinearities in the constraints into account while maintaining the linearity of the constraints in the sub-problem, the SQP method uses a quadratic model of Lagrangian function Λ as the objective.Let the Lagrangian function given by 11 ( , , ) ( ) ( ) ( ) In above equations, ui and vi represent the Lagrangian multipliers,  [30], the SQP framework can be extended easily to the nonlinear problem in ( 5) and ( 6), in this case, in order to model the problem it is necessary to linearize both the inequality and equality constraints to obtain subject to: where x k is the vector containing the design parameters at iteration k, T denotes the transpose operation; s x = xx k is the search direction; He k is taken as the Hessian matrix of the Lagrangian function at x k .
The numerical procedure starts with an initial guess of the manipulator chain solution and, during each iteration k, the quadratic programming problem is solved to provide a search direction s x .The solution s x can be used to generate a new iterate x k+1 , for some selection of the step-length parameter ψk, as x k+1 = x k + ψks x (11) To continue to the next iteration, a new estimate for the Lagrangian multipliers is necessary [31].A usual approach is to use optimal multipliers of the quadratic subproblem denoted by uqp and vqp.Thus, the updated multipliers u k+1 and v k+1 are obtained as follows In summary, a SQP technique solves the optimization problem defined by Eqs. ( 5) and ( 6) by computing the search direction by means of Eqs. ( 7)-( 12).

Generalized reduced gradient (GRG)
The GRG method was developed to solve nonlinear programming problems of the form minimize f(X) min max ,(m < n) are the nonlinear constraints [32].
Firstly, through linear approximation of the restrictions, the problem is transformed in to a sequence of linearized subproblems.At the optimal point, the approximated problem possesses the same solution as the original problem.Each subproblem with linear constraints is then solved with the reduced gradient method.The method reduces the dimension of the problem by representing part of the variables, called basics, by means of a subset of independent variables, called non-basics.The splitting of the variable X into (x, u), where x is the vector of basic variables and u is the vector of nonbasic variables, follows the non-degeneracy hypotheses: 1.The vector x has a dimension of m and the vector u has a dimension of (n-m).
2. The Jacobian matrix, g x J , of g=(g 1 ,...,g m ) with respect to x, is non-singular in X= (x,u).

OPTIMIZATION ON THE RESPONSE SURFACE MODELS
We use two test problem sets to compare GA and SA with SQP and GRG in our study.The first set consists of the seven quadratic models and the second of seven cubic models.These real models are selected from many application areas of RSM including engineering, chemistry, food, mechanical and biological sciences.

Ö. Yeniay
Here, quadratic models have five factors; however cubic models have three, four and five factors.In all response surface models, the purpose was to determine the optimum values for controlled variables to maximize the response variable.We implemented the conventional and heuristic methods in MATLAB 7.11 with the Global Optimization Toolbox 3.1 (Mathworks, Natick, MA) to solve our mathematical models.The GA and SA parameters used in our study were shown in Table1 and The results which were obtained for each of the problems with classical and heuristic algorithms are shown in Table 3.The results of the experiments on all 14 quadratic and cubic models show that for most of the models, SA and GA perform significantly better than GRG and SQP with regard to the maximum value of the response variable.The optimal value of response variables is given in bold fonts.Conventional optimization algorithms (GRG and SQP) fail to find the best global solution with the exception of three response surfaces (problems 2, 4 and 7).For these three problems whose model types are all quadratic, at least one of the heuristic methods produces the same results.In cubic models, neither GRG nor SQP could find the optimal response value.In contrast to SQP and GRG, SA and GA have ability to avoid being trapped at a local minimum.

CONCLUSION
This paper compared the well-known optimization algorithms GRG and SQP to GA and SA on 14 real common response surface models.To our knowledge, this is the first known comparative study of two heuristic approaches with gradient-based approaches on real response surface models.The results have shown that for most of the models GA and SA achieve significantly better performance than SQP and GRG to locate the global optimal point.On the basis of these results, we conclude that GA and SA explore the response surfaces more efficiently than GRG and SQP when the response surfaces are especially third-order and possess numerous local optimals.On the other hand, most of the commonly used software packages (Design Expert, Minitab, etc.) employ gradient-based algorithms for response optimization problems.For wellbehaved functions with a single peak or valley, these algorithms work very well.However, as we can conclude from our study, it will be difficult to use gradient-based optimization algorithms when the surface has many peaks, ridges, and valleys.With more than 3 factors and higher-order response models, finding the optimal will be much more difficult.For these reasons, GA and SA techniqes are found to be reliable, robust and more efficient algorithms for complex multimodal response surfaces and also they offer an alternative to the conventional gradient methods.
For the future work, we plan to apply popular heuristic algorithms such as GA, SA, ACO and PSO etc. on our response surface models and evaluate them in terms of their convergence rate, optimized results, and the number of parameters needed for adjustment.

REFERENCES
Jacobian matrix of the constraints.According to Nocedal and Wright This is due to nature ability of the SA allowing deteriorations with a large probability in the objective function.
T r is temperature reduction rate called cooling schedule and go to Step 2. The process is done until freezing point is reached.The major advantages of the SA are an ability to avoid becoming trapped in Ö. Yeniay local optimum.

Table 2 .
SA parameter settings

Table 3 .
Properties of the problems, optimization results and problem sources