A Lifting-Penalty Method for Quadratic Programming with a Quadratic Matrix Inequality Constraint

In this paper, a lifting-penalty method for solving the quadratic programming with a quadratic matrix inequality constraint is proposed. Additional variables are introduced to represent the quadratic terms. The quadratic programming is reformulated as a minimization problem having a linear objective function, linear conic constraints and a quadratic equality constraint. A majorization–minimization method is used to solve instead a l1 penalty reformulation of the minimization problem. The subproblems arising in the method can be solved by using the current semidefinite programming software packages. Global convergence of the method is proven under some suitable assumptions. Some examples and numerical results are given to show that the proposed method is feasible and efficient.


Introduction
In this paper, we consider the quadratic programming with a quadratic matrix inequality constraint of the following form: where q, x ∈ R n , Q, A ij ∈ S m , A is a linear operator, the cone K is a product of semidefinite cones, second-order cones, nonnegative orthants and Euclidean spaces, S m is the space of m × m symmetric matrices, and A 0 indicates that A is positive semidefinite.
Nonsmooth optimization methods suited for eigenvalue optimization were used to compute locally optimal solutions for the BMI problem in [26]. A linearized convex-concave semidefinite programming algorithm for solving the following problem: was proposed in [1], where f (x) is convex, X is a nonempty and closed convex set, and A(x) and B(x) are positive semidefinite convex. Based on the fact that the bilinear form X T Y + Y T X can be decomposed as a difference between two positive semidefinite convex mappings, the method was applied to BMI optimization formulations of the static state/output feedback controller design problems. Following the same line of the work in [1], an iterative procedure to search a local optimum of more general nonconvex problem was developed in [27], based on a positive semidefinite convex overestimate of a positive semidefinite nonconvex matrix mapping. These local methods may not be able to obtain a global optimum. It is easy to see that the problem (1) is equivalent to rank one constrained optimization problem, by introducing additional variable y ij that represents the quadratic term x i x j . Rank-constrained optimization problems are considered in [28,29]. In [30], polynomial-time checkable sufficient conditions, which guarantees that the semidefinite relaxations of quadratically constrained quadratic programs are exact, are given.
To attain global optimal solution, optimization approaches have been proposed. Based on the generalized Benders decomposition, a global approach is proposed for problems with BMI constraints in [31]. A slight general formulation of the BMI feasibility problem known as the following BMI eigenvalue problem (BMIEP): was dealt in [32], where A(x, y) = ∑ ij x i x j A ij + ∑ i x i A i0 + ∑ j y j A j0 + A 00 . They proposed robust Branch-and-Cut algorithms, which improves the first implemented branch-and-bound algorithm [33] for the BMIEP. Other branch-and-bound algorithms can refer to [34,35]. Though the approaches based on the generalized Benders decomposition and branch-and-bound algorithms are global methods, it is in general impractical to solve large-scale problems. A solution method of reduction of variables is proposed for BMI problems in system and control designs in [36]. The proposed method consists of a principle of variable classification, a procedure for problem transformation and a hybrid algorithm. The proposed method can address feasibility, single-objective, and multiobjective problems with BMI constraints. However, it can fail in two circumstances. The aim of this paper is to present a lifting-penalty method for the solution of problem (1). We reformulate a quadratic matrix inequality constraint as linear matrix inequality constraints and a single quadratic equality constraint, but instead of a rank one constraint or a quadratic matrix equality constraint. Then, we use a majorization-minimization method to solve instead a l 1 penalty reformulation of the minimization problem. For the fixed penalty problem, global convergence to a Karush-Kuhn-Tucker (KKT) point of the majorization-minimization method is proven. The organization of this paper is as follows. In Section 2, a lifting-penalty method, which can obtain an -optimal solution of the problem (1), and its global convergence are given. In Section 3, numerical results are reported to show that the proposed method is efficient.
Throughout this paper, we use the notations in Table 1.  The trace (sum of diagonal elements) of the matrix A λ min (A) The minimum eigenvalue of the matrix A N Ω (x) The normal cone to the set Ω at the point x ∈ Ω

The Lifting-Penalty Method
This section concerns a lifting-penalty method for solving the problem (1). Its motivation is simple to describe: Rather than solving the original problem (1) directly, we reformulate it as a minimization problem having a linear objective function, linear conic constraints and a quadratic equality constraint by lifting x to a symmetric matrix Y, and then we use l 1 penalty method to solve the reformulation of (1). For a given value of the penalty parameter, the majorization-minimization (MM) method is used to solve the penalty problem. Global convergence of the MM method is proven under strict feasibility and boundedness of the original problem (1). An -optimal solution to problem (1) is obtained by solving the penalized problem as long as the penalty parameter is chosen appropriately.
Letting Y = xx T , problem (1) can be stated equivalently as the following minimization problem: By Schur complement Theorem and the equivalence between the equality Y = xx T and the following system we can state problem (3) equivalently as follows: We assume throughout this paper that the feasible set of (1) is bounded. Hence, assume without loss of generality that x T x ≤ κ/2. By using a penalty function similar to the l 1 penalty function defined in [37], we consider the following penalized problem: where µ > 0 is the penalty parameter. To simplify the notation, we denote (b, −κ) and K × R + byb andK, respectively, and define the linear operatorÃ: (x, Y) → (Ax, −tr(Y)). Hence, we can rewrite constraints tr(Y) ≤ κ and Ax − b ∈ K more compactly asÃ(x, Y) −b ∈K.
Let Ω denote the feasible set for problem (5). Unfortunately, strict feasibility of Ω (i.e., Slater's constraint qualification) fails. Therefore, we cannot establish similar results to the exactness (Theorem 4.4 in [37]) of the l 1 penalty function. However, when the penalty parameter µ is large enough, it can be proven that an -optimal solution to problem (1) can be obtained by solving the penalized problem as long as the penalty parameter is sufficiently large.
Let Ω c denote the feasible set for (6), (6) and (5), respectively, and (x * c , Y * c ) be an optimal solution of the following problem: min Theorem 1. Let > 0 be a given constant, and assume that It follows from (8) and the assumption on µ that Based on Theorem 1, we give the following Algorithm 1 for finding an -optimal solution for problem (1).
Step 4. If tr(Y j+1 ) − x (j+1)T x j+1 ≤ 1 , stop; else, set j = j + 1, and go to Step 2. In Step 1, to obtain a feasible point of the problem (5), we use the MM method to solve the following problem: min More detailed discussion on the MM method can be found in [38][39][40][41]. In Step 2, we use the MM method to solve the problem (6) with µ = µ j . Let That is,f µ (x, Y, z) is the majorization function of f µ (x, Y) at z. Therefore, we can apply the following Algorithm 2 to the problem (6) for µ = µ j .
In Algorithm 2, linear conic programming problem (10) can be solved by software packages SeDuMi [42], SDPT3 [43] or SDPNAL [44]. The MM method has global convergence properties, which states that the cluster points of {(x k , Y k )} are all KKT points of (6) for any given µ. To prove it, we need a constraint qualification. We first give the following Lemma.

Lemma 1.
If strict feasibility of the problem (1) holds, then strict feasibility of the problem (6) is also ensured.
Proof. If the problem (1) is strictly feasible, then there exists a pointx ∈ R n such that where ri K is the relative interior of the cone K.
TakingȲ =xx T , we know that (x,Ȳ) satisfies Denote the vector of all eigenvalues of Z ∈ S m arranged in non-increasing order by λ(Z). By the definition of G(x,Ȳ) and Mirsky's Theorem ( [45], Cor 4.12), we have that for any From Schur complement Theorem and (13), we know that (x,Ỹ) is a strictly feasible point for (6).

Lemma 2.
If the feasible set of the problem (1) is nonempty and bounded, then, for any given µ > 0, Ω c and the solution sets of the problems (6) and (10) are nonempty and compact.
Proof. Because the feasible set of the problem (1) is a nonempty subset of Ω c , Ω c is nonempty. It is trivial to show that Ω c is closed. We will prove Ω c is bounded. For any (x, Y) ∈ Ω c , Y 0 together with tr(Y) ≤ κ implies that Y is bounded, and x T x ≤ tr(Y) ≤ κ implies that x is bounded. As a result, Ω c is nonempty and compact.
From the continuity of f µ (·) andf µ (·), we know that the solution sets of the problems (6) and (10) are also nonempty and compact.
By introducing the product topology on R n × S m with the induced inner product for any fixed z and any direction (d, D) ∈ R n × S m , we calculate the derivative off µ (x, Y, z) with respect to (x, Y) as follows:f The gradient ∇f µ (x, Y, z) off µ (x, Y, z) with respect to (x, Y), therefore, may be interpreted as the pair of matrices: Theorem 2. Suppose that Slater's constraint qualification for problem (1) holds, and the feasible set of (1) is bounded. For any given µ > 0, let {(x k , Y k )} be the sequence generated by Algorithm 2, then lim k→∞ f µ (x k , Y k ) = inf k f µ (x k , Y k ), and any cluster point of {(x k , Y k )} is a KKT point of (6).
Proof. From the boundedness of the feasible set of problem (1) and Lemma 2, we know that problem (10) has a nonempty and compact solution set and there exists a point (x k+1 , Y k+1 ) ∈ arg min (x,Y)∈Ω c {f µ (x, Y, x k )}.
By noting that f µ (·) is continuous and Because (x k+1 , Y k+1 ) is an optimal solution of (10), we have that We first consider the case ( Denote . From Lemma 1, we know that Slater's constraint qualification for (6) holds. This, together with convexity, implies that where (G k+1 i ) * andÃ * are the adjoint operators of (G k+1 i ) andÃ, respectively. It follows from (17) and (18) that (x k+1 , Y k+1 ) is a KKT point of problem (6).
Next, we assume that (x k+1 , Y k+1 ) = (x k , Y k ) for all k ≥ 0. Because {(x k , Y k )} is bounded, we know that {(x k , Y k )} has cluster points and any cluster point (x,Ȳ) ∈ Ω c . By the equality . Now, we prove that (x,Ȳ) is a KKT point of (6). For any k ≥ 0, we have that It follows from (16) From (19) and (20), we have that which implies that Hence, lim k→∞ (x k+1 − x k ) = 0. Let k j satisfy that x k j →x and Y k j →Ȳ, then lim j→∞ x k j = lim j→∞ x k j −1 =x. (22) Because (x k j , Y k j ) is an optimal solution, there exists D k j ∈ N Ω c (x k j , Y k j ) such that for any k j ≥ 1. Hence, for any (x,Ỹ) ∈ Ω c , which, from the convergence of the subsequence {(x k j , Y k j )} and (22), gives rise to together with Slater's constraint qualification, implies that (x,Ȳ) is a KKT point of problem (6).

Numerical Experiments
In this section, the examples and some preliminary numerical results taken by the lifting-penalty method (LPM) and a modified augmented Lagrangian method (MALM) [46] are given below.
All The problem amounts to solving a nonconvex feasibility problem S(K) 0 (see [3,47,48]). We solved it by solving the following non-strict optimization problem: From Figure 2 in [3], we know that the problem is nonconvex. For the starting point K (0) = (0, 0) T , our method ended with the final solution K = (−0.4293, 0.5514) T .
where A ij ∈ S m is generated by the Matlab function sprandn(m,m,r). We symmetrized the matrices by copying the upper triangular part to the lower one after creation.
In our experiment, 20 problem instances are randomly generated for each value of (n, m, d), and the additional variable Y is a d × d symmetric matrix, r = 0.2. Table 2 lists n, m, d and the average CPU time in seconds, respectively. From the results listed in Table 2, we find that our method requires less time than the LPM for most instances. Moreover, consuming time seems to be more sensitive to d, compared with n and m. Preliminary computational experiences show that our method is competitive with the LPM.

Conclusions
In this paper, a lifting-penalty method for solving a quadratic optimization problem involving a quadratic matrix inequality constraint is introduced. By introducing additional variables, we reformulate a quadratic matrix inequality constraint as linear matrix inequality constraints and a single quadratic equality constraint but instead of a rank one constraint or a quadratic matrix equality constraint. Its global convergence result has been given under mild assumptions. Then, the method was applied to a feasibility problem and a problem of maximizing the smallest eigenvalue of a symmetric matrix. The numerical results show that the proposed method is competitive with the LPM. Note, however, that linear conic optimization subproblems arising in our method have the same feasible set, so the development of an efficient method for solving a family of linear conic optimization problems with the same feasible set is our future work.