dCATCH—A Numerical Package for d-Variate near G-Optimal Tchakaloff Regression via Fast NNLS

Abstract

We provide a numerical package for the computation of a d-variate near G-optimal polynomial regression design of degree m on a finite design space $X\subset {\mathbb{R}}^d$, by few iterations of a basic multiplicative algorithm followed by Tchakaloff-like compression of the discrete measure keeping the reached G-efficiency, via an accelerated version of the Lawson-Hanson algorithm for Non-Negative Least Squares (NNLS) problems. This package can solve on a personal computer large-scale problems where $card\left(X\right)\times dim\left(P_{2m}^d\right)$ is up to ${10}^8$–${10}^9$, being $dim\left(P_{2m}^d\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{d}\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{2m}\right)$. Several numerical tests are presented on complex shapes in $d=3$ and on hypercubes in $d>3$.

1. Introduction

In this paper we present the numerical software package dCATCH [1] for the computation of a d-variate near G-optimal polynomial regression design of degree m on a finite design space $X\subset {\mathbb{R}}^d$. In particular, it is the first software package for general-purpose Tchakaloff-like compression of d-variate designs via Non-Negative Least Squares (NNLS), freely available on the Internet. The code is an evolution of the codes in Reference [2] (limited to $d=2,3$), with a number of features tailored to higher dimension and large-scale computations. The key ingredients are:

• use of d-variate Vandermonde-like matrices at X in a discrete orthogonal polynomial basis (obtained by discrete orthonormalization of the total-degree product Chebyshev basis of the minimal box containing X), with automatic adaptation to the actual dimension of ${\mathbb{P}}_m^d\left(X\right)$;
• few tens of iterations of the basic Titterington multiplicative algorithm until near G-optimality of the design is reached, with a checked G-efficiency of say $95\%$ (but with a design support still far from sparsity);
• Tchakaloff-like compression of the resulting near G-optimal design via NNLS solution of the underdetermined moment system, with concentration of the discrete probability measure by sparse re-weighting to a support $\subset X$, of cardinality at most ${\mathbb{P}}_{2m}^d\left(X\right)$, keeping the same G-efficiency;
• iterative solution of the large-scale NNLS problem by a new accelerated version of the classical Lawson-Hanson active set algorithm, that we recently introduced in Reference [3] for $2d$ and $3d$ instances and here we validate on higher dimensions.

Before giving a more detailed description of the algorithm, it is worth recalling in brief some basic notions of optimal design theory. Such a theory has its roots and main applications within statistics, but also strong connections with approximation theory. In statistics, a design is a probability measure $\mu$ supported on a (discrete or continuous) compact set $\mathsf{\Omega }\subset {\mathbb{R}}^d$. The search for designs that optimize some properties of statistical estimators (optimal designs) dates back to at least one century ago, and the relevant literature is so wide and still actively growing and monographs and survey papers are abundant in the literature. For readers interested in the evolution and state of the art of this research field, we may quote, for example, two classical treatises such as in References [4,5], the recent monograph [6] and the algorithmic survey [7], as well as References [8,9,10] and references therein. On the approximation theory side we may quote, for example, References [11,12].

The present paper is organized as follows—in Section 2 we briefly recall some basic concepts from the theory of Optimal Designs, for the reader’s convenience, with special attention to the deterministic and approximation theoretic aspects. In Section 3 we present in detail our computational approach to near G-optimal d-variate designs via Caratheodory-Tchakaloff compression. All the routines of the dCATCH software package here presented, are described. In Section 4 we show several numerical results with dimensions in the range 3–10 and a Conclusions section follows.

For the reader’s convenience we also display

Table 1. List of acronyms.

LS	Least Squares
NNLS	Non-Negative Least Squares
LH	Lawson-Hawson algorithm for NNLS
LHI	Lawson-Hawson algorithm with unconstrained LS Initialization
LHDM	Lawson-Hawson algorithm with Deviation Maximization acceleration

and

Table 2. dCATCH package content.

dCATCH	d-variate CAratheodory-TCHakaloff discrete measure compression
dCHEBVAND	d-variate Chebyshev-Vandermonde matrix
dORTHVAND	d-variate Vandermonde-like matrix in a weighted orthogonal polynomial basis
dNORD	d-variate Near G-Optimal Regression Designs
LHDM	Lawson-Hawson algorithm with Deviation Maximization acceleration

, describing the acronyms used in this paper and the content (subroutine names) of the dCATCH software package.

2. G-Optimal Designs

Let ${\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$ denote the space of d-variate real polynomials of total degree not greater than n, restricted to a (discrete or continuous) compact set $\mathsf{\Omega }\subset {\mathbb{R}}^d$, and let $\mu$ be a design, that is, a probability measure, with $supp\left(\mu \right)\subseteq \mathsf{\Omega }$. In what follows we assume that $supp\left(\mu \right)$ is determining for ${\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$ [13], that is, polynomials in ${\mathbb{P}}_m^d$ vanishing on $supp\left(\mu \right)$ vanish everywhere on $\mathsf{\Omega }$.

In the theory of optimal designs, a key role is played by the diagonal of the reproducing kernel for $\mu$ in ${\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$ (also called the Christoffel polynomial of degree m for $\mu$)

$$K_m^{\mu }(x,x)=\sum _{j=1}^{N_m}{p}_j^2\left(x\right),N_m=dim\left({\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)\right),$$

(1)

where $\left\{p_j\right\}$ is any $\mu$-orthonormal basis of ${\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$. Recall that $K_m^{\mu }(x,x)$ can be proved to be independent of the choice of the orthonormal basis. Indeed, a relevant property is the following estimate of the $L^{\infty }$-norm in terms of the $L_{\mu }^2$-norm of polynomials

$${\parallel p\parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)}\le \sqrt{\underset{x\in \mathsf{\Omega }}{max}{K}_m^{\mu }(x,x)}{\parallel p\parallel }_{L_{\mu }^2\left(\mathsf{\Omega }\right)},\forall p\in {\mathbb{P}}_m^d\left(\mathsf{\Omega }\right).$$

(2)

Now, by (1) and $\mu$-orthonormality of the basis we get

$${\int }_{\mathsf{\Omega }}{K}_m^{\mu }(x,x)d\mu =\sum _{j=1}^{N_m}{\int }_{\mathsf{\Omega }}{p}_j^2\left(x\right)d\mu =N_m,$$

(3)

which entails that ${max}_{x\in \mathsf{\Omega }}{K}_m^{\mu }(x,x)\ge N_m$.

Then, a probability measure ${\mu }_{*}={\mu }_{*}\left(\mathsf{\Omega }\right)$ is then called a G-optimal design for polynomial regression of degree m on $\mathsf{\Omega }$ if

$$\underset{\mu }{min}\underset{x\in \mathsf{\Omega }}{max}{K}_m^{\mu }(x,x)=\underset{x\in \mathsf{\Omega }}{max}{K}_m^{{\mu }_{*}}(x,x)=N_m.$$

(4)

Observe that, since ${\int }_{\mathsf{\Omega }}{K}_m^{\mu }(x,x)d\mu =N_m$ for every $\mu$, an optimal design has also the following property $K_m^{{\mu }_{*}}(x,x)=N_m$, ${\mu }_{*}$-a.e. in $\mathsf{\Omega }$.

Now, the well-known Kiefer-Wolfowitz General Equivalence Theorem [14] (a cornerstone of optimal design theory), asserts that the difficult min-max problem (4) is equivalent to the much simpler maximization problem

$$\underset{\mu }{max}det\left(G_m^{\mu }\right),G_m^{\mu }={\left({\int }_{\mathsf{\Omega }}{\varphi }_i\left(x\right){\varphi }_j\left(x\right)d\mu \right)}_{1\le i,j\le N_m},$$

where $G_m^{\mu }$ is the Gram matrix (or information matrix in statistics) of $\mu$ in a fixed polynomial basis $\left\{{\varphi }_i\right\}$ of ${\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$. Such an optimality is called D-optimality, and ensures that an optimal measure always exists, since the set of Gram matrices of probability measures is compact and convex; see for example, References [5,12] for a general proof of these results, valid for continuous as well as for discrete compact sets.

Notice that an optimal measure is neither unique nor necessarily discrete (unless $\mathsf{\Omega }$ is discrete itself). Nevertheless, the celebrated Tchakaloff Theorem ensures the existence of a positive quadrature formula for integration in $d{\mu }_{*}$ on $\mathsf{\Omega }$, with cardinality not exceeding $N_{2m}=dim\left({\mathbb{P}}_{2m}^d\left(\mathsf{\Omega }\right)\right)$ and which is exact for all polynomials in ${\mathbb{P}}_{2m}^d\left(\mathsf{\Omega }\right)$. Such a formula is then a design itself, and it generates the same orthogonal polynomials and hence the same Christoffel polynomial of ${\mu }_{*}$, preserving G-optimality (see Reference [15] for a proof of Tchakaloff Theorem with general measures).

We recall that G-optimality has two important interpretations in terms of statistical and deterministic polynomial regression.

From a statistical viewpoint, it is the probability measure on $\mathsf{\Omega }$ that minimizes the maximum prediction variance by polynomial regression of degree m, cf. for example, Reference [5].

On the other hand, from an approximation theory viewpoint, if we call ${\mathcal{L}}_m^{{\mu }_{*}}$ the corresponding weighted least squares projection operator $L^{\infty }\left(\mathsf{\Omega }\right)\to {\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)$, namely

$$\parallel f-{\mathcal{L}}_m^{{\mu }_{*}}{f\parallel }_{L_{{\mu }_{*}}^2\left(\mathsf{\Omega }\right)}=\underset{p\in {\mathbb{P}}_m^d\left(\mathsf{\Omega }\right)}{min}{\parallel f-p\parallel }_{L_{{\mu }_{*}}^2\left(\mathsf{\Omega }\right)},$$

(5)

by (2) we can write for every $f\in L^{\infty }\left(\mathsf{\Omega }\right)$

$$\begin{array}{cc}\hfill \parallel {\mathcal{L}}_m^{{\mu }_{*}}{f\parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)}\le & \sqrt{{\displaystyle \underset{x\in \mathsf{\Omega }}{max}}{K}_m^{{\mu }_{*}}(x,x)}\parallel {\mathcal{L}}_m^{{\mu }_{*}}{f\parallel }_{L_{{\mu }_{*}}^2\left(\mathsf{\Omega }\right)}=\sqrt{N_m}{\parallel {\mathcal{L}}_m^{{\mu }_{*}}f\parallel }_{L_{{\mu }_{*}}^2\left(\mathsf{\Omega }\right)}\hfill \\ & \le \sqrt{N_m}{\parallel f\parallel }_{L_{{\mu }_{*}}^2\left(\mathsf{\Omega }\right)}\le \sqrt{N_m}{\parallel f\parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)},\hfill \end{array}$$

(where the second inequality comes from ${\mu }_{*}$-orthogonality of the projection), which gives

$$\parallel {\mathcal{L}}_m^{{\mu }_{*}}\parallel =\underset{f\ne 0}{sup}\frac{\parallel {\mathcal{L}}_m^{{\mu }_{*}}{f\parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)}}{{\parallel f\parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)}}\le \sqrt{N_m},$$

(6)

that is a G-optimal measure minimizes (the estimate of) the weighted least squares uniform operator norm.

We stress that in this paper we are interested in the fully discrete case of a finite design space $\mathsf{\Omega }=X$, so that any design $\mu$ is identified by a set of positive weights (masses) summing up to 1 and integrals are weighted sums.

3. Computing near G-Optimal Compressed Designs

Since in the present context we have a finite design space $\mathsf{\Omega }=X=\{x_1,\cdots ,x_M\}\subset {\mathbb{R}}^d$, we may think a design $\mu$ as a vector of non-negative weights $u=(u_1,\cdots ,u_M)$ attached to the points, such that ${\parallel u\parallel }_1=1$ (the support of $\mu$ being identified by the positive weights). Then, a G-optimal (or D-optimal) design ${\mu }_{*}$ is represented by the corresponding non-negative vector $u_{*}$. We write $K_m^u(x,x)=K_m^{\mu }(x,x)$ for the Christoffel polynomial and similarly for other objects (spaces, operators, matrices) corresponding to a discrete design. At the same time, $L^{\infty }\left(\mathsf{\Omega }\right)={\ell }^{\infty }\left(X\right)$, and $L_{\mu }^2\left(\mathsf{\Omega }\right)={\ell }_u^2\left(X\right)$ (a weighted ${\ell }^2$ functional space on X) with ${\parallel f\parallel }_{{\ell }_u^2\left(X\right)}={\left({\sum }_{i=1}^M{u}_i{f}^2\left(x_i\right)\right)}^{1/2}$.

In order to compute an approximation of the desired $u_{*}$, we resort to the basic multiplicative algorithm proposed by Titterington in the ’70s (cf. Reference [16]), namely

$$u_i(k+1)=u_i\left(k\right)\frac{K_m^{u\left(k\right)}(x_i,x_i)}{N_m},1\le i\le M,k=0,1,2,\cdots ,$$

(7)

with initialization $u\left(0\right)={(1/M,\cdots ,1/M)}^T$. Such an algorithm is known to be convergent sublinearly to a D-optimal (or G-optimal by the Kiefer-Wolfowitz Equivalence Theorem) design, with an increasing sequence of Gram determinants

$$det\left(G_m^{u\left(k\right)}\right)=det\left(V^{T}diag\left(u\left(k\right)\right)V\right),$$

where V is a Vandermonde-like matrix in any fixed polynomial basis of ${\mathbb{P}}_m^d\left(X\right)$; cf., for example, References [7,10]. Observe that $u(k+1)$ is indeed a vector of positive probability weights if such is $u\left(k\right)$. In fact, the Christoffel polynomial $K_m^{u\left(k\right)}$ is positive on X, and calling ${\mu }_k$ the probability measure on X associated with the weights $u\left(k\right)$ we get immediately ${\sum }_i{u}_i(k+1)=\frac{1}{N_m}{\sum }_i{u}_i\left(k\right)K_m^{u\left(k\right)}(x_i,x_i)=\frac{1}{N_m}{\int }_X{K}_m^{u\left(k\right)}(x,x)d{\mu }_k=1$ by (3) in the discrete case $\mathsf{\Omega }=X$.

Our implementation of (7) is based on the functions

• $C=\mathrm{dCHEBVAND}(n,X)$
• $[U,jvec]=\mathrm{dORTHVAND}(n,X,u,jvec)$
• $[pts,w]=\mathrm{dNORD}(m,X,gtol)$

The function dCHEBVAND computes the d-variate Chebyshev-Vandermonde matrix $C=\left({\varphi }_j\left(x_i\right)\right)\in {\mathbb{R}}^{M\times N_n}$, where $\left\{{\varphi }_j\left(x\right)\right\}=\{T_{{\nu }_1}({\alpha }_1{x}_1+{\beta }_1)\cdots T_{{\nu }_d}({\alpha }_d{x}_d+{\beta }_d)\}$, $0\le {\nu }_i\le n$, ${\nu }_1+\cdots +{\nu }_d\le n$, is a suitably ordered total-degree product Chebyshev basis of the minimal box $[a_1,b_1]\times \cdots \times [a_d,b_d]$ containing X, with ${\alpha }_i=2/(b_i-a_i)$, ${\beta }_i=-(b_i+a_i)/(b_i-a_i)$. Here we have resorted to the codes in Reference [17] for the construction and enumeration of the required “monomial” degrees. Though the initial basis is then orthogonalized, the choice of the Chebyshev basis is dictated by the necessity of controlling the conditioning of the matrix. This would be on the contrary extremely large with the standard monomial basis, already at moderate regression degrees, preventing a successful orthogonalization.

Indeed, the second function dORTHVAND computes a Vandermonde-like matrix in a u-orthogonal polynomial basis on X, where u is the probability weight array. This is accomplished essentially by numerical rank evaluation for $C=\mathrm{dCHEBVAND}(n,X)$ and QR factorization

$$diag\left(\sqrt{u}\right)C_0=QR,U=C_0{R}^{-1},$$

(8)

(with Q orthogonal rectangular and R square invertible), where $\sqrt{u}=(\sqrt{u_1},\cdots ,\sqrt{u_M})$. The matrix $C_0$ has full rank and corresponds to a selection of the columns of C (i.e., of the original basis polynomials) via QR with column pivoting, in such a way that these form a basis of ${\mathbb{P}}_n^d\left(X\right)$, since $rank\left(C\right)=dim\left({\mathbb{P}}_n^d\left(X\right)\right)$. A possible alternative, not yet implemented, is the direct use of a rank-revealing QR factorization. The in-out parameter “jvec” allows to pass directly the column index vector corresponding to a polynomial basis after a previous call to dORTHVAND with the same degree n, avoiding numerical rank computation and allowing a simple “economy size” QR factorization of $diag\left(\sqrt{u}\right)C_0=diag\left(\sqrt{u}\right)C(:,jvec)$.

Summarizing, U is a Vandermonde-like matrix for degree n on X in the required u-orthogonal basis of ${\mathbb{P}}_n^d\left(X\right)$, that is

$$[p_1\left(x\right),\cdots ,p_{N_n}\left(x\right)]=[{\varphi }_{j_1}\left(x\right),\cdots ,{\varphi }_{j_{N_n}}\left(x\right)]R^{-1},$$

(9)

where $jvec=(j_1,\cdots ,j_{N_n})$ is the multi-index resulting from pivoting. Indeed by (8) we can write the scalar product ${(p_h,p_k)}_{{\ell }_u^2\left(X\right)}$ as

$${(p_h,p_k)}_{{\ell }_u^2\left(X\right)}=\sum _{i=1}^M{u}_i{p}_h\left(x_i\right)p_k\left(x_i\right)={\left(U^{T}diag\left(u\right)U\right)}_{hk}={\left(Q^{T}Q\right)}_{hk}={\delta }_{hk},$$

for $1\le h,k\le N_n$, which shows orthonormality of the polynomial basis in (9).

We stress that $rank\left(C\right)=dim\left({\mathbb{P}}_n^d\left(X\right)\right)$ could be strictly smaller than $dim\left({\mathbb{P}}_n^d\right)=\left(\genfrac{}{}{0pt}{}{n+d}{d}\right)$, when there are polynomials in ${\mathbb{P}}_n^d$ vanishing on X that do not vanish everywhere. In other words, X lies on a lower-dimensional algebraic variety (technically one says that X is not ${\mathbb{P}}_n^d$-determining [13]). This certainly happens when $card\left(X\right)$ is too small, namely $card\left(X\right)<dim\left({\mathbb{P}}_n^d\right)$, but think for example also to the case when $d=3$ and X lies on the 2-sphere $S^2$ (independently of its cardinality), then we have $dim\left({\mathbb{P}}_n^d\left(X\right)\right)\le dim\left({\mathbb{P}}_n^d\left(S^2\right)\right)={(n+1)}^2<dim\left({\mathbb{P}}_n^3\right)=(n+1)(n+2)(n+3)/6$.

Iteration (7) is implemented within the third function dNORD whose name stands for d-dimensional Near G-Optimal Regression Designs, which calls dORTHVAND with $n=m$. Near optimality is here twofold, namely it concerns both the concept of G-efficiency of the design and the sparsity of the design support.

We recall that G-efficiency is the percentage of G-optimality reached by a (discrete) design, measured by the ratio

$$G_m\left(u\right)=\frac{N_m}{ma{x}_{x\in X}{K}_m^u(x,x)},$$

knowing that $G_m\left(u\right)\le 1$ by (3) in the discrete case $\mathsf{\Omega }=X$. Notice that $G_m\left(u\right)$ can be easily computed after the construction of the u-orthogonal Vandermonde-like matrix U by dORTHVAND, as $G_m\left(u\right)=N_m/\left({max}_i\parallel ro{w}_i{\left(U\right)\parallel }_2^2\right).$

In the multiplicative algorithm (7), we then stop iterating when a given threshold of G-efficiency (the input parameter “gtol” in the call to dNORD) is reached by $u\left(k\right)$, since $G_m\left(u\left(k\right)\right)\to 1$ as $k\to \infty$, say for example $G_m\left(u\left(k\right)\right)\ge 95\%$ or $G_m\left(u\left(k\right)\right)\ge 99\%$. Since convergence is sublinear and in practice we see that $1-G_m\left(u\left(k\right)\right)=\mathcal{O}(1/k)$, for a $90\%$ G-efficiency the number of iterations is typically in the tens, whereas it is in the hundreds for $99\%$ one and in the thousands for $99.9\%$. When a G-efficiency very close to 1 is needed, one could resort to more sophisticated multiplicative algorithms, see for example, References [9,10].

In many applications however a G-efficiency of 90–$95\%$ could be sufficient (then we may speak of near G-optimality of the design), but though in principle the multiplicative algorithm converges to an optimal design ${\mu }_{*}$ on X with weights $u_{*}$ and cardinality $N_m\le card\left(supp\left({\mu }_{*}\right)\right)\le N_{2m}$, such a sparsity is far from being reached after the iterations that guarantee near G-optimality, in the sense that there is a still large percentage of non-negligible weights in the near optimal design weight vector, say

$$u\left(\overline{k}\right)\mathrm{such} \mathrm{that}{G}_m\left(u\left(\overline{k}\right)\right)\ge gtol.$$

(10)

Following References [18,19], we can however effectively compute a design which has the same G-efficiency of $u\left(\overline{k}\right)$ but a support with a cardinality not exceeding $N_{2m}=dim\left({\mathbb{P}}_{2m}^d\left(X\right)\right)$, where in many applications $N_{2m}\ll card\left(X\right)$, obtaining a remarkable compression of the near optimal design.

The theoretical foundation is a generalized version [15] of Tchakaloff Theorem [20] on positive quadratures, which asserts that for every measure on a compact set $\mathsf{\Omega }\subset {\mathbb{R}}^d$ there exists an algebraic quadrature formula exact on ${\mathbb{P}}_n^d\left(\mathsf{\Omega }\right))$, with positive weights, nodes in $\mathsf{\Omega }$ and cardinality not exceeding $N_n=dim({\mathbb{P}}_n^d\left(\mathsf{\Omega }\right)$.

In the present discrete case, that is, where the designs are defined on $\mathsf{\Omega }=X$, this theorem implies that for every design $\mu$ on X there exists a design $\nu$, whose support is a subset of X, which is exact for integration in $d\mu$ on ${\mathbb{P}}_n^d\left(X\right)$. In other words, the design $\nu$ has the same basis moments (indeed, for any basis of ${\mathbb{P}}_n^d\left(\mathsf{\Omega }\right)$)

$${\int }_X{p}_j\left(x\right)d\mu =\sum _{i=1}^M{u}_i{p}_j\left(x_i\right)={\int }_X{p}_j\left(x\right)d\nu =\sum _{\ell =1}^L{w}_{\ell }{p}_j\left({\xi }_{\ell }\right),1\le j\le N_n,$$

where $L\le N_n\le M$, $\left\{u_i\right\}$ are the weights of $\mu$, $supp\left(\nu \right)=\left\{{\xi }_{\ell }\right\}\subseteq X$ and $\left\{w_{\ell }\right\}$ are the positive weights of $\nu$. For $L<M$, which certainly holds if $N_n<M$, this represents a compression of the design $\mu$ into the design $\nu$, which is particularly useful when $N_n\ll M$.

In matrix terms this can be seen as the fact that the underdetermined $\left\{p_j\right\}$-moment system

$$U_n^{T}v=U_n^{T}u$$

(11)

has a non-negative solution $v={(v_1,\cdots ,v_M)}^T$ whose positive components, say $w_{\ell }=v_{i_{\ell }}$, $1\le \ell \le L\le N_n$, determine the support points $\left\{{\xi }_{\ell }\right\}\subseteq X$ (for clarity we indicate here by $U_n$ the matrix U computed by dORTHVAND at degree n). This fact is indeed a consequence of the celebrated Caratheodory Theorem on conic combinations [21], asserting that a linear combination with non-negative coefficients of M vectors in ${\mathbb{R}}^N$ with $M>N$ can be re-written as linear positive combination of at most N of them. So, we get the discrete version of Tchakaloff Theorem by applying Caratheodory Theorem to the columns of $U_n^T$ in the system (11), ensuring then existence of a non-negative solution v with at most $N_n$ nonzero components.

In order to compute such a solution to (11) we choose the strategy based on Quadratic Programming introduced in Reference [22], namely on sparse solution of the Non-Negative Least Squares (NNLS) problem

$$v={\mathrm{argmin}}_{z\in {\mathbb{R}}^M,z\ge 0}\parallel U_n^{T}z-U_n^T{u\parallel }_2^2$$

by a new accelerated version of the classical Lawson-Hanson active-set method, proposed in Reference [3] in the framework of design optimization in $d=2,3$ and implemented by the function LHDM (Lawson-Hanson with Deviation Maximization), that we tune in the present package for very large-scale d-variate problems (see the next subsection for a brief description and discussion). We observe that working with an orthogonal polynomial basis of ${\mathbb{P}}_n^d\left(X\right)$ allows to deal with the well-conditioned matrix $U_n$ in the Lawson-Hanson algorithm.

The overall computational procedure is implemented by the function

• $[pts,w,momerr]=\mathrm{dCATCH}(n,X,u)$,

where dCATCH stands for d-variate CAratheodory-TCHakaloff discrete measure compression. It works for any discrete measure on a discrete set X. Indeed, it could be used, other than for design compression, also in the compression of d-variate quadrature formulas, to give an example. The output parameter $pts=\left\{{\xi }_{\ell }\right\}\subset X$ is the array of support points of the compressed measure, while $w=\left\{w_{\ell }\right\}=\{v_{i_{\ell }}>0\}$ is the corresponding positive weight array (that we may call a d-variate near G-optimal Tchakaloff design) and $momerr=\parallel U_n^{T}v-U_n^T{u\parallel }_2$ is the moment residual. This function is called LHDM.

In the present framework we call dCATCH with $n=2m$ and $u=u\left(\overline{k}\right)$, cf. (10), that is, we solve

$$v={\mathrm{argmin}}_{z\in {\mathbb{R}}^M,z\ge 0}\parallel U_{2m}^{T}z-U_{2m}^{T}u\left(\overline{k}\right){\parallel }_2^2.$$

(12)

In such a way the compressed design generates the same scalar product of $u\left(\overline{k}\right)$ in ${\mathbb{P}}_m^d\left(X\right)$, and hence the same orthogonal polynomials and the same Christoffel function on X keeping thus invariant the G-efficiency

$${\mathbb{P}}_{2m}^d\left(X\right)\ni K_m^v(x,x)=K_m^{u\left(\overline{k}\right)}(x,x)\forall x\in X⟹G_m\left(v\right)=G_m\left(u\left(\overline{k}\right)\right)\ge gtol$$

(13)

with a (much) smaller support.

From a deterministic regression viewpoint (approximation theory), let us denote by $p_m^{opt}$ the polynomial in ${\mathbb{P}}_m^d\left(X\right)$ of best uniform approximation for f on X, where we assume $f\in C\left(D\right)$ with $X\subset D\subset R^d$, D being a compact domain (or even lower-dimensional manifold), and by $E_m(f;X)={inf}_{p\in {\mathbb{P}}_m^d\left(X\right)}{\parallel f-p\parallel }_{{\ell }^{\infty }\left(X\right)}={\parallel f-p_m^{opt}|}_{{\ell }^{\infty }\left(X\right)}$ and $E_m(f;D)={inf}_{p\in {\mathbb{P}}_m^d\left(D\right)}{\parallel f-p\parallel }_{L^{\infty }\left(D\right)}$ the best uniform polynomial approximation errors on X and D.

Then, denoting by ${\mathcal{L}}_m^{u\left(\overline{k}\right)}$ and ${\mathcal{L}}_m^{w}f={\mathcal{L}}_m^{v}f$ the weighted least squares polynomial approximation of f (cf. (5)) by the near G-optimal weights $u\left(\overline{k}\right)$ and w, respectively, with the same reasoning used to obtain (6) and by (13) we can write the operator norm estimates

$$\parallel {\mathcal{L}}_m^{u\left(\overline{k}\right)}\parallel ,\parallel {\mathcal{L}}_m^w\parallel \le \sqrt{{\tilde{N}}_m}\le \sqrt{\frac{N_m}{gtol}},{\tilde{N}}_m=\frac{N_m}{G_m\left(u\left(\overline{k}\right)\right)}=\frac{N_m}{G_m\left(v\right)}.$$

Moreover, since ${\mathcal{L}}_m^{w}p=p$ for any $p\in {\mathbb{P}}_m^d\left(X\right)$, we can write the near optimal estimate

$$\parallel f-{\mathcal{L}}_m^w{f\parallel }_{{\ell }^{\infty }\left(X\right)}\le \parallel f-p_m^{opt}{\parallel }_{{\ell }^{\infty }\left(X\right)}+\parallel p_m^{opt}-{\mathcal{L}}_m^w{p}_m^{opt}{\parallel }_{{\ell }^{\infty }\left(X\right)}+{\parallel {\mathcal{L}}_m^w{p}_m^{opt}-{\mathcal{L}}_m^{w}f\parallel }_{{\ell }^{\infty }\left(X\right)}$$

$$=\parallel f-p_m^{opt}{\parallel }_{{\ell }^{\infty }\left(X\right)}+\parallel {\mathcal{L}}_m^w{p}_m^{opt}-{\mathcal{L}}_m^w{f\parallel }_{{\ell }^{\infty }\left(X\right)}\le (1+\parallel {\mathcal{L}}_m^w\parallel )E_m(f;X)$$

$$\le \left(1+\sqrt{\frac{N_m}{gtol}}\right)E_m(f;X)\le \left(1+\sqrt{\frac{N_m}{gtol}}\right)E_m(f;D)\approx \left(1+\sqrt{N_m}\right)E_m(f;D).$$

Notice that ${\mathcal{L}}_m^{w}f$ is constructed by sampling f only at the compressed support $\left\{{\xi }_{\ell }\right\}\subset X$. The error depends on the regularity of f on $D\supset X$, with a rate that can be estimated whenever D admits a multivariate Jackson-like inequality, cf. Reference [23].

Accelerating the Lawson-Hanson Algorithm by Deviation Maximization (LHDM)

Let $A\in {\mathbb{R}}^{N\times M}$ and $b\in {\mathbb{R}}^N$. The NNLS problem consists of seeking $x\in {\mathbb{R}}^M$ that solves

$$x={\mathrm{argmin}}_{z\ge 0}{\parallel Az-b\parallel }_2^2.$$

(14)

This is a convex optimization problem with linear inequality constraints that define the feasible region, that is the positive orthant $\left\{x\in {\mathbb{R}}^M:x_i\ge 0\right\}$. The very first algorithm dedicated to problem (14) is due to Lawson and Hanson [24] and it is still one of the most often used. It was originally derived for solving overdetermined linear systems, with $N\gg M$. However, in the case of underdetermined linear systems, with $N\ll M$, this method succeeds in sparse recovery.

Recall that for a given point x in the feasible region, the index set $\left\{1,\cdots ,M\right\}$ can be partitioned into two sets: the active set Z, containing the indices of active constraints $x_i=0$, and the passive set P, containing the remaining indices of inactive constraints $x_i>0$. Observe that an optimal solution $x^{🟉}$ of (14) satisfies $A{x}^{🟉}=b$ and, if we denote by $P^{🟉}$ and $Z^{🟉}$ the corresponding passive and active sets respectively, $x^{🟉}$ also solves in a least square sense the following unconstrained least squares subproblem

$$x_{P^{🟉}}^{🟉}={\mathrm{argmin}}_y{\parallel A_{P^{🟉}}y-b\parallel }_2^2,$$

(15)

where $A_{P^{🟉}}$ is the submatrix containing the columns of A with index in $P^{🟉}$, and similarly $x_{P^{🟉}}^{🟉}$ is the subvector made of the entries of $x^{🟉}$ whose index is in $P^{🟉}$. The remaining entries of $x^{🟉}$, namely those whose index is in $Z^{🟉}$, are null.

The Lawson-Hanson algorithm, starting from a null initial guess $x=0$ (which is feasible), incrementally builds an optimal solution by moving indices from the active set Z to the passive set P and vice versa, while keeping the iterates within the feasible region. More precisely, at each iteration first order information is used to detect a column of the matrix A such that the corresponding entry in the new solution vector will be strictly positive; the index of such a column is moved from the active set Z to the passive set P. Since there’s no guarantee that the other entries corresponding to indices in the former passive set will stay positive, an inner loop ensures the new solution vector falls into the feasible region, by moving from the passive set P to the active set Z all those indices corresponding to violated constraints. At each iteration a new iterate is computed by solving a least squares problem of type (15): this can be done, for example, by computing a QR decomposition, which is substantially expensive. The algorithm terminates in a finite number of steps, since the possible combinations of passive/active set are finite and the sequence of objective function values is strictly decreasing, cf. Reference [24].

The deviation maximization (DM) technique is based on the idea of adding a whole set of indices T to the passive set at each outer iteration of the Lawson-Hanson algorithm. This corresponds to select a block of new columns to insert in the matrix $A_P$, while keeping the current solution vector within the feasible region in such a way that sparse recovery is possible when dealing with non-strictly convex problems. In this way, the number of total iterations and the resulting computational cost decrease. The set T is initialized to the index chosen by the standard Lawson-Hanson (LH) algorithm, and it is then extended, within the same iteration, using a set of candidate indices C chosen is such a way that the corresponding entries are likely positive in the new iterate. The elements of T are then chosen carefully within C: note that if the columns corresponding to the chosen indices are linearly dependent, the submatrix of the least squares problem (15) will be rank deficient, leading to numerical difficulties. We add k new indices, where k is an integer parameter to tune on the problem size, in such a way that, at the end, for every pair of indices in the set T, the corresponding column vectors form an angle whose cosine in absolute value is below a given threshold $thres$. The whole procedure is implemented in the function

• $[x,resnorm,exitflag]=\mathrm{LHDM}(A,b,options)$.

The input variable $options$ is a structure containing the user parameters for the LHDM algorithm; for example, the aforementioned k and $thres$. The output parameter x is the least squares solution, $resnorm$ is the squared 2-norm of the residual and $exitflag$ is set to 0 if the LHDM algorithm has reached the maximum number of iterations without converging and 1 otherwise.

In the literature, an accelerating technique was introduced by Van Benthem and Keenan [25], who presented a different NNLS solution algorithm, namely “fast combinatorial NNLS”, designed for the specific case of a large number of right-hand sides. The authors exploited a clever reorganization of computations in order to take advantage of the combinatorial nature of the problems treated (multivariate curve resolution) and introduced a nontrivial initialization of the algorithm by means of unconstrained least squares solution. In the following section we are going to compare such an approach, briefly named LHI, and the standard LH algorithm with the LHDM procedure just summarized.

4. Numerical Examples

In this section, we perform several tests on the computation of d-variate near G-optimal Tchakaloff designs, from low to moderate dimension d. In practice, we are able to treat, on a personal computer, large-scale problems where $card\left(X\right)\times dim\left(P_{2m}^d\right)$ is up to ${10}^8$–${10}^9$, with $dim\left(P_{2m}^d\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{d}\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{2m}\right)$. Recall that the main memory requirement is given by the $N_{2m}\times M$ matrix $U^T$ in the compression process solved by the LHDM algorithm, where $M=card\left(X\right)$ and $N_{2m}=dim\left(P_{2m}^d\left(X\right)\right)\le dim\left(P_{2m}^d\right)$.

Given the dimension $d>1$ and the polynomial degree m, the routine LHDM empirically sets the parameter k as follows $k=\lceil \left(\genfrac{}{}{0pt}{}{2m+d}{d}\right)/\left(m(d-1)\right)\rceil$, while the threshold is $thres=cos(\frac{\pi }{2}-\theta ),\theta \approx 0.22$. All the tests are performed on a workstation with a 32 GB RAM and an Intel Core i7-8700 CPU @ 3.20 GHz.

4.1. Complex 3d Shapes

To show the flexibility of the package dCATCH, we compute near G-optimal designs on a “multibubble” $D\subset {\mathbb{R}}^3$ (i.e., the union of a finite number of non-disjoint balls), which can have a very complex shape with a boundary surface very difficult to describe analytically. Indeed, we are able to implement near optimal regression on quite complex solids, arising from finite union, intersection and set difference of simpler pieces, possibly multiply-connected, where for each piece we have available the indicator function via inequalities. Grid-points or low-discrepancy points, for example, Halton points, of a surrounding box, could be conveniently used to discretize the solid. Similarly, thanks to the adaptation of the method to the actual dimension of the polynomial spaces, we can treat near optimal regression on the surfaces of such complex solids, as soon as we are able to discretize the surface of each piece by point sets with good covering properties (for example, we could work on the surface of a multibubble by discretizing each sphere via one of the popular spherical point configurations, cf. Reference [26]).

We perform a test at regression degree $m=10$ on the 5-bubble shown in Figure 1b. The initial support X consists in the $M=$ 18,915 points within 64,000 low discrepancy Halton points, falling in the closure of the multibubble. Results are shown in Figure 1 and

Table 3. Results for the multibubble numerical test: $compr=M/mean\left(cpts\right)$ is the mean compression ratio obtained by the three methods listed; $t_{LH}/t_{Titt}$ is the ratio between the execution time of LH and that of the Titterington algorithm; $t_{LH}/t_{LHDM}$ ($t_{LHI}/t_{LHDM}$) is the ratio between the execution time of LH (LHI) and that of LHDM; $cpts$ is the number of compressed Tchakaloff points and $momerr$ is the final moment residual.

Test			LH				LHI			LHDM
m	M	compr	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{Titt}}$	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	${\mathit{t}}_{\mathit{LHI}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	cpts	momerr
10	18,915	11/1	40.0/1	2.7/1	1755	$3.4\times {10}^{-8}$	3.2/1	1758	$3.2\times {10}^{-8}$	1755	$1.5\times {10}^{-8}$

.

4.2. Hypercubes: Chebyshev Grids

In a recent paper [19], a connection has been studied between the statistical notion of G-optimal design and the approximation theoretic notion of admissible mesh for multivariate polynomial approximation, deeply studied in the last decade after Reference [13] (see, e.g., References [27,28] with the references therein). In particular, it has been shown that near G-optimal designs on admissible meshes of suitable cardinality have a G-efficiency on the whole d-cube that can be made convergent to 1. For example, it has been proved by the notion of Dubiner distance and suitable multivariate polynomial inequalities, that a design with G-efficiency $\gamma$ on a grid X of ${\left(2km\right)}^d$ Chebyshev points (the zeros of $T_{2km}\left(t\right)=cos(2kmarccos\left(t\right))$, $t\in [-1,1]$), is a design for ${[-1,1]}^d$ with G-efficiency $\gamma (1-{\pi }^2/\left(8k^2\right))$. For example, taking $k=3$ a near G-optimal Tchakaloff design with $\gamma =0.99$ on a Chebyshev grid of ${\left(6m\right)}^d$ points is near G-optimal on ${[-1,1]}^d$ with G-efficiency approximately $0.99·0.86\approx 0.85$, and taking $k=4$ (i.e., a Chebyshev grid of ${\left(8m\right)}^d$ points) the corresponding G-optimal Tchakaloff design has G-efficiency approximately $0.99·0.92\approx 0.91$ on ${[-1,1]}^d$ (in any dimension d).

We perform three tests in different dimension spaces and at different regression degrees. Results are shown in Figure 2 and

Table 4. Results of numerical tests on $M={\left(2km\right)}^d$ Chebyshev’s nodes, with $k=4$, with different dimensions and degrees: $compr=M/mean\left(cpts\right)$ is the mean compression ratio obtained by the three methods listed; $t_{LH}/t_{Titt}$ is the ratio between the execution time of LH and that of Titterington algorithm; $t_{LH}/t_{LHDM}$ ($t_{LHI}/t_{LHDM}$) is the ratio between the execution time of LH (LHI) and that of LHDM; $cpts$ is the number of compressed Tchakaloff points and $momerr$ is the final moment residual.

Test				LH				LHI			LHDM
d	m	M	compr	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{Titt}}$	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	${\mathit{t}}_{\mathit{LHI}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	cpts	momerr
3	6	110,592	250/1	0.4/1	3.1/1	450	$5.0\times {10}^{-7}$	3.5/1	450	$3.4\times {10}^{-7}$	450	$1.4\times {10}^{-7}$
4	3	331,776	1607/1	0.2/1	2.0/1	207	$8.9\times {10}^{-7}$	3.4/1	205	$9.8\times {10}^{-7}$	207	$7.9\times {10}^{-7}$
5	2	1,048,576	8571/1	0.1/1	1.4/1	122	$6.3\times {10}^{-7}$	1.5/1	123	$3.6\times {10}^{-7}$	122	$3.3\times {10}^{-7}$

, using the same notation above.

4.3. Hypercubes: Low-Discrepancy Points

The direct connection of Chebyshev grids with near G-optimal designs discussed in the previous subsection suffers rapidly of the curse of dimensionality, so only regression at low degree in relatively low dimension can be treated. On the other hand, in sampling theory a number of discretization nets with good space-filling properties on hypercubes has been proposed and they allow to increase the dimension d. We refer in particular to Latin hypercube sampling or low-discrepancy points (Sobol, Halton and other popular sequences); see for example, Reference [29]. These families of points give a discrete model of hypercubes that can be used in many different deterministic and statistical applications.

Here we consider a discretization made via Halton points. We present in particular two examples, where we take as finite design space X a set of $M={10}^5$ Halton points, in $d=4$ with regression degree $m=5$, and in $d=10$ with $m=2$. In both examples, $dim\left(P_{2m}^d\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{d}\right)=\left(\genfrac{}{}{0pt}{}{2m+d}{2m}\right)=\left(\genfrac{}{}{0pt}{}{14}{4}\right)=1001$, so that the largest matrix involved in the construction is the $1001\times$ 100,000 Chebyshev-Vandermonde matrix C for degree $2m$ on X constructed at the beginning of the compression process (by dORTHVAND within dCATCH to compute $U_{2m}$ in (12)).

Results are shown in Figure 3 and

Table 5. Results of numerical tests on Halton points: $compr=M/mean\left(cpts\right)$ is the mean compression ratio obtained by the three methods listed; $t_{LH}/t_{Titt}$ is the ratio between the execution time of LH and that of Titterington algorithm; $t_{LH}/t_{LHDM}$ ($t_{LHI}/t_{LHDM}$) is the ratio between the execution time of LH (LHI) and that of LHDM; $cpts$ is the number of compressed Tchakaloff points and $momerr$ is the final moment residual.

Test				LH				LHI			LHDM
d	m	M	compr	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{Titt}}$	${\mathit{t}}_{\mathit{LH}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	${\mathit{t}}_{\mathit{LHI}}/{\mathit{t}}_{\mathit{LHDM}}$	cpts	momerr	cpts	momerr
10	2	10,000	10/1	41.0/1	1.9/1	990	1.1 $\times {10}^{-8}$	1.9/1	988	9.8 $\times {10}^{-9}$	990	9.4 $\times {10}^{-9}$
10	2	100,000	103/1	6.0/1	3.1/1	968	3.6 $\times {10}^{-7}$	2.8/1	973	2.7 $\times {10}^{-7}$	968	4.2 $\times {10}^{-7}$
4	5	10,000	10/1	20.2/1	2.3/1	997	9.7 $\times {10}^{-9}$	2.4/1	993	1.3 $\times {10}^{-8}$	997	2.1 $\times {10}^{-9}$
4	5	100,000	103/1	2.0/1	3.8/1	969	6.6 $\times {10}^{-7}$	3.8/1	964	6.3 $\times {10}^{-7}$	969	5.3 $\times {10}^{-7}$

, using the same notation as above.

Remark 1.

The computational complexity of dCATCH mainly depends on the QR decompositions, which clearly limit the maximum size of the problem and mainly determine the execution time. Indeed, the computational complexity of a QR factorization of a matrix of size $n_r\times n_c$, with $n_c\le n_r$, is high, namely $2(n_c^2{n}_r-n_c^3/3)\approx 2n_c^2{n}_r$ (see, e.g., Reference [30]).

Titterington algorithm performs a QR factorization of a $M\times N_m$ matrix at each iteration, with the following overall computational complexity

$$C_{Titt}\approx 2\overline{k}M{N}_m^2,$$

where $\overline{k}$ is the number of iterations necessary for convergence, that depends on the desired G-efficiency.

On the other hand, the computational cost of one iteration of the Lawson-Hanson algorithm, fixed the passive set P, is given by the solution of an LS problem of type (15), which approximately is $2N_{2m}{\left|P\right|}^2$ that is the cost of a QR decomposition of a matrix of size $N_{2m}\times \left|P\right|$. However, as experimental results confirm, the evolution of the set P along the execution of the algorithm may vary significantly depending on the experiment settings, so that the exact overall complexity is hard to estimate. Lower and upper bounds are available, but may lead to heavy under- and over-estimations, respectively; cf. Reference [31] for a discussion on complexity issues.

5. Conclusions

In this paper, we have presented dCATCH [1], a numerical software package for the computation of a d-variate near G-optimal polynomial regression design of degree m on a finite design space $X\subset {\mathbb{R}}^d$. The mathematical foundation is discussed connecting statistical design theoretic and approximation theoretic aspects, with a special emphasis on deterministic regression (Weighted Least Squares). The package takes advantage of an accelerated version of the classical NNLS Lawson-Hanson solver developed by the authors and applied to design compression.

As a few examples of use cases of this package we have shown the results on a complex shape (multibubble) in three dimensions, and on hypercubes discretized with Chebyshev grids and with Halton points, testing different combinations of dimensions and degrees which generate large-scale problems for a personal computer.

The present package, dCATCH works for any discrete measure on a discrete set X. Indeed, it could be used, other than for design compression, also in the compression of d-variate quadrature formulas, even on lower-dimensional manifolds, to give an example.

We may observe that with this approach we can compute a d-variate compressed design starting from a high-cardinality sampling set X, that discretizes a continuous compact set (see Section 4.2 and Section 4.3). This design allows an m-th degree near optimal polynomial regression of a function on the whole X, by sampling on a small design support. We stress that the compressed design is function-independent and thus can be constructed “once and for all” in a pre-processing stage. This approach is potentially useful, for example, for the solution of d-variate parameter estimation problems, where we may think to model a nonlinear cost function by near optimal polynomial regression on a discrete d-variate parameter space X; cf., for example, References [32,33] for instances of parameter estimation problems from mechatronics applications (Digital Twins of controlled systems) and references on the subject. Minimization of the polynomial model could then be accomplished by popular methods developed in the growing research field of Polynomial Optimization, such as Lasserre’s SOS (Sum of Squares) and measure-based hierarchies, and other recent methods; cf., for example, References [34,35,36] with the references therein.

From a computational viewpoint, the results shown in Table 3, Table 4 and Table 5 show relevant speed-ups in the compression stage, with respect to the standard Lawson-Hanson algorithm, in terms of the number of iterations required and of computing time within the Matlab scripting language. In order to further decrease the execution times and to allow us to tackle larger design problems, we would like in the near future to enrich the package dCATCH with an efficient C implementation of its algorithms and, possibly, a CUDA acceleration on GPUs.