Improved G-Optimal Designs for Small Exact Response Surface Scenarios: Fast and Efficient Generation via Particle Swarm Optimization

Abstract

G-optimal designs are those which minimize the worst-case prediction variance. Thus, such designs are of interest if prediction is a primary component of the post-experiment analysis and decision making. G-optimal designs have not attained widespread use in practical applications, in part, because they are difficult to compute. In this paper, we review the last two decades of algorithm development for generating exact G-optimal designs. To date, Particle Swarm Optimization (PSO) has not been applied to construct exact G-optimal designs for small response surface scenarios commonly encountered in industrial settings. We were able to produce improved G-optimal designs for the second-order model and several sample sizes under experiments with $K=1,2,3,4,$ and 5 design factors using an adaptation of PSO. Thereby, we publish updated knowledge on the best-known exact G-optimal designs. We compare computing cost/time and algorithm efficacy to all previous published results including those generated by the current state-of-the-art (SOA) algorithm, the $G\left(I_{\lambda }\right)$-coordinate exchange. PSO is hereby demonstrated to produce better designs than the SOA at commensurate cost. In all, the results of this paper suggest PSO should be adopted by more practitioners as a tool for generating exact optimal designs.

1. Introduction

In 1918 Kirstine Smith published guidance for how the residual error variance informs designing an experiment where the ensuing data are used to fit a polynomial model linear in the parameters [1]. This paper’s contribution provided what we now refer to as G-optimal designs for a $K=1$ factor experiment supporting a first up to sixth-degree polynomial. Atkinson and Bailey [2] note that Smith’s contribution is considered the seminal optimal design paper and was 30 years ahead of its time. Optimal design did not receive a rigorous theoretical development until the late 1950s with primary contributions from Kiefer and Wolfowitz [3,4,5]. These contributions focus on the theory of continuous optimal designs (often called approximate or asymptotic designs), that is, designs viewed as a probability measure. The core theory of continuous designs is mathematical in nature and includes the foundational General Equivalence Theorem which provides a way to verify the optimality of a candidate design.

Exact optimal designs, in contrast, are those that are optimal for an integer valued number of experimental runs N. Thus, an implemented experiment utilizes an exact design. Exact designs have no theory analogous to that of continuous designs with which to verify the optimality of a candidate design. Therefore, much of the research into generating exact designs has focused on algorithm development. Early algorithms for generating exact optimal designs date back to the 1970s and include the point-exchange and DETMAX [6,7]. Such algorithms discretize the design space via a candidate set (usually a high order factorial design). The most popular contemporary algorithm for generating exact optimal designs appears to be the Coordinate-Exchange (CEXCH) of Meyer and Nachtsheim [8] which, in contrast to the point exchange, honors a continuous design space and allows any point in the design region to be a component of the experiment.

CEXCH’s success and popularity notwithstanding, it is not without drawbacks. CEXCH is well-recognized to be a local optimizer and so several authors recommend applying CEXCH thousands of times in order to ensure the generation of a highly efficient design [9]. Because CEXCH starts with a single random candidate design followed by a local optimization, it is unlikely, especially in higher dimensional problems, that CEXCH will find the globally optimal design. This fact has motivated several authors to explore the class of meta-heuristic evolutionary algorithms for optimal design generation. Meta-heuristics include the Genetic Algorithm (GA), Simulated Annealing, and Particle Swarm Optimization (PSO) among others. See, for example applications in optimal design, [10,11,12,13,14,15,16,17,18,19]. Simulated Annealing appears to be widely popular, but has not been applied to the G-optimal design problem to date [19]. Meta-heuristics offer attractive solutions to the optimal design problem for two primary reasons. First, they make little to no assumptions regarding the nature of the objective (optimality criterion). Furthermore, second, as opposed to CEXCH, they attempt to search the space of candidate design matrices globally and are robust to entrapment in local optima thereby giving them a significant advantage over CEXCH to generate a highly efficient design in a single run. This benefit, however, often comes at a significant increase to computing cost vs. the CEXCH [11].

Among the meta-heuristics, PSO is a relative newcomer to optimal design, and most applications have been toward generating continuous designs or space-filling designs [13,14,15,16,17,18,20,21]. A comprehensive review of the last decade of research literature on PSO adaptations to experimental design is provided by Chen, Chen, and Wong (2022) [22]. These authors highlight recent applications of PSO for generating continuous optimal designs, minimax/maximin optimal designs (e.g., the G-optimal design problem, or optimal design for non-linear models), and exact optimal designs. Of the sixteen papers on PSO for experimental design discussed in Chen, Chen, and Wong (2022), only one works with exact designs and this is not the primary focus of the paper. Therefore, the literature is lacking demonstration of PSOs efficacy and efficiency to generating exact optimal designs, and specifically those to response surface scenarios commonly encountered by industrial practitioners. More recently, Walsh and Borkowski (2022) [23] present data from a detailed case-study on the efficiency and efficacy of PSO to generating exact optimal designs for 21 small exact response surface scenarios. In short PSO, and specific a version of PSO which uses a random local communication topology, is shown to generate highly efficient designs (efficiency >95%) with large probability (near 1), and an appreciably large probability of finding the global optimal designs for the D- and I-criteria in these scenarios. Therefore, we expect PSO to perform well when applied to the G-optimal exact design generation problem.

In light of the existing literature we designed a detailed study and applied PSO to generate exact G-optimal designs and offer the following contributions to the state-of-knowledge:

1.

A brief literature survey of the last 20 years of algorithm development and approaches for generating exact G-optimal designs in small exact response surface scenarios.

2.

Application of the PSO version with local communication topology, as described in Walsh and Borkowski (2022) to generating exact G-optimal designs for 29 design scenarios for $K=1,2,3,4,5$ experimental factors and a range of N (experiment sizes) [23].

3.

For most of the 29 design scenarios, PSO was able to find a better G-optimal design than those currently known, and we provide a detailed catalogue of these new designs in the Supplementary Material.

4.

Quantitative benchmarking of PSOs efficiency and efficacy to accomplish this task and performance comparison to state-of-practice and SOA CEXCH type algorithms as illustrated in Rodriquez et al. (2010) and Hernandez and Nachtsheim (2018) [11,24].

The remainder of this paper is organized as follows. In Section 2, we provide notation and define the exact G-optimal design generation problem. In Section 3, we present a literature review of recent research on generating exact G-optimal designs. In Section 4, develop an extension of PSO for generating G-optimal designs. In Section 5, we present the structure of our study for adapting PSO to generate exact G-optimal designs. In Section 6, we present the results where we compare G-PSO generated designs to those provided by [10,11,24]. In Section 7, we provide discussion, conclusions, and future research directions.

2. $\mathit{G}$-Optimal Design for Small Exact Response Surface Scenarios

2.1. Small Exact Response Surface Designs

As is standard at the optimization step of response surface methodology (RSM), we will be working with the second-order linear model under standard assumptions. Let N represent the number of design points and K represent the number of experimental factors [25]. A design point is an ${\mathbf{x}}^{\prime }:1\times K$ row-vector. In RSM all design factors are scaled to range [−1,1] and so the design space is the $\mathcal{X}={[-1,1]}^K$ hypercube [9,25]. Let $\mathbf{X}:N\times K$ represent the design matrix, while $\mathcal{X}$ denotes the space of candidate design points ${\mathbf{x}}^{\prime }$, a design matrix $\mathbf{X}$ is a collection of N such design points. Thus, the space of all candidate designs is an $NK$-dimensional hypercube and is denoted:

$$\mathbf{X}\in \underset{j=1}{\overset{N}{{\displaystyle \mathsf{✕}}}}\mathcal{X}=\underset{j=1}{\overset{N}{{\displaystyle \mathsf{✕}}}}{[-1,1]}^K={[-1,1]}^{NK}={\mathcal{X}}^N.$$

(1)

The second-order linear model which has $p=(\genfrac{}{}{0pt}{}{K+2}{2})$ linear coefficient parameters. In scalar form the model is written

$$y={\beta }_0+\sum _{i=1}^K{\beta }_i{x}_i+\sum _{i=1}^{K-1}\sum _{j=i+1}^K{\beta }_{ij}{x}_i{x}_j+\sum _{i=1}^K{\beta }_{ii}{x}_i^2+ϵ.$$

Let $\mathbf{F}\left(\mathbf{X}\right):N\times p$ represent the model matrix with rows given by the expansion vector ${\mathbf{f}}^{\prime }\left({\mathbf{x}}_i^{\prime }\right)=(1x_{i1}\cdots x_{i2}{x}_{i1}{x}_{i2}\cdots x_{i(K-1)}{x}_{iK}{x}_{i1}^2\cdots x_{iK}^2)$. Note that we will abbreviate the model matrix $\mathbf{F}$, but it is always a function of the design matrix $\mathbf{X}$. The model can be written in vector-matrix form as $\mathbf{y}=\mathbf{F}\beta +ϵ$ where we impose the standard ordinary least squares assumptions $ϵ\sim {\mathcal{N}}_N(0,{\sigma }^2{\mathbf{I}}_N)$ where ${\mathcal{N}}_N$ denotes the N-dimensional multivariate normal distribution. The ordinary least squares estimator of $\beta$ is $\widehat{\beta }={\left({\mathbf{F}}^{\prime }\mathbf{F}\right)}^{-1}{\mathbf{F}}^{\prime }\mathbf{y}$ which has variance $\mathrm{Var}\left(\widehat{\beta }\right)={\sigma }^2{\left({\mathbf{F}}^{\prime }\mathbf{F}\right)}^{-1}$. The total information matrix for $\beta$, specifically $\mathbf{M}\left(\mathbf{X}\right)={\mathbf{F}}^{\prime }\mathbf{F}$, plays an important role in optimal design of experiments—all optimal design objective functions are functions of this matrix [9].

The practitioner must choose a design from ${\mathcal{X}}^N$ to implement the experiment in practice. An optimality criterion is used to define which candidate designs $\mathbf{X}\in {\mathcal{X}}^N$ are ‘good’ designs. An optimization algorithm is required to search ${\mathcal{X}}^N$ to find the ‘best’, or optimal, design. Thus, an exact optimal design problem is defined by three components:

1.

The number of design points N that can be afforded in the experiment.

2.

The structure of the model one wishes to fit (here the second-order model).

3.

A criterion which defines an optimal design. This is a function of $\mathbf{M}\left(\mathbf{X}\right)$.

In the next subsection we define the G-criterion and G-optimal design.

2.2. G-Optimal Design

A G-optimal design is that design which minimizes the maximum (i.e., worst case) prediction variance over $\mathcal{X}$. The the variance of the mean predicted value is

$$\mathrm{Var}\left(\widehat{y}\left({\mathbf{x}}^{\prime }\right)\right)={\sigma }^2{\mathbf{f}}^{\prime }\left({\mathbf{x}}^{\prime }\right){\left({\mathbf{F}}^{\prime }\mathbf{F}\right)}^{-1}\mathbf{f}\left({\mathbf{x}}^{\prime }\right)$$

for any point of prediction ${\mathbf{x}}^{\prime }\in \mathcal{X}$. The scaled prediction variance (SPV) removes the scale parameter ${\sigma }^2$ and re-scales to N by multiplying by the factor $N/{\sigma }^2$. Thus, SPV is defined as (for a candidate design $\mathbf{X}$)

$$\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right|\mathbf{X}):=N{\mathbf{f}}^{\prime }\left({\mathbf{x}}^{\prime }\right){\left({\mathbf{F}}^{\prime }\mathbf{F}\right)}^{-1}\mathbf{f}\left({\mathbf{x}}^{\prime }\right).$$

(2)

The G-score of a candidate design $\mathbf{X}$ is defined as the maximum scaled prediction variance over all points of prediction ${\mathbf{x}}^{\prime }\in \mathcal{X}$

$$G\left(\mathbf{X}\right):=\underset{{\mathbf{x}}^{\prime }\in \mathcal{X}}{max}\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right|\mathbf{X}).$$

(3)

Equation (3), thus, shows that for a fixed candidate design $\mathbf{X}$, scoring the candidate on the G scale is itself an optimization problem. The G-optimal design ${\mathbf{X}}^{*}$ is that design which minimizes, over all designs $\mathbf{X}\in {\mathcal{X}}^N$, the maximum scaled prediction variance, namely

$$\begin{array}{cc}\hfill {\mathbf{X}}^{*}& :=\underset{\mathbf{X}\in {\mathcal{X}}^N}{argmin}G\left(\mathbf{X}\right)\hfill \\ \hfill & =\underset{\mathbf{X}\in {\mathcal{X}}^N}{argmin}\underset{{\mathbf{x}}^{\prime }\in \mathcal{X}}{max}\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right|\mathbf{X}).\hfill \end{array}$$

(4)

Equation (4) shows that finding the G-optimal design is a minimax problem. This optimization has proved notoriously difficult to solve because neither of the optimizations required to compute ${\mathbf{X}}^{*}$ are convex in large part due to the expansion of the design matrix into model matrix $\mathbf{F}$ [10,11,24].

The scale of G for an arbitrary design has known bounds. The General Equivalence Theorem of [3] demonstrates that the lower bound on $G\left(\mathbf{X}\right)$ is

$$G\left(\mathbf{X}\right)=\underset{{\mathbf{x}}^{\prime }\in \mathcal{X}}{max}\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right)\ge p.$$

(5)

That is, the smallest that the maximum scaled prediction variance of candidate design $\mathbf{X}$ may be is p, the number of parameters. This apparently gives a way to verify if a proposed exact G-optimal design is globally optimal, however, not all design scenarios have globally G-optimal designs that will achieve this lower bound. Further, if design ${\mathbf{X}}^{*}$ is globally G-optimal and achieves the result in Equation 5, then for this design $\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right|\mathbf{X})=p$ at all diagonals of the hat matrix $\mathbf{F}{\left({\mathbf{F}}^{\prime }\mathbf{F}\right)}^{-1}{\mathbf{F}}^{\prime }$ and $\mathrm{SPV}\left({\mathbf{x}}^{\prime }\right|\mathbf{X})\le p$ at all other points of prediction ${\mathbf{x}}^{\prime }\in \mathcal{X}$ [25]. It is customary to exploit this fact and score candidate designs on the G-efficiency scale,

$$G_{\mathrm{eff}}\left(\mathbf{X}\right)=100\frac{p}{G\left(\mathbf{X}\right)},$$

(6)

in order to gauge the quality of a candidate design $\mathbf{X}$ (larger $G_{\mathrm{eff}}$ on this scale implies a better design). Last, the relative efficiency of two candidate designs may be computed as

$$G_{\mathrm{releff}}({\mathbf{X}}_1,{\mathbf{X}}_2)=100\frac{G_{\mathrm{eff}}\left({\mathbf{X}}_1\right)}{G_{\mathrm{eff}}\left({\mathbf{X}}_2\right)}.$$

(7)

In the next section, we provide a review of the most literature for generating $\mathit{G}$-optimal designs for small exact response surface models.

3. Literature Review: Algorithm Development and Current Best-Known Exact $\mathit{G}$-Optimal Designs

The exact $\mathit{G}$-optimal design generation problem is recognized as the notoriously difficult mini-max problem expressed in Equation 4. It was not until recent decades that modern computing resources were cheap enough to enable the application of well-suited meta-heuristic optimization approaches to this problem. Borkowski [10] provided one of the earliest applications of Genetic Algorithms (GAs) for generating exact optimal designs over the hypercube supporting a second-order response surface model in $\mathit{K}=1,2,3$ design factors and for experiment sizes $\mathit{N}=3,4,5,6,7,8,9$, $\mathit{N}=6,7,8,9,10,11,12$, and $\mathit{N}=10,11,12,13,14,15,16$, respectively. Borkowski [10] adapted a GA to generate $\mathit{G}$-, $\mathit{D}$-, $\mathit{I}$-, and $\mathit{A}$-optimal designs and published a proposed catalog of exact optimal designs for each criterion. The results provided in [10] have become a ‘ground truth’ standard data set against which to compare results of newly developed algorithms to solve this problem. Until this point, the exact $\mathit{G}$-optimal designs of Borkowski (2003) have remained the best-known designs for these scenarios [10] and were subsequently reproduced by [24] via an augmented application of the coordinate exchange, and used as a benchmark dataset for new algorithms proposed by authors [11,26].

Next, [24] adapted the coordinate exchange (CEXCH) algorithm of [8] in conjunction with Brent’s minimization algorithm (used to score a candidate design on the $\mathit{G}$-scale) to generate exact $\mathit{G}$-optimal designs. These authors covered all scenarios or $\mathit{K}=2,3$ design factors as in Borkowski (2003) and were able to reproduce these designs with their new algorithm. These authors extend [10] and propose candidate exact $\mathit{G}$-optimal designs for $\mathit{K}=4$, $\mathit{N}=15,20,24$ and $\mathit{K}=5$, $\mathit{N}=21,26,30$. Note that, to this point, $\mathit{K}=4,5$ $\mathit{G}$-optimal designs had not been explored because of prohibitive computational cost. A third contribution of this work was a detailed comparison of the resulting $\mathit{G}$-CEXCH generated exact $\mathit{G}$-optimal design’s properties (via fraction of design space plots) with corresponding $\mathit{D}$- and $\mathit{I}$-optimal designs. The authors found that, for scenarios where post hoc prediction was the primary objective, the $\mathit{I}$-optimal designs exhibited smaller prediction variance than the corresponding $\mathit{G}$-optimal designs over large percentages of the design space (even though the $\mathit{G}$-optimal design has lower maximum prediction variance). This observation motivated the authors to recommend that the additional computing cost required for $\mathit{G}$-optimal designs may not be worth the effort and $\mathit{I}$-optimal designs might be preferred for many practical experimental scenarios. The $\mathit{G}$- vs. $\mathit{I}$-optimal design question is the subject of ongoing research in the broader literature [27].

Saleh and Pan (2015) recognized the lack of algorithm development on the $\mathit{G}$-optimal design problem and provide a hybridized point and coordinate exchange algorithm which utilizes clustering to explore the characteristics of SPV over the design space [26]. Their algorithm is termed cCEA and they apply it to linear model and generalized linear model scenarios. cCEA proceeds by first by generating a large set of candidate design points ${\mathbf{x}}^{\prime }$ to provide a covering of $\mathcal{X}$. These points are fed into a clustering algorithm to produce families of design points over $\mathcal{X}$ based on SPV. For a candidate design, these clusters are scored on the SPV scale. A point exchange algorithm is applied to the candidate and each cluster is rescored on SPV. The algorithm proceeds until no further reductions in maximum SPV are found. In the second step, standard CEXCH locally adjusts the candidate in order to find a design that further reduces the SPV. In short this algorithm attempts to get a design close to $\mathit{G}$-optimal quickly via the clustering and point exchange, and then refines this design locally using the standard coordinate exchange. With cCEA, these authors were able to nearly reproduce designs with scores equivalent to [10,24].

The most recent publication on generating exact $\mathit{G}$-optimal designs is offered by [11]. The primary focus of this paper is the computational cost associated with generating $\mathit{G}$-optimal designs. These authors noticed a correspondence between continuous ${\mathit{I}}_{\mathit{\lambda }}$-optimal designs and $\mathit{G}$-optimal designs. They propose a new algorithm that exploits the structural relationship between these two types of designs which first generates a continuous ${\mathit{I}}_{\mathit{\lambda }}$-optimal design as a starting point for the exact $\mathit{G}$-optimal design search. Then they apply the CEXCH algorithm to locally improve the candidate $\mathit{G}$-optimal design. We will refer to their algorithm as $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH. These authors are the first to completely run $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH and rerun the searches via GA by Borkowski (2003) in Matlab, and CEXCH by [24] in JMP (they also reported the architecture of their PC) in order to compare computing efficiency of their proposed algorithm to the others. They implemented 200 runs of each algorithm for each design scenario discussed in [10] and also ran two new scenarios: $\mathit{K}=4$, $\mathit{N}=17$ and $\mathit{K}=5$, $\mathit{N}=23$ with only $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH (they indicated that CEXCH would have taken 25 and 166 days, respectively, to complete the task, and so this algorithm was not run on these scenarios). These authors found that the GA was able to find the most efficient $\mathit{G}$-optimal designs in repeated runs, but that CEXCH and $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH produced designs with high relative efficiency vs. GA generated designs with relative-efficiencies of 90% or greater at a fraction of the cost [11]. Thus, the GA achieves this superiority at high computational cost exhibiting 2-orders of magnitude increase in the number of objective function evaluations as compared to the $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH. $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH was demonstrated to produce designs with high $\mathit{G}$-efficiency as well as standard CEXCH but with computing times at a fraction of CEXCH for the higher dimensional searches. Thus, they propose that $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH is a good choice for generating exact $\mathit{G}$-optimal designs as it is demonstrated to produce designs with high relative efficiency to GA generated designs and does this more efficiently than existing algorithms.

Last, in

Table 1. Summary of design scenarios, algorithms, and authors who have addressed the exact $\mathit{G}$-optimal design problem for the second order RSM model in the last 20 years.

# Exp. Factors	Experiment Sizes	Algorithms	Authors
$\mathit{K}$	$\mathit{N}$
1	3, 4, 5, 6, 7, 8, 9	GA $\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH	Borkowski (2003) Hernandez and Nachtsheim (2018)
2	6, 7, 8, 9, 10,11, 12	GA CEXCH cCEA ($\mathit{N}$ = 7 to 12)	Borkowski (2003) Rodriquez et al. (2010) Saleh and Pan (2015)
3	10, 11,12,13,14,15,16	GA CEXCH cCEA ($\mathit{N}$ = 11 to 16)	Borkowski (2003) Rodriquez et al. (2010) Saleh and Pan (2015)
4	15, 20, 24	CEXCH cCEA ($\mathit{N}$ = 24)	Rodriguez et al. (2010) Saleh and Pan (2015)
	16	cCEA	Saleh and Pan (2015)
	17	$\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH	Hernandez and Nachtsheim (2018)
5	21, 26, 30	CEXCH cCEA ($\mathit{N}$ = 26)	Rodriquez et al. (2010) Saleh and Pan (2015)
	23	$\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH	Hernandez and Nachtsheim (2018)

we present a summary of exact $\mathit{G}$-optimal design scenarios that authors have addressed in previous algorithm development and research, and that we will study further using PSO in this paper.

Evaluating the $\mathit{G}$-Score for Candidate Design $\mathbf{X}$

Any algorithm needs to address both optimizations expressed in Equation 4, or more specifically, given a candidate design $\mathbf{X}$, one must score it on the $\mathit{G}$-scale via Equation 3. There are two approaches to doing this implemented in the algorithms discussed in the previous section.

First, authors [10,11] exploit the symmetry of the SPV-surface for a $\mathit{G}$-optimal design and recommend using a $5^{\mathit{K}}$ point grid over $\mathcal{X}$ with each factor containing $x_j\in \{-1,0.5,0,0.5,1\}$ for $j=1,\cdots ,\mathit{K}$ grid-points. The full grid defined as ${\mathit{G}}_{\mathcal{X}}={\{-1,0.5,0,0.5,1\}}^{\mathit{K}}$ [11]. Once one has a candidate design, the SPV is evaluated on ${\mathit{G}}_{\mathcal{X}}$ and the maximum value is taken as an approximation of the $\mathit{G}$-score for the candidate design, while this approach computes an approximate $\mathit{G}$-score for a candidate design $\mathbf{X}$ both authors have noted that, due to the symmetry of the quadratic $\mathit{G}$-surface for $\mathit{G}$-optimal designs, the approximation is quite adequate giving small errors and supports finding highly $\mathit{G}$-optimal designs.

Authors [24,26] use a different mechanism to compute $\mathit{G}$ for a candidate design X. Ref. [26] use the clustering and point exchange phase of their algorithm. In [24], once a candidate is available (e.g., after every coordinate exchange) they employ Brent’s minimization algorithm to search for the maximum prediction variance for the candidate design over X.

Regarding the choice of method for scoring a candidate design on the $\mathit{G}$-scale in this paper: we implemented the $5^{\mathit{K}}$ grid approach recommended by [10,11] for several reasons. Similar to authors [10,11] we find that this approach yields $\mathit{G}$-optimal design candidates with small error on the $\mathit{G}$-eff scale. Further, we ran a pilot study using a nested PSO approach [20] but similar to [24] we observed that if the inner PSO search to score a candidate design on the $\mathit{G}$-eff scale mis-scored the candidate design, this error propagated through the outer PSO search and reduced the success of finding the globally optimal design.

In the next section we describe the PSO algorithm and provide an extension of PSO to generate $\mathit{G}$-optimal designs used to produce the results presented in this paper.

4. Particle Swarm Optimization for Generating Optimal Designs

PSO is a wildly popular meta-heuristic optimization approach which has been widely applied to great success across engineering applications and applied science problems. To date, the seminal paper by [28] has 73,647 citations (queried in Google Scholar on 8 August 2022). PSO has been demonstrated to perform very well for high-dimensional optimization of multimodal objective functions, see for example, [28,29,30,31,32,33,34,35,36,37,37]. PSOs strengths include:

1.

few-to-no assumptions about the properties of the objective function $f$ to be optimized,

2.

PSO is demonstrated to be robust to entrapment in local optima, and thereby is a good match to the exact optimal design generation problem,

3.

simplicity—the core function of the algorithm can be explained via two simple update equations, and

4.

in contrast to other meta-heuristics where studying a range of tuning parameters can yield more efficient searches for specific problems, PSO only has three tuning parameters and these have been studied extensively, both theoretically and empirically, with optimal values demonstrated for searches (such as ours) that reside in the common Cartesian product space with the typical Euclidean geometry, see [34,38,39,40] among others.

PSOs widespread application notwithstanding, we observe relatively few applications in Statistics and specifically Optimal Design. The first such application in 2013 by Chen et al. discussed PSO-generated latin-hypercube designs [14]. Application of PSO to generating space-filling designs is addressed by [13,17]. PSO for generating optimal designs for non-linear models is illustrated in [15,16,20]. PSO-generation of Bayesian continuous designs is discussed in [21] and PSO for constructing continuous optimal designs for mixture experiments is provided in [18]. More recently [23] provided a benchmarking study on PSO for generating small-exact response surface designs under the $\mathit{D}$- and $\mathit{I}$-criterion. These authors show that a version of PSO which utilizes a local communication topology is superior to standard PSO which uses global communication topology in the sense that it greatly increases the probability that PSO will find the globally optimal design in a single run of the algorithm [23]. We expect these results to be similarly realized for the $\mathit{G}$-optimality searches as conducted in this paper.

We present the full PSO algorithm, extended to optimize functions that take matrix inputs, in Algorithm 1. The primary difference of our exposition of PSO vs. the standard PSO literature is that our version in Algorithm 1 is a clear formulation that works on functions that take matrix inputs as opposed to vector inputs. A brief dictionary of all quantities is

• $S$:= number of candidate designs (i.e., particles) in the swarm,
• ${\mathbf{X}}_i:N\times K$:= candidate design $i$,
• ${\mathbf{P}}_{\mathrm{best},i}$:= the best design found by particle $i$,
• ${\mathbf{L}}_{\mathrm{best},i}$:= the best design found by the particles in particle is communication neighborhood,
• ${\mathbf{G}}_{\mathrm{best}}$:= the best best design found by the swarm; this is the proposed optimal design,
• ${\mathit{U}}_K({\mathbf{l}}_b,{\mathbf{u}}_b)$= $\mathit{K}$-dimensional multivariate uniform distribution with lower and upper bound vectors ${\mathbf{l}}_b$ and ${\mathbf{u}}_b$, respectively,
• $\mathbf{U}={\left\{u_{ij}\right\}}_{i=1,j=1}^{N,K}$:= random matrix with elements $u_{ij}\stackrel{i.i.d}{\sim }\mathit{U}(0,1)$,
• ⊙:= Hadamard product (elementwise multiplication).

Algorithm 1: PSO for Generating Exact Optimal Designs on the Hypercube

1: Input: Objective function $f$, number of particles $S$, search space bounds ${\mathbf{l}}_b=-1_K$ and ${\mathbf{u}}_b=1_{\mathit{K}}$ 2: // Randomly draw $S$ candidate designs ${\mathbf{X}}_i$ and initialize 3: for each$i=1,\dots ,S$do 4:   ${\mathbf{x}}_{ij}^{\prime }\sim {\mathit{U}}_K({\mathbf{l}}_b,{\mathbf{u}}_b)\mathrm{for}j=1,\cdots ,N$ giving candidate design ${\mathbf{X}}_i$ with rows ${\mathbf{x}}_{ij}^{\prime }$ 5:   ${\mathbf{v}}_{ij}^{\prime }\sim {\mathit{U}}_K\left(\frac{{\mathbf{l}}_b-{\mathbf{x}}_{ij}}{2},\frac{{\mathbf{u}}_b-{\mathbf{x}}_{ij}}{2}\right)\mathrm{for}j=1,\cdots ,N$ giving velocity matrix ${\mathbf{V}}_i$ with rows ${\mathbf{v}}_{ij}^{\prime }$ 6:   $\{v_{ijk}\leftarrow min\{v_{ijk},v_k^{\max }\}\}$ for $j=1,\cdots ,N$ // limit stepsize 7:   ${\mathbf{N}}_i\leftarrow \mathtt{genNeighbors}\left({\mathbf{X}}_i\right)$ // generate communication neighborhood for particle $i$ 8:   ${\mathbf{P}}_{\mathrm{best},i}\leftarrow {\mathbf{X}}_i$ //set initial personal best position 9: endfor   10: ${\mathbf{G}}_{\mathrm{best}}\leftarrow \underset{{\mathbf{X}}_i\in \{{\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_S\}}{\mathrm{argmin}}f\left({\mathbf{X}}_i\right)$// current best design among swarm   11: ${\mathbf{L}}_{\mathrm{best},i}\leftarrow \underset{{\mathbf{X}}_i\in N_i}{\mathrm{argmin}}f\left({\mathbf{X}}_i\right)$ for $i=1,\cdots ,S$ // current best design in local neighborhoods   12: // swarm search loop 13: while stopping criteria not met do (this is iteration over $t$) 14:   if ${\mathbf{G}}_{\mathrm{best}}\left(t\right)={\mathbf{G}}_{\mathrm{best}}(t-1)$ 15:     // if the solution does not improve, reform the local communication networks 16:     ${\mathbf{N}}_i\leftarrow \mathtt{genNeighbors}\left({\mathbf{X}}_i\right)$ 17:   endif 18:   for each $i=1,\dots ,S$ do // Update velocities and positions 19:     ${\mathbf{V}}_i\leftarrow \omega {\mathbf{V}}_i+c_1\mathbf{U}\odot \left({\mathbf{P}}_{\mathrm{best},i}-{\mathbf{X}}_i\right)+c_2\mathbf{U}\odot \left({\mathbf{L}}_{\mathrm{best},i}-{\mathbf{X}}_i\right)$ 20:     $\{v_{\mathrm{ijk}}\leftarrow min\{v_{ijk},v_k^{\max }\}\}$ for $j=1,\cdots ,N$ 21:     ${\mathbf{X}}_i\leftarrow {\mathbf{X}}_i+{\mathbf{V}}_i$ 22:     ${\mathbf{X}}_i\leftarrow \mathtt{confine}\left({\mathbf{X}}_i\right)$ // keep candidates in searchspace 23:     if $f\left({\mathbf{X}}_i\right)<f\left({\mathbf{P}}_{\mathrm{best},i}\right)$ // update knowledge about best known design to time $t$ 24:       ${\mathbf{P}}_{\mathrm{best},i}\leftarrow {\mathbf{X}}_i$ 25:       if $f\left({\mathbf{P}}_{\mathrm{best},i}\right)<f\left({\mathbf{G}}_{\mathrm{best}}\right)$ 26:         ${\mathbf{G}}_{\mathrm{best}}\leftarrow {\mathbf{P}}_{\mathrm{best},i}$ 27:       endif 28:     endif 29:   endfor 30:   for each $i=1,\dots ,S$ do // with personal best’s update, can update local best 31:     if $f\left({\mathbf{P}}_{\mathrm{best},i}\right)<f\left({\mathbf{L}}_{\mathrm{best},i}\right)$ 32:       ${\mathbf{L}}_{\mathrm{best},i}\leftarrow {\mathbf{P}}_{\mathrm{best},i}$ // update best known design in local neighborhoods 33:     endif 34:   endfor 35: endwhile   36: Output: Particle swarm solution—the best optimal design found ${\mathbf{G}}_{\mathrm{best}}$

We briefly identify core components of the algorithm as follows. Lines 4 and 5 show how the $S$ initial candidate designs are randomly constructed based on a multivariate uniform distribution. We employ Standard Particle Swarm 2007 which utilizes a local communication topology [30]. Therefore line 7 indicates a sub-routine genNeighbors which randomly initializes particle matrix is communication neighborhood. This local communication topology has been illustrated to increasing the chance that the swarm finds the global optimal design. The core velocity update equation is stated in line 19 which shows that particle is step size and direction is a weighting of the typical inertia, cognitive, and social components. We implement the optimal recommended values for the weighting parameters

$$\omega =\frac{1}{log\left(2\right)},c_1=c_2=\frac{1}{2}+log\left(2\right).$$

These values have been illustrated theoretically and empirically to provide an excellent balance between exploration and exploitation while guaranteeing that the swarm eventually converges to a consensus solution [30,34,41].

5. Study Structure: Experimental Design and PSO Run Parameters

We implemented Algorithm 1 in the Julia language [42]. Our machine is an Intel i7-6700K which has 4 cores and 8 compute threads running at 4.0GHz. Thus, we could send multiple runs of PSO to 7 of the cores at the same time. We ran $\mathit{G}$-PSO $n_{\mathrm{run}}=140$ times for all design scenarios (and searches for the optimal design that supports the second-order model) covered by [10]. These scenarios cover $K=1,2,3$ design factors with experiment sizes $N=3,4,5,6,7,8,9$, $N=6,7,8,9,10,11,12$, and $N=10,11,12,13,14,15,16$, respectively. Both [10,11] ran the GA for these scenarios, but, they did not find the same exact $\mathit{G}$-optimal designs (in part related to the stopping criterion implemented by [11]). Regarding $\mathit{G}$-optimality, [10] found the current best-known $\mathit{G}$-optimal designs (to this point). Hernandez and Nachtsheim (2018) [11] found highly $\mathit{G}$-efficient designs relative to those of [10], but used their GA generated designs to compare their results from the other two algorithms in their study. We also ran $\mathit{G}$-PSO $n_{\mathrm{run}}=210$ times for the additional scenarios in [24]. These scenarios cover $K=4,5$ with experiment sizes $N=15,20,24$ and $N=21,26,30$, respectively. Last, we ran $\mathit{G}$-PSO $n_{\mathrm{run}}=210$ for the additional scenarios in [11], specifically $K=4,N=17$ and $K=5,N=23$. Our stopping criteria for all PSO runs was either a non-zero change in the objective equal to the square root of machine epsilon (about 10 × 10⁻⁸, or if the objective score stagnated for 100 iterations, we terminated the run and started a new run [23].

In total we have covered all published exact $\mathit{G}$-optimal designs for 29 design scenarios requiring 4620 independent runs of PSO. In the next section we compare $\mathit{G}$-PSO results to those of [10,11,24].

Hernandez and Nachtsheim (2018) [11] is the only reference that reported computing cost among our references, and they offer two measures: (1) number of function evaluations over all runs for each design scenario and (2) computing time wall-clock for the set of searches for each algorithm and design scenario and so a way to check the efficiency of $\mathit{G}$-PSO relative to the state-of-the-art. We will compare the efficiency of PSO to the other algorithms by reporting number of function evaluations during the runs of PSO. This measure is, of course, senstive to the number of particles and the stopping criterion. We ran all PSO searches with $S=150$ particles.

6. Results

6.1. The $K=1,2,3$ Design Scenarios

The proposed $\mathit{G}$-optimal designs generated by the GA implemented by [10] were reproduced by $\mathit{G}$-CEXCH in [24] and heretofore have remained the best known exact $\mathit{G}$-optimal designs for the second-order model and each of the $K=1,2,3$ design scenarios. We provide a comparison of the $n_{\mathrm{run}}=140$ PSO search results (for each scenario) to the $\mathit{G}$-GA designs of [10] in Figure 1. The data presented in this graphic are efficiencies of the $\mathit{G}$-PSO designs relative to the single best found $\mathit{G}$-GA designs. Therefore, scores over 100 indicate that the PSO generated design is an improvement over the GA generated design. For the $K=1$ scenarios, which are low-dimensional optimization from 3 to 9 dimensions, the graphic illustrates that all PSO runs are finding designs with equivalent $\mathit{G}$-score relative to the GA designs. In the second panel the results for the $K=2$ designs now shows a distribution in the $\mathit{G}$-releff score, but, for each scenario, the PSO produced designs with 90% relative efficiency or higher for every single run. Further, PSO found a better $\mathit{G}$-optimal design for design sizes $N=9,10,11,12$. The last panel of the graphic indicates that, for $K=3$ factors and all experiment sizes $N$, PSO has identified an improved $\mathit{G}$-optimal design than those currently known, and the improvements are non-negligible, with increases ranging from 2 to 8% efficiency.

Hernandez and Nachtsheim [11] provides a comparison of $\mathit{G}\left(I_{\lambda }\right)$-CEXCH, $\mathit{G}$-CEXCH and $\mathit{G}$-GA and their relative ability to generate designs with the highest/best $\mathit{G}$-optimality. We note that, in their data, [11] did not compare their results to the $\mathit{G}$-GA designs as published by [10], rather, they re-ran the GA algorithm $n_{\mathrm{run}}=200$ times in order to compare relative costs of the algorithm. Thus, the $\mathit{G}$-GA optimal designs in [11] are equal to or less efficient than those of [10]. Nonetheless, we can use their data to compare all algorithms vs. the GA designs of [10] (which are the best known to date).

Table 2. Relative efficiencies of $\mathit{G}$-optimal designs for each algorithm relative to GA generated designs of [10].

Design Scenario		Best Design Efficiency Relative to G-GA)
$K$	$N$	$\mathit{G}$-PSO	$\mathit{G}\left(I_{\lambda }\right)$-CEXCH	$\mathit{G}$-CEXCH
	3	100.0	100.0	100.0
	4	100.0	96.2	98.7
	5	100.0	97.0	98.7
1	6	100.0	100.0	100.0
	7	100.0	98.8	99.7
	8	100.0	94.7	99.4
	9	100.0	100.0	89.4
	6	100.3	94.1	96.5
	7	100.1	95.5	97.9
	8	100.0	94.7	99.7
2	9	100.3	95.8	97.0
	10	101.7	93.2	97.5
	11	101.0	97.0	94.0
	12	103.9	95.1	101.2
	10	101.6	95.4	93.1
	11	104.2	96.9	92.9
	12	103.8	90.3	90.7
3	13	103.2	99.9	92.9
	14	100.5	100.0	87.6
	15	102.5	100.1	98.5
	16	108.1	100.2	100.1

reports the efficiencies of the optimal design relative to the GA design found by [10] for each scenario and algorithm. In all cases, it can be seen that PSO finds the best design vs. all other algorithms.

Regarding computing cost, run time wall-clock for the entire set $140\times 21=2940$ of PSO searches over $K=1,2,3$ factors and the $N$ aforementioned was approximately 30 min (recall we are using Julia as well as running independent PSO searches in parallel on separate CPU cores). Hernandez and Nachtsheim [11] provides algorithm cost in the form of number of function evaluations over $n_{\mathrm{run}}=200$ runs of $\mathit{G}\left(I_{\lambda }\right)$-CEXCH, $\mathit{G}$-CEXCH and $\mathit{G}$-GA.

Table 3. Algorithm cost comparison to values reported by [11]. Table value is log10(# $f$ evaluations). The first PSO column reports the observed number of function evaluations in 140 runs of PSO. The PSO columns report an estimate of the expected number of function evaluations in 200 PSO runs with a 95% confidence interval.

Design Scenario		G-PSO			$\mathit{G}\left({\mathit{I}}_{\mathit{\lambda }}\right)$-CEXCH	G-CEXCH	G-GA
$K$	$N$	$n_{\mathrm{run}}=140$	$n_{\mathrm{run}}=200$		$n_{\mathrm{run}}=200$
			estimate	95% CI
	3	6.000	6.155	(6.145, 6.165)	6.0	5.5	6.9
	4	6.535	6.690	(6.684, 6.695)	6.4	5.7	7.0
	5	6.681	6.835	(6.831, 6.840)	6.6	5.8	7.1
1	6	6.226	6.381	(6.373, 6.388)	6.4	5.9	7.2
	7	6.685	6.840	(6.835, 6.845)	6.8	6.0	7.2
	8	6.761	6.916	(6.912, 6.921)	6.8	6.0	7.3
	9	6.405	6.560	(6.553, 6.566)	6.9	6.1	7.4
	6	7.088	7.243	(7.240, 7.246)	7.2	7.3	8.4
	7	7.086	7.241	(7.238, 7.244)	7.4	7.3	8.5
	8	7.042	7.197	(7.194, 7.200)	7.2	7.5	8.6
2	9	7.119	7.274	(7.271, 7.277)	7.0	7.5	8.6
	10	7.163	7.318	(7.315, 7.321)	7.3	7.6	8.7
	11	7.221	7.376	(7.373, 7.378)	7.4	7.6	8.7
	12	7.196	7.351	(7.348, 7.354)	7.7	7.7	8.7
	10	7.437	7.592	(7.590, 7.594)	8.0	8.7	9.6
	11	7.511	7.666	(7.664, 7.668)	7.8	8.8	9.7
	12	7.544	7.699	(7.697, 7.701)	7.9	8.8	9.7
3	13	7.538	7.692	(7.691, 7.694)	7.5	8.9	9.7
	14	7.543	7.698	(7.696, 7.700)	7.6	9.0	9.8
	15	7.515	7.670	(7.668, 7.671)	7.6	9.2	9.8
	16	7.556	7.711	(7.709, 7.713)	7.6	9.9	9.8

contains comparison of the [11] cost data to the cost of running $\mathit{G}$-PSO over our $n_{\mathrm{run}}=140$ runs. We provide a comparison of $\mathit{G}$-PSO cost on [11]’s $n_{\mathrm{run}}=200$ run scale by estimating the expected number of function evaluations and a 95% confidence interval via Poisson statistics. Table 3 shows that GA is the most expensive algorithm, and often via 2 orders of magnitude for the higher-dimensional problems. $\mathit{G}\left(I_{\lambda }\right)$-CEXCH is slightly more costly than $\mathit{G}$-CEXCH for the lower dimensional problems, but is cheaper for the higher dimensional problems (often by an order of magnitude). Last, the 95% confidence interval on the expected number of function evaluations in 200 PSO searches indicates that, in all cases, $\mathit{G}$-PSO has approximately the same cost as the new $\mathit{G}\left(I_{\lambda }\right)$-CEXCH (with some scenarios being slightly higher, but many scenarios being slightly lower cost than $\mathit{G}\left(I_{\lambda }\right)$-CEXCH).

These results illustrate that $\mathit{G}$-PSO is approximately the same cost as the state-of-the-art algorithm $\mathit{G}\left(I_{\lambda }\right)$-CEXCH for generating $\mathit{G}$-optimal designs, while PSO generates highly optimal designs more efficiently, as it is demonstrated here to produce the current best-known exact $\mathit{G}$-optimal designs for these scenarios.

6.2. The $K=4,5$ Design Scenarios

Due to the computational cost, this number of experimental factors is the highest that the design community has gone to date. Searching for $\mathit{G}$-optimal designs for more factors will take a considerable time/computing investment. Note, however, that $K$ does not define the dimension of the optimization search: the dimension is $NK$ so the largest problem we study here is $K=5$, $N=30$ which is an optimization search in a 150 dimensional search space.

Table 4. $\mathit{G}$-efficiencies of published $\mathit{G}$-optimal designs for $K=4,5$ factors. Relative efficiency of PSO generated design to published designs is shown in column 3 (over 100 means PSO generated a better design). Author [24] used the $\mathit{G}$-CEXCH while [11] used the $\mathit{G}\left(I_{\lambda }\right)$-CEXCH.

Design Scenario		PSO Peformance		Published Design Quality
$K$	$N$	$\mathit{G}$-PSO Design Relative Efficiency	$\mathit{G}$-PSO	CEXCH Algorithm	$\mathit{G}$-CEXCH
4	14	145.41	71.09	CEXCH	48.89
	17	105.36	73.90	$\mathit{G}\left(I_{\lambda }\right)$-CEXCH	70.14
	20	123.18	80.20	CEXCH	65.11
	24	106.5	85.95	CEXCH	81.05
5	21	177.26	68.67	CEXCH	38.74
	23	100.24	73.19	$\mathit{G}\left(I_{\lambda }\right)$-CEXCH	73.02
	26	103.92	75.31	CEXCH	72.47
	30	100.47	76.16	CEXCH	75.80

contains the $\mathit{G}$-efficiencies of the best $\mathit{G}\left(I_{\lambda }\right)$-CEXCH, $\mathit{G}$-CEXCH, and $\mathit{G}$-PSO generated design, as well as the relative efficiency of the $\mathit{G}$-PSO design to the indicated CEXCH algorithm. In all cases PSO is found to generate better $\mathit{G}$-optimal designs, and in some cases with a significant improvement. We present distributions of the $\mathit{G}$-releff scores of the PSO designs to the corresponding CEXCH generated designs in Figure 2. In all cases it can be seen that PSO found an equivalent or better $\mathit{G}$-optimal design (evidenced by relative efficiencies over 100). We note that each of the CEXCH algorithms was run $n_{\mathrm{run}}=200$ times in the work of [11] and the PSO searches were run $n_{\mathrm{run}}=210$ times (a number evenly distributed on 7 computer cores). The graphic further illustrates PSO’s ability to seek highly optimal designs each run, evidenced by the distributions of $\mathit{G}$-releff being tightly packed at or over 100% relative efficiency. For many scenarios, there is apparently a high probability that PSO would generate a design with 95% efficiency or better in a single run.

Regarding the significant improvements in designs, first for the $K=4, N=15$ case PSO provided a design with 145% relative efficiency to the best-known design. Second, for the $K=4, N=20$ scenario, PSO produced a design with 123% improved efficiency. Furthermore, last, for the $K=5, N=21$ scenario, PSO found a design with 177% relative efficiency. We contacted the authors of [24] to investigate these large discrepancies. For the $K=4, N=15$ and $K=5, N=21$ scenarios, it was confirmed that the designs published by [24] were mis-scored on the $\mathit{G}$-scale (personal email correspondence with Dr. Bradley Jones, 20 June 2020). Given that [24] employs a separate optimization search to score each candidate design on the $\mathit{G}$-scale, these results illustrate the consequences of failing to find the maximum prediction variance for a candidate design, and, to our opinion, support the approach of using the $5^K$ grid ${\mathbf{G}}_{\mathcal{X}}$ to score candidate designs.

The information for a proper time comparison for generating designs with $K=4, 5$ factors via PSO vs. the other approaches does not exist due to algorithms being run on different machines and computing languages. Nonetheless, for information we report what data do exist for computing times on the $K=4, N=17$ and $K=5, N=23$ design scenarios. Hernandez and Nachtsheim [11] report that they were able to run $\mathit{G}\left(I_{\lambda }\right)$-CEXCH $n_{\mathrm{run}}=200$ times on the $K=4, N=17$ in 20.13 h on their computing platform (approx 6.0m for each run). They were not able to run $\mathit{G}$-CEXCH as they estimated that 200 runs would have taken 25 days on their machine. Our approach (PSO, Julia, $n_{\mathrm{run}}=210$ parallel PSO runs [30 runs per 7 cores]) took about 3 h which translates into approximately 6 m per each individual run. Hernandez and Nachtsheim [11] report that they were able to run $\mathit{G}\left(I_{\lambda }\right)$-CEXCH $n_{\mathrm{run}}=200$ times on the $K=5, N=23$ in 25.07 h on their computing platform (approx 7.5m for each run). They were not able to run $\mathit{G}$-CEXCH as they estimated that 200 runs would have taken 166 days on their machine. Our approach (PSO, Julia, $n_{\mathrm{run}}=210$ parallel PSO runs [30 runs per 7 cores]) took about 20h which translates into approximately 40 m per each individual run. These results imply that PSO has more difficulty scaling to higher dimension. The reason why PSO took approximately 6 times longer for the $K=5$ scenario than for the $K=4$ scenario is that the respective $5^K$ grids used to score each candidate design during the search have $5^5=3125$ and $5^4=625$ points, respectively, and each particle (i.e., each candidate design) must have the $\mathit{G}$-score evaluated at each of these grid points, at each iteration of the algorithm. Nonetheless, we believe this time increase to be of little hindrance to scaling PSO to efficient searches for higher dimensional designs due to the use of Julia and parallel computing, and the additional cost is easily mitigated via CPUs with more computer cores.

6.3. Prediction Variance Properties of the New $\mathit{G}$-Efficient Designs

We publish fraction of design space (FDS) plots of the highly $\mathit{G}$-efficient designs generated in this work in Figure 3 [43]. The FDS plot communicates the distribution of the designs prediction variance as a fraction of the entire design space. These plots are often used when analyzing and choosing between competing designs. Following the prescription of [44], the FDS plots communicate relative prediction variance (that is, not scaled by $N$) so that a comparison of designs can be conducted across different experiment sizes $N$.

For the $K=2$ cases, a clear benefit can be seen by increasing the size of the experiment beyond the saturated $\mathit{G}$-optimal design $N=6$, but the benefit seems to yield diminishing returns by $N=9$. Therefore, if the researcher needs a $\mathit{G}$-optimal design for $K=2$ factors, the $N=9$ design constructed in this study is a good choice. For $K=3$ factor experiments, the $\mathit{G}$-optimal design constructed in this paper for $N=16$, while having a better $\mathit{G}$-efficiency, has worse prediction variance than the $N=14,15$ designs, and hence these appear to be better choices of designs for the $K=3$ factor experiment case. The FDS plot for $K=4$ factors indicates that we realize substantial improvements to prediction variance for increasing the sample size in the range of all sizes studied, and so for these experiments the published $N=24$$\mathit{G}$-efficient design is a superior choice. Last, the prediction variance properties of the $N=26$$\mathit{G}$-efficient design found during this study has approximately the same prediction variance signature on the FDS plot as the largest $N=30$ point $\mathit{G}$-efficient design, and so would give equal quality predictions at lower cost.

7. Conclusions

In this paper, we summarized the last two decades of research into algorithm development for generating exact $\mathit{G}$-optimal designs. To date, PSO has not been applied to this problem and so we proposed an extension of PSO to generate exact $\mathit{G}$-optimal designs. We then ran PSO on all published design scenarios. For the 21 $N$ scenarios on $K=1,2,3$ design factors studied in [10,11,24], PSO found better $\mathit{G}$-optimal designs for 12 of the higher dimension scenarios, some of these improvements offering 5% or better efficiency than currently known designs, see Table 2. On our computing setup, the total run time for all 2940 PSO searches on the 21 $K=1,2,3$ design scenarios was less than 30 min with the the $n_{\mathrm{run}}=140$ PSO searches for the $K=1$ scenarios taking less than a minute, and the $n_{\mathrm{run}}=140$ PSO searches for the $K=3$, $N=16$ (i.e. a 48 dimensional optimization) took approximately 4 min, or about 12 s for each individual PSO search (using $S=150$ particles). The use of Julia and parallel computing definitely helps to run many PSO searches very quickly. We believe this speed may enable realistic searches for good $\mathit{G}$-optimal designs for practitioners.

There are not many published proposed exact $\mathit{G}$-optimal designs for $K=4,5$ design factor cases due to the expense of searching for optimal designs for these scenarios. To date, there are only 4 design scenarios for each $K$ covered by [11,24]. In all cases, $\mathit{G}$-PSO found as good or better $\mathit{G}$-optimal designs than those currently known, see Table 4. We reiterate, however, that the curse-of-dimensionality is at play here and higher dimensional searches require more particles, which is a major cost driver of the algorithm, and therefore for $\mathit{G}$-opt searches PSO has some difficulty scaling (in time/cost) to higher $K$.

PSO is distinct from the various coordinate exchange algorithms studied in the following way. CEXCH starts with a randomly drawn design and then optimizes this design locally in ${\mathcal{X}}^N$. In CEXCH, there is no ‘intelligence’ which seeks to search ${\mathcal{X}}^N$ more globally for the best possible optimum. PSO, in contrast, uses $S$ randomly drawn design matrices which are searching ${\mathcal{X}}^N$ for the best possible fitness on the objective. Further, these candidate designs remember and communicate their best positions, have a tendency to want to revisit these locations, and this increases the likelihood of PSO to find the global optimum. In this sense, and due to the demonstrated computational cost, we propose that PSO should now be viewed as state-of-the art for generating optimal designs.

The genetic algorithm has enjoyed large success in academic research and is demonstrated, to this point, to be the superior algorithm for generating optimal designs with the best optimality scores. The GA, however, is computationally expensive, and so it is not a great tool for the practitioner of experiments to generate candidate optimal designs for their problem. This work has demonstrated that PSO, in repeated runs, performs as well or better than GA in generating optimal designs at a fraction of the cost. Further, PSO has produced better designs than the SOA algorithm, the $\mathit{G}\left(I_{\lambda }\right)$-CEXCH, but at approximately the same cost, see Table 3. Therefore, we propose that PSO should be applied further in academic research in generating optimal designs instead of GA.

In conclusion, PSO is hereby demonstrated to be superior to existing algorithms for generating exact $\mathit{G}$-optimal designs. It costs roughly the same as the state-of-the-art algorithm, the $\mathit{G}\left(I_{\lambda }\right)$-CEXCH, but is more efficient at finding the global optimal design in all studied cases.