Confidence Sets for Statistical Classification (II): Exact Confidence Sets

Abstract

Classification has applications in a wide range of fields including medicine, engineering, computer science and social sciences among others. Liu et al. (2019) proposed a confidence-set-based classifier that classifies a future object into a single class only when there is enough evidence to warrant this, and into several classes otherwise. By allowing classification of an object into possibly more than one class, this classifier guarantees a pre-specified proportion of correct classification among all future objects. However, the classifier uses a conservative critical constant. In this paper, we show how to determine the exact critical constant in applications where prior knowledge about the proportions of the future objects from each class is available. As the exact critical constant is smaller than the conservative critical constant given by Liu et al. (2019), the classifier using the exact critical constant is better than the classifier by Liu et al. (2019) as expected. An example is provided to illustrate the method.

1. Introduction

Classification has applications in a wide range of fields including medicine, engineering, computer science and social sciences among others. For overviews, the reader is referred to the books by [1,2,3,4,5]. In the recent paper, Liu et al. (2019) [6] proposed a new classifier based on confidence sets. It constructs a confidence set for the the unknown parameter c, the true class of each future object, and classifies the object as belonging to the set of classes given by the confidence set. Hence, this approach classifies a future object into a single class only when there is enough evidence to warrant this, and into several classes otherwise. By allowing classification of an object into potentially more than one class, this classifier guarantees a pre-specified proportion of correct classification among all future objects with a pre-specified confidence $\gamma$ about the randomness in the training data based on which the classifier is constructed.

However, the classifier of Liu et al. (2019) uses a conservative critical constant $\lambda$ and so the resultant confidence sets may be larger than necessary. The purpose of this paper is to determine the exact critical constant $\lambda$ and therefore to improve the classifier of Liu et al. (2019) in situations where one has prior knowledge about the proportions of the (infinite) future objects belonging to the k possible classes.

The layout of the paper is as follows. Section 2 gives a very brief review of the classifier of Liu et al. (2019), and then considers the determination of the exact critical constant $\lambda$ under the additional knowledge/assumption given above. An illustrative example is given in Section 3 to demonstrate the advantage of the improved classifier proposed in this paper when the additional assumption holds. Section 4 contains conclusions and discussions. Finally, some mathematical details are provided in the Appendix A. As the same setting and notation as in the work by Liu et al. (2019) are used, it is recommended to read this paper in conjunction with the one by Liu et al. (2019).

2. Methodology

2.1. Methodology

Let the p-dimensional data vector ${\mathbf{x}}_l={(x_{l1},\dots ,x_{lp})}^T$ denote the feature measurement on an object from the lth class, which has multivariate normal distribution $N({\mathit{\mu }}_l,{\mathsf{\Sigma }}_l)$, $l=1,\dots ,k$; here, k denotes the total number of classes, which is a known number. The available training dataset is given by $\mathcal{T}=\{{\mathbf{x}}_{l1},\dots ,{\mathbf{x}}_{l{n}_l};l=1,\dots ,k\}$, where ${\mathbf{x}}_{l1},\dots ,{\mathbf{x}}_{l{n}_l}$ are i.i.d. observations from the lth class with distribution $N({\mathit{\mu }}_l,{\mathsf{\Sigma }}_l)$, $l=1,\dots ,k$. The classification problem is to make inference about c, the true class of a future object, based on the feature measurement $\mathbf{y}={(y_1,\dots ,y_p)}^T$ observed on the object, which is only known to belong to one of the k classes and so follows one of the k multivariate normal distributions. In statistical terminology, c is the unknown parameter of interest that takes a possible value in the simple parameter space $C=\{1,\dots ,k\}$. We emphasize that c is treated as non-random in both the work of Liu et al. (2019) and here.

A classifier that classifies an object with measurement $\mathbf{y}$ into one single class in $C=\{1,\dots ,k\}$ can be regarded as a point estimator of c. The classifier of Liu et al. (2019) provides a set ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)\subseteq C$ as plausible values of c. Depending on $\mathbf{y}$ and the training dataset $\mathcal{T}$, ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ may contain only a single value, in which case $\mathbf{y}$ is classified into one single class given by ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$. When ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ contains more than one value in C, $\mathbf{y}$ is classified as possibly belonging to the several classes given by ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$. Hence, in statistical terms, the classifier uses the confidence set approach. The inherent advantage of the confidence set approach over the point estimation approach is the guaranteed $1-\alpha$ proportion of confidence sets that contain the true classes.

Specifically, the set ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)\subseteq C$ was constructed by Liu et al. (2019) as

$${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)=\left\{l\in C:{(\mathbf{y}-\widehat{{\mathit{\mu }}_l})}^T{\widehat{\mathsf{\Sigma }}}_l^{-1}(\mathbf{y}-\widehat{{\mathit{\mu }}_l})\le \lambda \right\},$$

(1)

where $\widehat{{\mathit{\mu }}_l}=\frac{1}{n_l}{\sum }_{m=1}^{n_l}{\mathbf{x}}_{lm}$ and ${\widehat{\mathsf{\Sigma }}}_l=\frac{1}{n_l-1}{\sum }_{m=1}^{n_l}({\mathbf{x}}_{lm}-\widehat{{\mathit{\mu }}_l}){({\mathbf{x}}_{lm}-\widehat{{\mathit{\mu }}_l})}^T,l=1,\dots ,k$, are, respectively, the usual estimators of the unknown ${\mathit{\mu }}_l$ and ${\mathsf{\Sigma }}_l$ based on the training dataset $\mathcal{T}=\{{\mathbf{x}}_{l1},\dots ,{\mathbf{x}}_{l{n}_l};l=1,\dots ,k\}$, and $\lambda$ is a suitably chosen critical constant whose determination is considered next. The intuition behind the definition of ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ in Equation (1) is that a future object $\mathbf{y}$ is likely to be from class l if and only if ${(\mathbf{y}-\widehat{{\mathit{\mu }}_l})}^T{\widehat{\mathsf{\Sigma }}}_l^{-1}(\mathbf{y}-\widehat{{\mathit{\mu }}_l})\le \lambda$.

Note that the proportion of the future confidence sets ${\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)$ ($j=1,2,\dots$) that include the true classes $c_j$ of ${\mathbf{y}}_j$ ($j=1,2,\dots$) is given by ${lim\; inf}_{N\to \infty }\frac{1}{N}{\sum }_{j=1}^N{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}$. Thus, it is desirable that

$$\underset{N\to \infty }{lim\; inf}\frac{1}{N}\sum _{j=1}^N{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}\ge 1-\alpha$$

(2)

where $1-\alpha$ is a pre-specified large (close to 1) proportion, e.g., 0.95. While the constraint in Equation (2) is difficult to deal with, Liu et al. (2019) showed that a sufficient condition for guaranteeing Equation (2) is

$$\underset{c_j\in C}{inf}{E}_{{\mathbf{y}}_j|\mathcal{T}}{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}\ge 1-\alpha$$

(3)

where $E_{{\mathbf{y}}_j|\mathcal{T}}$ denotes the conditional expectation with respect to the random variable ${\mathbf{y}}_j$ conditioning on the training dataset $\mathcal{T}$ (or, equivalently, $\{({\widehat{\mathit{\mu }}}_1,{\widehat{\mathsf{\Sigma }}}_1),\dots ,({\widehat{\mathit{\mu }}}_k,{\widehat{\mathsf{\Sigma }}}_k)\}$).

Since the value of the expression on the left hand side of the inequality in Equation (3) (and in Equation (2) as well) depends on $\mathcal{T}$ and $\mathcal{T}$ is random, the inequality in Equation (3) cannot be guaranteed for each observed $\mathcal{T}$. We therefore guarantee Equation (3) with a large (close to 1) probability $\gamma$ with respect to the randomness in $\mathcal{T}$:

$$P_{\mathcal{T}}\left\{\underset{c_j\in C}{inf}{E}_{{\mathbf{y}}_j|\mathcal{T}}{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}\ge 1-\alpha \right\}=\gamma ,$$

(4)

which in turn guarantees that

$$P_{\mathcal{T}}\left\{\underset{N\to \infty }{lim\; inf}\frac{1}{N}\sum _{j=1}^N{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}\ge 1-\alpha \right\}\ge \gamma .$$

(5)

Computer code in R was provided by Liu et al. (2019) to compute the $\lambda$ that solves Equation (4), which allows the confidence sets ${\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)$ in Equation (1) to be constructed for each future object.

The interpretation of Equations (5) and (6) below is that, based on one observed training dataset $\mathcal{T}$, one constructs confidence sets ${\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)$ for the $c_j$s of all future ${\mathbf{y}}_j$ ($j=1,2,\cdots$) and claims that at least $1-\alpha$ proportion of these confidence sets do contain the true $c_j$s. Then, we are $\gamma$ confident with respect to the randomness in the training dataset $\mathcal{T}$ that the claim is correct.

A natural question is how to find the exact critical constant $\lambda$ that solves the equation

$$P_{\mathcal{T}}\left\{\underset{N\to \infty }{lim\; inf}\frac{1}{N}\sum _{j=1}^N{I}_{\left\{c_j\in {\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)\right\}}\ge 1-\alpha \right\}=\gamma$$

(6)

which is an improvement to the conservative $\lambda$ that solves Equation (4) as given by Liu et al. (2019). Next, we show how to find the exact critical constant $\lambda$ under an additional assumption which is satisfied in some applications.

Assume that, among the N future objects that need to be classified, $N_l$ objects are actually from the lth class with the distribution $N({\mathit{\mu }}_l,{\mathsf{\Sigma }}_l)$, $l=1,\dots ,k$. The additional assumption we make is that

$$\underset{N\to \infty }{lim}\frac{N_l}{N}=r_l,l=1,\dots ,k$$

(7)

where the $r_l$s are assumed to be known constants in the interval $[0,1]$. Intuitively, this assumption means that we know the proportions of the future objects that belong to each of the k classes, even though we do not know the true class of each individual future object.

The assumption in Equation (7) is reasonable in some applications. For example, when screening for a particular disease among a specific population for preventive purpose, there are $k=2$ classes: having the disease ($l=1$) or not having the disease ($l=2$). If we know the prevalence of the disease, d, in the overall population, then $r_1=d$ and $r_2=1-d$, even though we do not know whether an individual subject has the disease or not.

It is shown in the Appendix A that, under the assumption in Equation (7), Equation (6) is equivalent to

$$P_{{\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\}}\left\{\sum _{l=1}^k{r}_l{P}_{{\mathbf{w}}_l|{\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\}}\left\{{({\mathbf{w}}_l-{\mathbf{u}}_l)}^T{\left(\frac{1}{n_l-1}\sum _{m=1}^{n_l-1}{\mathbf{v}}_{lm}{\mathbf{v}}_{lm}^T\right)}^{-1}({\mathbf{w}}_l-{\mathbf{u}}_l)\le \lambda \right\}\ge 1-\alpha \right\}=\gamma$$

(8)

where

$${\mathbf{w}}_l\sim N(\mathbf{0},I_p),{\mathbf{u}}_l\sim N(\mathbf{0},I_p/n_l),{\mathbf{v}}_{lm}\sim N(\mathbf{0},I_p),m=1,\cdots ,n_l-1$$

(9)

and all the ${\mathbf{w}}_l$s, ${\mathbf{u}}_l$s and ${\mathbf{v}}_{lm}$s are independent, $P_{{\mathbf{w}}_l|{\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\}}\{·\}$ denotes the conditional probability about ${\mathbf{w}}_l$ conditioning on $({\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\})$, and $P_{{\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\}}\{·\}$ denotes the probability about $({\mathbf{u}}_l,\left\{{\mathbf{v}}_{lm}\right\})$.

2.2. Algorithm for Computing the Exact $\lambda$

We now consider how to compute the critical constant $\lambda$ that solves Equation (8). Similar to Liu et al. (2019), this is accomplished by simulation in the following way. From the distributions given in Equation (9), in the sth repeat of simulation, $s=1,\dots ,S$, generate independent

$${\mathbf{u}}_l^s\sim N(\mathbf{0},I_p/n_l),{\mathbf{v}}_{l1}^s,\dots ,{\mathbf{v}}_{l(n_l-1)}^s\sim N(\mathbf{0},I_p);l=1,\dots ,k.$$

and find the $\lambda ={\lambda }_s$ so that

$$\sum _{l=1}^k{r}_l{P}_{{\mathbf{w}}_l|{\mathbf{u}}_l^s,\left\{{\mathbf{v}}_{lm}^s\right\}}\left\{{({\mathbf{w}}_l-{\mathbf{u}}_l^s)}^T{\left(\frac{1}{n_l-1}\sum _{m=1}^{n_l-1}{\mathbf{v}}_{lm}^s{{\mathbf{v}}_{lm}^s}^T\right)}^{-1}({\mathbf{w}}_l-{\mathbf{u}}_l^s)\le {\lambda }_s\right\}=1-\alpha .$$

(10)

Repeat this S times to get ${\lambda }_1,\dots ,{\lambda }_S$ and order these as ${\lambda }_{\left[1\right]}\le \dots \le {\lambda }_{\left[S\right]}$. It is well known (cf. [7]) that ${\lambda }_{\left[\gamma S\right]}$ converges to the required critical constant $\lambda$ with probability one as $S\to \infty$. Hence, ${\lambda }_{\left[\gamma S\right]}$ is used as the required critical constant $\lambda$ for a large S value, e.g., 10,000.

To find the ${\lambda }_s$ in Equation (10) for each s, we use simulation in the following way. Generate independent random vectors $\{{\mathbf{w}}_{lq}:q=1,\dots ,Q;l=1,\dots ,k\}$ from $N(\mathbf{0},I_p)$, where Q is the number of simulations for finding ${\lambda }_s$. For each given value of ${\lambda }_s>0$, the expression on the left-side of Equation (10) can be computed by approximating each of the k probabilities involved using the corresponding proportions out of the Q simulations. It is also clear that this expression is monotone increasing in ${\lambda }_s$. Hence, the ${\lambda }_s$ that solves Equation (10) can be found by using a searching algorithm; for example, the bi-section method is used in our R code. To approximate reasonably accurately the probabilities with the proportions, a large Q value, e.g., 10,000, should be used.

It is noteworthy from Equations (8) and (9) that $\lambda$ depends only on $\gamma ,\alpha ,p,k,n_1,\dots ,n_k,r_1,\dots ,r_k$ (and the numbers of simulations S and Q, which determine the numerical accuracy of $\lambda$ due to simulation randomness). It is also worth emphasizing that only one $\lambda$ needs to be computed based on the observed training dataset $\mathcal{T}$, which is then used for constructing the confidence sets ${\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)$ and classifying accordingly all future objects.

It is expected that larger values of S and Q will produce more accurate $\lambda$ value, one can use the method discussed by Liu et al. (2019) to assess how the accuracy of $\lambda$ depends on the values of S and Q. Similar to the work by Liu et al. (2019), it is recommended to set $S=$ 10,000 and $Q=$ 10,000 for reasonable computation time and accuracy of $\lambda$ due to simulation randomness.

3. An Illustrative Example

As in the work of Liu et al. (2019), the famous iris dataset introduced by Fisher (1936) [8] is used in this section to illustrate the method proposed in this paper. The dataset contains $k=3$ classes representing the three species/classes of Iris flowers (1 = setosa; 2 = versicolor; and 3 = virginica), and has $n_i=50$ observations from each class in $\mathcal{T}$. Each observation gives the measurements (in centimeters) of the four variables: sepal length and width, and petal length and width.

We focus on the case that only the first two measurements, sepal length and width, are used for classification in order to easily illustrate the method since the acceptance sets ${\mathcal{A}}_l=\left\{\mathbf{y}\in R^p:{(\mathbf{y}-\widehat{{\mathit{\mu }}_l})}^T{\widehat{\mathsf{\Sigma }}}_l^{-1}(\mathbf{y}-\widehat{{\mathit{\mu }}_l})\le \lambda \right\},l=1,2,3$ are two-dimensional and thus can be easily plotted in this case. Based on the fifty observations on $p=2$ measurements from each of the three classes, the ${\widehat{\mathit{\mu }}}_l$ and ${\widehat{\mathsf{\Sigma }}}_l$ were given by Liu et al. (2019).

For $\alpha =5\%$ and $\gamma =95\%$, the critical constant $\lambda$ that solves Equation (4) was computed by Liu et al. (2019) to be ${\lambda }_{con}=9.175$ using $S=$ 10,000 and $Q=$ 10,000. The corresponding acceptance sets, based on which the confidence set ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ in Equation (1) can be constructed directly (cf. [6]), are given by

$${\mathcal{A}}_l^{con}=\left\{\mathbf{y}\in R^p:{(\mathbf{y}-{\widehat{\mathit{\mu }}}_l)}^T{\widehat{\mathsf{\Sigma }}}_l^{-1}(\mathbf{y}-{\widehat{\mathit{\mu }}}_l)\le {\lambda }_{con}\right\},l=1,2,3$$

and plotted in Figure 1 by the dotted ellipsoidal region centered at ${\widehat{\mathit{\mu }}}_l$, marked by “+”.

Figure 1. The exact (solid) and conservative (dotted) acceptance sets for the three classes.

Now, assume that we have the knowledge about the proportions of the three species among all the Iris flowers $(r_1,r_2,r_3)$ and the Iris flowers that need to be classified reflect this composition. For the same $\alpha =5\%$, $\gamma =95\%$, $S=$ 10,000 and $Q=$ 10,000 and with, for example, $(r_1,r_2,r_3)=(0.3,0.4,0.3)$, the exact critical constant $\lambda$ that solves Equation (6) is computed by our R program to be ${\lambda }_{exa}=7.737$. As expected, ${\lambda }_{exa}$ is smaller than ${\lambda }_{con}$ and, as a result, the corresponding confidence set ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ in Equation (1) with $\lambda ={\lambda }_{exa}$ and acceptance sets ${\mathcal{A}}_l^{exa}=\left\{\mathbf{y}\in R^p:{(\mathbf{y}-{\widehat{\mathit{\mu }}}_l)}^T{\widehat{\mathsf{\Sigma }}}_l^{-1}(\mathbf{y}-{\widehat{\mathit{\mu }}}_l)\le {\lambda }_{exa}\right\},l=1,2,3$, are also smaller than the ${\mathcal{A}}_l^{con}$ given by Liu et al. (2019).

The acceptance sets ${\mathcal{A}}_l^{exa},l=1,2,3$ are plotted in Figure 1 by the solid ellipsoidal regions. For example, if a future object has $\mathbf{y}=(4.79,2.35)$, marked by a solid dot in Figure 1, then the conservative confidence set of Liu et al. (2019) classifies the object as from Classes 2 and 3 since this $\mathbf{y}$ belongs to both ${\mathcal{A}}_2^{con}$ and ${\mathcal{A}}_3^{con}$. However, the new exact confidence set of this paper classifies the object as from Class 2 only since this $\mathbf{y}$ belongs to ${\mathcal{A}}_2^{exa}$ but not ${\mathcal{A}}_1^{exa}$ or ${\mathcal{A}}_3^{exa}$. This demonstrates the advantage of the new confidence set using ${\lambda }_{exa}$ in this paper over the conservative confidence set using ${\lambda }_{con}$ by Liu et al. (2019). We have also computed the value of ${\lambda }_{exa}$ for several other given $(r_1,r_2,r_3)$. For example, ${\lambda }_{exa}=7.706$ for $(r_1,r_2,r_3)=(1/3,1/3,1/3)$, ${\lambda }_{exa}=7.865$ for $(r_1,r_2,r_3)=(0.1,0.45,0.45)$, and ${\lambda }_{exa}=8.019$ for $(r_1,r_2,r_3)=(0.1,0.7,0.2)$. The conservative ${\lambda }_{con}=9.175$ is considerably, ranging from 14% to 19%, larger than these ${\lambda }_{exa}$ values.

One can download from http://www.personal.soton.ac.uk/wl/Classification/ the R computer program ExactConfidenceSetClassifier.R that implements this simulation method of computing the critical constant ${\lambda }_{exa}$. The computation of one ${\lambda }_{exa}$ using $(S,Q)=$ (10,000, 10,000) takes about 13 h on an ordinary Window’s PC (Core(TM2) Duo CPU P8400@2.26 GHz).

However, it must be emphasized the new confidence set is valid only if the assumption in Equation (7) is true. If the assumption does not hold, then the conservative confidence set of Liu et al. (2019) should be used in order for the statement in Equation (5) to hold.

4. Conclusions

The probability statement in Equation (5) allows that the confidence sets by Liu et al. (2019) have the nice interpretation that, with confidence level $\gamma$ about the randomness in the training dataset $\mathcal{T}$, at least $1-\alpha$ proportion of the confidence sets ${\mathcal{C}}_{\mathcal{T}}\left({\mathbf{y}}_j\right)$, $j=1,2,\dots$ contain the true classes $c_j$, $j=1,2,\dots$ of the future objects ${\mathbf{y}}_j$, $j=1,2,\dots$. However, the confidence set given by Liu et al. (2019) is conservative in that the $\lambda$ in the confidence set in Equation (1) is computed to solve the equation in Equation (4), which implies the constraint in Equation (5). This paper considers how to compute the $\lambda$ in the confidence set in Equation (1) so that the probability in Equation (5) is equal to $\gamma$, i.e. from the Equation (6). The confidence sets using the $\lambda$ that solves the Equation (6) have the confidence level equal to $\gamma$ and so are exact. We show that this can be accomplished under the extra assumption given in Equation (7), which may be sensible in some applications.

As the ${\lambda }_{exa}$ that solves Equation (6) is smaller than the ${\lambda }_{con}$ that solves Equation (4) used by Liu et al. (2019), the new confidence sets are smaller and so better than the confidence sets given by Liu et al. (2019).

One wonders whether there are other sensible assumptions that allow the $\lambda$ to be solved from Equation (6). This warrants further research.

If ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ for a future object $\mathbf{y}$ is empty then, since $\mathbf{y}$ must be from one of the k classes, ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ can be augmented to include the class that has the largest posterior probability using the naive Bayesian classifier as in the work by Liu et al. (2019). The probability statement in Equation (5) clearly holds under this augmentation to ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ only when ${\mathcal{C}}_{\mathcal{T}}\left(\mathbf{y}\right)$ is empty.

There are applications in which information about the proportions $r_l$ would be known with uncertainty. For example, the training set may be a representative sample from the population and as such the proportion of each class can be estimated, or the proportions might have been estimated by a previous independent auxiliary dataset. If one replaces the $r_l$s in Equation (8) by these estimates then the $\lambda$ solved in Equation (8) will depend on these estimates and so be random. As a result, the probability statement in Equation (5) is no longer valid. How to deal with these applications warrants further research.

Finally, the classifier of Liu et al. (2019) is developed from the idea of Lieberman et al. [9,10]. The same idea was also used by, for example, Mee et al. (1991) [11], Han et al. (2016) [12], Liu et al. (2016) [13] and Peng et al. (2019) [14], who all used conservative critical constants as did Liu et al. (2019). The idea of this paper can be applied to all these works to compute exact critical constants under suitable extra assumptions.