Analysis on Optimal Error Exponents of Binary Classification for Source with Multiple Subclasses

Abstract

We consider a binary classification problem for a test sequence to determine from which source the sequence is generated. The system classifies the test sequence based on empirically observed (training) sequences obtained from unknown sources $P_1$ and $P_2$. We analyze the asymptotic fundamental limits of statistical classification for sources with multiple subclasses. We investigate the first- and second-order maximum error exponents under the constraint that the type-I error probability for all pairs of distributions decays exponentially fast and the type-II error probability is upper bounded by a small constant. In this paper, we first give a classifier which achieves the asymptotically maximum error exponent in the class of deterministic classifiers for sources with multiple subclasses, and then provide a characterization of the first-order error exponent. We next provide a characterization of the second-order error exponent in the case where only $P_2$ has multiple subclasses but $P_1$ does not. We generalize our results to classification in the case that $P_1$ and $P_2$ are a stationary and memoryless source and a mixed memoryless source with general mixture, respectively.

1. Introduction

1.1. Background

The problem of learning sources from training sequences and estimating the source from which a test sequence is generated is known as a classification problem. Recently, this problem has been actively studied in a field such as machine learning, and it is desirable to conduct studies that guarantee the performance of the system. In the field of information theory, studies have been conducted mainly to analyze the performance of optimal tests. When the number of sources is two, the binary classification problem can be regarded as a binary hypothesis testing problem, using training sequences. In the setting of binary hypothesis testing, it is assumed that the sources are known, but in real-world applications, the sources are not known in general. Therefore, it is of importance to consider the binary classification problem.

Hypothesis testing includes approaches such as the Bayesian test [1,2] and the Neyman–Pearson test [3,4,5]. In this paper, we take the latter approach to formulate the best asymptotic error exponent (the exponential part of an error probability).

There are a lot of studies related to the classification problem. We state important points, which are deeply connected to this study in some previous studies. Gutman [3] has shown that type-based (empirical distribution-based) tests asymptotically achieve the maximum type-II error exponent for stationary Markov sources, while the type-I error probability exponentially converges to zero as the length of a test sequence goes to infinity. Zhou et al. [5] derived second-order approximations of the maximum type-I error exponent for stationary and memoryless sources when the type-II error probability is upper bounded by a small constant. On the other hand, for the hypothesis testing problem, Han and Nomura [6] characterized a first-order maximum error exponent when each sequence is generated from a mixed memoryless source, which is a mixture of stationary and memoryless sources. In addition, they also characterized a second-order maximum error exponent in the case where one source is a stationary and memoryless source and the other source is a mixed memoryless source.

1.2. Contributions

In this paper, we investigate the binary classification problem for stationary memoryless sources with multiple subclasses. The class of sources with multiple subclasses is important in binary classification because there are many such settings in real-world applications. For example, newspaper articles with science headlines consist of topics of physics, chemistry, biology, etc. We assume that sources (subclasses) are characterized by a mixture with some unknown prior distribution (cf. Equation (2)), and the overall sources can be regarded as mixed memoryless sources [6]. The purpose of this paper is to characterize the first- and second-order maximum error exponents in a single-letter form (the term “single-letter form” means an expression which does not depend on lengths of sequences n or N (cf. the formulas for error exponents in Theorems 2–4)).

To this end, we generalize Gutman’s classifier [3], which was shown to be first- and second-order optimal for memoryless sources (with no multiple subclasses) in [5]. This classifier uses training sequences from one of the two sources, as in [3,4,5], making a type-based decision for a source (subclass) with the smallest skewed Jensen–Shannon divergence [7] among the subclasses. We show that this classifier asymptotically achieves the maximum type-II error exponent in the class of deterministic classifiers for a given pair of distributions when the type-I error probability decays exponentially fast for all pairs of distributions in Theorem 1. We also demonstrate that the structure of this classifier leads to a reversed and more relaxed relation; the maximization of the type-I error exponent when the type-II error probability is upper bounded by a small constant $ϵ$$(0\le ϵ<1)$ for sources with multiple subclasses in Theorem 2. In addition, using the Berry–Esseen theorem [8], we derive the second-order maximum error exponent in the case where only one of sources has subclasses in Theorem 3. Finally, the fact that the classifier uses the test sequence from one of two sources motivates us to consider a more general case; the first source is a source with no multiple subclasses, but the second source is given by a general mixture [6,9]. That is, the number of subclasses is not necessarily finite, and the prior distribution of subclasses may not be discrete (cf. Equation (75)). We give characterizations of the first- and second-order maximum error exponents in Theorem 4.

1.3. Related Work

Ziv [10] proposed a classifier based on empirical entropy and discussed the relationship between binary classification and universal source coding. Hsu and Wang [4] characterized the maximum error exponent with mismatched empirically observed statistics. In their achievability proof, a generalization of Gutman’s classifier is also used. Kelly et al. [11] investigated binary classification with large alphabets. Unnikrishnan and Huang [12] investigated the type-I error probability of binary classification using the analysis of weak convergence. Generalizing the binary classification problem, He et al. [13] discussed the binary distribution detection problem, in which a different generalization of Gutman’s classifier is also discussed.

There are also studies which take a Bayesian approach. Merhav and Ziv [1] analyzed the weighted sum of type-I and type-II error probabilities, and subsequently, Saito and Matsushima [2,14] gave a different result via the analysis for the Bayes code.

1.4. Organization

The rest of this paper is organized as follows: In Section 2, we define the notation used in this paper and describe the details of the source and system model. Moreover, we state the problem setting, defining the first- and second-order maximum error exponents. In Section 3, we first give a classifier which achieves the asymptotically maximum error exponent in the class of deterministic classifiers for sources with multiple subclasses. Next, we characterize the first- and second-order maximum error exponents and the detailed proofs for the first-order representation. In Section 4, we generalize the obtained results to the classification of a mixed memoryless source with general mixture. In Section 5, we present numerical examples. Finally, in Section 6, we provide some concluding remarks and future work.

2. Problem Formulation

2.1. Notation

The set of non-negative real numbers is denoted by ${\mathbb{R}}^{+}$. Calligraphic $\mathcal{X}$ stands for a finite alphabet. Upper-case X denotes a random variable taking values in $\mathcal{X}$, and lower-case $x\in \mathcal{X}$ denotes its realization. Throughout this paper, logarithms are of base e. For integers a and b such that $a\le b,[a,b]$ denotes the set $\{a,a+1,\cdots ,b\}$. The set of all probability distributions on a finite set $\mathcal{X}$ is denoted as $\mathcal{P}\left(\mathcal{X}\right)$. Notation regarding the method of types [15] is as follows: Given a vector $x_1^n=(x_1,x_2,\cdots ,x_n)\in {\mathcal{X}}^n$, the type is denoted as

$$\begin{array}{c}\hfill q_{x_1^n}\left(a\right)=\frac{1}{n}\sum _{i=1}^n\mathbf{1}\{x_i=a\},a\in \mathcal{X}.\end{array}$$

(1)

The set of types formed from length-n sequences with alphabet $\mathcal{X}$ is denoted as ${\mathcal{P}}_n\left(\mathcal{X}\right)$. The probability that n independent drawings from a probability distribution $Q\in \mathcal{P}\left(\mathcal{X}\right)$ give $\mathit{x}\in {\mathcal{X}}^n$ is denoted by $Q\left(\mathit{x}\right)$.

2.2. Source with Multiple Subclasses

Consider a source consisting of multiple subclasses. Each subclass is distributed according to a given probability (weight). Let ${\left\{P_1^i\right\}}_{i\in \mathcal{S}}$ be a family of probability distributions on a finite alphabet $\mathcal{X}$, where $\mathcal{S}=\{1,\cdots ,s\}$ is a probability space with probability measure $v\left(i\right),i\in \mathcal{S}$. That is, the probability of $\mathit{x}\in {\mathcal{X}}^n$ is given by

$$\begin{array}{c}\hfill P_1\left(\mathit{x}\right)=\sum _{i=1}^{s}v\left(i\right)P_1^i\left(\mathit{x}\right),\end{array}$$

(2)

where the i-th subclass $P_1^i$ is a stationary and memoryless source. That is, for $\mathit{x}=(x_1,x_2,\cdots ,x_n)\in {\mathcal{X}}^n$,

$$\begin{array}{c}\hfill P_1^i\left(\mathit{x}\right)=\prod _{j=1}^n{P}_1^i\left(x_j\right)\end{array}$$

(3)

(for notational simplicity, we denote both the multi-letter and single-letter probabilities by $P_1^i$ with a slight abuse of notation).

In view of (2), the sequence $\mathit{x}$ can be regarded as an output from a mixed memoryless source $P_1(·)$, and it is called a test sequence. Similarly, let ${\left\{P_2^i\right\}}_{i\in \mathcal{U}}$ be a family of probability distributions, where $\mathcal{U}=\{1,\cdots ,u\}$ is a probability space with probability measure $w\left(i\right),i\in \mathcal{U}$. For these mixed sources, if the sources are known, that is, the addressed problem is hypothesis testing, the first- and second-order error exponents were analyzed by Han and Nomura [6]. In this paper, we assume that the sources are unknown and training sequences are available to learn about the source. Sets of training sequences are denoted by ${\mathit{t}}_1=\{{\mathit{t}}_1^1,\cdots ,{\mathit{t}}_1^s\}$ and ${\mathit{t}}_2=\{{\mathit{t}}_2^1,\cdots ,{\mathit{t}}_2^u\}$, where ${\mathit{t}}_i^j\in {\mathcal{X}}^N$ of length N is output from subclass j and $N=\lceil n\gamma \rceil$ for some fixed $\gamma \in {\mathbb{R}}^{+}$. Then, the joint probabilities of training sequences are, respectively,

$$\begin{array}{c}\hfill P_1\left({\mathit{t}}_1\right)=\prod _{i=1}^s{P}_1^i\left({\mathit{t}}_1^i\right),\end{array}$$

(4)

$$\begin{array}{c}\hfill P_2\left({\mathit{t}}_2\right)=\prod _{j=1}^u{P}_2^j\left({\mathit{t}}_2^j\right).\end{array}$$

(5)

We define the class of sources with multiple subclasses on a probability space $\mathcal{S}$ with probability measure $v(·)$ as

$$\begin{array}{c}\hfill {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right):=\left\{P={\{P^i,v\left(i\right)\}}_{i\in \mathcal{S}}:P^i\in \mathcal{P}\left(\mathcal{X}\right)\right\},\end{array}$$

(6)

which means $P_1\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)$, where the set of weights ${\left\{v\left(i\right)\right\}}_{i\in \mathcal{S}}$ is implicitly fixed. Similarly, we define the class of sources with multiple subclasses on a probability space $\mathcal{U}$ with probability measure $w(·)$ as

$$\begin{array}{c}\hfill {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right):=\left\{P={\{P^i,w\left(i\right)\}}_{i\in \mathcal{U}}:P^i\in \mathcal{P}\left(\mathcal{X}\right)\right\},\end{array}$$

(7)

which means $P_2\in {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$.

2.3. System Model

The binary classification problem assumed in this paper is shown in Figure 1. It consists of two phases: (I) learning phase and (II) classification phase. We explain the details of each phase.

(I) Learning phase: Determine the classifier by learning with the training sequences ${\mathit{t}}_1=\{{\mathit{t}}_1^1,\cdots ,{\mathit{t}}_1^s\}$ and ${\mathit{t}}_2=\{{\mathit{t}}_2^1,\cdots ,{\mathit{t}}_2^u\}$ generated from unknown source $P_1\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)$ and $P_2\in {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, respectively.

(II) Classification phase: It represents the phase in which we judge whether the test sequence $\mathit{x}\in {\mathcal{X}}^n$ generated from $P_1\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)$ or $P_2\in {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ according to the classifier determined in (I).

2.4. Maximum Error Exponent

In this section, we define two error probabilities that arise in a binary classification problem and formulate the maximum error exponents. In the binary classification problem, a test is described as a partition of the space ${\varphi }^n:{\mathcal{X}}^n\times {\mathcal{X}}^{(s+u)N}\to \{1,2\}$. The type-I and type-II error probabilities of a given test ${\varphi }^n$ are denoted as ${\beta }_1\left({\varphi }^n\right|P_1,P_2)$ and ${\beta }_2\left({\varphi }^n\right|P_1,P_2)$, respectively. That is,

$$\begin{array}{c}\hfill {\beta }_1\left({\varphi }^n\right|P_1,P_2):={\mathbb{P}}_1\{{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=2\},\end{array}$$

(8)

$$\begin{array}{c}\hfill {\beta }_2\left({\varphi }^n\right|P_1,P_2):={\mathbb{P}}_2\{{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=1\}.\end{array}$$

(9)

Here, ${\mathbb{P}}_{\theta }\{·\}$ is the joint probability of training and testing sequences when the underlying parameter is $\theta$, given by

$$\begin{array}{cc}\hfill & {\mathbb{P}}_1\{{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=\ell \}:=\sum _{(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2):{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=\ell }{P}_1\left(\mathit{x}\right)\prod _{i=1}^s{P}_1^i\left({\mathit{t}}_1^i\right)\prod _{j=1}^u{P}_2^j\left({\mathit{t}}_2^j\right)\ell \in \{1,2\},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & {\mathbb{P}}_2\{{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=\ell \}:=\sum _{(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2):{\varphi }^n(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)=\ell }{P}_2\left(\mathit{x}\right)\prod _{i=1}^s{P}_1^i\left({\mathit{t}}_1^i\right)\prod _{j=1}^u{P}_2^j\left({\mathit{t}}_2^j\right)\ell \in \{1,2\}.\hfill \end{array}$$

(11)

We consider the problem of maximizing the type-I error exponent when the type-II error probability is upper bounded by a small constant $ϵ\in [0,1)$. In this study, we characterize the following quantities (the first- and second-order maximum type-I error exponent).

Definition 1

(First-order maximum error exponent). For any pair of distributions $\mathit{P}=(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ and $ϵ\in [0,1)$, we define

$$\begin{array}{cc}\hfill \widehat{\lambda }\left(ϵ\right)& :=sup\left\{\lambda \in {\mathbb{R}}^{+}|\begin{array}{c}\exists {\left\{{\varphi }^n\right\}}_{n=1}^{\infty }\mathrm{s}.\mathrm{t}.\mathrm{for}\mathrm{all}\mathrm{sufficiently}\mathrm{large}n\hfill \\ {\beta }_1\left({\varphi }^n\right|\tilde{\mathit{P}})\le exp(-n\lambda )(\forall \tilde{\mathit{P}}\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)),\hfill \\ {\displaystyle \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\varphi }^n\right|\mathit{P})\le ϵ}\hfill \end{array}\right\},\hfill \end{array}$$

(12)

where the weights of $\tilde{\mathit{P}}=({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ are the sames as the weights of $\mathit{P}=(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$.

Definition 2

(Second-order maximum error exponent). For any pair of distributions $\mathit{P}=(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ and $ϵ\in [0,1)$, we define

$$\begin{array}{c}\hfill \widehat{r}(ϵ,\lambda ):=sup\left\{r|\begin{array}{c}\exists {\left\{{\varphi }^n\right\}}_{n=1}^{\infty }\mathrm{s}.\mathrm{t}.\mathrm{for}\mathrm{all}\mathrm{sufficiently}\mathrm{large}n\hfill \\ {\beta }_1\left({\varphi }^n\right|\tilde{\mathit{P}})\le exp(-n\lambda -\sqrt{n}r)(\forall \tilde{\mathit{P}}\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)),\hfill \\ {\displaystyle \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\varphi }^n\right|\mathit{P})\le ϵ}\hfill \end{array}\right\},\end{array}$$

(13)

where the weights of $\tilde{\mathit{P}}=({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ are the sames as the weights of $\mathit{P}=(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$.

Remark 1.

In (12) and (13), the type-I error probability is constrained for any $\tilde{\mathit{P}}\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ for technical reasons. In more detail, this condition is required in the proof of the converse part. This condition was also imposed by Gutman [3], Hsu and Wang [4], and Zhou et al. [5].

In Definitions 1 and 2, we focus on universal tests that perform well for all pairs of distributions with respect to the type-I error probability, and at the same time, constrain the type-II error probability with respect to a particular pair of distributions. We obtain the same result when the weights of $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ are not fixed.

3. Main Result

3.1. A Test to Achieve Maximum Error Exponent

The rule for estimating whether a test sequence generated from $P_1\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)$ or $P_2\in {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ is called a decision rule. One of the goals of the classification problem is to design an optimal decision rule which achieves a maximum error exponent based on training sequences. In this section, we present a decision rule that asymptotically achieves the maximum type-II error exponent for any pair of distributions when the type-I error exponent is lower bounded by a constant for all pairs of distributions (cf. Theorem 1).

To define a test that is asymptotically optimum, we define two generalizations of the Jensen–Shannon divergence. These generalizations are related to some variational definitions in [7,16]. For any pair of distributions $(Q_1,Q_2)\in \mathcal{P}{\left(\mathcal{X}\right)}^2$ and any number $\gamma \in {\mathbb{R}}^{+}$, let the generalized Jensen–Shannon divergence be

$$\begin{array}{cc}\hfill & \mathrm{GJS}(Q_1,Q_2,\gamma ):=\gamma D\left(Q_1\left|\right|\frac{\gamma Q_1+Q_2}{1+\gamma }\right)+D\left(Q_2\left|\right|\frac{\gamma Q_1+Q_2}{1+\gamma }\right),\hfill \end{array}$$

(14)

where $D\left(p\right|\left|q\right)$ denotes the Kullback–Leibler divergence for $p\in \mathcal{P}\left(\mathcal{X}\right)$ and $q\in \mathcal{P}\left(\mathcal{X}\right)$ defined as

$$\begin{array}{c}\hfill D\left(p\right|\left|q\right):=\sum _{i\in \mathcal{X}}p\left(i\right)log\frac{p\left(i\right)}{q\left(i\right)}.\end{array}$$

(15)

The generalized Jensen–Shannon divergence $\mathrm{GJS}(Q_1,Q_2,\gamma )$ corresponds to a skewed $\alpha$-Jensen–Shannon divergence for $\alpha =\frac{\gamma }{1+\gamma }$. Additionally, for $(Q_1,Q_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{X}\right)$, we define the minimized generalized Jensen–Shannon divergence by

$$\begin{array}{c}\hfill \mathrm{MGJS}(Q_1,Q_2,\gamma ):=\underset{i\in \mathcal{S}}{min}\mathrm{GJS}(Q_1^i,Q_2,\gamma ).\end{array}$$

(16)

Given a threshold $\lambda \in {\mathbb{R}}^{+}$ (including $\lambda =0$), the decision rule to achieve the maximum error exponent is given by

$$\begin{array}{c}\hfill {\mathrm{\Lambda }}_2^n=\left\{(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)\in {\mathcal{X}}^n\times {\mathcal{X}}^{sN}\times {\mathcal{X}}^{uN}|\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )\ge \tilde{\lambda }\right\},\end{array}$$

(17)

where $\tilde{\lambda }=\lambda +\eta \left(n\right)$, $\eta \left(n\right):=\frac{2logs+\left|\mathcal{X}\right|log(n+N+1)}{n}$ and ${\mathrm{\Lambda }}_i^n,i\in \{1,2\}$ is the set of $(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)$ determined to be class i by the test ${\mathrm{\Lambda }}^n$. By definition, the discriminant function $\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )$, appearing on the right-hand side of (17), can also be expressed as

$$\begin{array}{c}\hfill \mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )=\underset{i\in \mathcal{S}}{min}\left\{\gamma D(q_{{\mathit{t}}_1}\left|\right|q_{{\mathbf{y}}_1^i})+D(q_{\mathit{x}}\left|\right|q_{{\mathbf{y}}_1^i})\right\},\end{array}$$

(18)

where ${\mathbf{y}}_1^i:=\mathit{x}{\mathit{t}}_1^i$. From (17) and (18), ${\mathrm{\Lambda }}^n$ is a type-based test and implicitly depends on $\lambda$. In addition, this test uses training sequences asymmetrically; only sequence ${\mathit{t}}_1$ is used, but not ${\mathit{t}}_2$ (cf. refs. [3,4,5]).

Theorem 1.

For any given $\lambda \in {\mathbb{R}}^{+}$ and any sequence of tests $\left\{{\varphi }^n\right\}$ such that ${\beta }_1\left({\varphi }^n\right|\tilde{\mathit{P}})\le exp(-n\lambda ),\forall \tilde{\mathit{P}}=({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, the sequence of tests $\left\{{\mathrm{\Lambda }}^n\right\}$ given by (17) satisfies for any pair of distributions $\mathit{P}=(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$,

$$\begin{array}{cc}\hfill {\beta }_1\left({\mathrm{\Lambda }}^n\right|\tilde{\mathit{P}})& \le exp(-n\lambda )(\forall \tilde{\mathit{P}}\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}-\frac{1}{n}log{\beta }_2\left({\varphi }^n\right|\mathit{P})& \le \underset{n\to \infty }{lim}-\frac{1}{n}log{\beta }_2\left({\mathrm{\Lambda }}^n\right|\mathit{P}).\hfill \end{array}$$

(20)

Proof. 

Equation (19) is derived in Section 3.3.1. The proof of (20) follows from Corollary 1. Although there is a deviation between the exponents in Corollary 1 and for the test ${\mathrm{\Lambda }}^n$, the deviation vanishes asymptotically. □

Theorem 1 shows that the test ${\mathrm{\Lambda }}^n$ can asymptotically achieve the maximum type-II error exponent among the tests ${\varphi }^n$ for which the type-I error exponent is greater than or equal to $\lambda$. This test also has a reversed and more relaxed property; it achieves $\widehat{\lambda }\left(ϵ\right)$, the maximum type-I error exponent when the type-II error probability is upper bounded by a constant $ϵ(0\le ϵ<1)$ (see the achievability proof of Theorem 2 in Section 3.3.1).

3.2. First-Order Maximum Error Exponent

In this section, we characterize the first-order maximum error exponent in a single-letter form for sources with multiple subclasses.

Theorem 2.

For any pair of distributions $(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, we have

$$\begin{array}{c}\hfill \widehat{\lambda }\left(ϵ\right)=sup\left\{\overline{\lambda }|\sum _{\{j\in \mathcal{U}:\mathrm{MGJS}(P_1,P_2^j,\gamma )<\overline{\lambda }\}}w\left(j\right)\le ϵ\right\}.\end{array}$$

(21)

It should be noted that $\widehat{\lambda }\left(ϵ\right)$ depends on ${\left\{w\left(j\right)\right\}}_{j\in \mathcal{U}}$, but not on ${\left\{v\left(i\right)\right\}}_{i\in \mathcal{S}}$.

Proof. 

The proof is provided in Section 3.3. □

Remark 2.

If $\mathcal{S}$ and $\mathcal{U}$ are singletons (that is, $s=1$ and $u=1$), Theorem 2 reduces to the following formula given by Zhou et al. [5]:

$$\begin{array}{c}\hfill \widehat{\lambda }\left(ϵ\right)=\mathrm{GJS}(P_1,P_2,\gamma )(0\le \forall ϵ<1),\end{array}$$

(22)

which means that $\widehat{\lambda }\left(ϵ\right)$ does not depend on ϵ and the strong converse holds in this case, unlike in the case $\left|\mathcal{S}\right|,\left|\mathcal{U}\right|>1$. On the other hand, for general $\mathcal{S}$ and $\mathcal{U}$ but in the spacial case of $ϵ=0$, formula (21) reduces to

$$\begin{array}{c}\hfill \widehat{\lambda }\left(0\right)=\underset{i\in \mathcal{S},j\in \mathcal{U}}{min}\mathrm{GJS}(P_1^i,P_2^j,\gamma ).\end{array}$$

(23)

3.3. Proof of Theorem 2

We divide the proof of Theorem 2 into two parts: the achievability (direct) part and the converse part.

3.3.1. Achievability Part

In the achievability proof, we use the type-based test ${\mathrm{\Lambda }}^n$ given by (17). Fix any

$$\begin{array}{c}\hfill \lambda <sup\left\{\overline{\lambda }|\sum _{\{j\in \mathcal{U}:\mathrm{MGJS}(P_1,P_2^j,\gamma )<\overline{\lambda }\}}w\left(j\right)\le ϵ\right\}.\end{array}$$

(24)

Then, for any pair of distributions $(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ and for all pairs of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, we show

$$\begin{array}{cc}\hfill {\beta }_1\left({\mathrm{\Lambda }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2)& \le exp(-n\lambda ),\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\mathrm{\Lambda }}^n\right|P_1,P_2)& \le ϵ.\hfill \end{array}$$

(26)

First, we prove (25). For preliminaries, we define the following sets used in the proof:

$$\begin{array}{cc}\hfill {\mathcal{T}}_n\left(q_{\mathit{x}}\right)& :=\left\{{\mathit{x}}^{\prime }\in {\mathcal{X}}^n:q_{{\mathit{x}}^{\prime }}=q_{\mathit{x}}\right\},\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\tilde{\mathrm{\Lambda }}}_2^n& :=\left\{(\mathit{x},{\mathit{t}}_1):\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )\ge \tilde{\lambda }\right\},\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \mathrm{\Gamma }\left(\mathrm{\Lambda }\right)& :=\left\{(q_{\mathit{x}},q_{{\mathit{t}}_1}):(\mathit{x},{\mathit{t}}_1)\in \mathrm{\Lambda }\right\},\hfill \end{array}$$

(29)

where ${\tilde{\mathrm{\Lambda }}}_2^n$ is the projection of ${\mathrm{\Lambda }}_2^n\subseteq {\mathcal{X}}^n\times {\mathcal{X}}^{sN}\times {\mathcal{X}}^{uN}$ onto the space ${\mathcal{X}}^n\times {\mathcal{X}}^{sN}$. To evaluate the probability of a source sequence being in ${\mathcal{T}}_n\left(q_{\mathit{x}}\right)$, the following relationship holds from the method of types [15].

Lemma 1.

Suppose that the sequence $\mathit{x}$ is sampled independently from the source $P\in \mathcal{P}\left(\mathcal{X}\right)$. Then,

$$\begin{array}{c}\hfill \frac{1}{|{\mathcal{P}}_n\left(\mathcal{X}\right)|}exp\left\{-nD\left(q_{\mathit{x}}\right|\left|P\right)\right\}\le P\left({\mathcal{T}}_n\left(q_{\mathit{x}}\right)\right)\le exp\left\{-nD\left(q_{\mathit{x}}\right|\left|P\right)\right\},\end{array}$$

(30)

where ${\mathcal{P}}_n\left(\mathcal{X}\right)$ denotes the set of types formed from length-n sequences with alphabet $\mathcal{X}$ and $|{\mathcal{P}}_n{\left(\mathcal{X}\right)|\le (n+1)}^{\left|\mathcal{X}\right|}$.

Then, an upper bound on the type-II error probability of the test ${\mathrm{\Lambda }}^n=({\mathrm{\Lambda }}_1^n,{\mathrm{\Lambda }}_2^n)$ for all pairs of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ can be evaluated as follows:

$$\begin{array}{cc}\hfill & {\beta }_1\left({\mathrm{\Lambda }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2)\hfill \end{array}$$

$$\begin{array}{c}\hfill =\sum _{(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)\in {\mathrm{\Lambda }}_2^n}{\tilde{P}}_1\left(\mathit{x}\right)\prod _{r=1}^s{\tilde{P}}_1^r\left({\mathit{t}}_1^r\right)\prod _{j=1}^u{\tilde{P}}_2^j\left({\mathit{t}}_2^j\right)\\ =\sum _{(\mathit{x},{\mathit{t}}_1)\in {\tilde{\mathrm{\Lambda }}}_2^n}\sum _{i=1}^s{\tilde{P}}_1^i\left(\mathit{x}\right)\prod _{r=1}^s{\tilde{P}}_1^r\left({\mathit{t}}_1^r\right)v\left(i\right)\hfill \\ =\sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}\sum _{i=1}^s{\tilde{P}}_1^i\left({\mathcal{T}}_n\left(q_{\mathit{x}}\right)\right)\prod _{r=1}^s{\tilde{P}}_1^r\left({\mathcal{T}}_N\left(q_{{\mathit{t}}_1^r}\right)\right)v\left(i\right)\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & \le \sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}\sum _{i=1}^{s}exp\left\{-nD(q_{\mathit{x}}\left|\right|{\tilde{P}}_1^i)-\sum _{r=1}^{s}ND(q_{{\mathit{t}}_1^r}\left|\right|{\tilde{P}}_1^r)\right\}\hfill \\ \hfill & \le \sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}\sum _{i=1}^{s}exp\left\{-nD(q_{\mathit{x}}\left|\right|{\tilde{P}}_1^i)-ND(q_{{\mathit{t}}_1^i}\left|\right|{\tilde{P}}_1^i)\right\}\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & =\sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}\sum _{i=1}^{s}exp\left\{-nD(q_{\mathit{x}}\left|\right|q_{{\mathbf{y}}_1^i})-ND(q_{{\mathit{t}}_1^i}\left|\right|q_{{\mathbf{y}}_1^i})-n(1+\gamma )D\left(\frac{q_{\mathit{x}}+\gamma q_{{\mathit{t}}_1^i}}{1+\gamma }\left|\right|{\tilde{P}}_1^i\right)\right\},\hfill \end{array}$$

(33)

where (31) is derived from (4), (32) follows from Lemma 1 and (33) is derived in Appendix A. Minimizing exponents in (33) with respect to $i\in \mathcal{S}$, we further obtain

$$\begin{array}{cc}\hfill & {\beta }_1\left({\mathrm{\Lambda }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2)\hfill \\ \hfill & \le \sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}sexp\left\{-n\underset{i\in \mathcal{S}}{min}\left[D(q_{\mathit{x}}\left|\right|q_{{\mathbf{y}}_1^i})+\gamma D(q_{{\mathit{t}}_1^i}\left|\right|q_{{\mathbf{y}}_1^i})\right]\right\}\hfill \\ \hfill & ·exp\left\{-n\underset{i\in \mathcal{S}}{min}(1+\gamma )D\left(\frac{q_{\mathit{x}}+\gamma q_{{\mathit{t}}_1^i}}{1+\gamma }\left|\right|{\tilde{P}}_1^i\right)\right\}\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill & \le \sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}sexp\left\{-n\tilde{\lambda }\right\}·exp\left\{-n\underset{i\in \mathcal{S}}{min}(1+\gamma )D\left(\frac{q_{\mathit{x}}+\gamma q_{{\mathit{t}}_1^i}}{1+\gamma }\left|\right|{\tilde{P}}_1^i\right)\right\}\hfill \\ \hfill & =exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}·\sum _{(q_{\mathit{x}},q_{{\mathit{t}}_1})\in \mathrm{\Gamma }\left({\mathrm{\Lambda }}_2^n\right)}exp\left\{-n\underset{i\in \mathcal{S}}{min}(1+\gamma )D\left(\frac{q_{\mathit{x}}+\gamma q_{{\mathit{t}}_1^i}}{1+\gamma }\left|\right|{\tilde{P}}_1^i\right)\right\}\hfill \\ \hfill & \le exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}·\sum _{Q\in {\mathcal{P}}_{n+N}\left(\mathcal{X}\right)}exp\left\{\underset{i\in \mathcal{S}}{min}-(n+N)D\left(Q\right||{\tilde{P}}_1^i)\right\}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & \le exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}{(n+N+1)}^{\left|\mathcal{X}\right|}·\sum _{Q\in {\mathcal{P}}_{n+N}\left(\mathcal{X}\right)}\underset{i\in \mathcal{S}}{max}{\tilde{P}}_1^i\left({\mathcal{T}}_{n+N}\left(Q\right)\right)\hfill \\ \hfill & \le exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}{(n+N+1)}^{\left|\mathcal{X}\right|}·\sum _{Q\in {\mathcal{P}}_{n+N}\left(\mathcal{X}\right)}\sum _{i=1}^s{\tilde{P}}_1^i\left({\mathcal{T}}_{n+N}\left(Q\right)\right)\hfill \\ \hfill & =exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}{(n+N+1)}^{\left|\mathcal{X}\right|}·\sum _{i=1}^s\sum _{Q\in {\mathcal{P}}_{n+N}\left(\mathcal{X}\right)}{\tilde{P}}_1^i\left({\mathcal{T}}_{n+N}\left(Q\right)\right)\hfill \\ \hfill & =exp\left\{-n\left[\tilde{\lambda }-\frac{logs}{n}\right]\right\}{(n+N+1)}^{\left|\mathcal{X}\right|}s\hfill \\ \hfill & =exp\left\{-n\left[\tilde{\lambda }-\frac{2logs}{n}-\frac{\left|\mathcal{X}\right|log(n+N+1)}{n}\right]\right\}\hfill \\ \hfill & =exp\left\{-n\lambda \right\},\hfill \end{array}$$

(36)

where (34) is derived from (16) and (28), and (35) follows from Lemma 1.

Next, we demonstrate (26). For preliminaries, we define a new typical set and show some properties for the proof. For any given $Q\in \mathcal{P}\left(\mathcal{X}\right)$, define the following typical set:

$$\begin{array}{c}\hfill {\mathcal{B}}_n\left(Q\right):=\left\{\mathit{x}\in {\mathcal{X}}^n:\underset{x\in \mathcal{X}}{max}|q_{\mathit{x}}\left(x\right)-Q\left(x\right)|\le \sqrt{\frac{logn}{n}}\right\}.\end{array}$$

(37)

By using [6, Lemma 22], for ${\mathit{t}}_1\sim Q_1$, $\mathit{x}\sim Q_2$, generated from memoryless sources $(Q_1,Q_2)\in \mathcal{P}{\left(\mathcal{X}\right)}^2$, we have

$$\begin{array}{c}\hfill Pr\left\{{\mathit{t}}_1\notin {\mathcal{B}}_N\left(Q_1\right)\mathrm{or}\mathit{x}\notin {\mathcal{B}}_n\left(Q_2\right)\right\}\le \frac{(1+{\gamma }^2)\left|\mathcal{X}\right|}{{\gamma }^2{n}^2}.\end{array}$$

(38)

For any given $x\in \mathcal{X}$ and any pair of distributions $(Q_1,Q_2)\in \mathcal{P}{\left(\mathcal{X}\right)}^2$, we define information density as

$$\begin{array}{c}\hfill {\iota }_i\left(x\right|Q_1,Q_2,\gamma ):=log\frac{(1+\gamma )Q_i\left(x\right)}{\gamma Q_1\left(x\right)+Q_2\left(x\right)}(i=1,2).\end{array}$$

(39)

Furthermore, for any pair of distributions $(Q_1,Q_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{X}\right)$, define the function $i^{\ast }(Q_1,Q_2)$, as the index of the subclass of $Q_1$, as follows:

$$\begin{array}{cc}\hfill i^{\ast }(Q_1,Q_2)& :=\underset{i\in \mathcal{S}}{arg\; min}\left\{\gamma D\left(Q_1^i\left|\right|\frac{\gamma Q_1^i+Q_2}{1+\gamma }\right)+D\left(Q_2\left|\right|\frac{\gamma Q_1^i+Q_2}{1+\gamma }\right)\right\}\hfill \\ \hfill & =\underset{i\in \mathcal{S}}{arg\; min}\mathrm{GJS}(Q_1^i,Q_2,\gamma ).\hfill \end{array}$$

(40)

Hereafter, we denote $i^{\ast }(Q_1,Q_2)$ simply as $i^{\ast }\left(Q_2\right)$ when the first argument is clear from the context.

Lemma 2.

Assume that $(Q_1,Q_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{X}\right)$. For $q_{{\mathit{t}}_1^i}\in {\mathcal{B}}_N\left(Q_1^i\right)(i\in \mathcal{S})$, $q_{\mathit{x}}\in {\mathcal{B}}_n\left(Q_2\right)$, we have

$$\begin{array}{c}\hfill i^{\ast }(q_{{\mathit{t}}_1},q_{\mathit{x}})\to i^{\ast }(Q_1,Q_2)(n\to \infty ).\end{array}$$

(41)

Proof. 

The proof is provided in Appendix B. □

Lemma 3

(Zhou et al. [5]). Assume that $(Q_1,Q_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times \mathcal{P}\left(\mathcal{X}\right)$. For $q_{{\mathit{t}}_1^i}\in {\mathcal{B}}_N\left(Q_1^i\right)(i\in \mathcal{S})$, $q_{\mathit{x}}\in {\mathcal{B}}_n\left(Q_2\right)$, by applying the Taylor expansion to $\mathrm{GJS}(q_{{\mathit{t}}_1^i},q_{\mathit{x}},\gamma )$ around $(Q_1^i,Q_2)$, we have

$$\begin{array}{c}\hfill \mathrm{GJS}(q_{{\mathit{t}}_1^i},q_{\mathit{x}},\gamma )=\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^i\right|Q_1^i,Q_2,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|Q_1^i,Q_2,\gamma )+O\left(\frac{logn}{n}\right),\end{array}$$

(42)

where $t_{1,m}^i$ denotes the m-th symbol of sequence ${\mathit{t}}_1^i$ and $x_m$ denotes the m-th symbol of sequence $\mathit{x}$.

Note that the probability ${\mathbb{P}}_2$ is calculated by assuming that the test sequence $\mathit{x}$ is generated from $P_2$ (cf. Equation (11)). An upper bound on the type-II error probability can be evaluated as follows:

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\mathrm{\Lambda }}^n\right|P_1,P_2)\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}{\mathbb{P}}_2\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )<\lambda +\eta \left(n\right)\right\}\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )<\lambda +\eta \left(n\right)\right\}\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )<\lambda +\eta \left(n\right),\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\}\hfill \\ \hfill & +\underset{n\to \infty }{lim\; sup}\sum _{i=1}^s\sum _{j=1}^{u}w\left(j\right)·{\mathbb{P}}_2^j\left\{\mathit{x}\notin {\mathcal{B}}_n\left(P_2^j\right)\mathrm{or}{\mathit{t}}_1^i\notin {\mathcal{B}}_N\left(P_1^i\right)\right\}\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\left\{\mathrm{GJS}(q_{{\mathit{t}}_1^{i^{\ast }\left(q_{\mathit{x}}\right)}},q_{\mathit{x}},\gamma )<\lambda +\eta \left(n\right),\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\},\hfill \end{array}$$

(43)

where Equation (43) follows from Equations (14), (38) and (40). By using Lemma 3, Equation (43) can be expanded as follows:

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\mathrm{\Lambda }}^n\right|P_1,P_2)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\{\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )<\lambda \hfill \\ & +O\left(\frac{logn}{n}\right),\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\}\hfill \\ \hfill & \le \sum _{j=1}^u\underset{n\to \infty }{lim\; sup}w\left(j\right){\mathbb{P}}_2^j\{\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )<\lambda \hfill \end{array}$$

$$\begin{array}{c}\hfill +O\left(\frac{logn}{n}\right),\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\}\\ =\sum _{j=1}^u\underset{n\to \infty }{lim\; sup}w\left(j\right){\mathbb{P}}_2^j\left\{\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\}\hfill \\ ·{\mathbb{P}}_2^j\{\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )<\lambda \hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill & +O\left(\frac{logn}{n}\right)\left|\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\},\hfill \end{array}$$

(45)

where Equation (44) is derived from Fatou’s lemma. It follows from (38) that

$$\begin{array}{c}\hfill {\mathbb{P}}_2^j\left\{{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),\forall i\in \mathcal{S}\right\}\to 1(n\to \infty ).\end{array}$$

(46)

Here, Equation (14) can be also expressed as follows:

$$\begin{array}{cc}\hfill \mathrm{GJS}(Q_1,Q_2,\gamma )& =\gamma {\mathbb{E}}_{Q_1}\left[{\iota }_1\left(X\right|Q_1,Q_2,\gamma )\right]+{\mathbb{E}}_{Q_2}\left[{\iota }_2\left(X\right|Q_1,Q_2,\gamma )\right].\hfill \end{array}$$

(47)

Therefore, by the weak law of large numbers, for any given $\delta >0$,

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}Pr\left\{\right|\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )\hfill \\ \hfill & -\mathrm{GJS}(P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )|\le \delta \}=1.\hfill \end{array}$$

(48)

From this result of the weak law of large numbers and Lemma 2, combining (45)–(48) gives

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\mathrm{\Lambda }}^n\right|P_1,P_2)& \le \sum _{\{j\in \mathcal{U}:\mathrm{GJS}(P_1^{i^{\ast }\left(P_2^j\right)},P_2^j,\gamma )<\lambda \}}w\left(j\right).\hfill \end{array}$$

(49)

Thus, by (24), we can see that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\mathrm{\Lambda }}^n\right|P_1,P_2)\le ϵ.\end{array}$$

(50)

3.3.2. Converse Part

For any pair of distributions $(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ and for all pairs of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, fix any test ${\varphi }^n$ satisfying that

$$\begin{array}{cc}\hfill {\beta }_1\left({\varphi }^n\right|{\tilde{P}}_1,{\tilde{P}}_2)& \le exp(-n\lambda ),\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\varphi }^n\right|P_1,P_2)& \le ϵ.\hfill \end{array}$$

(52)

We show

$$\begin{array}{c}\hfill \lambda \le sup\left\{\overline{\lambda }|\sum _{\{j\in \mathcal{U}:\mathrm{MGJS}(P_1,P_2^j,\gamma )<\overline{\lambda }\}}w\left(j\right)\le ϵ\right\}.\end{array}$$

(53)

We first give some lemmas, which are useful in the proof of the converse part.

Lemma 4.

Let ${\varphi }^n$ be a test in which the decision rule depends only on $(\mathit{x},{\mathit{t}}_1,{\mathit{t}}_2)\in {\mathcal{X}}^n\times {\mathcal{X}}^{sN}\times {\mathcal{X}}^{uN}$. Then, for any given $\kappa \in [0,1]$, we can construct a type-based test ${\mathrm{\Omega }}^n$ satisfying

$$\begin{array}{cc}\hfill {\beta }_1\left({\varphi }^n\right|{\tilde{P}}_1,{\tilde{P}}_2)& \ge \kappa {\beta }_1\left({\mathrm{\Omega }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2),\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill {\beta }_2\left({\varphi }^n\right|{\tilde{P}}_1,{\tilde{P}}_2)& \ge (1-\kappa ){\beta }_2\left({\mathrm{\Omega }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2)\hfill \end{array}$$

(55)

for any pair of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$.

Proof. 

Lemma 4 can be proved in the same way as (Lemma 7 [5]), the proof of which is inspired by (Lemma 2 [3]). □

Remark 3.

As in the proof of (Lemma 2 [3]) and (Lemma 7 [5]), a type-based test ${\mathrm{\Omega }}^n$ specified in Lemma 4 is obtained by tailoring ${\varphi }^n$ and satisfies Equations (54) and (55) for all $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$. In other words, the construction of ${\mathrm{\Omega }}^n$ is universal, which is in the same spirit of (Lemma 2 [3]). This claim is slightly stronger than the one in (Lemma 7 [5]).

Lemma 5.

For any $\lambda \in {\mathbb{R}}^{+}$, any type-based test ${\mathrm{\Omega }}^n$ satisfying the condition that for all pairs of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$,

$$\begin{array}{c}\hfill {\beta }_1\left({\mathrm{\Omega }}^n\right|{\tilde{P}}_1,{\tilde{P}}_2)\le exp(-n\lambda ),\end{array}$$

(56)

we have that for any pair of distributions $(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$

$$\begin{array}{c}\hfill {\beta }_2\left({\mathrm{\Omega }}^n\right|P_1,P_2)\ge {\mathbb{P}}_2\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )<\lambda -\rho \left(n\right)\right\},\end{array}$$

(57)

where $\rho \left(n\right):=\frac{\left|\mathcal{X}\right|log(n+1)+(s+u)\left|\mathcal{X}\right|log(N+1)-log{v}^{\ast }}{n}$ with $v^{\ast }:=min\{v\left(i\right):v\left(i\right)>0,i\in \mathcal{S}\}$.

Proof. 

The proof is provided in Appendix C. □

The type-based test ${\mathrm{\Omega }}^n$ specified in Lemma 4 satisfies Equations (54) and (55) for all $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$. If we set $\kappa =1/n$ in Lemma 4, and combine it with Lemma 5, we can derive the following relation:

Corollary 1.

For any given $\lambda \in {\mathbb{R}}^{+}$, any test ${\varphi }^n$ satisfying the condition that for all pairs of distributions $({\tilde{P}}_1,{\tilde{P}}_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$

$$\begin{array}{c}\hfill {\beta }_1\left({\varphi }^n\right|{\tilde{P}}_1,{\tilde{P}}_2)\le exp(-n\lambda ),\end{array}$$

(58)

we have that for any pair of distributions $(P_1,P_2)\in {\mathcal{P}}_{\mathcal{S}}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$

$$\begin{array}{cc}\hfill & {\beta }_2\left({\varphi }^n\right|P_1,P_2)\ge \left(1-\frac{1}{n}\right)·{\mathbb{P}}_2\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )+\rho \left(n\right)+\frac{logn}{n}<\lambda \right\}.\hfill \end{array}$$

(59)

Proof. 

The proof is provided in Appendix D. □

By (59), a lower bound on the type-II error probability can be evaluated as follows:

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\beta }_2\left({\varphi }^n\right|P_1,P_2)\hfill \\ \hfill & \ge \underset{n\to \infty }{lim\; sup}\left(1-\frac{1}{n}\right){\mathbb{P}}_2\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )+O\left(\frac{logn}{n}\right)<\lambda \right\}\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )+O\left(\frac{logn}{n}\right)<\lambda \right\}\hfill \\ \hfill & \ge \underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right){\mathbb{P}}_2^j\left\{\mathrm{MGJS}(q_{{\mathit{t}}_1},q_{\mathit{x}},\gamma )+O\left(\frac{logn}{n}\right)<\lambda ,\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\}\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\sum _{j=1}^{u}w\left(j\right)·{\mathbb{P}}_2^j\{\frac{1}{n}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill & +O\left(\frac{logn}{n}\right)<\lambda ,\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\}\hfill \\ \hfill & \ge \sum _{j=1}^u\underset{n\to \infty }{lim\; inf}w\left(j\right)·{\mathbb{P}}_2^j\{\frac{1}{n}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill & +O\left(\frac{logn}{n}\right)<\lambda ,\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\}\hfill \\ \hfill & \ge \sum _{j=1}^u\underset{n\to \infty }{lim\; inf}w\left(j\right){\mathbb{P}}_2^j\left\{\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\}\hfill \\ \hfill & ·{\mathbb{P}}_2^j\{\frac{\gamma }{N}\sum _{m=1}^N{\iota }_1\left(t_{1,m}^{i^{\ast }\left(q_{\mathit{x}}\right)}\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )+\frac{1}{n}\sum _{m=1}^n{\iota }_2\left(x_m\right|P_1^{i^{\ast }\left(q_{\mathit{x}}\right)},P_2^j,\gamma )<\lambda \hfill \\ \hfill & +O\left(\frac{logn}{n}\right)\left|\mathit{x}\in {\mathcal{B}}_n\left(P_2^j\right),{\mathit{t}}_1^i\in {\mathcal{B}}_N\left(P_1^i\right),\forall i\in \mathcal{S}\right\}\hfill \\ \hfill & =\sum _{\{j\in \mathcal{U}:\mathrm{GJS}(P_1^{i^{\ast }\left(P_2^j\right)},P_2^j,\gamma )<\lambda \}}w\left(j\right),\hfill \end{array}$$

(62)

where Equations (60) and (61) are derived from Lemma 3 and Fatou’s lemma, respectively. Equation (62) follows from Equations (45)–(48). By Equation (52), Equation (62) indicates that

$$\begin{array}{c}\hfill \lambda \le sup\left\{\overline{\lambda }|\sum _{\{j\in \mathcal{U}:\mathrm{GJS}(P_1^{i^{\ast }\left(P_2^j\right)},P_2^j,\gamma )<\overline{\lambda }\}}w\left(j\right)\le ϵ\right\}.\end{array}$$

(63)

3.4. Second-Order Maximum Error Exponent

In this section, we characterize the second-order maximum error exponent. For simplicity, we assume that only $P_2$ has subclasses, but $P_1$ does not ($s=1$). First, from Theorem 2 with $s=1$, the first-order maximum error exponent in this setting is characterized as follows: for any pair of distributions $(P_1,P_2)\in \mathcal{P}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$, we have

$$\begin{array}{c}\hfill \widehat{\lambda }\left(ϵ\right)=sup\left\{\overline{\lambda }|\sum _{\{j\in \mathcal{U}:\mathrm{GJS}(P_1,P_2^j,\gamma )<\overline{\lambda }\}}w\left(j\right)\le ϵ\right\},\end{array}$$

(64)

where $\mathrm{MGJS}(P_1,P_2^j,\gamma )$ in (21) is replaced by $\mathrm{GJS}(P_1,P_2^j,\gamma )$.

Next, we provide a characterization of the second-order maximum error exponent in Definition 2 with $s=1$ in the case where only $P_2$ has subclasses. By definition, $\widehat{r}(ϵ,\lambda )=+\infty$ if $\lambda <\widehat{\lambda }\left(ϵ\right)$ and $\widehat{r}(ϵ,\lambda )=-\infty$ if $\lambda >\widehat{\lambda }\left(ϵ\right)$. Therefore, in the discussion of the second-order error exponent, we focus on the case $\lambda =\widehat{\lambda }\left(ϵ\right)$.

Theorem 3.

For any pair of distributions $(P_1,P_2)\in \mathcal{P}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathcal{U}}\left(\mathcal{X}\right)$ and $ϵ\in [0,1)$,

$$\begin{array}{cc}\hfill \widehat{r}(ϵ,\lambda )& =sup\left\{r|\sum _{\left\{j\in \mathcal{U}:\mathrm{GJS}(P_1,P_2^j,\gamma )<\lambda \right\}}w\left(j\right)+\sum _{\left\{j\in \mathcal{U}:\mathrm{GJS}(P_1,P_2^j,\gamma )=\lambda \right\}}{\mathrm{\Phi }}_j\left(r\right)w\left(j\right)\le ϵ\right\},\hfill \end{array}$$

(65)

where $\lambda =\widehat{\lambda }\left(ϵ\right)$,

$$\begin{array}{cc}\hfill {\mathrm{\Phi }}_j\left(r\right)& :=G\left(\frac{r}{\sqrt{V(P_1,P_2^j,\gamma )}}\right),\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill G\left(a\right)& :=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^a{e}^{-\frac{x^2}{2}}dx,\hfill \end{array}$$

(67)

which is the cumulative distribution function of the standard Gaussian distribution, and for any pair of distributions $(Q_1,Q_2)\in \mathcal{P}{\left(\mathcal{X}\right)}^2$,

$$\begin{array}{c}\hfill V(Q_1,Q_2,\gamma ):=\gamma {\mathrm{Var}}_{Q_1}\left[{\iota }_1\left(X\right|Q_1,Q_2,\gamma )\right]+{\mathrm{Var}}_{Q_2}\left[{\iota }_2\left(X\right|Q_1,Q_2,\gamma )\right],\end{array}$$

(68)

where ${\mathrm{Var}}_Q[·]$ represents the variances with respect to $Q\in \mathcal{P}\left(\mathcal{X}\right)$.

Proof. 

The proof is provided in Appendix E. □

Remark 4.

If $\mathcal{U}$ is a singleton ($u=1$), Theorem 3 reduces to $\widehat{r}(ϵ,\lambda )=\sqrt{V(P_1,P_2^j,\gamma )}{G}^{-1}\left(ϵ\right)$ for $\lambda =\widehat{\lambda }\left(ϵ\right)=\mathrm{GJS}(P_1,P_2^j,\gamma )$, which is the same result given by Zhou et al. [5].

Remark 5.

We can summarize the two terms on the right-hand side of (65) into the following single term called the canonical equation [6]:

$$\begin{array}{c}\hfill \sum _{j\in \mathcal{U}}w\left(j\right)\underset{n\to \infty }{lim}{\mathrm{\Phi }}_j(\sqrt{n}(\lambda -\mathrm{GJS}(P_1,P_2^j,\gamma ))+r).\end{array}$$

(69)

We focus on the case $\lambda =\widehat{\lambda }\left(ϵ\right)$. From Theorem 2, it holds that

$$\begin{array}{c}\hfill \sum _{\{j\in \mathcal{U}:\mathrm{MGJS}(P_1,P_2^j,\gamma )<\lambda \}}w\left(j\right)\le ϵ\end{array}$$

(70)

and

$$\begin{array}{c}\hfill \sum _{\{j\in \mathcal{U}:\mathrm{MGJS}(P_1,P_2^j,\gamma )\le \lambda \}}w\left(j\right)\ge ϵ.\end{array}$$

(71)

Here, let us consider the following canonical equation for r

$$\begin{array}{c}\hfill \sum _{j\in \mathcal{U}}w\left(j\right)\underset{n\to \infty }{lim}{\mathrm{\Phi }}_j(\sqrt{n}(\widehat{\lambda }\left(ϵ\right)-\mathrm{GJS}(P_1,P_2^j,\gamma ))+r)=ϵ.\end{array}$$

(72)

Thus, in view of (70) and (71), this equation always has the solution $r=r\left(ϵ\right)$. If

$$\begin{array}{c}\hfill \sum _{\left\{j\in \mathcal{U}:\mathrm{GJS}(P_1,P_2^j,\gamma )=\widehat{\lambda }\left(ϵ\right)\right\}}w\left(j\right)=0,\end{array}$$

(73)

the solution is not unique ($r\left(ϵ\right)=+\infty$). By using the solution $r\left(ϵ\right)$, Equation (65) with $\lambda =\widehat{\lambda }\left(ϵ\right)$ can be expressed in a simpler form

$$\begin{array}{c}\hfill \widehat{r}(ϵ,\lambda )=r\left(ϵ\right).\end{array}$$

(74)

4. Generalization to Mixed Memoryless Sources with General Mixture

In this section, we consider the classification problem in the case where $P_1$ does not have subclasses and $P_2$ is given by a general mixture model. The general mixture model considered in this problem represents an extension of the source with multiple subclasses defined in Section 2.2. Since the decision rule that achieves the maximum error exponent can be operated using only one of the training sequences, we assume in this section that only the training sequence ${\mathit{t}}_1$ is available. Then, we provide a characterization of the maximum error exponents in a single-letter form under this setting. First, we define the source referred to as a mixed memoryless source with general mixture [6,9] as follows. Let $\mathrm{\Theta }$ be an arbitrary probability space with a general probability measure $w\left(\theta \right),\theta \in \mathrm{\Theta }$. Then, the probability of $\mathit{x}\in {\mathcal{X}}^n$ is given by

$$\begin{array}{c}\hfill P_2\left(\mathit{x}\right)={\int }_{\theta \in \mathrm{\Theta }}{P}_2^{\theta }\left(\mathit{x}\right)dw\left(\theta \right),\end{array}$$

(75)

where $P_2^{\theta }$ is a stationary and memoryless source for each $\theta \in \mathrm{\Theta }$. That is, for $\mathit{x}=(x_1,x_2,\cdots ,x_n)\in {\mathcal{X}}^n$

$$\begin{array}{c}\hfill P_2^{\theta }\left(\mathit{x}\right)=\prod _{i=1}^n{P}_2^{\theta }\left(x_i\right).\end{array}$$

(76)

When a test sequence is output from $P_2$, the probability distribution of the sequence takes the form of (75). Here, type-I and type-II error probabilities of a test ${\varphi }^n=({\varphi }_1^n,{\varphi }_2^n)$ are given by

$$\begin{array}{c}\hfill {\beta }_1\left({\varphi }^n\right|P_1,P_2)=\sum _{(\mathit{x},{\mathit{t}}_1)\in {\varphi }_2^n}{P}_1\left(\mathit{x}\right)P_1\left({\mathit{t}}_1\right),\end{array}$$

(77)

$$\begin{array}{c}\hfill {\beta }_2\left({\varphi }^n\right|P_1,P_2)=\sum _{(\mathit{x},{\mathit{t}}_1)\in {\varphi }_1^n}{P}_2\left(\mathit{x}\right)P_1\left({\mathit{t}}_1\right).\end{array}$$

(78)

Theorem 4.

(First-and second-order maximum error exponents) For any pair of distributions $(P_1,P_2)\in \mathcal{P}\left(\mathcal{X}\right)\times {\mathcal{P}}_{\mathrm{\Theta }}\left(\mathcal{X}\right)$ and $ϵ\in [0,1)$, we have

$$\begin{array}{c}\hfill \widehat{\lambda }\left(ϵ\right)=sup\left\{\overline{\lambda }|{\int }_{\{\theta \in \mathrm{\Theta }:\mathrm{GJS}(P_1,P_2^{\theta },\gamma )<\overline{\lambda }\}}dw\left(\theta \right)\le ϵ\right\}\end{array}$$

(79)

and

$$\begin{array}{cc}\hfill \widehat{r}(ϵ,\lambda )& =sup\left\{r|{\int }_{\left\{\theta \in \mathrm{\Theta }:\mathrm{GJS}(P_1,P_2^{\theta },\gamma )<\lambda \right\}}dw\left(\theta \right)+{\int }_{\left\{\theta \in \mathrm{\Theta }:\mathrm{GJS}(P_1,P_2^{\theta },\gamma )=\lambda \right\}}{\mathrm{\Phi }}_{\theta }\left(r\right)dw\left(\theta \right)\le ϵ\right\}.\hfill \end{array}$$

(80)

Proof. 

We can prove this theorem in the same way as Theorems 2 and 3. □

5. Numerical Calculation

5.1. First-Order Maximum Error Exponent

In this section, we present a numerical example to illustrate the first-order maximum type-I error exponent $\widehat{\lambda }\left(ϵ\right)$ characterized in Theorem 2.

A numerical example of the first-order maximum error exponent is given by calculating the right-hand side of (64) for the following settings. We assume that $\mathcal{X}=\{0,1\}$. We fix the set of probabilities and weights

$$\begin{array}{c}\hfill P_1=\left\{\left\{\mathrm{Bern}\left(0.389\right),v\left(1\right)=\frac{1}{3}\right\},\left\{\mathrm{Bern}\left(0.322\right),v\left(2\right)=\frac{1}{3}\right\},\left\{\mathrm{Bern}\left(0.256\right),v\left(3\right)=\frac{1}{3}\right\}\right\}\end{array}$$

(81)

and

$$\begin{array}{c}\hfill P_2=\left\{\left\{\mathrm{Bern}\left(0.301\right),w\left(1\right)=\frac{1}{3}\right\},\left\{\mathrm{Bern}\left(0.244\right),w\left(2\right)=\frac{1}{6}\right\},\left\{\mathrm{Bern}\left(0.223\right),w\left(3\right)=\frac{1}{2}\right\}\right\},\end{array}$$

(82)

where $\mathrm{Bern}(·)$ denotes the Bernoulli distribution. The relation among $\widehat{\lambda }\left(ϵ\right)$, $ϵ$ and $\gamma$ is shown in Figure 2. Additionally, for $\gamma =2$, the behavior of $\widehat{\lambda }\left(ϵ\right)$ is depicted in Figure 3. When $ϵ$ becomes larger, the value of $\widehat{\lambda }\left(ϵ\right)$ also increases like a step function. The step increases when $ϵ=\frac{1}{6}$ and $ϵ=\frac{1}{2}$. We can also confirm that $\widehat{\lambda }\left(ϵ\right)$ is right-continuous in $ϵ$. This is due to the limit that the superior of the type-II error probability is constrained in Definition 1.

5.2. Second-Order Maximum Error Exponent

As in the previous subsection, we present a numerical example to illustrate the second-order maximum type-I error exponent $\widehat{r}(ϵ,\lambda )$ characterized in Theorem 3.

A numerical example of the second-order maximum error exponent is given by calculating the right-hand side of (65) for the following settings. We assume that $\gamma =2$, $\mathcal{X}=\{0,1\}$. We fix $\lambda =\widehat{\lambda }\left(ϵ\right)$ and the set of probabilities and weights

$$\begin{array}{c}\hfill P_1=\mathrm{Bern}\left(0.268\right)\end{array}$$

(83)

and

$$\begin{array}{c}\hfill P_2=\left\{\left\{\mathrm{Bern}\left(0.301\right),w\left(1\right)=\frac{1}{3}\right\},\left\{\mathrm{Bern}\left(0.244\right),w\left(2\right)=\frac{1}{6}\right\},\left\{\mathrm{Bern}\left(0.223\right),w\left(3\right)=\frac{1}{2}\right\}\right\},\end{array}$$

(84)

where $P_2$ is the same as the setting in the previous subsection. The behavior of $\widehat{r}(ϵ,\lambda )$ is shown in Figure 4. The value of $\widehat{r}(ϵ,\lambda )$ takes the inverse of the cumulative distribution function of the standard Gaussian for each interval of $ϵ$ such that $0\le ϵ<\frac{1}{6}$, $\frac{1}{6}\le ϵ<\frac{1}{2}$ and $\frac{1}{2}\le ϵ<1$. In contrast to the first-order $\widehat{\lambda }\left(ϵ\right)$, $\widehat{r}(ϵ,\lambda )$ is no longer right continuous in $ϵ$.

6. Conclusions

For binary classification of sources with multiple subclasses, we characterized the first- and second-order maximum error exponents. First, we revealed the first-order maximum error exponent in the case where $P_1$ and $P_2$ are sources with multiple subclasses. In order to derive this representation, we gave a classifier which achieves the asymptotically maximum error exponent in the class of deterministic classifiers for sources with multiple subclasses.

Next, we showed the second-order maximum error exponent in the case where only one of sources has subclasses. The most important key technique to derive the second-order maximum error exponent is to apply the Berry–Esseen theorem [8] instead of the weak law of large numbers. One may wonder whether we can also derive the second-order approximation in the case where $P_1$ is also a source with multiple subclasses. To this end, we need to evaluate Lemma 2 more rigorously. This is future work.

In addition, for binary classification using only a training sequence generated from $P_1$ in the case where $P_1$ does not have subclasses and $P_2$ is given by a general mixture model, we generalized the analysis for the first- and second-order error exponents. From these results, we revealed the asymptotic performance limits of statistical classification for sources with multiple subclasses.

In this paper, we considered a binary classification problem, but in practice, multiclass classification is of importance. In the case where each class is a memoryless source (without multiple subclasses), the first- and second-order maximum error exponents were analyzed in [5]. Extending the obtained results to multiclass classification for sources with multiple subclasses is also a subject of future studies.