Equivalence between LC-CRF and HMM, and Discriminative Computing of HMM-Based MPM and MAP

Abstract

Practitioners have used hidden Markov models (HMMs) in different problems for about sixty years. Moreover, conditional random fields (CRFs) are an alternative to HMMs and appear in the literature as different and somewhat concurrent models. We propose two contributions: First, we show that the basic linear-chain CRFs (LC-CRFs), considered as different from HMMs, are in fact equivalent to HMMs in the sense that for each LC-CRF there exists an HMM—that we specify—whose posterior distribution is identical to the given LC-CRF. Second, we show that it is possible to reformulate the generative Bayesian classifiers maximum posterior mode (MPM) and maximum a posteriori (MAP), used in HMMs, as discriminative ones. The last point is of importance in many fields, especially in natural language processing (NLP), as it shows that in some situations dropping HMMs in favor of CRFs is not necessary.

1. Introduction

Let $Z_{1:N}=(Z_1,\dots ,Z_N)$ be a stochastic sequence, with $Z_n=(X_n,Y_n)$. The random variables $X_1$, $\dots ,X_N$ take their values in a finite set $\mathsf{\Lambda }$, while $Y_1$, $\dots ,Y_N$ take their values either in a discrete or continuous set $\mathsf{\Omega }$. Realizations of $X_{1:N}=(X_1,\dots ,X_N)$ are hidden while realizations of $Y_{1:N}=(Y_1,\dots ,Y_N)$ are observed, and the problem we deal with is to estimate $X_{1:N}=x_{1:N}$ from $Y_{1:N}=y_{1:N}$. We deal with Bayesian methods of estimation, which requires some probabilistic model. Probabilistic model is a distribution—or a family of distributions—which is denoted by ${p(z}_{1:N})$, or ${p(x}_{1:N},y_{1:N})$. We are interested in the case of dependent $Z_1,\dots ,Z_N$. The simplest model taking into account this dependence is the well-known hidden Markov model (HMM) [1,2,3,4,5], whose distribution is given with

$$p\left(x_{1:N},y_{1:N}\right)=p\left(x_1\right)p\left(y_1|x_1\right){\prod }_{n=1}^{N-1}p\left(x_{n+1}|x_n\right)p\left(y_{n+1}|x_{n+1}\right)$$

(1)

In the whole paper, we consider HMMs such that $p\left(x_{n+1}|x_n\right)$ and $p\left(y_{n+1}|x_{n+1}\right)$ in Equation (1) are non-null. HMMs allow recursive fast computation of Bayesian estimators, called “classifiers” in this paper, and recalled below. In spite of their simplicity, HMMs are very robust and provide quite satisfactory results in many applications.

Moreover, conditional random fields (CRFs) [6,7] also allow estimating $X_{1:N}=x_{1:N}$ from $Y_{1:N}=y_{1:N}$. Their definition is different from the definition of HMMs in that in CRFs, one directly considers $p\left(x_{1:N}\right|y_{1:N})$, and neither ${p(x}_{1:N},y_{1:N})$ nor $p\left(y_{1:N}\right|x_{1:N})$ is needed to perform the estimation. The distribution of general LC-CRFs is written as such:

$$p\left(x_{1:N}\right|y_{1:N})=p\left(x_1|y_{1:N}\right){\prod }_{n=1}^{N-1}p\left(x_{n+1}|x_n,y_{1:N}\right)$$

(2)

In this paper, we consider the following basic LC-CRF:

$$p\left(x_{1:N}|y_{1:N}\right)=\frac{1}{\kappa \left(y_{1:N}\right)}exp\left[{\sum }_{n=1}^{N-1}{V}_n\left(x_n,x_{n+1}\right)+{\sum }_{n=1}^N{U}_n\left(x_n,y_n\right)\right]$$

(3)

with $\kappa \left(y_{1:N}\right)$ the normalizing constant.

Authors usually consider the two families HMMs and LC-CRFs as different [6,7,8,9,10,11,12,13]. They classify the former in the category of “generative models”, while they classify the latter in the category of “discriminative” models.

In this paper, we investigate the relationship between HMMs (Equation (1)) and LC-CRFs (Equation (3)). We immediately see that if $p\left(x_{1:N},y_{1:N}\right)$ is of the form (Equation (1)), then $p\left(x_{1:N}{|y}_{1:N}\right)$ is of the form (Equation (3)). Indeed, $p\left(x_{n+1}|x_n\right)$ and $p\left(y_{n+1}|x_{n+1}\right)$ in Equation (1) being non-zero, one can take $V_1\left(x_1,x_2\right)=Log[{p(x}_1,x_2)]$, $V_n\left(x_n,x_{n+1}\right)=Log\left[p\left(x_{n+1}|x_n\right)\right]$ for $n=2$, …, $N-1$, $U_n\left(x_n,y_n\right)=Log\left[p\left(y_n|x_n\right)\right]$, for $n=1$, …, $N$, and $\kappa \left(y_{1:N}\right)=p\left(y_{1:N}\right)$. Thus, the posterior distribution $p\left(x_{1:N}{|y}_{1:N}\right)$ of each HMM (Equation (1)) with the positivity conditions assumed above is a LC-CRF (Equation (3)). Our first contribution is to show the converse: for a given LC-CRF $p\left(x_{1:N}|y_{1:N}\right)$ of form (Equation (3)), there is an HMM $q\left(x_{1:N},y_{1:N}\right)$ of form (Equation (3)) such that $q\left(x_{1:N}|y_{1:N}\right)=p\left(x_{1:N}|y_{1:N}\right)$. Moreover, we give exact computation of the HMM parameters $q\left(x_1\right)$, $q\left(x_{n+1}|x_n\right)$, $q\left(y_{n+1}|x_{n+1}\right)$, from the LC-CRF parameters $V_n\left(x_n,x_{n+1}\right)$, $U_n\left(x_n,y_n\right)$. Such a general result, without additional constraints on LC-CRF, is new.

Our second contribution is related to the computation of Bayesian classifiers. In some situations, CRFs are preferred over HMMs because the Bayesian classifiers based on CRFs are computed without considering $p\left(y_n|x_n\right)$ distributions, which are difficult to model. This is particularly the case with automatic natural language processing (NLP). Indeed, when considering an HMM (Equation (1)) and calculating the Bayesian classifiers “maximum posterior margins” (MPM) and “maximum posterior” (MAP) in the standard way, we use $p\left(y_n|x_n\right)$ and this is the reason why LC-CRFs are preferred over HMMs. In this paper, we show that the HMM-based MPM and MAP can also be computed without using $p\left(y_n|x_n\right)$. Also recall that each Bayesian “discriminative” classifier based on LC-CRF (Equation (3)) is identical to the “generative” Bayesian classifier based on an HMM (Equation (1)), since the posterior distribution $p\left(x_{1:N}|y_{1:N}\right)$ of an HMM gives the one of a LC-CRF. Indeed, Bayesian classifiers only depend on the posterior distribution. Thus, the distinction between “generative” classifiers and “discriminative” classifiers is misleading, they are all “discriminative”, but they can be computed in a “generative” way, using $p\left(y_n|x_n\right)$, or in a discriminative manner, without using $p\left(y_n|x_n\right)$. Thus, we can say that our second contribution is to show that the HMM-based MPM and MAP, usually computed in a generative manner, can also be computed in a discriminative manner. This is important because it shows that abandoning HMMs in favor of CRFs in the aforementioned situations is not justified. Indeed, attached to the first contribution, it shows that the use of the MPM or MAP based on HMM (Equation (1)) is as interesting as the use of the MPM or MAP based on LC-CRF (Equation (3)).

Let us give some more technical details on the two contributions.

• We establish an equivalence between HMMs (Equation (1)) and basic linear-chain CRFs (Equation (3)), which completes the results presented in [14].

Let us notice that wanting to compare the two models directly is somewhat misleading. Indeed, HMMs and CRFs are defined with distributions on different spaces. To be precise, we adopt the following definition:

Definition 1.

Let $X_1$, $\dots ,X_N$, $Y_1$, $\dots ,Y_N$ be the two stochastic sequences defined above.

(i) 

We will call “model” a distribution $p\left(x_{1:N},y_{1:N}\right)$;

(ii) 

We will call “conditional model” a distribution $p\left(x_{1:N}|y_{1:N}\right)$;

(iii) 

We will say that a model $p\left(x_{1:N},y_{1:N}\right)$ is “equivalent” to a conditional model $q\left(x_{1:N}|y_{1:N}\right)$ if there exists a distribution $r\left(y_{1:N}\right)$ such that $p\left(x_{1:N},y_{1:N}\right)=q\left(x_{1:N}|y_{1:N}\right)r\left(y_{1:N}\right)$;

(iv) 

We will say that a family of models A is “equivalent” to a family of conditional models B if for each model $p\left(x_{1:N},y_{1:N}\right)$ in A there exists an equivalent conditional model $q\left(x_{1:N}|y_{1:N}\right)$ in B.

According to Definition 1, HMMs are particular “models”, while CRFs are particular “conditional models”. Then a particular HMM model cannot be equal to a particular conditional CRF model, but it can be equivalent to the latter.

Our contribution is to show that the family of LC-CRFs (Equation (3)) is equivalent to the family of HMMs (Equation (1)). In addition, we specify, for each LC-CRF $q\left(x_{1:N}|y_{1:N}\right)$, a particular HMM $p\left(x_{1:N},y_{1:N}\right)$ such that $p\left(x_{1:N}|y_{1:N}\right)=q\left(x_{1:N}|y_{1:N}\right)$.

Finally, the core of our first contribution is the following. Let $p\left(x_{1:N},y_{1:N},\theta \right)$ be an HMM (Equation (1)), with parameters $\theta$. Taking $r\left(y_{1:N}\right)=p\left(y_{1:N},\theta \right)$, it is immediate to see that $p\left(x_{1:N}|y_{1:N},\theta \right)$ is an equivalent CRF. The converse is not immediate. Is a given CRF $p\left(x_{1:N}|y_{1:N},\theta \right)$ equivalent to a certain HMM? If yes, can we find $r\left(y_{1:N}\right)$ such that $p\left(x_{1:N}|y_{1:N},\theta \right)r\left(y_{1:N}\right)$ is an HMM? Moreover, can we give its (Equation (1)) form? Answering to these questions in a simple linear-chain CRF case is our first contribution. More precisely, we show that the family of LC-CRFs (Equation (3)) is equivalent to the family of HMMs (Equation (1)), and we specify, for each LC-CRF $p\left(x_{1:N}|y_{1:N}\right)$, a particular HMM $q\left(x_{1:N},y_{1:N}\right)$ given in the form (Equation (1)), such that $p\left(x_{1:N}|y_{1:N}\right)=q\left(x_{1:N}|y_{1:N}\right)$.

2.

We show that the “generative” estimators MPM and MAP in HMM are computable in a “discriminative” manner, exactly as in LC-CRF.

One of the interests of HMMs and CRFs is that in both of them there exist Bayesian classifiers, which allow estimating $x_{1:N}$ from $y_{1:N}$ in a reasonable computer time. As examples, let us consider the “maximum of posterior margins” (MPM) defined with:

$$\left[g\left(y_{1:N}\right)={\widehat{x}}_{1:N}=\left({\widehat{x}}_1,\dots ,{\widehat{x}}_N\right)\right]⟺[\forall n=1,\dots ,N,p({\widehat{x}}_n\left|y_{1:N}\right)=\underset{x_n}{sup}(p\left(x_n|y_{1:N}\right))]$$

(4)

and the “maximum a posteriori” (MAP), defined with

$$\left[g\left(y_{1:N}\right)={\widehat{x}}_{1:N}\right]⟺[p({\widehat{x}}_{1:N}\left|y_{1:N}\right)=\underset{x_{1:N}}{sup}(p\left(x_{1:N}|y_{1:N}\right))]$$

(5)

Note that likely to any other Bayesian classifier, MPM and MAP are independent from $p\left(y_{1:N}\right)$. This means that in any generative model $p(x_{1:N},y_{1:N})$, any related Bayesian classifier is strictly the same as the one related to the equivalent (in the meaning of Definition 1) CRF model $p\left(x_{1:N}\right|y_{1:N})$. We see that the distinction between “generative” and “discriminative” classifiers is not justified: all Bayesian classifiers are discriminative. However, in HMM the related MPM and MAP classifiers are computed using $p\left(y_n|x_n\right)$, while this is not the case in LC-CRF. We show that both MPM and MAP in HMM can also be computed in a “discriminative” way, without using $p\left(y_n|x_n\right)$. Thus, the feasibility of using MPM and MAP in HMM is strictly the same as that of their use in LC-CRF, which is our second contribution. One of the consequences is that the use of MPM and MAP in the two families HMMs and LC-CRFs presents exactly the same interest, in particular in NLP. This shows that abandoning HMMs in favor of LC-CRFs in NLP because of the “generative” nature [6,7,8,9,15,16,17,18] of their related Bayesian classifiers was not justified.

2. Related Works

Concerning the first contribution, several authors noticed similarities between LC-CRFs and HMMs in different previous works. Our first remark is to notice that trying to compare the two families directly is somewhat incorrect, as they are defined on different spaces. In particular, they cannot be equal. It is well-known that the posterior distribution $p\left(x_{1:N}|y_{1:N}\right)$ of an HMM $p\left(x_{1:N},y_{1:N}\right)$ is a LC-CRF. Conversely, showing that for a given CRF $p\left(x_{1:N}|y_{1:N}\right)$ it is possible to find an HMM $q\left(x_{1:N},y_{1:N}\right)$ such that $q\left(x_{1:N}|y_{1:N}\right)=p\left(x_{1:N}|y_{1:N}\right)$ is more difficult and at our knowledge, there is no general results, except [14], published. However, let us mention [19] where authors show a similar result to ours assuming an additional constraint on the LC-CRF considered. In [7], authors comment similarities and differences between LC-CRFs and HMMs considered here; however, the problem of searching an HMM equivalent to a given LC-CRF is not addressed. In this paper we show how to compute, from the LC-CRF $p\left(x_{1:N}|y_{1:N}\right)$ given with Equation (3), an HMM $q\left(x_{1:N},y_{1:N}\right)$ verifying $q\left(x_{1:N}|y_{1:N}\right)=p\left(x_{1:N}|y_{1:N}\right)$, without any additional constraints. Concerning the second contribution, the authors generally distinguish between “discriminative” classifiers, linked to discriminative models, and “generative” classifiers, linked to generative models. As mentioned above, our contribution is to show that the MPMs and MAPs based on generative HMMs can also be considered as discriminative classifiers. To our knowledge, there is no work on such conversions, except [20].

3. Equivalence between HMMs and Simple Linear-Chain CRFs

We will use the following Lemma:

Lemma 1. 

Let $W_{1:N}=(W_1,\dots ,W_N)$ be random sequence, taking its values in a finite set $\mathsf{\Delta }$. Then

(i) 

$W_{1:N}$ is a Markov chain if and only if (iff) there exist $N-1$ functions ${\phi }_1,\dots ,{\phi }_{N-1}$ from ${\mathsf{\Delta }}^2$ to ${\mathrm{R}}^{+}$ such that

$$p(w_1,\dots ,w_N)\propto {\phi }_1(w_1,w_2)\dots {\phi }_{N-1}(w_{N-1},w_N)$$

(6)

where “$\propto$” means “proportional to”;

(ii) 

For the HMM defined with ${\phi }_1$, …, ${\phi }_{N-1}$ verifying Equation (6), $p\left(w_1\right)$ and $p\left(w_{n+1}|w_n\right)$ are given with

$$p\left(w_1\right)=\frac{{\beta }_1\left(w_1\right)}{\sum _{w_1}{\beta }_1\left(w_1\right)};p\left(w_{n+1}|w_n\right)=\frac{{\phi }_n(w_n,w_{n+1}){\beta }_{n+1}\left(w_{n+1}\right)}{{\beta }_n\left(w_n\right)}$$

(7)

where ${\beta }_1\left(w_1\right)$, …, ${\beta }_N\left(w_N\right)$ are defined with the following backward recursion:

$${\beta }_N\left(w_N\right)=1,{\beta }_n\left(w_n\right)={\sum }_{w_{n+1}}{\phi }_n(w_n,w_{n+1}){\beta }_{n+1}\left(w_{n+1}\right)$$

(8)

Proof of Lemma.

• Let $W_{1:N}$ be Markov: $p\left(w_1,\dots ,w_N\right)=p\left(w_1\right)p\left(w_2|w_1\right)p\left(w_3|w_2\right)\dots p\left(w_N|w_{N-1}\right)$. Then (Equation (6)) is verified by ${\phi }_1\left(w_1,w_2\right)=p\left(w_1\right)p\left(w_2|w_1\right)$, ${\phi }_2\left(w_2,w_3\right)=p\left(w_3|w_2\right)$, …, ${\phi }_{N-1}\left(w_{N-1},w_N\right)=p\left(w_N|w_{N-1}\right)$.
• Conversely, let $p\left(w_1,\dots ,w_N\right)$ verifies (Equation (6)). Thus $p\left(w_1,\dots ,w_N\right)=K{\phi }_1(w_1,w_2)\dots {\phi }_{N-1}(w_{N-1},w_N)$ with $K$ constant. This implies that for each $n=1$, …, $N-1$ we have

  $$\begin{gathered} p\left(w_{n+1}|w_1,\dots ,w_n\right)=\frac{p\left(w_1,\dots ,w_n,w_{n+1}\right)}{p\left(w_1,\dots ,w_n\right)}= \\ \frac{{\sum }_{\left(w_{n+2},\dots ,w_N,\right)}{\phi }_1(w_1,w_2)\dots {{\phi }_n\left(w_n,w_{n+1}\right){\phi }_{n+1}\left(w_{n+1},w_{n+2}\right)\dots \phi }_{N-1}(w_{N-1},w_N)}{{\sum }_{\left({w_{n+1},w}_{n+2},\dots ,w_N,\right)}{\phi }_1(w_1,w_2)\dots {{\phi }_n\left(w_n,w_{n+1}\right){\phi }_{n+1}\left(w_{n+1},w_{n+2}\right)\dots \phi }_{N-1}(w_{N-1},w_N)}= \\ \frac{{\phi }_n\left(w_n,w_{n+1}\right){\sum }_{\left(w_{n+2},\dots ,w_N,\right)}{{\phi }_{n+1}\left(w_{n+1},w_{n+2}\right)\dots \phi }_{N-1}(w_{N-1},w_N)}{{\sum }_{\left({w_{n+1},w}_{n+2},\dots ,w_N,\right)}{{\phi }_n\left(w_n,w_{n+1}\right){\phi }_{n+1}\left(w_{n+1},w_{n+2}\right)\dots \phi }_{N-1}(w_{N-1},w_N)}=p\left(w_{n+1}|w_n\right) \end{gathered}$$

  (9)

  which shows that $p\left(w_1,\dots ,w_N\right)$ is Markov.

Moreover, let us set ${\beta }_n\left(w_n\right)={\sum }_{\left({w_{n+1},w}_{n+2},\dots ,w_N\right)}{{\phi }_n\left(w_n,w_{n+1}\right)\dots \phi }_{N-1}(w_{N-1},w_N)$ for $n=1$, …, $N-1$. On the one hand, we see that ${\beta }_n\left(w_n\right)=\sum _{w_{n+1}}{\phi }_n(w_n,w_{n+1}){\beta }_{n+1}\left(w_{n+1}\right)$. On the other hand, according to (Equation (9)) we have $p\left(w_{n+1}|w_n\right)=\frac{{\phi }_n(w_n,w_{n+1}){\beta }_{n+1}\left(w_{n+1}\right)}{{\beta }_n\left(w_n\right)}$. As $p\left(w_1\right)=\frac{{\beta }_1\left(w_1\right)}{\sum _{w_1}{\beta }_1\left(w_1\right)}$, (Equation (7)) and (Equation (8)) are verified, which ends the proof. □

Proposition 1 below shows that the LC-CRF defined with (Equation (3)) is equivalent to an HMM defined with Equation (1). In addition, $p\left(x_1\right)$, $p\left(x_{n+1}|x_n\right)$, and $p\left(y_n|x_n\right)$ in Equation (1) defining an equivalent HMM are computed from $V_n\left(x_n,x_{n+1}\right)$ and $U_n\left(x_n,y_n\right)$. To the best of our knowledge, except some first weaker results in [19], these results are new.

Proposition 1.

Let $Z_{1:N}=(Z_1,\dots ,Z_N)$ be a stochastic sequence, with$Z_n=(X_n,Y_n)$. Each $(X_n,Y_n)$ takes its values in $\mathsf{\Lambda }\times \mathsf{\Omega }$, with $\mathsf{\Lambda }$ and $\mathsf{\Omega }$ finite. If $Z_{1:N}$ is a LC-CRF with the distribution $p\left(x_{1:N}|y_{1:N}\right)$ defined by

$$p\left(x_{1:N}|y_{1:N}\right)=\frac{1}{\kappa \left(y_{1:N}\right)}exp\left[{\sum }_{n=1}^{N-1}{V}_n\left(x_n,x_{n+1}\right)+{\sum }_{n=1}^N{U}_n\left(x_n,y_n\right)\right]$$

(10)

then (Equation (10)) is the posterior distribution of the HMM

$$q\left(x_{1:N},y_{1:N}\right)=q_1\left(x_1\right)q\left(y_1|x_1\right){\prod }_{n=1}^{N-1}q\left(x_{n+1}|x_n\right)q\left(y_{n+1}|x_{n+1}\right)$$

(11)

with

$$q\left(x_1{,y}_1\right)=\frac{{\beta }_1\left(x_1{,y}_1\right)}{\sum _{\left(x_1{,y}_1\right)}{\beta }_1\left(x_1{,y}_1\right)}$$

(12)

and, for $n=1,\dots ,N-1$:

$$q\left(x_{n+1}|x_n\right)=\frac{\psi \left(x_{n+1}\right){exp[V}_n\left(x_n,x_{n+1}\right)]}{\sum _{x_{n+1}}\psi \left(x_{n+1}\right){exp[V}_n\left(x_n,x_{n+1}\right)]}$$

(13)

$$q\left(y_{n+1}|x_{n+1}\right)=\frac{exp\left[U_{n+1}\left(x_{n+1},y_{n+1}\right)\right]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}{\psi \left(x_{n+1}\right)}$$

(14)

where

$$\psi \left(x_{n+1}\right)={\sum }_{y_{n+1}}exp\left[U_{n+1}\left(x_{n+1},y_{n+1}\right)\right]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)$$

(15)

and ${\beta }_1\left(x_1,y_1\right)$, …,${\beta }_N\left(x_N,y_N\right)$are given by the backward recursion

$${\beta }_N\left(x_N,y_N\right)=1$$

$${\beta }_n\left(x_n,y_n\right)={\sum }_{\left(x_{n+1},y_{n+1}\right)}{\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)+U_{n+1}\left(x_{n+1},y_{n+1}\right)]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)$$

(16)

Proof of Proposition 1.

Let us consider functions ${\mathsf{\phi }}_1$, …, ${\mathsf{\phi }}_N$ defined on ${[\mathsf{\Lambda }\times \mathsf{\Omega }]}^2$ by

$${\phi }_1\left(x_1,y_1,x_2,y_2\right)=exp\left[V_1\left(x_1,x_2\right)+U_1\left(x_1,y_1\right)+U_2\left(x_2,y_2\right)\right]$$

(17)

$${\phi }_n\left(x_n,y_n,x_{n+1},y_{n+1}\right)={\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)+U_{n+1}\left(x_{n+1},y_{n+1}\right)],\mathrm{for}n=2,\dots ,N-1$$

(18)

According to the lemma, they define a Markov chain $Z_{1:N}=(Z_1,\dots ,Z_N)$, with $Z_n=(X_n,Y_n)$. Let us denote its distribution by ${q(z}_{1:N})=q\left(x_{1:N},y_{1:N}\right)$. As $q\left(x_{1:N},y_{1:N}\right)=Kexp\left[{\sum }_{n=1}^{N-1}{V}_n\left(x_n,x_{n+1}\right)+{\sum }_{n=1}^N{U}_n\left(x_n,y_n\right)\right]$ with $K$ constant, we have $q\left(x_{1:N}|y_{1:N}\right)=p\left(x_{1:N}|y_{1:N}\right)$. Let us show that $q\left(x_{1:N},y_{1:N}\right)$ verifies (Equations (11)–(16)). According to the lemma, considering ${\beta }_1\left(x_1,y_1\right)$, …, ${\beta }_N\left(x_N,y_N\right)$ defined with (Equation (16)), and $\psi \left(x_{n+1}\right)$ defined with (Equation (15)), we have

$$\begin{gathered} q\left(x_{n+1},y_{n+1}|x_n,y_n\right)=\frac{{\phi }_n\left(x_n,y_n,x_{n+1},y_{n+1}\right){\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}{\sum _{\left(x_{n+1},y_{n+1}\right)}{\phi }_n\left(x_n,y_n,x_{n+1},y_{n+1}\right){\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}= \\ \frac{{\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)+U_{n+1}\left(x_{n+1},y_{n+1}\right)]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}{\sum _{\left(x_{n+1},y_{n+1}\right)}{\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)+U_{n+1}\left(x_{n+1},y_{n+1}\right)]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}= \end{gathered}$$

(19)

$$\begin{gathered} \frac{{\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]\mathrm{e}\mathrm{x}\mathrm{p}[U_{n+1}\left(x_{n+1},y_{n+1}\right)]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}{\sum _{x_{n+1}}{\psi \left(x_{n+1}\right)\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]}= \\ \left[\frac{{\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]}{\sum _{x_{n+1}}{\psi \left(x_{n+1}\right)\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]}\right]\left[\exp \left[U_{n+1}\left(x_{n+1},y_{n+1}\right)\right]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)\right]= \\ \left[\frac{\psi \left(x_{n+1}\right){\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]}{\sum _{x_{n+1}}{\psi \left(x_{n+1}\right)\mathrm{e}\mathrm{x}\mathrm{p}[V}_n\left(x_n,x_{n+1}\right)]}\right]\left[\frac{\exp \left[U_{n+1}\left(x_{n+1},y_{n+1}\right)\right]{\beta }_{n+1}\left(x_{n+1},y_{n+1}\right)}{\psi \left(x_{n+1}\right)}\right]= \\ q\left(x_{n+1}|x_n\right)q\left(y_{n+1}|x_{n+1}\right) \end{gathered}$$

(12) being directly implied by the lemma, this ends the proof. □

4. Discriminative Classifiers in Generative HMMs

One of the interests of HMMs and some CRFs with hidden discrete finite data lies in possibilities of analytic fast computation of Bayesian classifiers. As examples of classic Bayesian classifiers, let us consider MPM (Equation (4)) and MAP (Equation (5)). However, in some domains such as NLP, CRFs are preferred to HMMs for the following reasons. As HMM is a generative model, MPM and MAP used in HMM are also called “generative”, and people consider that HMM-based MPM and MAP require the knowledge of $p\left(y_n|x_n\right)$. Then people consider it as improper to use the HMM-based MPM and MAP in situations where the distributions $p\left(y_n|x_n\right)$ are hard to handle. We show that this reason is not valid. More precisely, we show two points:

(i)

First, we notice that regardless of the distribution $p\left(x_{1:N},y_{1:N}\right)$, all Bayesian classifiers are independent from $p\left(y_{1:N}\right)$, so that the distinction between « generative » and « discriminative » classifiers is misleading: they are all discriminative;

(ii)

Second, we show “discriminative” computation of MPM and MAP in HMMs is not intrinsic to HMMs but is due to its particular classic parameterization Equation (1). In other words, changing the parametrization, it is possible to compute the HMM-based MPM and MAP without using $p\left(y_{1:N}|x_{1:N}\right)$ or $p\left(y_{1:N}\right)$.

The first point is rather immediate: we note that Bayesian classifier $g_L$ is defined by a loss function $L:{\mathsf{\Omega }}^2\to {{R}}^{+}$ through

$$\left[g_L\left(y_{1:N}\right)={\widehat{x}}_{1:N}\right]⟺[E\left[L\left(g_L\left(y_{1:N}\right),X_{1:N}\right)|y_{1:N}\right]=\underset{x_{1:N}}{inf}E\left[L\left(x_{1:N},X_{1:N}\right)|y_{1:N}\right]]$$

(20)

it is thus immediate to notice that $g_L\left(y_{1:N}\right)$ only depends on $p\left(x_{1:N}|y_{1:N}\right)$. This implies that it is the same in a generative model or its equivalent (within the meaning of Definition 1) discriminative model.

We show (ii) by separately considering the MPM and MAP cases.

4.1. Discriminative Computing of HMM-Based MPM

To show (ii), let us consider Equation (1) with $p\left(y_n|x_n\right)=\frac{p\left(y_n\right)p\left(x_n|y_n\right)}{p\left(x_n\right)}$. It becomes,

$$p\left(x_{1:N},y_{1:N}\right)=p\left(x_1|y_1\right){\prod }_{n=2}^{T}p\left(x_n|x_{n-1}\right)\frac{p\left(x_n|y_n\right)}{p\left(x_n\right)}{\prod }_{n=1}^{N}p\left(y_n\right)$$

(21)

We see that (Equation (21)) is of the form $p\left(x_{1:N},y_{1:N}\right)=h(x_{1:N},y_{1:N}){\prod }_{n=1}^{N}p\left(y_n\right)$, where $h(x_{1:N},y_{1:N})$ does not depend on $p\left(y_1\right)$, …, $p\left(y_N\right)$. This implies that $h\left(x_n,y_{1:N}\right)={\sum }_{(x_{1:n-1},x_{n+1:N})}h\left(x_{1:N},y_{1:N}\right)$ does not depend on $p\left(y_1\right)$, …, $p\left(y_N\right)$ either. Then $p\left(x_n|y_{1:N}\right)=\frac{p\left(x_n,y_{1:N}\right)}{p\left(y_{1:N}\right)}=\frac{h\left(x_n,y_{1:N}\right){\prod }_{n=1}^{N}p\left(y_n\right)}{[{\sum }_{x_n}h\left(x_n,y_{1:N}\right)]{\prod }_{n=1}^{N}p\left(y_n\right)}=\frac{h\left(x_n,y_{1:N}\right)}{[{\sum }_{x_n}h\left(x_n,y_{1:N}\right)]}$, so that

$$p\left(x_n|y_{1:N}\right)=\frac{{\sum }_{(x_{1:n-1},x_{n+1:N})}p\left(x_1|y_1\right){\prod }_{t=2}^{N}p\left(x_n|x_{n-1}\right)\frac{p\left(x_n|y_n\right)}{p\left(x_n\right)}}{{\sum }_{\left(x_{1:N}\right)}p\left(x_1|y_1\right){\prod }_{t=2}^{N}p\left(x_n|x_{n-1}\right)\frac{p\left(x_n|y_n\right)}{p\left(x_n\right)}}$$

(22)

neither depends on $p\left(y_1\right)$, …, $p\left(y_N\right)$. Thus, the HMM-based classifier MPM also verifies the “discriminative classifier” definition.

How to compute $p\left(x_n|y_{1:N}\right)$? It is classically computable using “forward” probabilities ${\alpha }_n\left(x_n\right)$ and “backward” ones ${\beta }_n\left(x_n\right)$ defined with

$${\alpha }_n\left(x_n\right)=p\left(x_n,y_{1:n}\right)$$

(23)

$${\beta }_n\left(x_n\right)=p\left(y_{n+1:N}|x_n\right)$$

(24)

then

$$p\left(x_n|y_{1:N}\right)=\frac{{\alpha }_n\left(x_n\right){\beta }_n\left(x_n\right)}{{\sum }_{x_n}{\alpha }_n\left(x_n\right){\beta }_n\left(x_n\right)}$$

(25)

with all ${\alpha }_n\left(x_n\right)$ and ${\beta }_n\left(x_n\right)$ computed using the following forward and backward recursions [21]:

$${\alpha }_1\left(x_1\right)=p\left(x_1\right)p\left(y_1|x_1\right);{\alpha }_{n+1}\left(x_{n+1}\right)={\sum }_{x_n}p\left(x_{n+1}|x_n\right)p\left(y_{n+1}|x_{n+1}\right){\alpha }_n(x_n)$$

(26)

$${\beta }_N\left(x_N\right)=1;{\beta }_n\left(x_n\right)={\sum }_{x_{n+1}}p\left(x_{n+1}|x_n\right)p\left(y_{n+1}|x_{n+1}\right){\beta }_{n+1}\left(x_{n+1}\right)$$

(27)

Setting $p\left(y_n|x_n\right)=\frac{p\left(y_n\right)p\left(x_n|y_n\right)}{p\left(x_n\right)}$ and recalling that $p\left(x_n|y_{1:N}\right)$ does not depend on $p\left(y_1\right)$, …, $p\left(y_N\right)$, we can arbitrarily modify them. Let us consider the uniform distribution over $\mathsf{\Omega }$, so that $p\left(y_1\right)=\dots =p\left(y_N\right)=\frac{1}{\#\mathsf{\Omega }}=c$. Then (Equation (26)) and (Equation (27)) become

$${\alpha }_{n+1}^{\mathrm{*}}\left(x_{n+1}\right)={\sum }_{x_n}p\left(x_{n+1}|x_n\right)\frac{cp\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}{\alpha }_n^{\mathrm{*}}\left(x_n\right)$$

(28)

$${\beta }_n^{\mathrm{*}}\left(x_n\right)={\sum }_{x_{n+1}}p\left(x_{n+1}|x_n\right)\frac{cp\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}{\beta }_{n+1}^{\mathrm{*}}\left(x_{n+1}\right)$$

(29)

and we still have

$$p\left(x_n|y_{1:N}\right)=\frac{{\alpha }_n^{\mathrm{*}}\left(x_n\right){\beta }_n^{\mathrm{*}}\left(x_n\right)}{{\sum }_{x_n}{\alpha }_n^{\mathrm{*}}{\beta }_n^{\mathrm{*}}\left(x_n\right)}$$

(30)

Finally, we see that $p\left(x_n|y_{1:N}\right)$ is independent from $c$, so that we can take $c=1$. Then we can state the following proposition.

Proposition 2.

Let $X_1$,$\dots ,X_N$,$Y_1$,$\dots ,Y_N$ be an HMM Equation (1). Let us define “discriminative forward” quantities ${\alpha }_1^D\left(x_1\right)$, …,${\alpha }_N^D\left(x_N\right)$, and “discriminative backward” ones ${\beta }_1^D\left(x_1\right)$, …,${\beta }_N^D\left(x_N\right)$by the following forward and backward recursions:

$${\alpha }_1^D\left(x_1\right)=p\left(x_1|y_1\right);{\alpha }_{n+1}^D\left(x_{n+1}\right)={\sum }_{x_n}p\left(x_{n+1}|x_n\right)\frac{p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}{\alpha }_n^D(x_n)$$

(31)

$${\beta }_N^D\left(x_N\right)=1;{\beta }_n^D\left(x_n\right)={\sum }_{x_{n+1}}p\left(x_{n+1}|x_n\right)\frac{p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}{\beta }_{n+1}^D\left(x_{n+1}\right)$$

(32)

then

$$p\left(x_n|y_{1:N}\right)=\frac{{\alpha }_n^D\left(x_n\right){\beta }_n^D\left(x_n\right)}{{\sum }_{x_n}{\alpha }_n^D\left(x_n\right){\beta }_n^D\left(x_n\right)}$$

(33)

Consequently, we can compute the MPM classifier in a discriminative manner, only using $p\left(x_1\right)$, …, $p\left(x_N\right)$, $p\left(x_2|x_1\right)$, …, $p\left(x_N|x_{N-1}\right)$, and $p\left(x_1|y_1\right)$, …, $p\left(x_N|y_N\right)$.

Note that this result is similar to the result in [21], with a different proof.

Remark 1.

Let us notice that according to Equations (31) and (32), it is possible to compute ${\alpha }_n^D\left(x_n\right)$and${\beta }_n^D\left(x_n\right)$ by a very slight adaptation of classic computing programs giving classic ${\alpha }_n\left(x_n\right)=p\left(x_n,y_{1:n}\right)$ and ${\beta }_n\left(x_n\right)=p\left(y_{n+1:N}|x_n\right)$ with recursions (Equations (26) and (27)). All we have to do is to replace$p\left(y_{n+1}|x_{n+1}\right)$ with $\frac{p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}$. Of course, ${\alpha }_n^D\left(x_n\right)\ne p\left(x_n,y_{1:n}\right)$ and${\beta }_n^D\left(x_n\right)\ne p\left(y_{n+1:N}|x_n\right)$, but (Equation (33)) holds and thus $p\left(x_n|y_{1:N}\right)$ is computable allowing MPM.

Remark 2.

We see that we can compute the MPM in HMM only using $p\left(x_1\right)$, …, $p\left(x_N\right)$, $p\left(x_2|x_1\right)$, …,$p\left(x_N|x_{N-1}\right)$, and $p\left(x_1|y_1\right)$, …, $p\left(x_N|y_N\right)$. This means that in supervised classification, where we have a learn sample, we can use any parametrization to estimate them. For example, we can model them with logistic regression, as currently done in CRFs. It is of importance to note that such a parametrization is unusual; however, what is important is that the model remains the same.

Remark 3.

When using the MPM to deal with a concrete problem, Proposition 2 implies that talking about a comparison between LC-CRFs and HMMs is somewhat incorrect. Indeed, there is only one model. However, there are two different parameterizations, and this can produce two different results. Indeed, the estimation of the parameters can give two models of the same nature (posterior distribution of an HMM) but unequally situated with respect to the optimal model, the best suited to the data. It would therefore be more correct to speak of a comparison of two parametrizations of HMM, each associated with its own parameter estimator. The same is true concerning the MAP discussed in the next paragraph.

4.2. Discriminative Computing of HMM-Based Map: Discriminative Viterbi

Let $X_1$, $\dots ,X_N$, $Y_1$, $\dots ,Y_N$ be an HMM (Equation (1)). The Bayesian MAP classifier (Equation (5)) based on such HMM is computed with the following Viterbi algorithm [22]. For each $n=1$, $\dots ,N$, and each $x_n$, let $x_{1:n-1}^{max}\left(x_n\right)=(x_1^{max},\dots ,x_{n-1}^{max})\left(x_n\right)$ be the path $x_1^{max}$, …, $x_{n-1}^{max}$ verifying

$$p(x_{1:n-1}^{max}\left(x_n\right),x_n,y_{1:n})=\underset{x_{1:n-1}}{sup}p(x_{1:n-1},x_n,y_{1:n})$$

(34)

We see that $x_{1:n-1}^{max}\left(x_n\right)$ is a path maximizing $p(x_{1:n-1},x_n|y_{1:n})$ over all paths ending in $x_n$. Then having the paths $x_{1:n-1}^{max}\left(x_n\right)$ and the probabilities $p(x_{1:n-1}^{max}\left(x_n\right),x_n,y_{1:n})$ for each $x_n$, one determines, for each $x_{n+1}$, the paths $x_{1:n}^{max}\left(x_{n+1}\right)$ and the probabilities $p(x_{1:n}^{max}\left(x_{n+1}\right),x_{n+1},y_{1:n+1})=p\left(x_{1:n-1}^{max}\left(x_n^{max}\right),{x_n^{max},x}_{n+1},y_{1:n+1}\right)$, searching $x_n^{max}$ with

$$\begin{gathered} p\left(x_{1:n-1}^{max}\left(x_n^{max}\right),{x_n^{max},x}_{n+1},y_{1:n+1}\right))= \\ \underset{x_n}{\mathit{sup}}\left[p\right(x_{1:n-1}^{max}\left(x_n\right),x_n,y_{1:n}\left)p\right(x_{n+1}\left|x_n\right)p\left(y_{n+1}\right|x_{n+1})] \end{gathered}$$

(35)

Setting in Equation (35) $p\left(y_{n+1}|x_{n+1}\right)=\frac{p\left(y_{n+1}\right)p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}$, we see that $x_n^{max}$ which verifies Equation (35) is the same that $x_n^{max}$ which maximizes $p(x_{1:n-1}^{max}\left(x_n\right),x_n,y_{1:n})p\left(x_{n+1}\right|x_n)\frac{p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}$, so that we can suppress $p\left(y_{n+1}\right)$. In other words, we can replace Equation (35) with

$$\begin{gathered} p\left(x_{1:n-1}^{max}\left(x_n^{max}\right),{x_n^{max},x}_{n+1},y_{1:n+1}\right))= \\ \underset{x_n}{\mathit{sup}}[p(x_{1:n-1}^{max}\left(x_n\right),x_n,y_{1:n})p(x_{n+1}|x_n)\frac{p\left(x_{n+1}|y_{n+1}\right)}{p\left(x_{n+1}\right)}] \end{gathered}$$

(36)

Finally, we propose the following discriminative version of the Viterbi algorithm:

-

Set $x_1^{max}=\underset{x_n}{\mathit{argmax}}\left[p\left(x_1|y_1\right)\right]$;

-

For each $n=1$, …, $N-1$, and each $x_{n+1}$, apply (3.17) to find a path $x_{1:n}^{max}\left(x_{n+1}\right)$ from the paths $x_{1:n-1}^{max}\left(x_n\right)$ (for all $x_n$), and the probabilities $p\left(x_{1:n}^{max}\left(x_{n+1}\right),x_{n+1},y_{1:n+1}\right)$ (for all $x_{n+1}$);

-

End setting $x_{1:N}^{max}=\underset{x_N}{\mathit{argmax}}\left[p\left(x_{1:N-1}^{max}\left(x_N\right),x_N,y_{1:N}\right)\right]$.

As with MPM above, we see that we can find $x_{1:N}^{max}$ with the only use of $p\left(x_1\right)$, …, $p\left(x_N\right)$, $p\left(x_2|x_1\right)$, …, $p\left(x_N|x_{N-1}\right)$, and $p\left(x_1|y_1\right)$, …, $p\left(x_N|y_N\right)$, exactly as in CRF case. As above, it appears that dropping HMMs in some NLP tasks on the grounds that MAP is a “generative” classifier, is not justified. In particular, in supervised stationary framework, distributions $p\left(x_n\right)$, $p\left(x_{n+1}|x_n\right)$, and $p\left(x_n|y_n\right)$ can be estimated in the same way as in LC-CRFs case.

5. Discussion and Conclusions

We have proposed two results. First, we have shown that the basic LC-CRF (Equation (3)) is equivalent to a classical HMM (Equation (1)) in that one can find an HMM whose posterior distribution is exactly the given LC-CRF. More precisely, we specified the way to calculate the parameters $p\left(x_1\right)$, $p\left(x_{n+1}|x_n\right)$, and $p\left(y_n|x_n\right)$ which define Equation (1) from the parameters $V_n\left(x_n,x_{n+1}\right)$, $U_n\left(x_n,y_n\right)$ which define Equation (3). Second, noting that all Bayesian classifiers are discriminative in the sense that they do not depend on the observation distribution, we showed that classifiers based on HMMs, usually considered as “generative”, can also be considered as discriminative classifiers. More specifically, we have proposed discriminative methods for computing classic maximum posterior mode (MPM) and maximum a posteriori (MAP) classifiers based on HMMs. The first result shows that LC-CRFs are as general as classical HMMs. The second shows that at the application level, HMMs offer strictly the same processing power, at least with regard to MPM and MAP, as LC-CRFs.

The practical interest of our contributions is as follows. Until now, some authors considered LC-CRFs and HMMs to be equivalent without presenting rigorous proof, and others considered LC-CRFs to be more general [19]. Partly because of this uncertainty, CRFs have often been preferred over HMMs. We have proved, at least in the particular framework considered, that the two models were equivalent, and the abandonment of HMMs in favor of CRFs was not always justified. In other words, faced with a particular application, there is no reason to choose CRFs systematically. However, we also cannot say that HMMs should be chosen. Finally, our contribution is likely to encourage practitioners to consider both models, or rather, according to Remark 3, both parametrizations of the same model, on an equal footing.

We considered basic LC-CRFs and HMMs, which are a limited, yet widely used framework. LC-CRFs and HMMs can be extended in different directions. The general question to study is to search an extension of HMM which would be equivalent to the general CRF (Equation (2)). This seems to be a hard problem. However, one can study some existing extensions of HMMs and wonder what kind of CRFs they would be equivalent to. For example, HMMs have been extended to “pairwise Markov models” (PMMs [23]), and the question is therefore what kind of CRFs would be equivalent to PMMs? Another kind of extension consists of adding a latent process $U_{1:N}$ to the pair ($X_{1:N},Y_{1:N})$. In the case of CRFs this leads to hidden CRFs (HCRFs [24]), and in the case of HMMs this leads to triplet Markov models (TMMs [25]). Finally, CRFs were generalized to semi-Markov CRFs [26], and HMMs were generalized to hidden semi-Markov models, with explicit distribution of the exact sojourn time in a given state [27], or with explicit distribution of the minimal sojourn time in a given state [28]. The study of the relations between these different extensions of CRFs and HMMs is an interesting perspective for further investigations.