Markov Observation Models and Deepfakes

Abstract

Herein, expanded Hidden Markov Models (HMMs) are considered as potential deepfake generation and detection tools. The most specific model is the HMM, while the most general is the pairwise Markov chain (PMC). In between, the Markov observation model (MOM) is proposed, where the observations form a Markov chain conditionally on the hidden state. An expectation-maximization (EM) analog to the Baum–Welch algorithm is developed to estimate the transition probabilities as well as the initial hidden-state-observation joint distribution for all the models considered. This new EM algorithm also includes a recursive log-likelihood equation so that model selection can be performed (after parameter convergence). Once models have been learnt through the EM algorithm, deepfakes are generated through simulation, while they are detected using the log-likelihood. Our three models were compared empirically in terms of their generative and detective ability. PMC and MOM consistently produced the best deepfake generator and detector, respectively.

1. Introduction

Hidden Markov Models (HMMs) were introduced in papers by Baum and Petrie [1] and Baum and Eagon [2]. Traditional HMMs have enjoyed tremendous modelling success in applications like computational finance (see e.g., Petropoulos et al. [3]), single-molecule kinetic analysis (see Nicolai [4]), animal tracking (see Sidrow et al. [5]), forecasting commodity futures (see Date et al. [6]) and protein folding (see Stigler et al. [7]). The unobservable hidden HMM states ${\displaystyle X}$ are a discrete-time Markov chain, and the observations process ${\displaystyle Y}$ is some distorted, corrupted partial information or measurement of the current state of ${\displaystyle X}$, satisfying the following condition:

$${\displaystyle P\left(Y_n\in A|X_n,X_{n-1},\dots ,X_1\right)=P\left(Y_n\in A|X_n\right).}$$

These emission probabilities, ${\displaystyle P\left(Y_n\in A|X_n\right)}$, have a conditional probability mass function $y\to b_{X_n}\left(y\right)$.

Perhaps the most common challenges in HMMs are calibrating the model, decoding the hidden sequence from the observation sequence and real-time belief propagation, i.e., filtering. The first problem is solved recursively in the HMM setting by the Baum–Welch re-estimation algorithm, which is an application of the Expectation-Maximization (EM) algorithm, predating the EM algorithm. The second, decoding problem is solved by the Viterbi algorithm (see Viterbi [8], Rabiner [9], Shinghal and Toussaint [10]), which is a dynamic programming algorithm. The filtering problem is also solved effectively after calibration using a recursive algorithm that is similar to part of the Baum–Welch algorithm. In practice, there can be numeric problems, like a multitude of local maxima to trap the Baum-Welch algorithm, or inefficient matrix operations when the state size is large but the hidden state resides in a small subset most of the time. In these cases, it can be advisable to use particle filters or other alternative methods, which are not the subject of this paper (see instead Cappé et al. [11] for more information). The forward and backward propagation probabilities of the Baum–Welch algorithm also tend to become very small over time, a phenomenon known as the small-number problem. While satisfactory results can sometimes be obtained by (often logarithmic) rescaling, this small-number problem is still a severe limitation of the Baum–Welch algorithm. However, the independent emission form of the observation modelling undertaken in HMMs can be even more fundamentally limiting.

The autoregressive HMM (AR-HMM) and, more generally, the pairwise Markov chain (PMC) were introduced to allow more extensive and practical observation models. For the AR-HMM, the observations take the following structure:

$$Y_n={\beta }_0^{\left(X_n\right)}+{\beta }_1^{\left(X_n\right)}{Y}_{n-1}+\dots +{\beta }_p^{\left(X_n\right)}{Y}_{n-p}+{\epsilon }_n,$$

(1)

where ${\left\{{\epsilon }_n\right\}}_{n=1}^{\infty }$ is a (usually zero-mean Gaussian) i.i.d. sequence of random variables, and the autoregressive coefficients are functions of the current hidden state $X_n$. The AR-HMM has experienced strong success in applications like speech recognition (see Bryan and Levinson [12]), the diagnosis of blood infections (see Stanculescu et al. [13]) and the study of climate patterns (see Xuan [14]). One advantage of the AR-HMM is that the Baum–Welch algorithm can still be used (see Bryan and Levinson [12]).

The general PMC model from Pieczynski [15] only assumes that $(X,Y)$ is jointly Markov. Derrode and Pieczynski [16], Derrode and Pieczynski [17] and Kuljus and Lember [18] explain the generality of the PMC and give some interesting subclasses of this model. It is now well understood how to filter and decode PMCs. In fact, Kuljus and Lember [18] solve the decoding problem in great generality, while Derrode and Pieczynski [17] use Baum–Welch-like recursions to produce the filter. Both Derrode and Pieczynski [16] and Derrode and Pieczynski [17] assume reversibility of the PMC and have the observations living in a continuous space. To our knowledge, the Baum–Welch rate re-estimation algorithm has not been validated in general for PMCs. Our first goal is to develop and validate this Baum–Welch algorithm for PMCs, while at the same time estimating hidden initial states and overcoming the small-number problem mentioned above by using alternative variables in our forward and backward recursions. Our resulting EM algorithm will apply to many big data problems.

Our second goal is to show the applicability of HMMs and PMCs, as well as a model called the Markov Observation Model (MOM), which falls part way between HMMs and PMCs in deepfake detection and generation. The key to producing and detecting deepfakes is to bring in an element that is easily calculated, yet often overlooked in HMMs and PMCs: likelihood. During training, as well as during detection, likelihood can be used in the place of the discriminator in a Generative Adversarial Network (GAN), while simulation plays the part of the generator. Naturally, the expectation-maximization algorithm also plays a key role in this deepfake application, as explained below.

Our third goal is subtler. Just because the PMC model is more general than the HMM, and the Baum–Welch algorithm can be extended to learn either model, does not mean one should pronounce the death of the HMM. The problem is that the additional generality leads, in general, to a more complicated likelihood with a multitude of maxima for the EM algorithm to become trapped in or choose from. It can become a virtually impossible task to learn a global, or even a useful, maximum. Hence, the performance of the PMC model as a hidden Markov structure can be sub-optimal compared to the performance of the HMM or MOM, as we shall show empirically. Alternatively, the global maximum of the PMC may not be what is wanted. For these reasons, we promote the MOM and, in fact, show that it performs the best in simple deepfake detection, while the PMC generates the best deepfakes.

The HMM and nonlinear filtering theory (NFT) can each be thought of as nonlinear generalization of the Kalman filter (see Kalman [19], Kalman and Bucy [20]). The recent analogues (see [21]) of the celebrated Fujisaki–Kallianpur–Kunita and the Duncan–Mortensen–Zakai equations (see [22,23,24,25,26] for some original and general results) of NFT to continuous-time Markov chain observations provide further evidence of the closeness of the HMM and NFT. The hidden state, called the signal in NFT, can be a general Markov process model and live in a general state space, but there is no universal EM algorithm for identifying the model, like the Baum–Welch algorithm, nor a dynamic programming algorithm for identifying a most likely hidden-state path, like the Viterbi algorithm. Rather, the goals in NFT are usually to compute filters, predictors and smoothers, for which there are no exact closed-form solutions, except in isolated cases (see [27]), and approximations have to be used. Like HMMs, nonlinear filtering has enjoyed widespread application. For instance, the subfield of nonlinear particle filtering, also known as sequential Monte Carlo, has a number of powerful algorithms (see Elfring [28], Pitt and Shephard [29], Del Moral et al. [30], Kouritzin [31], Chopin and Papaspiliopoulos [32]) and has been applied to numerous problems in areas like Bayesian inference (Chopin [33], Kloek and van Dijk [34], van Dijk and Kloek [35]), bioinformatics (Hajiramezanali et al. [36]), economics and mathematical finance (Creal [37]), intracellular movement (Maroulas and Nebenführ [38]), fault detection (D’Amato et al. [39]), pharmacokinetics (Bonate [40]), geosciences (Van Leeuwen et al. [41]), and many other fields. Still, like in HMMs, the observations in nonlinear filter models are largely limited to distorted, corrupted, partial observations of the signal. NFT is used successfully in deepfake generation and detection herein. However, the simplicity of the EM and likelihood algorithms for HMMs, MOMs and PMCs are compelling advantages here in the deepfake application but likely also in some of these other applications of NFT.

The layout of this paper is as follows: In the next section, we explain the models, in particular the Markov observation models, and how they can be simulated. In Section 3 the filter and likelihood calculations are derived. In Section 4, EM techniques are used to derive an analog to the Baum–Welch algorithm for identifying the system (probability) parameters. In particular, joint recursive formulas for the hidden-state and observation transition probabilities, as well as the initial hidden-state-observation joint distribution, are derived. Section 5 contains our deepfake application and results. Section 6 is devoted to connecting the limit points of the EM-type algorithm to the maxima of the conditional likelihood, given the observations. Finally, Section 7 clarifies our contributions, makes our most basic conclusions and suggests some future work the author hopes will be undertaken.

2. Models and Simulation

Let $N\in \mathbb{N}$ be some final time. We first clarify the HMM assumption of independent emission probabilities.

Under the HMM,

$$P(Y_1=y_1,\dots ,Y_N=y_N\mid {\left\{X_i\right\}}_{i=1}^N)=\prod _{i=1}^N{b}_{X_i}\left(y_i\right),\forall y_i,$$

(2)

where $y\to b_x\left(y\right)$ is a probability mass function for each x. Otherwise, the HMM and PMC are explained elsewhere.

Next, we explain how the MOM generalizes the HMM and fits into the PMC. Suppose ${\displaystyle O}$ is some discrete observation space. In the MOM, like in the HMM, the hidden state is a homogeneous Markov chain ${\displaystyle X}$ on some discrete (finite or countable) state space ${\displaystyle E}$ with one-step transition probabilities ${\displaystyle p_{x\to x^{\prime }}}$ for ${\displaystyle x,x^{\prime }\in E}$. Contrary to the HMM, the MOM allows self-dependence in the observations (this is illustrated by rightward arrows between the Ys in Figure 1). In particular, MOM observations Y are a (conditional) Markov chain, given the hidden state with the following transition probabilities:

$${\displaystyle P\left(Y_{n+1}=y|{\{X_i=x_i\}}_{i=0}^{n+1},{\{Y_j=y_j\}}_{j=0}^n\right)=q_{y_n\to y}\left(x_{n+1}\right)\forall x_0,\dots ,x_N\in E;y,y_n\in O}$$

(3)

These do not affect the hidden-state transitions, in the sense that

$$P(X_{n+1}=\widehat{x}|X_n=x,{\left\{X_i\right\}}_{i<n},{\left\{Y_j\right\}}_{j\le n})=p_{x\to \widehat{x}},\forall x,\widehat{x}\in E,n\in {\mathbb{N}}_0$$

(4)

Still, (3) implies that

$$P\left(Y_{n+1}=y|{\left\{X_i\right\}}_{i=0}^{n+1},{\left\{Y_j\right\}}_{j\le n}\right)=P\left(Y_{n+1}=y|X_{n+1},Y_n\right),\forall y\in O$$

(5)

i.e., that the new observation only depends upon the new hidden state (as well as the past observation). Equations (3) and (4) imply that the hidden-state observation pair ${\displaystyle \left(\begin{array}{c}X\\ Y\end{array}\right)}$ is jointly Markov with joint one-step transition probabilities:

$${\displaystyle P\left(X_{n+1}=x,Y_{n+1}=y|X_n=x_n,Y_n=y_n\right)=p_{x_n\to x}{q}_{y_n\to y}\left(x\right)\forall x,x_n\in E;y,y_n\in O.}$$

The joint Markov property then implies that

${\displaystyle P\left(X_{n+1}=x,Y_{n+1}=y|X_1=x_1,Y_1=y_1,X_2=x_2,Y_2=y_2,\dots ,X_n=x_n,Y_n=y_n\right)=p_{x_n\to x}{q}_{y_n\to y}\left(x\right).}$

Notice that this generalizes the emisson probability to

$${\displaystyle P\left(Y_n=y|X_n,X_{n-1},\dots ,X_1;Y_{n-1},\dots ,Y_1\right)=P\left(Y_n=y|Y_{n-1},X_n\right)=q_{Y_{n-1}\to y}\left(X_n\right)}$$

(6)

so the MOM generalizes the HMM by just taking $q_{Y_{n-1}\to y}\left(X_n\right)=b_{X_n}\left(y\right)$, a state-dependent probability mass function. To see how the MOM is related to the AR-HMM, we rewrite (1) as

$$\underset{{\mathcal{Y}}_n}{\underbrace{\left[\begin{array}{c}{Y}_n\\ Y_{n-1}\\ Y_{n-2}\\ ⋮\\ Y_{n-p+1}\end{array}\right]}}=\left[\begin{array}{ccccc}{\beta }_1^{\left(X_n\right)}& {\beta }_2^{\left(X_n\right)}& {\beta }_3^{\left(X_n\right)}& \dots & {\beta }_p^{\left(X_n\right)}\\ 1& 0& 0& \dots & 0\\ 0& 1& 0& \dots & 0\\ ⋮& & \ddots & ⋮\\ 0& 0& 0& \dots 1& 0\end{array}\right]\underset{{\mathcal{Y}}_{n-1}}{\underbrace{\left[\begin{array}{c}{Y}_{n-1}\\ Y_{n-2}\\ Y_{n-3}\\ ⋮\\ Y_{n-p}\end{array}\right]}}+\left[\begin{array}{c}{\beta }_0^{\left(X_n\right)}+{\epsilon }_n\\ 0\\ 0\\ ⋮\\ 0\end{array}\right],$$

(7)

which, given the hidden state $X_n$, gives an explicit formula for ${\mathcal{Y}}_n$ in terms of only ${\mathcal{Y}}_{n-1}$ and some independent noise ${\epsilon }_n$. Hence, $\left\{{\mathcal{Y}}_n\right\}$ is obviously conditionally Markov, and $\left\{(X_n,{\mathcal{Y}}_n)\right\}$ is an MOM. We have not claimed that this subsumes the AR-HMM yet, because $\left\{{\epsilon }_n\right\}$ is usually Gaussian in the AR-HMM, and we handle the case of discrete noise herein. This will be further discussed in Section 7.

A subtly that arises with the MOM over the HMM is that we need an enlarged initial distribution, since we have a $Y_0$ that is not observed (see Figure 1). Rather, we think of starting up the observation process at time 1, even though there were observations to be made prior to this time. Further, since we generally do not know the model parameters, we need a means to estimate this initial distribution

$${\displaystyle P\left(X_0\in d{x}_0,Y_0\in d{y}_0\right)=\mu \left(d{x}_0,d{y}_0\right).}$$

It is worth noting that the MOM resembles the stationary PMC under Condition (H) in Pieczynski [15], which forces the hidden state to be Markov by Proposition 2.2 of Pieczynski [15].

Simulation

Any PMC is characterized by an initial distribution $\mu$ on $E\times O$ and a joint transition probability $p_{x,y\to \widehat{x},\widehat{y}}$ for its hidden state and observations. In particular,

$$p_{x,y\to \widehat{x},\widehat{y}}=p_{x\to \widehat{x}}{q}_{y\to \widehat{y}}\left(\widehat{x}\right)$$

(8)

for the MOM and

$$p_{x,y\to \widehat{x},\widehat{y}}=p_{x\to \widehat{x}}{b}_{\widehat{x}}\left(\widehat{y}\right)$$

(9)

for the HMM. In any case, the marginal transitions are denoted as

$$p_{x,y\to \widehat{x}}=\sum _{\widehat{y}}{p}_{x,y\to \widehat{x},\widehat{y}}\mathrm{and}{p}_{x,y\to \widehat{y}}=\sum _{\widehat{x}}{p}_{x,y\to \widehat{x},\widehat{y}}.$$

(10)

$\mu$, p characterize a $(\mu ,p)$-PMC. The initial distribution $\mu$ gives the distribution of $(X_0,Y_0)$ for the MOM and PMC, while the initial distribution ${\mu }_X$ gives the distribution of $X_1$ for the HMM by convention. This convention makes sense since the MOM and PMC have observation history to model in some unknown $Y_0$. In the case of the HMM, an initial $(X_1,Y_1)$ can then be drawn from $\mu (x,y)={\mu }_X\left(x\right)b_x\left(y\right)$.

The simulation of the HMM, MOM and PMC observations is performed in the same way: Begin by drawing $(X_0,Y_0)$ ($(X_1,Y_1)$ for the HMM) from $\mu$, continue the simulation using $p_{x,y\to \widehat{x},\widehat{y}}$ and then finally throw out the hidden state X (as well as $Y_0$ for MOM and PMC) to leave the observation process Y.

3. Likelihood, Filter and Predictor

A PMC is parameterized by its initial distribution $\mu$ and joint transition probability p for its hidden state and observations. Its ability to fit a given sequence of observations $Y_1,\dots ,Y_n$ up to time n is naturally judged by its likelihood:

$$L_n=L_n^{\mu ,p}=P(Y_1,\dots ,Y_n)=P^{\mu ,p}(Y_1,\dots ,Y_n)\mathrm{for}\mathrm{all}n\ge 1\mathrm{with}{L}_0=1.$$

(11)

Here, $P^{\mu ,p}$ is a probability measure, where ${\displaystyle \left(\begin{array}{c}X\\ Y\end{array}\right)}$ is a $(\mu ,p)$-PMC. Therefore, given several $({\mu }_1,p_1),..,({\mu }_m,p_m)$ PMC models, perhaps found by different runs of an expectation-maximization algorithm, as well as an observation $Y_1,\dots ,Y_N$ data sequence, one can use the likelihoods ${\left\{L_n^{{\mu }_i,p_i}\right\}}_{i=1}^m$ to judge which model best fits the data. Each run of the EM algorithm would converge to a local maximum of the likelihood function, and then the likelihood function could be used to determine which of these produces a higher maximum. Since the MOM and HMM are PMCs (with specific p given in (8) and (9)), this test extends to judging the best MOM and best HMM.

In applications like filtering, the hidden state has significance, and estimating (the distribution of) it is important. The (optimal) filter is the (conditional) hidden-state probability mass function

$${\pi }_n\left(x\right)\circeq P(X_n=x|Y_1,\dots ,Y_n)\forall x\in E,n\ge 1.$$

(12)

We first work with the PMC, and then extract the MOM and HMM from these calculations. The likelihood and filter can computed together in real time using the forward probability

$$\left\{\begin{array}{ccc}{\displaystyle {\alpha }_0\left(x,y\right)}\hfill & =\hfill & P\left(Y_0=y,X_0=x\right)\hfill \\ {\alpha }_n\left(x\right)\hfill & =\hfill & P\left(Y_1,\dots ,Y_n,X_n=x\right),1\le n\le N\hfill \end{array}\right.,$$

(13)

which is motivated from the Baum–Welch algorithm. Then, it follows from (11)–(13) that

$${\pi }_n\left(x\right)={\displaystyle \frac{{\alpha }_n\left(x\right)}{{\displaystyle \sum _{\xi }}{\alpha }_n\left(\xi \right)}}={\displaystyle \frac{{\alpha }_n\left(x\right)}{L_n}}\mathrm{so} L_n=\sum _{\xi }{\alpha }_n\left(\xi \right)\forall n\ge 1\mathrm{and}{\pi }_0(x,y)={\alpha }_0\left(x,y\right).$$

(14)

Moreover, we obtain, based on the multiplication rule, the joint Markov property and (13), the following:

$$\begin{array}{ccc}\hfill {\displaystyle }& & {\alpha }_n\left(x\right)\hfill \\ \hfill & =& P\left(Y_1,\dots ,Y_n,X_n=x\right)\hfill \\ \hfill & =& \sum _{x_{n-1}}P\left(Y_1,\dots ,Y_n,X_{n-1}=x_{n-1},X_n=x\right)\hfill \\ \hfill & =& \sum _{x_{n-1}}P\left(Y_1,\dots ,Y_{n-1},X_{n-1}=x_{n-1}\right)P(X_n=x,Y_n|Y_1,\dots ,Y_{n-1},X_{n-1}=x_{n-1})\hfill \\ \hfill & =& \sum _{x_{n-1}}{\alpha }_{n-1}\left(x_{n-1}\right)p_{x_{n-1},Y_{n-1}\to x,Y_n},\hfill \end{array}$$

(15)

which can be solved recursively for ${\displaystyle n=2,3,\dots ,N-1,N}$, starting (according to (13)) at

$${\alpha }_1\left(x_1\right)=\sum _{x_0}\sum _{y_0}\mu (x_0,y_0)p_{x_0,y_0\to x_1,Y_1}.$$

(16)

Recall that ${\alpha }_0=\mu$ is assigned differently. On a computer, we do not recurse ${\alpha }_n$, due to risk of underflow (the small-number problem), but rather we revert back to the filter ${\pi }_n$. Using (14) and (15), one finds that the forward recursion for $\pi$ is

$${\displaystyle {\rho }_n\left(x\right)=\sum _{x_{n-1}}{\pi }_{n-1}\left(x_{n-1}\right)p_{x_{n-1},Y_{n-1}\to x,Y_n},{\pi }_n\left(x\right)=\frac{{\rho }_n\left(x\right)}{a_n},a_n=\sum _{x_n}{\rho }_n\left(x_n\right),}$$

(17)

which can be solved forward for ${\displaystyle n=2,3,\dots ,N-1,N}$, starting at

$${\displaystyle {\pi }_1\left(x\right)=\frac{{\displaystyle \sum _{x_0}}{\displaystyle \sum _{y_0}}\mu (x_0,y_0)p_{x_0,y_0\to x,Y_1}}{a_1},a_1=\sum _{x_1}\sum _{x_0}\sum _{y_0}\mu (x_0,y_0)p_{x_0,y_0\to x_1,Y_1}.}$$

(18)

This immediately implies that $L_1=a_1$, and then, by using (14), (17) and induction, that

$$L_n=a_1{a}_2\dots a_n⇝L_n=L_{n-1}{a}_n,L_0=1.$$

(19)

Thus, the filter and likelihood can be computed in real time (after initialization) via the recursions in (17) and (19).

Once the filter has been computed, predictors can also be computed using Chapman–Kolmogorov-type equations. For example, it follows from the multiplication rule and the Markov property that the one-step predictor is

$$\begin{array}{ccc}\hfill P(Y_{n+1}=y_{n+1}\mid Y_1,\dots ,Y_n)& =& \sum _{x_n,x_n+1}{\displaystyle \frac{P(Y_{n+1}=y_{n+1},X_{n+1}=x_{n+1},X_n=x_n,Y_1,\dots ,Y_n)}{P(Y_1,\dots ,Y_n)}}\hfill \\ \hfill & =& \sum _{x_n,x_n+1}P(Y_{n+1}=y_{n+1},X_{n+1}=x_{n+1}\mid X_n=x_n,Y_1,\dots ,Y_n)P(X_n=x_n\mid Y_1,\dots ,Y_n)\hfill \\ \hfill & =& \sum _{x_n,x_n+1}{p}_{x_n,Y_n\to x_{n+1},y_{n+1}}{\pi }_n\left(x_n\right),\hfill \end{array}$$

(20)

which reduces to

$$\begin{array}{c}\hfill P(Y_{n+1}=y_{n+1}\mid Y_1,\dots ,Y_n)=\sum _{x_n,x_n+1}{p}_{x_n\to x_{n+1}}{q}_{Y_n\to y_{n+1}\left(x_{n+1}\right)}{\pi }_n\left(x_n\right),\end{array}$$

(21)

and

$$\begin{array}{c}\hfill P(Y_{n+1}=y_{n+1}\mid Y_1,\dots ,Y_n)=\sum _{x_n,x_n+1}{p}_{x_n\to x_{n+1}}{b}_{x_{n+1}}\left(y_{n+1}\right){\pi }_n\left(x_n\right)\end{array}$$

(22)

respectively in the cases of the MOM and HMM.

In non-real-time applications, we strengthen our hidden-state estimates to include future observations via the joint path filter

$${\Pi }_{n-1,n}(x,\widehat{x})=P(X_{n-1}=x,X_n=\widehat{x}\mid Y_1,\dots ,Y_N),$$

(23)

which is a joint pmf for $n=2,\dots ,N$. To compute the joint path filter, we first let

$$\left\{\begin{array}{ccc}{\displaystyle {\beta }_0\left(x_0,x_1,y\right)}\hfill & =\hfill & P\left(Y_1,\dots ,Y_N|X_0=x_0,X_1=x_1,Y_0=y\right)\hfill \\ {\displaystyle {\beta }_n\left(x_n,x_{n+1}\right)}\hfill & =\hfill & P\left(Y_{n+1},\dots ,Y_N|X_n=x_n,X_{n+1}=x_{n+1},Y_n\right),\forall 0<n<N-1\hfill \\ {\displaystyle {\beta }_{N-1}\left(x_{N-1},x_N\right)}\hfill & =\hfill & {\displaystyle P\left(Y_N|X_{N-1}=x_{N-1},X_N=x_N,Y_{N-1}\right)=\frac{p_{x_{N-1},Y_{N-1}\to x_N,Y_N}}{p_{x_{N-1},Y_{N-1}\to x_N}}}\hfill \end{array}\right.,$$

(24)

where the last equality follows from the definition of conditional probability, and the normalized versions of $\beta$:

$${\chi }_n(x,\widehat{x})={\displaystyle \frac{{\beta }_n(x,\widehat{x})}{a_{n+1}\dots a_N}},\forall n=1,\dots ,N-1\mathrm{and}{\chi }_0(x,\widehat{x},y)={\displaystyle \frac{{\beta }_0(x,\widehat{x},y)}{a_1\dots a_N}}.$$

(25)

Notice that we include an extra variable y in ${\alpha }_0,{\beta }_0$. This is because we do not see the first observation $Y_0$, so we have to consider all possibilities and treat it like another hidden state. Then, based on (11), (13), the Markov property, (19) and (14), the following is obtained:

$$\begin{array}{ccc}\hfill {\Pi }_{n-1,n}(x,\widehat{x})& =& {\displaystyle \frac{P(X_{n-1}=x,X_n=\widehat{x},Y_1,\dots ,Y_N)}{P(Y_1,\dots ,Y_N)}}\hfill \\ \hfill & =& {\displaystyle \frac{{\alpha }_{n-1}\left(x\right)P(X_n=\widehat{x},Y_n,\dots ,Y_N|X_{n-1}=x,Y_1,\dots ,Y_{n-1})}{L_N}}\hfill \\ \hfill & =& {\displaystyle \frac{{\pi }_{n-1}\left(x\right)P(X_n=\widehat{x},Y_n,\dots ,Y_N|X_{n-1}=x,Y_{n-1})}{a_n\dots a_N}}\hfill \end{array}$$

(26)

so based on (24)–(26),

$$\begin{array}{ccc}\hfill & & {\Pi }_{n-1,n}(x,\widehat{x})\hfill \\ \hfill & =& {\displaystyle \frac{{\pi }_{n-1}\left(x\right)P(X_n=\widehat{x},Y_n,\dots ,Y_N,X_{n-1}=x,Y_{n-1})P(X_n=\widehat{x},X_{n-1}=x,Y_{n-1})}{a_n\dots a_{N}P(X_n=\widehat{x},X_{n-1}=x,Y_{n-1})P(X_{n-1}=x,Y_{n-1})}}\hfill \\ \hfill & =& {\displaystyle \frac{{\pi }_{n-1}\left(x\right)P(Y_n,\dots ,Y_N|X_n=\widehat{x},X_{n-1}=x,Y_{n-1})P(X_n=\widehat{x}|X_{n-1}=x,Y_{n-1})}{a_n\dots a_N}}\hfill \\ \hfill & =& {\displaystyle {\pi }_{n-1}\left(x\right){\chi }_{n-1}(x,\widehat{x})p_{x,Y_{n-1}\to \widehat{x}}}\hfill \end{array}$$

(27)

for ${\displaystyle n=2,3,\dots ,N}$. This means that there are two ways to compute the (marginal) path filter directly from (27):

$${\displaystyle {\Pi }_n\left(x\right)=P(X_n=x\mid Y_1,\dots ,Y_N)={\pi }_n\left(x\right)\sum _{x_{n+1}}{\chi }_n(x,x_{n+1})p_{x,Y_n\to x_{n+1}}}$$

(28)

for ${\displaystyle n=1,2,\dots ,N-1}$, and

$${\displaystyle {\Pi }_n\left(x\right)=P(X_n=x\mid Y_1,\dots ,Y_N)=\sum _{x_{n-1}}{\chi }_{n-1}\left(x_{n-1},x\right)p_{x_{n-1},Y_{n-1}\to x}{\pi }_{n-1}\left(x_{n-1}\right)}$$

(29)

for ${\displaystyle n=2,3,\dots ,N}$. These all become computationally effective by a backward recursion for $\chi$. It also follows from (24), the definition of conditional probability, the Markov property, partitioning and our transition probabilities that

$$\begin{array}{ccc}\hfill {\beta }_n\left(x_n,x\right)& =& P\left(Y_{n+1},\dots ,Y_N\mid X_n=x_n,X_{n+1}=x,Y_n\right)\hfill \\ \hfill & =& P\left(Y_{n+2},\dots ,Y_N\mid X_n=x_n,X_{n+1}=x,Y_{n+1},Y_n\right)P\left(Y_{n+1}\mid X_n=x_n,X_{n+1}=x,Y_n\right)\hfill \\ \hfill & =& P\left(Y_{n+2},\dots ,Y_N\mid X_{n+1}=x,Y_{n+1}\right){\displaystyle \frac{p_{x_n,Y_n\to x,Y_{n+1}}}{p_{x_n,Y_n\to x}}}\hfill \\ \hfill & =& \sum _{x^{\prime }\in E}P\left(Y_{n+2},\dots ,Y_N\mid X_{n+2}=x^{\prime },X_{n+1}=x,Y_{n+1}\right)\hfill \\ \hfill & & \ast P\left(X_{n+2}=x^{\prime }\mid X_{n+1}=x,Y_{n+1}\right){\displaystyle \frac{p_{x_n,Y_n\to x,Y_{n+1}}}{p_{x_n,Y_n\to x}}}\hfill \\ \hfill & =& {\displaystyle \frac{p_{x_n,Y_n\to x,Y_{n+1}}}{p_{x_n,Y_n\to x}}}\sum _{x^{\prime }}{\beta }_{n+1}(x,x^{\prime })p_{x,Y_{n+1}\to x^{\prime }},\hfill \end{array}$$

(30)

so normalizing by (25), the following can be obtained:

$$\begin{array}{ccc}\hfill {\chi }_n\left(x_n,x\right)& =& {\displaystyle \frac{p_{x_n,Y_n\to x,Y_{n+1}}}{a_{n+1}{p}_{x_n,Y_n\to x}}}\sum _{x^{\prime }}{\chi }_{n+1}(x,x^{\prime })p_{x,Y_{n+1}\to x^{\prime }},\hfill \end{array}$$

(31)

which can be solved backward for ${\displaystyle n=N-1,N-2,\dots ,3,2,1}$, starting from

$${\chi }_N\left(x_N,x_{N+1}\right)=1.$$

(32)

The $n=0$ value for $\pi$ and $\chi$ becomes

$$\begin{array}{ccc}\hfill {\chi }_0\left(x_0,x_1,y\right)& =& {\displaystyle \frac{p_{x_0,y\to x_1,Y_1}}{a_1{p}_{x_0,y\to x_1}}}\sum _{x^{\prime }}{\chi }_1(x_1,x^{\prime })p_{x_1,Y_1\to x^{\prime }},\hfill \end{array}$$

(33)

$$\begin{array}{ccc}\hfill {\pi }_0\left(x,y\right)& =& {\alpha }_0\left(x,y\right)=\mu \left(x,y\right)\hfill \end{array}$$

(34)

to account for the fact that we do not see $Y_0$ as the data turns on at time 1. With ${\chi }_0$ in hand, we can estimate the joint distribution of $(X_0,Y_0)$, which are the remaining hidden variables. It follows from Bayes’ rule, (11), (19), the multiplication rule, (24) and (25) that

$$\begin{array}{ccc}\hfill {\Pi }_0(x,y)& =& P(X_0=x,Y_0=y\mid Y_1,\dots ,Y_N)\hfill \\ \hfill & =& {\displaystyle \frac{P(Y_1,\dots ,Y_N\mid X_0=x,Y_0=y)P(X_0=x,Y_0=y)}{L_N}}\hfill \\ \hfill & =& {\displaystyle \frac{{\displaystyle \sum _{x_1}}P(Y_1,..,Y_N\mid X_1=x_1,X_0=x,Y_0=y)P(X_1=x_1\mid X_0=x,Y_0=y)\mu (x,y)}{a_1\dots a_N}}\hfill \\ \hfill & =& \mu (x,y)\sum _{x_1}{\chi }_0(x,x_1,y)p_{x,y\to x_1}.\hfill \end{array}$$

(35)

for all $x\in E$, $y\in O$.

The pathspace filter and likelihood algorithm is given in Algorithm 1.

Algorithm 1: Path filter and likelihood for PMC

    The first part of Algorithm 1, up to the first set of outputs, runs in real time, as the observations arrive, and provides the real-time filter and likelihood. For real-time applications, one would stop there, or else add predictors not included in Algorithm 1 but given as an example in (20). Otherwise, one can refine the estimates of the hidden states based on future observations, which then provides the pathspace filters and is the key to learning a model. This is the second part of Algorithm 1, and is explained below. But first, we note that the recursions developed so far are easily tuned to an MOM or HMM.

3.1. MOM Adjustments

For the MOM, we use (8). We leave (13), (14) and (19) unchanged, so (17) and (18) become

$${\displaystyle {\rho }_n\left(x\right)=q_{Y_{n-1}\to Y_n}\left(x\right)\sum _{x_{n-1}\in E}{\pi }_{n-1}\left(x_{n-1}\right)p_{x_{n-1}\to x},{\pi }_n\left(x\right)=\frac{{\rho }_n\left(x\right)}{a_n},a_n=\sum _{x_n}{\rho }_n\left(x_n\right),}$$

(36)

for all ${\displaystyle x\in E}$, which can be solved forward for ${\displaystyle n=2,3,\dots ,N-1,N}$, starting at

$${\displaystyle {\pi }_1\left(x\right)=\frac{{\displaystyle \sum _{x_0}}{\displaystyle \sum _{y_0}}\mu (x_0,y_0)p_{x_0\to x}{q}_{y_0\to Y_1}\left(x\right)}{a_1},a_1=\sum _{x_1}\sum _{x_0}\sum _{y_0}\mu (x_0,y_0)p_{x_0\to x_1}{q}_{y_0\to Y_1}\left(x_1\right).}$$

(37)

The backward recursions change a little more, starting with (24) and (25), which change to

$$\left\{\begin{array}{ccc}{\displaystyle {\beta }_0\left(x_1,y\right)}\hfill & =\hfill & P\left(Y_1,\dots ,Y_N|X_1=x_1,Y_0=y\right)\hfill \\ {\displaystyle {\beta }_n\left(x_{n+1}\right)}\hfill & =\hfill & P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1},Y_n\right),\forall 0<n<N-1\hfill \\ {\displaystyle {\beta }_{N-1}\left(x_N\right)}\hfill & =\hfill & P\left(Y_N|X_N=x_N,Y_{N-1}\right)=q_{Y_{N-1}\to Y_N}\left(x_N\right)\hfill \end{array}\right.$$

(38)

and the normalized versions

$${\chi }_n\left(\widehat{x}\right)={\displaystyle \frac{{\beta }_n\left(\widehat{x}\right)}{a_{n+1}\dots a_N}},\forall n=1,\dots ,N-1\mathrm{and}{\chi }_0(\widehat{x},y)={\displaystyle \frac{{\beta }_0(\widehat{x},y)}{a_1\dots a_N}}$$

(39)

since

$$P(Y_{n+1},\dots ,Y_N|X_n=x_n,X_{n+1}=x_{n+1},Y_n)=P(Y_n,\dots ,Y_N|X_{n+1}=x_{n+1},Y_n)$$

(40)

by Lemma 1 (to follow). Then, (27) becomes

$$\begin{array}{c}\hfill {\displaystyle {\Pi }_{n-1,n}(x,\widehat{x})={\pi }_{n-1}\left(x\right){\chi }_{n-1}\left(\widehat{x}\right)p_{x\to \widehat{x}}}\end{array}$$

(41)

for ${\displaystyle n=2,3,\dots ,N}$. This then implies the obvious simplifications of (28) and (29) to

$${\displaystyle {\Pi }_n\left(x\right)={\pi }_n\left(x\right)\sum _{x_{n+1}}{\chi }_n\left(x_{n+1}\right)p_{x\to x_{n+1}}\mathrm{and}{\Pi }_n\left(x\right)={\chi }_{n-1}\left(x\right)\sum _{x_{n-1}}{p}_{x_{n-1}\to x}{\pi }_{n-1}\left(x_{n-1}\right)}$$

(42)

for ${\displaystyle n=1,2,\dots ,N-1}$ and ${\displaystyle n=2,3,\dots ,N}$, respectively. Then, (31) becomes

$$\begin{array}{ccc}\hfill {\chi }_n\left(x\right)& =& {\displaystyle \frac{q_{Y_n\to Y_{n+1}}\left(x\right)}{a_{n+1}}}\sum _{x^{\prime }}{\chi }_{n+1}\left(x^{\prime }\right)p_{x\to x^{\prime }}\hfill \end{array}$$

(43)

by (5), which is solved backwards starting from ${\chi }_N\left(x_{N+1}\right)=1$. The values at $n=0$ become

$$\begin{array}{ccc}\hfill {\chi }_0\left(x_1,y\right)& =& {\displaystyle \frac{q_{y\to Y_1}\left(x_1\right)}{a_1}}\sum _{x^{\prime }}{\chi }_1\left(x^{\prime }\right)p_{x_1\to x^{\prime }},{\pi }_0\left(x,y\right)=\mu \left(x,y\right)\hfill \end{array}$$

(44)

and

$$\begin{array}{c}\hfill {\Pi }_0(x,y)=\mu (x,y)\sum _{x_1}{\chi }_0(x_1,y)p_{x\to x_1}.\end{array}$$

(45)

for all $x\in E$, $y\in O$.

3.2. HMM Adjustments

For the HMM, we use (9). We have a MOM with the specific

$$q_{y\to \widehat{y}}\left(\widehat{x}\right)=b_{\widehat{x}}\left(\widehat{y}\right)$$

(46)

that also starts at $n=1$ with $\mu (x,y)={\mu }_X\left(x\right)b_x\left(y\right)$, instead of $n=0$. This creates modest changes or simplifications for the filter startup:

$$\begin{array}{c}\hfill {\rho }_1\left(x\right)=b_x\left(Y_1\right){\mu }_X\left(x\right),a_1=\sum _x{\rho }_1\left(x\right),{\pi }_1\left(x\right)={\displaystyle \frac{{\rho }_1\left(x\right)}{a_1}}.\end{array}$$

(47)

But otherwise, (36) holds with just the substitution $q_{y\to \widehat{y}}\left(\widehat{x}\right)=b_{\widehat{x}}\left(\widehat{y}\right)$.

To handle the backward recursion, we first reduce the general definition of $\beta$ in (24), using (2), to

$$\left\{\begin{array}{ccc}{\displaystyle {\beta }_n\left(x_{n+1}\right)}\hfill & =\hfill & P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1}\right),\forall 0<n<N-1\hfill \\ {\displaystyle {\beta }_{N-1}\left(x_N\right)}\hfill & =\hfill & P\left(Y_N|X_N=x_N\right)=b_{x_N}\left(Y_N\right)\hfill \end{array}\right.$$

(48)

and the normalized versions

$${\chi }_n\left(x\right)={\displaystyle \frac{{\beta }_n\left(x\right)}{a_{n+1}\dots a_N}},\forall n=1,\dots ,N-1.$$

(49)

There are no ${\alpha }_0$, ${\pi }_0$, ${\beta }_0$ or ${\chi }_0$ variables for the HMM. The HMM’s backward-recursion simplifications are based on the following result.

Lemma 1.

For the MOM and the HMM,

$$P\left(Y_{n+1},\dots ,Y_N|X_n=x_n,X_{n+1}=x_{n+1},Y_n\right)=\left\{\begin{array}{cc}P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1},Y_n\right)\hfill & \mathit{for}\mathit{MOM}\hfill \\ P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1}\right)\hfill & \mathit{for}\mathit{HMM}\hfill \end{array}\right..$$

Proof. 

For the MOM, we have

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{P\left(Y_n,\dots ,Y_N,X_n=x_n,X_{n+1}=x_{n+1}\right)}{P\left(Y_n,X_n=x_n,X_{n+1}=x_{n+1}\right)}}\hfill \\ \hfill & =& {\displaystyle \frac{{\sum }_{x_{n+2},\dots ,x_N}P\left(X_n=x_n,Y_n\right)p_{x_n\to x_{n+1}}{q}_{Y_n\to Y_{n+1}}\left(x_{n+1}\right)p_{x_{n+1}\to x_{n+2}}\dots q_{Y_{N-1}\to Y_N}\left(x_N\right)p_{x_{N-1}\to x_N}}{P\left(X_n=x_n,Y_n\right)p_{x_n\to x_{n+1}}}}\hfill \\ \hfill & =& \sum _{x_{n+2},\dots ,x_N}{q}_{Y_n\to Y_{n+1}}\left(x_{n+1}\right)p_{x_{n+1}\to x_{n+2}}{q}_{Y_{n+1}\to Y_{n+2}}\left(x_{n+2}\right)\dots p_{x_{N-1}\to x_N}{q}_{Y_{N-1}\to Y_N}\left(x_N\right)\hfill \\ \hfill & =& P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1},Y_n\right).\hfill \end{array}$$

(50)

In the case of the HMM, this becomes. However, it follows from the multiplication rule, the tower property and (2) that

$$\begin{array}{ccc}\hfill & & P\left(Y_{n+1},\dots ,Y_N|X_n=x_n,X_{n+1}=x_{n+1},Y_n\right)\hfill \\ \hfill & =& \sum _{x_{n+2},\dots ,x_N}{b}_{x_{n+1}}\left(Y_{n+1}\right)p_{x_{n+1}\to x_{n+2}}{b}_{x_{n+2}}\left(Y_{n+2}\right)\dots p_{x_{N-1}\to x_N}{b}_{x_N}\left(Y_N\right)\hfill \\ \hfill & =& P\left(Y_{n+1},\dots ,Y_N|X_{n+1}=x_{n+1}\right)\hfill \end{array}$$

(51)

which establishes the desired dependence.    □

Finally, the initial probability estimate comes from Bayes rule, (11), (24) and (25):

$$\begin{array}{ccc}\hfill {\displaystyle {\Pi }_1\left(x\right)}& =& P(X_1=x|Y_1,\dots ,Y_N)\hfill \\ \hfill & =& {\displaystyle \frac{P(Y_1,\dots ,Y_N|X_1=x)P(X_1=x)}{P(Y_1,\dots ,Y_N)}}\hfill \\ \hfill & =& {\displaystyle \frac{{\beta }_1\left(x\right){\mu }_X\left(x\right)}{L_N}}\hfill \\ \hfill & =& {\chi }_1\left(x\right){\mu }_X\left(x\right).\hfill \end{array}$$

(52)

4. Probability Estimation via EM Algorithm

In this section, we develop a recursive expectation-maximization algorithm that can be used to create convergent estimates for the transition and initial probabilities of our models. We leave the theoretical justification of convergence to Section 6.

The main goal of developing an EM algorithm is to find ${\displaystyle p_{x,y\to \widehat{x},\widehat{y}}}$ for all ${\displaystyle x,\widehat{x}\in E}$, ${\displaystyle y,\widehat{y}\in O}$ and $\mu (x,y)$ for all ${\displaystyle x\in E}$, $y\in O$. Noting that every time step is considered to be a transition in a discrete-time Markov chain, we would ideally set the following:

$$\begin{array}{ccc}& & {\displaystyle p_{x,y\to \widehat{x},\widehat{y}}=\frac{\mathrm{Expected} \mathrm{transitions} (x,y) \mathrm{to} (\widehat{x},\widehat{y}) \mathrm{given} \mathrm{observations}}{\mathrm{Eepected} \mathrm{occurrences} \mathrm{of} (x,y) \mathrm{given} \mathrm{observations}}}\hfill \\ \hfill & =& {\displaystyle \frac{1_{Y_1=\widehat{y}}P(Y_0=y,X_0=x,X_1=\widehat{x}|Y_1,\dots ,Y_N)+{\displaystyle \sum _{n=2}^N}{1}_{Y_{n-1}=y,Y_n=\widehat{y}}P(X_{n-1}=x,X_n=\widehat{x}|Y_1,\dots ,Y_N)}{P(Y_0=y,X_0=x|Y_1,\dots ,Y_N)+{\displaystyle \sum _{n=2}^N}{1}_{Y_{n-1}=y}P(X_{n-1}=x|Y_1,\dots ,Y_N)}},\hfill \end{array}$$

(53)

which means that we must compute ${\displaystyle P(Y_0=y,X_0=x,X_1=\widehat{x}|Y_1,\dots ,Y_N)}$, ${\displaystyle P(Y_0=y,X_0=x|Y_1,\dots ,Y_N)}$ and, using (23) and (28), ${\displaystyle {\Pi }_n=\left(x\right)}$ for all $0\le n\le N$, and ${\displaystyle {\Pi }_{n-1,n}(x,\widehat{x})}$ for all ${\displaystyle 1\le n\le N}$, to get this transition probability estimate. Now, by Bayes’ rule, ((11), (19)), ((24), (25)) and ((13), (14)), we obtain the following:

$$\begin{array}{ccc}\hfill & & P(Y_0=y,X_0=x,X_1=\widehat{x}|Y_1,\dots ,Y_N)\hfill \\ \hfill & =& {\displaystyle \frac{P(Y_1,\dots ,Y_N|X_1=\widehat{x},X_0=x,Y_0=y)P(X_1=\widehat{x},X_0=x,Y_0=y)}{a_1\dots a_N}}\hfill \\ \hfill & =& {\displaystyle {\chi }_0(x,\widehat{x},y)p_{x,y\to \widehat{x}}{\pi }_0(x,y)}\hfill \end{array}$$

(54)

so

$$\begin{array}{ccc}\hfill {\Pi }_{0,1}(x,\widehat{x})& =& {\displaystyle \sum _y{\pi }_0(x,y)p_{x,y\to \widehat{x}}{\chi }_0(x,\widehat{x},y)}\hfill \end{array}$$

(55)

and so

$${\displaystyle {\Pi }_0\left(x\right)=\sum _{y,\widehat{x}}{\pi }_0(x,y)p_{x,y\to \widehat{x}}{\chi }_0(x,\widehat{x},y).}$$

(56)

${\pi }_n$ and ${\chi }_n$ are computed recursively in (17) and (31) using the prior estimates of ${\displaystyle p_{x,y\to \widehat{x}},\widehat{y}}$ and $\mu$.

Expectation-maximization algorithms use these types of formulas and prior estimates to produce better estimates. We take estimates for ${\displaystyle p_{x,y\to \widehat{x},\widehat{y}}}$, and $\mu (x,y)$ and obtain new estimates for these quantities iteratively using (53), (54), (27), (35) and (28):

$$p_{x,y\to \widehat{x},\widehat{y}}^{\prime }={\displaystyle \frac{1_{Y_1=\widehat{y}}{\pi }_0(x,y)p_{x,y\to \widehat{x}}{\chi }_0(x,\widehat{x},y)+{\displaystyle \sum _{n=1}^{N-1}}{1}_{Y_n=y,Y_{n+1}=\widehat{y}}{\pi }_n\left(x\right)p_{x,y\to \widehat{x}}{\chi }_n(x,\widehat{x})}{{\pi }_0(x,y){\displaystyle \sum _{x_1}}{p}_{x,y\to x_1}{\chi }_0(x,x_1,y)+{\displaystyle \sum _{n=1}^{N-1}}{1}_{Y_n=y}{\pi }_n\left(x\right){\displaystyle \sum _{x_{n+1}}}{p}_{x,y\to x_{n+1}}{\chi }_n\left(x,x_{n+1}\right)}},$$

(57)

and using (35),

$$\begin{array}{ccc}\hfill {\mu }^{\prime }(x,y)& =& \sum _{x_1}{\chi }_0(x,x_1,y)p_{x,y\to x_1}\mu (x,y).\hfill \end{array}$$

(58)

Remark 1.

(1) Different iterations of $p_{x,y\to \widehat{x},\widehat{y}},\mu (x,y)$ will be used on the left- and right-hand sides of (57) and (58). The new estimates on the left are denoted as $p_{x,y\to \widehat{x},\widehat{y}}^{\prime },{\mu }^{\prime }(x,y)$.

(2) Setting the marginal $p_{x,y\to \widehat{x}}=0$ or probability $\mu (x,y)=0$ will result in it staying zero for all updates. This effectively removes this parameter from the EM optimization update, and should be avoided unless it is known that one of these should be 0.

(3) If there are no successive observations with $Y_n=y$ and $Y_{n+1}=\widehat{y}$ in the actual observation sequence, then all new estimates $p_{x,y\to \widehat{x},\widehat{y}}^{\prime }$ will either be set to 0 or close to it. They might not be exactly zero, due to the first term in the numerator of (57), where we could have an estimate of $Y_0=y$ and an observed $Y_1=\widehat{y}$.

We now have everything required for our EM algorithms, which are given for the PMC, MOM and HMM cases in Algorithms 2, 3 and 4 respectively.

These algorithms start with the initial estimates $p_{x,y\to \widehat{x},\widehat{y}}^1$, ${\mu }^1(x,y)$ of $p_{x,y\to \widehat{x},\widehat{y}}$, $\mu (x,y)$, and refine them successively to new estimates $p_{x,y\to \widehat{x},\widehat{y}}^2$, ${\mu }^2(x,y)$; $p_{x,y\to \widehat{x},\widehat{y}}^3$, ${\mu }^3(x,y)$; etc. It is important to know that our estimates $\{p_{x,y\to \widehat{x},\widehat{y}}^k,{\mu }^k(x,y)\}$ improve as $k\to \infty$.

Lemma 3 (below) will be used to ensure that an initially positive estimate stays positive as k increases, which is important in our proofs in Section 6. The following lemma follows easily from (31)–(33), (17), (18), (34), induction and the fact that ${\displaystyle \sum _{x^{\prime }}}{p}_{x,Y_{n+1}\to x^{\prime }}=1$. A sensible initialization of our EM algorithm would ensure that the condition $p_{x,Y_n\to \widehat{x},Y_{n+1}}>0$ holds.

Lemma 2.

Suppose $p_{x,Y_n\to \widehat{x},Y_{n+1}}>0$ for all $x,\widehat{x}\in E$ and $n\in \{1,\dots ,N-1\}$. Then,

1. 

${\chi }_m(x,\widehat{x})>0$ for all $x,\widehat{x}\in E$ and $m\in \{1,\dots ,N-1\}$.

2. 

${\chi }_0(x,\widehat{x},y)>0$ for any $x,\widehat{x}\in E,y\in O$, such that $p_{x,y\to \widehat{x},Y_1}>0$.

3. 

${\pi }_m\left(x\right)>0$ for all $x\in E$ and $m\in \{1,\dots ,N\}$ if, in addition, ${\displaystyle \sum _{x_0,y_0}}\mu (x_0,y_0)p_{x_0,y_0\to \widehat{x},Y_1}>0$ for all $\widehat{x}\in E$.

4. 

${\pi }_0(x,y)>0$ if $\mu (x,y)>0$.

The following result is the key to ensuring that our non-zero parameters stay non-zero. It follows from the prior lemma, as well as (57), (58) and (31).

Lemma 3.

Suppose $N\ge 2$, $p_{x,Y_n\to \widehat{x},Y_{n+1}}>0$ for all $x,\widehat{x}\in E$ and $n\in \{1,\dots ,N-1\}$. Then,

1. 

$p_{x,y\to \widehat{x},\widehat{y}}^{\prime }>0$ if $p_{x,y\to \widehat{x},\widehat{y}}>0$; $\{Y_n=y,Y_{n+1}=\widehat{y}\}$ occurs; and ${\displaystyle \sum _{x_0,y_0}}\mu (x_0,y_0)p_{x_0,y_0\to x,Y_1}>0$ for all $x,x_0\in E$.

2. 

${\mu }^{\prime }(x,y)>0$ if $\mu (x,y)>0$ and there exists $\widehat{x}$ such that $p_{x,y\to \widehat{x},Y_1}>0$.

Algorithm 2: EM algorithm for PMC

Algorithm 3: EM algorithm for MOM

Algorithm 4: EM algorithm for HMM

5. Deepfake Application

Motivated by [42], we considered our three hidden models in deepfake generation and detection. In particular, we used the models’ EM, simulation and Bayes’ factor capabilities to generate and detect deepfake real coin-flip sequences, and then compared them to determine which of the three is the best at both generation and detection.

We first created 137 real sequences of 400 coin flips by generating independent fair Bernoulli trials. Another 137 hand fake sequences of 200 coin flips were created by students with knowledge of undergraduate probability. They were told to make them look real to try to fool both humans and machines. Note that we worked with coin flip sequences with a length of 200, except for the training with real sequences, where a length of 400 was used so that length was not a defining factor of these real sequences. This added length to the real sequences did not bias either of the HMM, MOM or PMC over the others, as it was consistent for all.

We used HMM, MOM and PMC simulation with a single hidden-state variable taking s possible values (henceforth referred to as s states) to generate deepfake sequences of 200 coin flips based on the 137 real sequences. To do this, we first learnt each of the 137 real sequences using the EM algorithms with $s+1$ hidden states for each model, creating three collections of 137 parameter sets for each s. Then, we simulated a sequence from each set of parameters, throwing the hidden states away, creating three collections of 137 observation coin-flip sequences for each s. These were the HMM-, MOM- and PMC-type deepfake sequences. Note that learning was conducted based on the 400 long real sequences (to remove noise from the parameters), but we created 200 long deepfake sequences.

Once all five sets of (real, fake and deepfake) data had been collected, we ran 100 training and testing trials at each selected s and averaged over these trials. For each trial, we randomly and independently split each of the 137 (hand) fake sequences into 110 training and 27 testing sequences, i.e., an 80-to-20 split. Conversely, we regenerated the 137 independent sets of real sequences and 3 deepfake sequences using, respectively, independent random number and Markov chain simulation with their models, but still divided these sets into 110 training and 27 testing sequences. We then trained the HMM, MOM and PMC with s hidden states on each of these sets of 110 training sequences. Note that since the deepfake sequences were generated with $s+1$ hidden states, the actual model generating these sequences could not be identified. At this point, we had 110 sets of HMM parameters (i.e., HMM models) for each of the real, hand fake, HMM, MOM and PMC different training sequences in that trial. Similarly, we had 550 sets of MOM and PMC parameters.

Detection for each testing sequence was carried out using all the models. In a trial, each of the five sets of 27 sequences was run against the 550 HMM, 550 MOM and 550 PMC models. A sequence was then predicted by the HMM to be real, hand fake, HMM-generated, MOM-generated or PMC-generated based on HMM likelihood with s hidden states. In particular, a sequence was predicted to be real if the sum of the log-likelihood over the 110 real HMM models was higher than that over the 110 hand fake, 110 HMM, 110 MOM and 110 PMC HMM models. In the same way, it was predicted to be hand fake, HMM, MOM or PMC by the HMM. This same procedure was repeated for the MOM and for the PMC, and then for the remaining 99 trials, using the regeneration method mentioned above. The results were averaged and put into

Table 1. Generative and detection ability with $s=3$. Blue highlight indicates this detection method is the best detector, while orange indicates the generation method is the most difficult to detect bythis detection method.

	Real (%)	Handfake (%)	HMM (%)	MOM (%)	PMC (%)	Overall (%)
HMM detection	99.96	93.36	76.89	78.25	59.79	81.65
Standard deviation	0.357	3.590	25.343	9.841	27.386	10.076
MOM detection	99.03	89.39	98.39	91.31	77.11	91.11
Standard deviation	2.250	0.612	2.347	9.370	5.129	2.148
PMC detection	100	70.14	95.18	90.04	88.07	88.69
Standard deviation	0.0	2.243	1.990	3.491	5.519	1.402
Overall detection	99.66	84.30	90.15	86.53	74.99	87.15
Standard deviation	0.759	1.425	8.510	4.677	9.343	3.466

,

Table 2. Generative and detection ability with $s=5$.

	Real (%)	Handfake (%)	HMM (%)	MOM (%)	PMC (%)	Overall (%)
HMM detection	100	94.79	73.61	64.89	63.25	79.31
Standard deviation	0	3.383	27.013	24.905	19.987	11.739
MOM detection	98.79	89.29	95.32	87.90	79.96	90.30
Standard deviation	2.101	0.001	3.685	11.203	9.868	3.040
PMC detection	96.71	70.82	89.54	84.18	92.32	86.71
Standard deviation	2.470	1.688	1.917	3.526	4.607	1.218
Overall detection	98.5	84.97	86.16	78.99	78.51	85.44
Standard deviation	1.081	1.260	9.110	9.179	7.587	4.062

and

Table 3. Generative and detection ability with $s=7$.

	Real (%)	Handfake (%)	HMM (%)	MOM (%)	PMC (%)	Overall (%)
HMM detection	100	95.00	41.5	55.68	33.89	65.21
Standard deviation	0	3.003	29.270	28.099	22.608	12.141
MOM detection	98.76	89.29	96.96	90.52	90.82	93.29
Standard deviation	2.166	0.001	3.419	12.049	7.998	2.531
PMC detection	99.82	73.25	95.75	94.21	88.32	90.27
Standard deviation	0.782	2.298	1.736	2.723	5.464	1.230
Overall detection	99.53	85.85	78.07	80.14	71.01	82.92
Standard deviation	0.768	1.260	9.989	10.231	8.198	4.154

in the cases $s=3,5$ and 7, respectively.

6. Convergence of Probabilities

In this section, we establish the convergence properties of the transition probabilities and the initial distribution $\{p_{x,y\to \widehat{x},\widehat{y}}^k,{\mu }^k(x,y)\}$ that we derived in Section 4. Our method adapts the ideas of Baum et al. [43], Liporace [44] and Wu [45] to our setting.

We think of the transition probabilities and initial distribution as parameters, and let $\Theta$ denote all of the non-zero transition and initial distribution probabilities in $p,\mu$. Let $e=\left|E\right|$ and $o=\left|O\right|$ be the cardinalities of the hidden and observation spaces, and set $d^{\prime }=e+o$. Then, $p_{x,y\to \widehat{x},\widehat{y}}:{(E\times O)}^2\to [0,1]$ has a domain space of cardinality ${\left(d^{\prime }\right)}^2$, and $\mu (x,y)\in {[0,1]}^{E\otimes O}$ has a domain space of cardinality $e\times o$. Combined, this leads to ${\left(d^{\prime }\right)}^2+e\times o$ parameters. However, we are removing the values that will be set to zero and adding sum to one constraints to consider a constrained optimization problem on ${(0,\infty )}^d$ for some $d\le {\left(d^{\prime }\right)}^2+e\times o$. Removing these zero possibilities gives us the necessary regularity for our re-estimation procedure. However, it is not enough to just remove them at the beginning. We have to ensure that zero parameters will not creep in during our interations, or else we will be doing such things as taking logarithms of 0. Lemma 3 suggests that estimates not initially set to zeros will not occur as zero in later iterations. In general, we will assume the following:

Definition 1.

A sequence of estimates $\{p^k,q^k,{\mu }^k\}$ is zero-separating if

1. 

$p_{x,y\to \widehat{x},\widehat{y}}^1>0$ iff $p_{x,y\to \widehat{x},\widehat{y}}^k>0$ for all $k=1,2,3,\dots$,

2. 

${\mu }^1(x,y)>0$ iff ${\mu }^k(x,y)>0$ for all $k=1,2,3,\dots$.

Here, iff stands for if and only if.

This means that we can potentially optimize over the $p,\mu$ that we initially do not set to zero. Henceforth, we factor the zero $p,\mu$ out of $\Theta$, consider $\Theta \subset {(0,\infty )}^d$ with $d\le d^{\prime }$ and define the parameterized mass functions

$$\begin{array}{c}\hfill p_{y_0,y_1,\dots ,y_N}(x;\Theta )=p_{x_0,y_0\to x_1,y_1}{p}_{x_1,y_1\to x_2,y_2}\dots p_{x_{N-1},y_{N-1}\to x_N,y_N}\mu (x_0,y_0)\end{array}$$

(59)

in terms of the non-zero values only. The observable likelihood

$$\begin{array}{ccc}\hfill P_{Y_1,\dots ,Y_N}(\Theta )& =& \sum _{x_0,x_1,\dots ,x_N}\sum _{y_0}{p}_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;\Theta )\hfill \end{array}$$

(60)

is not changed by removing the zero values of $p,\mu$, and this removal allows us to define the re-estimation function

$$\begin{array}{ccc}\hfill Q_{Y_1,\dots ,Y_N}(\Theta ,{\Theta }^{\prime })& =& \sum _{x_0,\dots ,x_N}\sum _{y_0}{p}_{y_0,Y_1,\dots ,Y_N}(x_0,\dots ,x_N;\Theta )ln{p}_{y_0,Y_1,\dots ,Y_N}(x_0,\dots ,x_N;{\Theta }^{\prime }).\hfill \end{array}$$

(61)

Note: Here, and in the sequel, the summation in $P,Q$ above is only over the non-zero combinations. We would not include an $x_i,x_{i+1}$ pair where $p_{x_i,Y_j\to x_{i+1},Y_{j+1}}=0$, nor an $x_0,y_0$ pair where $\mu (x_0,y_0)=0$. Hence, our parameter space is

$$\Gamma =\{\Theta \in {(0,\infty )}^d:\sum _{\widehat{x},\widehat{y}}{p}_{x,y\to \widehat{x},\widehat{y}}=1,\sum _{x,y}\mu (x,y)=1\}.$$

Later, we will consider the extended parameter space

$$K=\{\Theta \in {[0,1]}^d:\sum _{\widehat{x},\widehat{y}}{p}_{x,y\to \widehat{x},\widehat{y}}=1,\sum _{x,y}\mu (x,y)=1\}$$

as limit points. Note that in both $\Gamma$ and K, $\Theta$ is only over the $p_{x,y\to \widehat{x},\widehat{y}}$ and $\mu (x,y)$ that are not just set to 0 (before limits).

Then, equating $Y_0$ with $y_0$ to ease notation, one obtains the following:

$$\begin{array}{ccc}\hfill Q(\Theta ,{\Theta }^{\prime })& =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\mu (x_0,y_0)\hfill \\ \hfill & & \left[\sum _{m=1}^{N}ln{p}_{x_{m-1},Y_{m-1}\to x_m,Y_m}^{\prime }+ln{\mu }^{\prime }(x_0,y_0)\right].\hfill \end{array}$$

(62)

The re-estimation function is used to interpret the EM algorithm we derived earlier. We impose the following condition to ensure everything is well defined.

• (Zero) The EM estimates are zero-separating.

The following result is motivated by Theorem 3 of Liporace [44].

Theorem 1.

Suppose (Zero) holds. The expectation-maximization solutions (57) and (58) derived in Section 4 are the unique critical point of the re-estimation function ${\Theta }^{\prime }\to Q(\Theta ,{\Theta }^{\prime })$, subject to ${\Theta }^{\prime }$ forming probability mass functions. This critical point is a maximum taking value in ${(0,1]}^d$ for d explained above.

We consider it as an optimization problem over the open set ${(0,\infty )}^d$, but with the constraint that we have mass functions, so the values have to be in the set ${(0,1]}^d$.

Proof. 

One obtains based on (62), as well as the constraint ${\sum }_{\widehat{x},\widehat{y}}{p}_{x,y\to \widehat{x},\widehat{y}}^{\prime }=1$, that the maximum must satisfy

$$\begin{array}{ccc}\hfill 0& =& {\displaystyle \frac{\partial }{\partial p_{x,y\to \widehat{x},\widehat{y}}^{\prime }}}\left\{Q(\Theta ,{\Theta }^{\prime })-\lambda \left(\sum _{\xi ,\theta }{p}_{x,y\to \xi ,\theta }^{\prime }-1\right)\right\}\hfill \\ \hfill & =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\sum _{m=1}^N{\displaystyle \frac{1_{x_{m-1}=x,Y_{m-1}=y}{1}_{x_m=\widehat{x},Y_m=\widehat{y}}}{p_{x,y\to \widehat{x},\widehat{y}}^{\prime }}}\mu (x_0,y_0)-\lambda \hfill \end{array}$$

(63)

where $\lambda$ is a Lagrange multiplier and $Y_{m-1}=y$ means $Y_0=y_0$ when $m=1$. Multiplying by $p_{x,y\to \widehat{x},\widehat{y}}^{\prime }$, summing over $\widehat{x},\widehat{y}$ and then using (11), (35) and (28) and then (19), (14) and (25), one determines that

$$\begin{array}{ccc}\hfill \lambda & =& \sum _{m=1}^N\sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]1_{x_{m-1}=x,Y_{m-1}=y}\mu (x_0,y_0)\hfill \\ \hfill & =& P(X_0=x,Y_0=y,Y_1,\dots ,Y_N)+\sum _{m=2}^N{1}_{Y_{m-1}=y}P(X_{m-1}=x,Y_1,\dots ,Y_N)\hfill \\ \hfill & =& {\Pi }_0(x,y)L_N+\sum _{m=2}^N{1}_{Y_{m-1}=y}{\Pi }_{m-1}\left(x\right)L_N\hfill \\ \hfill & =& \sum _{x_1}{\beta }_0(x,x_1,y)p_{x,y\to x_1}{\alpha }_0(x,y)+\sum _{m=2}^N\sum _{x_m}{1}_{Y_{m-1}=y}{\beta }_{m-1}(x,x_m)p_{x,Y_{m-1}\to x_m}{\alpha }_{m-1}\left(x\right).\hfill \end{array}$$

(64)

Substituting (64) into (63) and repeating the argument in (64), but with (27) instead of (28), one determines that

$$\begin{array}{ccc}\hfill {\displaystyle p_{x,y\to \widehat{x},\widehat{y}}^{\prime }}& =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\sum _{m=1}^N{\displaystyle \frac{1_{x_{m-1}=x,Y_{m-1}=y,x_m=\widehat{x},Y_m=\widehat{y}}}{\lambda }}\mu (x_0,y_0)\hfill \\ \hfill & =& {\displaystyle \frac{1_{Y_1=\widehat{y}}P(X_0=x,Y_0=y,X_1=\widehat{x},Y_1,\dots ,Y_N)+{\displaystyle \sum _{m=2}^N}{1}_{Y_{m-1}=y,Y_m=\widehat{y}}P(X_{m-1}=x,X_m=\widehat{x},Y_1,\dots ,Y_N)}{{\displaystyle \sum _{x_1}}{\beta }_0(x,x_1,y)p_{x,y\to x_1}{\alpha }_0(x,y)+{\displaystyle \sum _{m=2}^N}{\displaystyle \sum _{x_m}}{1}_{Y_{m-1}=y}{\beta }_{m-1}(x,x_m)p_{x,Y_{m-1}\to x_m}{\alpha }_{m-1}\left(x\right)}}\hfill \\ \hfill & =& {\displaystyle \frac{1_{Y_1=\widehat{y}}{\chi }_0(x,\widehat{x},y)p_{x,y\to \widehat{x}}{\pi }_0(x,y)+{\displaystyle \sum _{m=2}^N}{1}_{Y_{m-1}=y,Y_m=\widehat{y}}{\chi }_{m-1}(x,\widehat{x})p_{x,Y_{m-1}\to \widehat{x}}{\pi }_{m-1}\left(x\right)}{{\displaystyle \sum _{x_1}}{\chi }_0(x,x_1,y)p_{x,y\to x_1}{\pi }_0(x,y)+{\displaystyle \sum _{m=2}^N}{\displaystyle \sum _{x_m}}{1}_{Y_{m-1}=y}{\chi }_{m-1}(x,x_m)p_{x,Y_{m-1}\to x_m}{\pi }_{m-1}\left(x\right)}}.\hfill \end{array}$$

(65)

To explain the first term in the numerator in the last equality, we use the multiplication rule and (24) to find

$$P(X_0=x,Y_0=y,X_1=\widehat{x},Y_1,\dots ,Y_N)={\beta }_0(x,\widehat{x},y)P(X_0=x,Y_0=y,X_1=\widehat{x})={\chi }_0(x,\widehat{x},y)L_N{\pi }_0(x,y)p_{x,y\to \widehat{x}}$$

from which it will follow easily.

Finally, for a maximum, one also requires

$$\begin{array}{ccc}\hfill 0& =& {\displaystyle \frac{\partial }{\partial {\mu }^{\prime }(x,y)}}\left\{Q(\Theta ,{\Theta }^{\prime })-\lambda \left(\sum _{\xi \in E,\theta \in O}{\mu }^{\prime }(\xi ,\theta )-1\right)\right\}\hfill \\ \hfill & =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]{\displaystyle \frac{1_{x_0=x}{1}_{y_0=y}}{{\mu }^{\prime }(x,y)}}\mu (x_0,y_0)-\lambda ,\hfill \end{array}$$

(66)

where $\lambda$ is a Lagrange multiplier. Multiplying by ${\mu }^{\prime }(x,y)$ and summing over $x,y$, one obtains that

$$\begin{array}{ccc}\hfill \lambda & =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\mu (x_0,y_0)\hfill \\ \hfill & =& P(Y_1,\dots ,Y_N)\hfill \\ \hfill & =& L_N.\hfill \end{array}$$

(67)

Substituting (67) into (66), one obtains by (35) that

$$\begin{array}{ccc}\hfill {\displaystyle {\mu }^{\prime }(x,y)}& =& {\displaystyle \frac{{\displaystyle \sum _{x_0,\dots ,x_N}}{\displaystyle \sum _{y_0}}\left[{\displaystyle \prod _{n=1}^N}{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]1_{x_0=x}{1}_{y_0=y}\mu (x_0,y_0)}{L_N}}\hfill \\ \hfill & =& {\displaystyle \frac{P(X_0=x,Y_0=y,Y_1,\dots ,Y_N)}{L_N}}\hfill \\ \hfill & =& {\pi }_0(x,y){\displaystyle \sum _{x_1}}{\chi }_0(x,x_1,y)p_{x,y\to x_1}.\hfill \end{array}$$

(68)

Now, we have established that the EM algorithm of Section 4 corresponds to the unique critical point of ${\Theta }^{\prime }\to Q(\Theta ,{\Theta }^{\prime })$. Moreover, all mixed partial derivatives of Q in the components of ${\Theta }^{\prime }$ are 0, while

$$\begin{array}{ccc}& & {\displaystyle \frac{{\partial }^2{Q}_{Y_1,Y_2,\dots ,Y_N}(\Theta ,{\Theta }^{\prime })}{{\partial p_{x,y\to \widehat{x},\widehat{y}}^{\prime }}^2}}\hfill \\ \hfill & =& -\sum _{y_0;x_0,\dots ,x_N}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\sum _{m=1}^N{\displaystyle \frac{1_{X_{m-1}=x,Y_{m-1}=y,x_m=\widehat{x},Y_m=\widehat{y}}}{{p_{x,y\to \widehat{x},\widehat{y}}^{\prime }}^2}}\mu (x_0,y_0)\hfill \end{array}$$

(69)

and

$$\begin{array}{ccc}& & {\displaystyle \frac{{\partial }^2{Q}_{Y_1,Y_2,\dots ,Y_N}(\Theta ,{\Theta }^{\prime })}{{\partial {\mu }^{\prime }(x,y)}^2}}\hfill \\ \hfill & =& -\sum _{y_0;x_0,\dots ,x_N}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\sum _{m=1}^N{\displaystyle \frac{1_{y_0=y,x_0=x}}{{{\mu }^{\prime }(x,y)}^2}}\mu (x_0,y_0).\hfill \end{array}$$

(70)

Hence, the Hessian matrix is diagonal with negative values along its axis, and the critical point is a maximum. □

The upshot of this result is that if the EM algorithm produces parameters $\left\{{\Theta }^k\right\}\subset \Gamma$, then $Q({\Theta }^k,{\Theta }^{k+1})\ge Q({\Theta }^k,{\Theta }^k)$.

Now, we have the following result, based on Theorem 2.1 of Baum et al. [43], that establishes that the observable likelihood is also increasing i.e., $P\left({\Theta }^{k+1}\right)\ge P\left({\Theta }^k\right)$.

Lemma 4.

Suppose (Zero) holds. $Q(\Theta ,{\Theta }^{\prime })\ge Q(\Theta ,\Theta )$ implies $P\left({\Theta }^{\prime }\right)\ge P(\Theta )$. Moreover, $Q(\Theta ,{\Theta }^{\prime })>Q(\Theta ,\Theta )$ implies $P\left({\Theta }^{\prime }\right)>P(\Theta )$.

Proof. 

$ln\left(t\right)$ for $t>0$ has convex inverse $exp\left(t\right)$. Hence, by Jensen’s inequality,

$$\begin{array}{ccc}\hfill {\displaystyle }& & {\displaystyle \frac{Q(\Theta ,{\Theta }^{\prime })-Q(\Theta ,\Theta )}{P(\Theta )}}\hfill \\ \hfill & =& lnexp\left[{\displaystyle \sum _{x_0,x_1,\dots ,x_N}}{\displaystyle \sum _{y_0}}ln\left({\displaystyle \frac{p_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;{\Theta }^{\prime })}{p_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;\Theta )}}\right){\displaystyle \frac{p_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;\Theta )}{P(\Theta )}}\right]\hfill \\ \hfill & \le & ln\left({\displaystyle \frac{{\displaystyle \sum _{x_0,x_1,\dots ,x_N}}\sum _{y_0}{p}_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;\Theta )\frac{p_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;{\Theta }^{\prime })}{p_{y_0,Y_1,\dots ,Y_N}(x_0,x_1,\dots ,x_N;\Theta )}}{P(\Theta )}}\right)\hfill \\ \hfill & =& ln\left({\displaystyle \frac{P\left({\Theta }^{\prime }\right)}{P(\Theta )}}\right)\hfill \end{array}$$

(71)

and the result follows. □

The stationary points of P and Q are also related.

Lemma 5.

Suppose (Zero) holds. A point $\Theta \in \Gamma$ is a critical point of $P(\Theta )$ if, and only if, it is a fixed point of the re-estimation function, i.e., $Q(\Theta ;\Theta )={max}_{{\Theta }^{\prime }}Q(\Theta ;{\Theta }^{\prime })$, since Q is differentiable on ${(0,\infty )}^d$ in ${\Theta }^{\prime }$.

Proof. 

The following derivatives are equal:

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial P_{Y_1,\dots ,Y_N}(\Theta )}{\partial p_{x,y\to \widehat{x},\widehat{y}}}}& =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]\sum _{m=1}^N{\displaystyle \frac{1_{x_{m-1}=x,Y_{m-1}=y,x_m=\widehat{x},Y_m=\widehat{y}}}{p_{x_{m-1}\to x_m}}}\mu (x_0,y_0)\hfill \\ \hfill & =& {\displaystyle \frac{\partial Q_{Y_1,Y_2,\dots ,Y_N}(\Theta ,{\Theta }^{\prime })}{p_{x,y\to \widehat{x},\widehat{y}}}}{|}_{{\Theta }^{\prime }=\Theta },\hfill \end{array}$$

(72)

which are defined since $p_{x,y\to \widehat{x},\widehat{y}}\ne 0$. Similarly,

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial P_{Y_1,\dots ,Y_N}(\Theta )}{\partial \mu (x,y)}}& =& \sum _{x_0,\dots ,x_N}\sum _{y_0}\left[\prod _{n=1}^N{p}_{x_{n-1},Y_{n-1}\to x_n,Y_n}\right]1_{(x_0,y_0)=(x,y)}\hfill \\ \hfill & =& {\displaystyle \frac{\partial Q_{Y_1,Y_2,\dots ,Y_N}(\Theta ,{\Theta }^{\prime })}{\partial {\mu }^{\prime }(x,y)}}{|}_{{\Theta }^{\prime }=\Theta }.\hfill \end{array}$$

(73)

□

We can rewrite (65), (68) in recursive form, with the values of $\pi$ and $\chi$ substituted in, to find that

$${\Theta }^{k+1}=M\left({\Theta }^k\right),$$

where M is a continuous function. Moreover, $P:K\to [0,1]$ is continuous and satisfies $P\left({\Theta }^k\right)\le P\left(M\left({\Theta }^k\right)\right)$ from above. Now, we have established everything we need for the following result, which follows from the proof of Theorem 1 of Wu [45].

Theorem 2.

Suppose (Zero) holds. Then, ${\left\{{\Theta }^k\right\}}_{k=1}^{\infty }$ is relatively compact, all its limit points (in K) are stationary points of P, producing the same likelihood $P\left({\Theta }^{*}\right)$, say, and $P\left({\Theta }^k\right)$ converges monotonically to $P\left({\Theta }^{*}\right)$.

Wu [45] provides several interesting results in the context of general EM algorithms to guarantee convergence to local or global maxima under certain conditions. However, the point of this paper is to introduce a new model and algorithms with just enough theory to justify the algorithms. Hence, we do not consider theory under any special cases here, but rather refer the reader to Wu [45].

7. Discussion and Conclusions

We have established a new expectation-maximization (EM) algorithm to converge to the parameters of general pairwise Markov chains and Markov observation models that generalizes the Baum–Welch algorithm for hidden Markov models. Our extension not only expands the model itself, but also identifies the initial distribution and solves the small-number problem. We have shown that the likelihood, filter, and (observation) predictor are all easily computable in real time using a recursion like the forward equation in the EM algorithm (after the parameters have converged). We have shown that the pathspace filter for conditional distribution of the hidden state, given all the observations, is also computable using the results of both the forward and backward equations. We invented a GAN-like setup using the likelihoods of known models (with a voting scheme) for detection and simulation (throwing away the hidden component) for the generation part. Finally, we have shown how all our new technology might be combined to solve interesting problems like deepfake generation and detection. Work that is currently underway appears shows great promise for the application of these methods in areas like fraud detection, statistical process control and deepfake detection. It seems like the quality of the results obtained in these domains will not be satisfactory with any existing approaches in the literature, which will surely validate this present work as more than just theory.

I was asked a couple of intriguing questions by the anonymous reviewers, which I will begin to discuss within this paragraph on potential future work. All our development focused on the discrete-space case. However, the classical Baum–Welch algorithm for HMMs also holds in the continuous (nearly) Gaussian case. A similar generalization to the one we made here should establish an EM algorithm for (nearly) Gauss–Markov coupled hidden-state observation pairs. Then, one would be in a position to properly establish our method for establishing the EM algorithm in the usual AR-HMM with Gaussian noise using the representation (7). Continuing in this direction, one could wonder whether there are EM-based forward–backward equations to estimate the parameters in an ARMA-HMM or ARIMA-HMM, both of which would satisfy an equation like the following:

$$Y_n={\beta }_0^{\left(X_n\right)}+{\beta }_1^{\left(X_n\right)}{Y}_{n-1}+\dots +{\beta }_p^{\left(X_n\right)}{Y}_{n-p}+{\epsilon }_n+{\theta }_1^{\left(X_n\right)}{\epsilon }_{n-1}+\dots +{\theta }_q^{\left(X_n\right)}{\epsilon }_{n-q}$$

(74)

where $\left\{{\beta }_i^x\right\}$ and $\left\{{\theta }_i^x\right\}$ are parameters that depend upon the state of a hidden Markov chain, and $\left\{{\epsilon }_i\right\}$ is an i.i.d. noise sequence. (Here, the parameters $\theta$ would take different values or the equation might be rearranged if we had an ARIMA model instead of an ARMA model). These observation equations are not naturally Markov. Indeed, they are close to the ARFIMA models that are used to simulate long-range-dependent sequences. However, the ARMA-HMM and ARIMA-HMM still have linear observation equations with a finite number of parameters and dependence upon a hidden Markov chain. It would be intriguing to investigate whether the EM method can be extended to handle these cases, and whether there are analogs to the forward and backward equations that can be can be combined to estimate all the parameters in these models.