# Handling Covariates in Markovian Models with a Mixture Transition Distribution Based Approach

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Markovian Models

#### 2.1. The Markov Chain

#### 2.2. The Mixture Transition Distribution Model

#### 2.3. Hidden Markov Model

## 3. Covariates in Markovian Modeling

#### Covariates as Additional Explanatory Factors

**D**. Then, the model will estimate, simultaneously, separate models for each distinctive value of the covariate by simply counting the number of observed transitions for each modality of the covariate. This approach is similar to how covariates are handled in nonparametric Kaplan–Meier estimation of survival curves, where separate curves are fitted for each value of the covariate. An example of a single transition matrix for a Markov chain with 3 states and two binary covariates (e.g., “gender” and “be married”) is reported in Figure 1. This approach is quite easy to use, but the size of the resulting matrix can easily explode involving too large a number of parameters. For instance, with just three states and two dichotomous covariates, the number of rows is $3\times 2\times 2=12$ for a total number of free parameters to estimate equal to 24.

## 4. Application: Health Conditions among Older Adults in Switzerland

#### 4.1. Data

#### 4.2. The Hidden Markov Model

#### 4.3. HMM with Education at the Hidden Level

#### 4.4. HMM with Education at the Visible Level

## 5. Conclusions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Raftery, A.E. A Model for High-order Markov Chains. J. R. Stat. Soc. Ser. B
**1985**, 47, 528–539. [Google Scholar] [CrossRef] - Berchtold, A.; Raftery, A. The Mixture Transition Distribution Model for High-Order Markov Chains and Non-Gaussian Time Series. Stat. Sci.
**2002**, 17, 328–359. [Google Scholar] [CrossRef] - Brémaud, P. Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues; Springer: New York, NY, USA, 1999. [Google Scholar]
- Dempster, A.P.; Laird, N.M.; Rubin, D.B. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. R. Stat. Soc. Ser. B (Methodol.)
**1977**, 39, 1–38. [Google Scholar] - Bartolucci, F.; Farcomeni, A. A note on the mixture transition distribution and hidden Markov models. J. Time Ser. Anal.
**2010**, 31, 132–138. [Google Scholar] [CrossRef] [Green Version] - Berchtold, A. Estimation in the Mixture Transition Distribution Model. J. Time Ser. Anal.
**2001**, 22, 379–397. [Google Scholar] [CrossRef] [Green Version] - Zucchini, W.; MacDonald, I.L. Hidden Markov Models for Time Series. An Introduction Using R; CRC Monographs on Statistics & Applied Probability; CRC Press/Chapman & Hall: New York, NY, USA, 2009; p. 275. [Google Scholar]
- McLachlan, G.J.; Peel, D. Finite Mixture Models; Wiley Series in Probability and Statistics; John Wiley & Sons: New York, NY, USA, 2000; p. 456. [Google Scholar]
- Le Strat, Y.; Carrat, F. Monitoring epidemiologic surveillance data using hidden Markov models. Stat. Med.
**1999**, 18, 3463–3478. [Google Scholar] [CrossRef] - Shirley, K.E.; Small, D.S.; Lynch, K.G.; Maisto, S.A.; Oslin, D.W. Hidden Markov models for alcoholism treatment trial data. Ann. Appl. Stat.
**2010**, 4, 366–395. [Google Scholar] [CrossRef] [Green Version] - Baum, L.E.; Petrie. Statistical inference for probabilistic functions of finite state markov chains. Ann. Math. Stat.
**1966**, 37, 1554–1563. [Google Scholar] [CrossRef] - Bijleveld, C.C.; Mooijaart, A. Latent Markov Modelling of Recidivism Data. Statistica Neerlandica
**2003**, 57, 305–320. [Google Scholar] [CrossRef] - Bartolucci, F.; Pennoni, F.; Francis, B. A latent Markov model for detecting patterns of criminal activity. J. R. Stat. Soc. Ser. A (Stat. Soc.)
**2007**, 170, 115–132. [Google Scholar] [CrossRef] - Visser, I.; Raijmakers, M.E.J.; Molenaar, P.C.M. Fitting hidden Markov models to psychological data. Sci. Program.
**2002**, 10, 185–199. [Google Scholar] [CrossRef] [Green Version] - Elliott, R.J.; Hunterb, W.C.; Jamieson, B.M. Drift and volatility estimation in discrete time. J. Econ. Dyn. Control
**1998**, 22, 209–218. [Google Scholar] [CrossRef] - Hayashi, T. A discrete-time model of high-frequency stock returns. Quant. Financ.
**2004**, 4, 140–150. [Google Scholar] [CrossRef] - Netzer, O.; Lattin, J.M.; Srinivasan, V. A Hidden Markov Model of Customer Relationship Dynamics. Mark. Sci.
**2008**, 27, 185–204. [Google Scholar] [CrossRef] [Green Version] - Bolano, D.; Berchtold, A.; Burge, E. The heterogeneity of disability trajectories in later life: Dynamics of activities of daily living performance among nursing home residents. J. Aging Health
**2018**. [Google Scholar] [CrossRef] - Han, S.Y.; Liefbroer, A.C.; Elzinga, C.H. Mechanisms of family formation: An application of Hidden Markov Models to a life course process. Adv. Life Course Res.
**2019**, 43, 100265. [Google Scholar] [CrossRef] - Rabiner, L.R. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE
**1989**, 77, 257–286. [Google Scholar] [CrossRef] - Viterbi, A.J. Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm. IEEE Trans. Inf. Theory
**1967**, 16, 260–269. [Google Scholar] [CrossRef] [Green Version] - Baum, L.E.; Petrie, T.; Soules, G.; Weiss, N. A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains. Ann. Math. Stat.
**1970**, 41, 164–171. [Google Scholar] [CrossRef] - Visser, I.; Raijmakers, M.E.J.; Molenaar, P.C.M. Confidence intervals for hidden Markov model parameters. Br. J. Math. Stat. Psychol.
**2000**, 53, 317–327. [Google Scholar] [CrossRef] - Zhivko, T.; Berchtold, A. Bootstrap Validation of the Estimated Parameters in Mixture Models Used for Clustering. J. Soc. FranÇaise Stat.
**2019**, 160, 114–129. [Google Scholar] - Berchtold, A.; Raftery, A. The Mixture Transition Distribution (MTD) Model for High-Order Markov Chains and Non-Gaussian Time Series; Technical Report 360; Department of Statistics, University of Washington: Seattle, WA, USA, 1999. [Google Scholar]
- Bartolucci, F.; Montanari, G.E.; Pandolfi, S. Three-Step Estimation of latent Markov Models with Covariates. Comput. Stat. Data Anal.
**2015**, 170, 115–132. [Google Scholar] [CrossRef] [Green Version] - Bartolucci, F.; Farcomeni, A.; Pennoni, F. Latent Markov Models for Longitudinal Data; Statistics in the Social and Behavioral Sciences; Chapman and Hall/CRC Press: New York, NY, USA, 2012; p. 252. [Google Scholar]
- Bartolucci, F.; Farcomeni, A.; Pennoni, F. Latent Markov models: A review of a general framework for the analysis of longitudinal data with covariates. Test
**2014**, 23, 433–465. [Google Scholar] [CrossRef] [Green Version] - Maitre, O.; Berchtold, A.; Emery, K.; Burschor, O. march: Markov Chains, R Package Version 3.1. 2019. Available online: https://CRAN.R-project.org/package=march (accessed on 1 January 2020).
- Voorpostel, M.; Tillmann, R.; Lebert, F.; Kuhn, U.; Lipps, O.; Ryser, V.A.; Schmid, F.; Rothenbühler, M.; Boris, W. Swiss Household Panel Userguide (1999–2016), Wave 18, December 2017. Lausanne FORS; 2017; Available online: https://forscenter.ch/wp-content/uploads/2018/08/shp_user_guide_w18.pdf (accessed on 1 January 2020).
- Schwarz, G.E. Estimating the dimension of a model. Ann. Stat.
**1978**, 6, 461–464. [Google Scholar] [CrossRef] - Raftery, A.E. Bayesian Model Selection in Social Research. Sociol. Methodol.
**1995**, 25, 111–163. [Google Scholar] [CrossRef] - Berchtold, A. The Double Chain Markov Model. Commun. Stat. Theory Methods
**1999**, 28, 2569–2589. [Google Scholar] [CrossRef] [Green Version] - Bolano, D.; Berchtold, A. General framework and model building in the class of Hidden Mixture Transition Distribution models. Comput. Stat. Data Anal.
**2016**, 93, 131–145. [Google Scholar] [CrossRef] [Green Version] - Le, N.D.; Martin, D.; Raftery, A.E. Modeling Flat Stretches, Bursts, and Outliers in Time Series Using Mixture Transition Distribution Models. J. Am. Stat. Assoc.
**1996**, 91, 1504–1515. [Google Scholar]

Sample Availability: Data used in the empirical example is drawn from the first 14 waves of the Swiss Household Panel (SHP 1999–2012). SHP data is freely accessible at https://forsbase.unil.ch/project/study-public-overview/15632/0/ after signing a user agreement. The variables used in this studies are named pXXC01 and educatXX that correspond respectively to the health condition and level of education in the year XX. The list of ids of 1331 individuals included in the analysis is available from the author. |

**Figure 3.**Hidden transition matrix and first hidden state distribution. Educational level as covariate at the hidden level.

No. of Hidden States | Free Parameters | Log-Likelihood | BIC |
---|---|---|---|

2 | 11 | −10,532.6 | 21,167.2 |

3 | 20 | −8893.17 | 17,971.82 |

4 | 31 | −8887.315 | 18,062.1 |

5 | 44 | −8782.427 | 17,972.87 |

SRH | Hidden State 1 | Hidden State 2 | Hidden State 3 |
---|---|---|---|

P | 0.013 | 0.000 | 0.000 |

B | 0.082 | 0.002 | 0.002 |

M | 0.649 | 0.098 | 0.016 |

W | 0.245 | 0.841 | 0.421 |

E | 0.011 | 0.059 | 0.561 |

SRH | Hidden State 1 | Hidden State 2 | Hidden State 3 |
---|---|---|---|

P | 29 | 0 | 0 |

B | 181 | 10 | 3 |

M | 1383 | 655 | 29 |

W | 426 | 5471 | 798 |

E | 22 | 414 | 1228 |

**Table 4.**3-state HMM model with educational level as covariate at hidden level using the MTD-based approach.

Education | F | G | VG |
---|---|---|---|

Low | 0.888 | 0.112 | 0.000 |

Medium | 0.381 | 0.619 | 0.000 |

High | 0.000 | 0.689 | 0.311 |

Covariates | Free | Log-Likelihood | BIC | p-Value Likelihood |
---|---|---|---|---|

Parameters | Ratio Test | |||

No covariate | 20 | −8893.17 | 17,971.8 | |

Covariate at hidden level | ||||

Into a unique transition matrix | 34 | −8868.57 | 18,052.4 | ≤0.001 |

MTD approach | 29 | −8873.92 | 18,016.8 | ≤0.005 |

SRH | F | G | VG |
---|---|---|---|

P | 0.006 | 0.000 | 0.000 |

B | 0.057 | 0.000 | 0.004 |

M | 0.491 | 0.097 | 0.022 |

W | 0.418 | 0.862 | 0.545 |

E | 0.027 | 0.041 | 0.428 |

Weight | F | G | VG |
---|---|---|---|

${\theta}_{{S}_{i}}$ | 0.4549 | 0.996 | 1.000 |

${\theta}_{edu}$ | 0.5451 | 0.004 | 0.000 |

Level of Education | |||
---|---|---|---|

SRH | Low | Medium | High |

P | 0.020 | 0.023 | 0.011 |

B | 0.076 | 0.090 | 0.098 |

M | 0.905 | 0.618 | 0.586 |

W | 0.000 | 0.259 | 0.304 |

E | 0.000 | 0.010 | 0.000 |

Level of Education | |||
---|---|---|---|

SRH | Low | Medium | High |

P | 0.013 | 0.015 | 0.009 |

B | 0.067 | 0.075 | 0.080 |

M | 0.717 | 0.561 | 0.543 |

W | 0.190 | 0.331 | 0.356 |

E | 0.012 | 0.018 | 0.012 |

© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bolano, D.
Handling Covariates in Markovian Models with a Mixture Transition Distribution Based Approach. *Symmetry* **2020**, *12*, 558.
https://doi.org/10.3390/sym12040558

**AMA Style**

Bolano D.
Handling Covariates in Markovian Models with a Mixture Transition Distribution Based Approach. *Symmetry*. 2020; 12(4):558.
https://doi.org/10.3390/sym12040558

**Chicago/Turabian Style**

Bolano, Danilo.
2020. "Handling Covariates in Markovian Models with a Mixture Transition Distribution Based Approach" *Symmetry* 12, no. 4: 558.
https://doi.org/10.3390/sym12040558