Variational Bayesian Variable Selection for High-Dimensional Hidden Markov Models

Abstract

The Hidden Markov Model (HMM) is a crucial probabilistic modeling technique for sequence data processing and statistical learning that has been extensively utilized in various engineering applications. Traditionally, the EM algorithm is employed to fit HMMs, but currently, academics and professionals exhibit augmenting enthusiasm in Bayesian inference. In the Bayesian context, Markov Chain Monte Carlo (MCMC) methods are commonly used for inferring HMMs, but they can be computationally demanding for high-dimensional covariate data. As a rapid substitute, variational approximation has become a noteworthy and effective approximate inference approach, particularly in recent years, for representation learning in deep generative models. However, there has been limited exploration of variational inference for HMMs with high-dimensional covariates. In this article, we develop a mean-field Variational Bayesian method with the double-exponential shrinkage prior to fit high-dimensional HMMs whose hidden states are of discrete types. The proposed method offers the advantage of fitting the model and investigating specific factors that impact the response variable changes simultaneously. In addition, since the proposed method is based on the Variational Bayesian framework, the proposed method can avoid huge memory and intensive computational cost typical of traditional Bayesian methods. In the simulation studies, we demonstrate that the proposed method can quickly and accurately estimate the posterior distributions of the parameters with good performance. We analyzed the Beijing Multi-Site Air-Quality data and predicted the PM2.5 values via the fitted HMMs.

1. Introduction

Hidden Markov Models (HMMs) are a statistical model used to describe the evolution of observable events that depend on internal factors or states, which are not directly observable and called hidden states. Each hidden state can transition to another hidden state, including itself, with a certain probability, while we cannot observe them directly, we infer their presence and transitions between them based on observable outputs. HMMs have been widely used in various applications, including speech recognition, bioinformatics, natural language processing, and financial markets. In practice, HMMs often face high-dimensional issues, that is, a large number of covariates (or high-dimensional covariates) and multiple states result in high-dimensional parameters existing in the HMMs. The high-dimensional issue may result in the overfitting for the HMMs. The challenge then becomes identifying important variables or parameters in different hidden states. Thus, efficient parameter estimation and hidden Markov chain recovering are significant for the high-dimensional HMMs.

Currently, there have been many methods for estimating parameter estimation and recovering hidden Markov chains, including recursive algorithms [1,2,3,4] and traditional Bayesian methods [5,6]. Bayesian inference [7,8] is a versatile framework that utilizes sophisticated hierarchical data models for learning and consistently quantifies uncertainty in unknown parameters through the posterior distribution [9]. However, computing the posterior is demanding even for moderately intricate models and frequently necessitates approximation. Moreover, traditional Bayesian methods (e.g., Markov chain Monte Carlo (MCMC)) for the HMMs is often considered a black-box method by many statisticians due to its reliance on simulations to produce computable results, which is generally inefficient and unnecessary. Traditional Markov Chain Monte Carlo (MCMC) methods may also exhibit slow convergence and extended running times, as documented in prior studies [10,11,12].

The Variational Bayesian approach is an alternative to traditional MCMC algorithm in high-dimensional issue. Variational inference (VI) based on Bayesian method [13,14,15,16] can approximate posterior distributions quickly [17], since VI uses the Kullback–Leibler (KL) divergence to measure the difference between the variational posterior and the true posterior, and transforms the statistical inference problem into a mathematical optimization problem by minimizing the KL divergence. Therefore, the variational approach is as close as possible to the true posterior distribution according to the KL divergence. Wang and Blei [17] have proved that the variational posterior is consistent to the true posterior distribution. Moreover, there have been many efficient optimization algorithms to approximate complex probability densities such as coordinate-ascent (CAVI) [18] and gradient-based methods [15,19]. Currently, there exist many VI research studies for the HMMs [20,21,22,23,24]. For example, MacKay [20] was the pioneering proponent of the application of variational methods to HMMs, with a focus only on cases with discrete observations. Despite the limited comprehension of the state-removal phenomenon, which is that removing certain states from an HMM for simplifying the HMM while preserving its essential statistical properties does not significantly affect the the ability of the HMM to represent the underly stochastic process, variational methods are gaining popularity for HMMs within the machine learning community. C. A. McGrory [21] extended the deviance information criterion for Bayesian model selection within the Hidden Markov Model framework, utilizing a variational approximation. Nicholas J. Foti [22] devised an SVI algorithm to learn HMM parameters in settings of time-dependent data. Since VI can be seen as a special instance of the EM algorithm [23], Gruhl [23] integrates both approaches and uses the multivariate Gaussian output distribution of VI to train the HMM. Ding [24] employed variational inference techniques to investigate nonparametric Bayesian Hidden Markov Models built on Dirichlet processes. These processes enable an unbounded number of hidden states and adopt an infinite number of Gaussian components to handle continuous observations. However, variational inference has not been fully explored in HMMs, especially in HMMs with a high-dimensional covariate. It is important to note that research frequently entails using high-dimensional covariate datasets in real-world applications. When high-dimensional HMMs contain a large number of parameters, there is a possibility of overfitting the given data set. The challenge then becomes identifying which covariates have a substantial impact on the interpretation of observations and state shifts in each hidden state of the model, and which covariates have a negligible effect. This process is important for improving the predictive accuracy of the model, reducing the risk of overfitting, and improving the interpretability of the model.

In this article, we develop a Variational Bayesian method for variable selection. We utilize the double-exponential shrinkage prior [25,26,27,28] as the prior of coefficients in each hidden state models, to screen vital variables that affect each hidden state and obtain hidden Markov regression models. We use mean-field variational inference to identify variational densities for approximating complex posterior densities, via minimizing the difference between the approximate probability density and the actual posterior probability density. Moreover, we adopt the Monte Carlo Co-ordinate Ascent VI (MC-CAVI) [29] algorithm to compute the necessary expectations within the CAVI. Since the variational inference is a fast alternative to the MCMC method and can avoid large memory and intensive computational cost compared to traditional Bayesian methods, the proposed approach inherits the good properties of variational inference, and can quickly and accurately estimate the posterior distributions and the unknown parameters. In the simulation studies and real data analysis, the proposed method outperforms the common methods in term of variable selection and prediction.

The main contributions of this article are as follows: First, the proposed method can perform variable selection for high-dimensional HMMs, and offer the advantage of fitting the model and investigating specific factors that impact the response variable changes simultaneously. Since the proposed method uses double-exponential shrinkage prior, which has the feature of being able to select important variables, the proposed method can simultaneously select important variables to the response variable and estimate the corresponding parameters. Second, since the proposed method is based on the Variational Bayesian framework, the proposed method can avoid huge memory and intensive computational cost of the traditional Bayesian methods, especially for the high-dimensional issue. Finally, we demonstrate that the proposed method can quickly and accurately estimate the posterior distributions of the parameters with good performance in the simulation studies. Moreover, we analyze Beijing Multi-Site Air-Quality data and predict the PM2.5 values well via the fitted HMMs.

The rest of the article is organized as follows: Section 2 introduce the Hidden Markov Model with high-dimensional covariate and shrinkage priors in the Bayesian inference. In Section 3, we propose an efficient Variational Bayesian estimation method with the double-exponential shrinkage prior for variable selection of the high-dimensional HMMs (HDVBHMM). In Section 4, we conduct simulation studies to investigate the finite sample performances of the proposed method. In Section 5, Beijing Multi-Site Air-Quality data are analyzed and the efficiency of the proposed method is verified. Section 6 concludes our work. Technical details are presented in the Appendix A.

2. Model and Notation

2.1. Hidden Markov Model

In this section, we first introduce Hidden Markov Models (HMMs). The HMMs are a type of doubly stochastic process that occurs over discrete time intervals and includes observations $y_t$ and latent states $z_t$. In a traditional Hidden Markov Model without covariates, the observation $y_t$ depends only on the current potential state $z_t$. The conditional distribution of the observation $y_t$ when given the potential state $z_t=k$ can be expressed as:

$$y_t\mid z_t=k\sim F_k\left({\theta }_k\right),$$

where $F_k\left({\theta }_k\right)$ denotes a certain family of distributions, such as the normal distribution $N\left({\mu }_k,{\sigma }_k^2\right)$. Extended HMM models can include covariates $x_t\in R^p$. That is, the set of observations is $y=\left(y_1,\dots ,y_T\right)$ and $x=\left(x_1,\dots ,x_T\right)$. Specifically, for $y=\left(y_1,\dots ,y_T\right)$, $z=\left(z_1,\dots ,z_T\right)$ and $x=\left(x_1,\dots ,x_T\right)$, the model expression is as follows:

$$y_t\mid x_t,z_t=k,\beta ,{\sigma }^2\sim N\left(x_t^{\top }{\beta }_k,{\sigma }^2\right)\mathrm{for}t=1,\dots ,T,$$

where the symbol N represents the normal distribution, ${\sigma }^2$ denotes the variance of $y_t$, and $\beta ={({\beta }_1,\dots ,{\beta }_K)}^{\top }$ is the coefficient of the covariate at all hidden states. In the article, we consider the high-dimensional issue of the covariate. We denote the dummy variable corresponding to $z_t$ as the $v_t={(v_{t1},\dots ,v_{tK})}^{\top }$, where $v_{tk}=1$ and other elements being zero if $z_t=k$. Thus,

$$P\left(y_t\mid x_t,\beta ,{\sigma }^2\right)=\prod _{k=1}^{K}P{\left(y_t\mid x_t,z_t=k,\beta ,{\sigma }^2\right)}^{v_{tk}}=\prod _{k=1}^{K}P{\left(y_t\mid x_t,{\beta }_k,{\sigma }^2\right)}^{v_{tk}}.$$

In hidden Markov chains, each hidden state $z_t$ is independent from $z_1,\dots ,z_{t-2}$ and $z_{t+1},\dots ,z_T$ conditionally on $z_{t-1}$. Therefore, we can assume that the probability distribution of $z_1$ is given by $z_1\sim P\left(z_1|\pi \right)={\left({\pi }_1,{\pi }_2,\dots ,{\pi }_K\right)}^{\top }$, where ${\sum }_{k=1}^K{\pi }_k=1$ and ${\pi }_k>0$. The conditional probability of $z_t$ given $z_{t-1}$ is assumed as:

$$P\left(z_t\mid z_{t-1},A\right)=\prod _{k=1}^K\prod _{j=1}^K{A}_{jk}^{v_{t-1,j}{v}_{tk}},$$

where A is the transition matrix with elements $A_{ij}$ for $i,j=1,\dots ,K$, ${\sum }_{j=1}^K{A}_{ij}=1$ and $A_{ij}>0$, and $A_{ij}$ represents the probability of transitioning from state i to state j. Thus, the joint distribution is as follows:

$$\begin{array}{ccc}\hfill & \hfill & P\left(\mathit{y},\mathit{z}\mid \mathit{x},\pi ,A,\beta ,{\sigma }^2\right)\hfill \\ \hfill & =\hfill & P\left(z_1|\pi \right)\prod _{t=2}^{T}P\left(z_t\mid z_{t-1},A\right)\prod _{t=1}^{T}P\left(y_t\mid x_t,\beta ,{\sigma }^2\right)\hfill \\ \hfill & =\hfill & \left(\prod _{k=1}^K{\pi }_k^{v_{1k}}\right)\left(\prod _{t=2}^T\prod _{k=1}^K\prod _{j=1}^K{A}_{jk}^{v_{t-1,j}{v}_{tk}}\right)\left(\prod _{t=1}^T\prod _{k=1}^{K}P{\left(y_t\mid x_t,\beta ,{\sigma }^2,z_t=k\right)}^{v_{tk}}\right).\hfill \end{array}$$

(1)

2.2. Prior Selection in the HMMs

To make Variational Bayesian inference, we require specifying the prior of the parameters $\pi$, A, $\beta$ and ${\sigma }^2$. Based on the characteristics of $\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_K\right)$, ${\sum }_{k=1}^K{\pi }_k=1$ and ${\pi }_k>0$, Dirichlet distribution is applied to the prior distribution of $\pi$ as follows:

$$\pi \sim Dir\left({\alpha }^{\left(\pi \right)}\right),$$

(2)

where ${\alpha }^{\left(\pi \right)}={\left({\alpha }_1^{\left(\pi \right)},\dots ,{\alpha }_K^{\left(\pi \right)}\right)}^{\top }$, ${\sum }_{k=1}^K{\alpha }_k^{\left(\pi \right)}=1$, and ${\alpha }_k^{\left(\pi \right)}>0$. In the model, A denotes the transition matrix of the hidden state $\mathit{z}$ and can be expressed as follows:

$$A=\left(\begin{array}{ccc}{A}_{11}& \dots & A_{1K}\\ ⋮& \ddots & ⋮\\ A_{K1}& \dots & A_{KK}\end{array}\right).$$

Like Nicholas [22], we specify the prior of the jth row of the transition matrix A as:

$$A_j\sim Dir\left({\alpha }_j^{\left(A\right)}\right)\mathrm{for}j=1,\dots ,K,$$

(3)

where ${\alpha }_j^{\left(A\right)}={\left({\alpha }_{j1}^{\left(A\right)},{\alpha }_{j2}^{\left(A\right)},\dots ,{\alpha }_{jK}^{\left(A\right)}\right)}^{\top }$, ${\sum }_{k=1}^K{\alpha }_{jk}^{\left(A\right)}=1$, and ${\alpha }_{jk}^{\left(A\right)}>0$. Since ${\sigma }^2$ is variance of the y, we specify the prior of the ${\sigma }^2$ as

$${\sigma }^2\sim f\left({\sigma }^2\right)=\frac{1}{{\sigma }^2}.$$

(4)

In a high-dimensional and sparse issue, we consider the double-exponential shrinkage prior [25,27] as the prior of $\beta$, defined as follows:

$$\begin{array}{cc}\hfill & {\beta }_k\mid {\sigma }^2,{\tau }_1,\dots ,{\tau }_p^2\sim N_p\left(0,{\sigma }^2{D}_{\tau }\right),\hfill \\ \hfill & D_{\tau }=diag\left({\tau }_1^2,\dots ,{\tau }_p^2\right),\hfill \\ \hfill & {\tau }_m^2\sim Exp\left(\frac{{\lambda }^2}{2}\right)\mathrm{for}m=1,2,\dots ,p,\hfill \\ \hfill & {\lambda }^2\sim \Gamma (r,\delta ),\hfill \end{array}$$

(5)

where $\Gamma$ represents gamma distribution and $Exp(·)$ represents the exponential distribution. The above prior can select important variables of the HMMs in each hidden state. Bayesian approaches can be used to solve the parameter estimation question with the above prior information. However, in high-dimensional data, the traditional Bayesian methods (e.g., MCMC) require huge memory and intensive computational cost. The Variational Bayesian approach is an alternative to the traditional MCMC algorithm in high-dimensional issue. Next, we introduce the proposed Variational Bayesian inference for high-dimensional HMMs.

3. Variational Bayesian Inference for the HMMS

3.1. Mean Field Variational

Mean-field Variational Bayesian inference is a prevalent approach in variational inference, and aims to identify an approximate density by minimizing the difference between the approximate probability density and the actual posterior probability density, while being bounded by the Kullback–Leibler divergence. In this subsection, we proposed the mean-field variational inference for HMMS with the high-dimensional covariates.

Let $\mathit{D}$ be an observed data set, $\mathit{D}=\{\mathit{y},\mathit{x}\}$ with response set $\mathit{y}=\left\{y_i\mid i=1,\dots ,n\right\}$ and covariate set $\mathit{x}=\left\{x_i\mid i=1,\dots ,n\right\}$, and $\mathit{\theta }=\{\pi ,A,\beta ,{\sigma }^2,{\tau }_m^2,{\lambda }^2\}$. The $\mathit{\theta }$ and $\mathit{z}$ include all parameters in the HMMs. We focus on the posterior distribution of parameters $\mathit{\theta }$ and the hidden state $z_t$. Assume that there is an approximate density family $\mathit{Q}$ containing possible densities over the parameters $\mathit{\theta },\mathit{z}$. Minimizing the Kullback–Leibler (KL) divergence between the member of the family $q(\mathit{\theta },\mathit{z})$ and the true posterior $P(\mathit{\theta },\mathit{z}\mid \mathit{D})$ is to obtain the optimal density approximation of the true posterior, with variational inference prioritizing optimization rather than sampling. That is,

$$q^{*}(\mathit{\theta },\mathit{z})=arg\underset{q(\mathit{\theta },\mathit{z})\in \mathit{Q}}{min}KL(q(\mathit{\theta },\mathit{z})\parallel P(\mathit{\theta },\mathit{z}\mid \mathit{D})),$$

where the KL-divergence is:

$$KL(q(\mathit{\theta },\mathit{z})\parallel P(\mathit{\theta },\mathit{z}\mid \mathit{D}))=\int q(\mathit{\theta },\mathit{z})log\left\{\frac{q(\mathit{\theta },\mathit{z})}{P(\mathit{\theta },\mathit{z}|\mathit{D})}\right\}d(\mathit{\theta },\mathit{z}).$$

The KL-divergence can be further written as:

$$\begin{array}{cc}& KL\left(q\right(\mathit{\theta },\mathit{z})\parallel P(\mathit{\theta },\mathit{z}\mid \mathit{D}\left)\right)\hfill \\ \hfill =& E_q[logq(\mathit{\theta },\mathit{z})]-E_q[logP(\mathit{\theta },\mathit{z}\mid \mathit{D})]\hfill \\ \hfill =& E_q[logq(\mathit{\theta },\mathit{z})]-E_q[logP(\mathit{\theta },\mathit{z},\mathit{D})]+logP\left(\mathit{D}\right),\hfill \end{array}$$

where $logP\left(\mathit{D}\right)$ is a constant, $E_q$ denotes the expected value of $\mathit{\theta }$ and z drawn from the distribution q. Thus, minimizing the KL divergence is equivalent to maximizing the following evidence lower bound (ELBO):

$$ELBO\left(q\right)=E_q[logP(\mathit{\theta },\mathit{z},\mathit{D})]-E_q[logq(\mathit{\theta },\mathit{z})].$$

(6)

From another perspective, the ELBO comprises the negative KL divergence and $logP\left(\mathit{D}\right)$.

According to the mean-field variational framework [30,31], the parameters are assumed to be posterior independent of each other and to be controlled by a separate factor in the variational density. In the HMMs, $q(\mathit{\theta },\mathit{z})$ is decomposed as:

$$q(\mathit{\theta },\mathit{z})=q\left(\pi \right)q\left(A\right)q\left({\sigma }^2\right)q\left({\tau }_m^2\right)q\left({\lambda }^2\right)\prod _{t=1}^{T}q\left(z_t\right).$$

(7)

Each parameter ${\mathit{\theta }}_i$ and latent state $\mathit{z}$ is governed by its own variational factor. The forms of $q\left({\mathit{\theta }}_i\right)$ and $q\left(\mathit{z}\right)$ are unknown, but the form of the hypothesized factorization is determined. In the optimization process, the optimal solutions of these variational factors $q\left({\mathit{\theta }}_{\mathit{i}}\right)$ and $q\left(\mathit{z}\right)$ are obtained by maximizing the ELBO of Equation (6) by the coordinate ascent method. Based on the consistency of the Variational Bayesian [17], the variational densities over the mean-field family are still consistent to the posterior densities, even though the mean field approximating family can be a brutal approximation. More generally, one can consider structured variational distributions involving partial factorizations that correspond to tractable substructures of parameters [32]. In this article, we only consider the mean field framework. To express the variational posterior formula concisely, we define $\varphi =\left\{\mathit{\theta },\mathit{z}\right\}$ and rewrite $q(\mathit{\theta },\mathit{z})$ as $q\left(\varphi \right)$.

3.2. The Coordinate Ascent Algorithm for Optimizing the ELBO

Based on the variational density decomposition, we can obtain each factor of the variational density via maximizing the ELBO. Let $q_i\left({\varphi }_i\right)$ for $i=1,2,\dots ,b$ be the ith factor of the variational density in (7). The common approaches to maximize the ELBO mainly include a Coordinate Ascent Variational Inference (CAVI) and a gradient-based approach [33]. The CAVI approach sequentially optimizes each factor of the variational density of the mean field to obtain a local maximizer for the ELBO, while keeping the others fixed. Based on the CAVI approach, we can obtain the optimal variational density $q_i^{*}\left({\varphi }_i\right)$ as follows:

$$q_i^{*}\left({\varphi }_i\right)\propto exp\left\{E_{-i}\left[logP\left({\varphi }_{i_{-}},{\varphi }_i,{\varphi }_{i_{+}},\mathit{D}\right)\right]\right\},$$

(8)

where $i_{-}$ (or $i_{+}$) refers to the ordered indexes that are less than (or greater than) i. Let ${\varphi }_{-i}:=({\varphi }_{i_{-}},{\varphi }_{i_{+}})$. The vector ${\varphi }_{-i}$ represents the vector $\varphi$ with the ith component ${\varphi }_i$ removed. The $E_{-i}$ denotes the expectation with respect to ${\varphi }_{-i}$.

Based on the joint distribution (1), the priors (2)–(5) and Formula (8), we can derive all variational posteriors (see Appendix A for details). The variational posterior of the $\pi$ is:

$$q^{*}\left(\pi \right)\sim Dir\left({\alpha }_{\left(\pi \right)}\right),$$

(9)

where ${\alpha }_{\left(\pi \right)}=E\left(z_1\right)+{\alpha }^{\left(\pi \right)}$. The variational posterior of the $A_j$ is:

$$q^{*}\left(A_j\right)\sim Dir\left({\alpha }_{\left(A_j\right)}\right)\mathrm{for}j=1,\dots ,K,$$

(10)

where ${\alpha }_{\left(A_j\right)}={\sum }_{t=2}^{T}E\left(v_{t-1,j}{v}_{tk}\right)+{\alpha }_{jk}^{\left(A\right)}$. The variational posterior of the ${\beta }_k$ is:

$$q^{*}\left({\beta }_k\right)\sim N_p\left({\beta }_k;{\mu }_k,{\Sigma }_k\right),$$

(11)

where

$$\begin{array}{ccc}{\Sigma }_k\hfill & =\hfill & {\left(E\left(\frac{1}{{\sigma }_2}\right)\sum _{t=1}^{T}E\left(v_{tk}\right)x_t{x}_t^{\top }+E\left(\frac{1}{{\sigma }^2}\right)E\left(D_{\tau }^{-1}\right)\right)}^{-1},\hfill \\ {\mu }_k\hfill & =\hfill & {\Sigma }_k\left(E\left(\frac{1}{{\sigma }^2}\right)\sum _{t=1}^T{y}_t·E\left(v_{tk}\right)·x_t\right).\hfill \end{array}$$

The variational posterior of the ${\sigma }^2$ is:

$$q^{*}\left({\sigma }^2\right)\sim \mathrm{Inverse-Gamma}\left({\alpha }_{\left({\sigma }^2\right)},{\beta }_{\left({\sigma }^2\right)}\right),$$

(12)

where ${\alpha }_{\left({\sigma }^2\right)}=\frac{T}{2}$ and ${\beta }_{\left({\sigma }^2\right)}=\frac{1}{2}{\sum }_{t=1}^T{\sum }_{k=1}^{K}E\left(v_{tk}\right)\left[{\left(y_t-x_t^{\top }{\mu }_k\right)}^2+x_t^{\top }{\Sigma }_k{x}_t\right].$ The variational posterior of the ${\tau }_m^2$ is:

$$q^{*}\left({\tau }_m^2\right)\sim \mathrm{Generalized-Inverse-Gaussian}\left(C_{{\tau }_m},a_{{\tau }_m},b_{{\tau }_m}\right),$$

(13)

where $a_{{\tau }_m}=E\left({\lambda }^2\right)$, $b_{{\tau }_m}=E(1/{\sigma }_2){\sum }_{k=1}^{K}E\left({\beta }_{km}^2\right)$, and $C_{{\tau }_m}=1-K/2$. The variational posterior of the ${\lambda }^2$ is:

$$q^{*}\left({\lambda }^2\right)\sim \Gamma \left({\alpha }_{\left({\lambda }^2\right)},{\beta }_{\left({\lambda }^2\right)}\right),$$

(14)

where ${\alpha }_{\left({\lambda }^2\right)}=p+r$ and ${\beta }_{\left({\lambda }^2\right)}=\delta +\frac{1}{2}{\sum }_{m=1}^{p}E\left({\tau }_m^2\right).$

Based on the dependencies of hidden states, we divide the posterior of z into three parts. The variational posterior of the $z_1$ is:

$$q^{*}\left(z_1\right)\sim Mult\left(P_{\left(z_1\right)}\right),$$

(15)

where the Mult represents multinomial distribution, $P_{\left(z_1\right)}={(P_{\left(z_1\right)1},\dots ,P_{\left(z_1\right)K})}^{\top }$ and

$$P_{\left(z_1\right)k}=exp\left\{E\left[log{\pi }_k\right]\right\}exp\left\{E\left[logP\left(y_1\mid x_1,{\beta }_k,{\sigma }^2\right)\right]\right\}\prod _{j=1}^{K}exp\left\{E\left[v_{2j}\right]E\left[log{A}_{kj}\right]\right\}.$$

The variational posterior of the $z_t$ for $t=2,\dots ,T-1$ is:

$$q^{*}\left(z_t\right)\sim Mult\left(P_{\left(z_t\right)}\right)\mathrm{for}t=2,\dots ,T-1,$$

(16)

where $P_{\left(z_t\right)}={(P_{\left(z_t\right)1},\dots ,P_{\left(z_t\right)K})}^{\top }$ and

$$\begin{array}{ccc}{P}_{\left(z_t\right)k}\hfill & =\hfill & exp\left(E\left[logP\left(y_t\mid x_t,{\beta }_k,{\sigma }^2\right)\right]\right)·\prod _{j=1}^{K}exp\left\{E\left[log{A}_{jk}\right]E\left(v_{t-1,j}\right)\right\}\hfill \\ \hfill & \hfill & ·\prod _{j=1}^{K}exp\left\{E\left[log{A}_{kj}\right]E\left(v_{t+1,j}\right)\right\}.\hfill \end{array}$$

The variational posterior of the $z_T$ is:

$$q^{*}\left(z_T\right)\sim Mult\left(P_{\left(z_T\right)}\right),$$

(17)

where $P_{\left(z_T\right)}={(P_{\left(z_T\right)1},\dots ,P_{\left(z_T\right)K})}^{\top }$ and

$$P_{\left(z_T\right)k}=exp\left\{E\left[logP\left(y_T\mid x_T,{\beta }_k,{\sigma }^2\right)\right]\right\}·\prod _{j=1}^{K}exp\left\{E\left[log{A}_{jk}\right]E\left(v_{T-1},j\right)\right\}.$$

The expectation $E\left[logP\left(y_t\mid x_t,{\beta }_k,{\sigma }^2\right)\right]$ in the above variational posteriors (15)–(17) is expressed as follows:

$$E\left[logP\left(y_t\mid x_t,{\beta }_k,{\sigma }^2\right)\right]=-\frac{1}{2}log\left(2\pi \right)-\frac{1}{2}E[log\left({\sigma }^2\right)]-\frac{1}{2}E\left(\frac{1}{{\sigma }^2}\right)\left[{(y_t-x_t^{\top }{\mu }_k)}^2+x_t^{\top }{\Sigma }_k{x}_t\right].$$

Note that the expectation part of some parameter posterior formulas is difficult to derive analytically. One feasible method is to use Monte Carlo (MC) sampling to approximate the expectation part that cannot be derived analytically, that is, the Monte Carlo Coordinate Ascent VI (MC-CAVI) [29] algorithm. The MC-CAVI recursion approaches have been proved to be convergent to the maximizer of the ELBO with arbitrarily high probability under regularity conditions. In the article, we also use MC-CAVI to obtain the intractable expectations.

3.3. Implementation

Assume that the expectations $E_{-i}\left[logP\left({\varphi }_{i-},{\varphi }_i,{\varphi }_{i+},\mathit{D}\right)\right]$ for $i\in I$ within an index set I can be analytically obtained across all updates of the variational density $q^{*}\left(\varphi \right)$, and cannot be analytically obtained for $i\notin I$. For the MC-CAVI method, intractable integrals can be approximated using the MC methods if $i\notin I$. Specifically, for $i\notin I$, the samples with the sample size $N\ge 1$ are drawn from the current $q_{-i}^{*}\left({\varphi }_{-i}\right)$ to obtain the expectation estimations as follows:

$${\widehat{E}}_{-i}\left[logP\left({\varphi }_{i-},{\varphi }_i,{\varphi }_{i+},\mathit{D}\right)\right]=\frac{{\sum }_{n=1}^{N}logP\left({\varphi }_{i_{-}}^{\left(n\right)},{\varphi }_i^{\left(n\right)},{\varphi }_{i_{+}}^{\left(n\right)},\mathit{D}\right)}{N}.$$

The Algorithm 1 summarizes the implementation of MC-CAVI, where the $q_{i,k}\left({\varphi }_i\right)$ denotes the density of the ith density factor after it has undergone the kth updates, and $q_{-i,k}\left({\varphi }_{-i}\right)$ refers to the density of all density factors except the ith factors after the kth updates to the factors preceding the ith factor and the $k-1$ updates to the blocks following it.

Algorithm 1 Main iteration steps of MC-CAVI

Necessary: Number of iteration cycles T.

Necessary: Quantity of Monte Carlo samples denoted as N.

Necessary: ${\mathbb{E}}_{-i}\left[logP\left({\varphi }_{i_{-}},{\varphi }_i,{\varphi }_{i_{+}},\mathit{D}\right)\right]$ in closed form for $i\in \mathcal{I}$.

1.  Initialize $q_{i,0}\left({\varphi }_i\right)\mathrm{for}i=1,\dots ,b$.

2.  for $k=1\dots T$:

3.          for $i=1\dots b$:

4.                 If $i\in \mathcal{I}$:

5.                       Set $q_{i,k}\left({\varphi }_i\right)\propto exp\left\{{\mathbb{E}}_{-i,k}\left[logP\left({\varphi }_{i_{-}},{\varphi }_i,{\varphi }_{i_{+}},x,y\right)\right]\right\}$;

6.                 If $i\notin \mathcal{I}$:

7.                       Obtain N samples $\left({\varphi }_{i_{-},k}^{\left(n\right)},{\varphi }_{i_{+},k-1}^{\left(n\right)}\right)$ from $q_{-i,k}\left({\theta }_{-i}\right)$ for $n=1,2,\dots ,N$;

8.                       Set $q_{i,k}\left({\varphi }_i\right)\propto exp\left\{\frac{{\sum }_{n=1}^{N}logp\left({\varphi }_{i_{-},k}^{\left(n\right)},{\varphi }_i,{\varphi }_{i_{+},k-1}^{\left(n\right)},\mathit{D}\right)}{N}\right\}$;

9.           end

10.  end.

Combining with the MC-CAVI algorithm, we can summarize the implementation algorithm for variational posteriors for all parameters as follows in Algorithm 2. Based on the Algorithm 2, we can adopt the variational posterior means of the parameters as the estimators.

Algorithm 2 Variational Bayesian Algorithm for the high-dimensional HMMs

Data Input: $\left\{\left(x_t,y_t\right)\right\},t=1,\dots ,T$;

Hyperparameter Input: ${\alpha }^{\left(\pi \right)}$, $r>0$, $\delta >0$, and  ${\alpha }_{jk}^{\left(A\right)}$ for $k,j=1,\dots K$;

Initialize: ${\alpha }_{\left(\pi \right)}$, ${\alpha }_{\left(A_j\right)}$, ${\alpha }_{\left({\sigma }^2\right)}$, ${\beta }_{\left({\sigma }^2\right)}$, ${\beta }_{\left({\lambda }^2\right)}$, ${\Sigma }_k$ and ${\mu }_k$ for $k=1,\dots K$,

$a_{{\tau }_m}$ and $b_{{\tau }_m}$ for $m=1,\dots ,p$, iteration-index $\ell =1$, a sufficiently small $ϵ={10}^{-6}$

and a maximum iteration times $M=1000$;

While the absolute change of the iterated ELBO $|L^{\ell }-L^{\ell -1}|>ϵ$ and $\ell <M$ do:

Update ${\alpha }_{\left(\pi \right)}$ and $q^{*}\left(\pi \right)$ according to Equation (9);

Estimate $E\left[log{\pi }_k\right]$ by the MC method;

for $j=1,\dots ,K$:

Update ${\alpha }_{\left(A_j\right)}$ and $q^{*}\left(A_j\right)$ according to Equation (10);

Estimate $E\left[log{A}_{jk}\right]$ by the MC method;

end

for $k=1,\dots ,K$:

Update ${\Sigma }_k$, ${\mu }_k$ and $q^{*}\left({\beta }_k\right)$ according to Equation (11);

end

Update ${\alpha }_{\left({\sigma }^2\right)}$, $\beta \left({\sigma }^2\right)$ and $q^{*}\left({\sigma }^2\right)$ according to Equation (12);

Estimate $E\left[log\left({\sigma }^2\right)\right]$ by the MC method;

for $m=1,\dots ,p$:

Update $a_{{\tau }_m}$, $b_{{\tau }_m}$ and $q^{*}\left({\tau }_m^2\right)$ according to Equation (13);

end

Update ${\beta }_{\left({\lambda }^2\right)}$ and $q^{*}\left({\lambda }^2\right)$ according to Equation (14);

Update $P_{\left(z_1\right)}$ and $q^{*}\left(z_1\right)$ according to Equation (15);

Update $P_{\left(z_t\right)}$ and $q^{*}\left(z_t\right)$ according to Equation (16);

Update $P_{\left(z_T\right)}$ and $q^{*}\left(z_T\right)$ according to Equation (17);

Compute the ELBO using the Formula (6), denoted as $L^{\ell }$,

and the absolute change of the iterated ELBO $|L^{\ell }-L^{\ell -1}|$;

$\ell \to \ell +1$;

Output: the variational densities $q^{*}\left(\pi \right)$, $q^{*}\left(A_j\right)$ for $j=1,\dots ,K$, $q^{*}\left({\beta }_k\right)$ for $k=1,\dots ,K$,

$q^{*}\left({\sigma }^2\right)$, $q^{*}\left({\lambda }^2\right)$, $q^{*}\left({\tau }_m^2\right)$ for $m=1,\dots ,p$, and $q^{*}\left(z_t\right)$ for $t=1,\dots ,T$;

and the posterior modes of parameters ${\beta }_k$ for $k=1,\dots ,K$.

4. Simulation Studies

In this section, we carry out simulation studies to investigate the finite sample performances of the proposed method, denoted as HDVBHMM. To evaluate the prediction performance, we compare the proposed method with some commonly used and popular methods, including Back Propagation Neural Network (BP), Long Short-Term Memory (LSTM), and Random Forest. The experimental code can be found via the github link (https://github.com/LiuWei-hub/VBHDHMM, accessed on 23 March 2024).

We consider the dataset $\{x_t,y_t:t=1,\dots ,T\}$, where T is the number of the discrete time intervals, the covariate $x_t$ is generated from the Gaussian distribution $N_p\left(0,2I_p\right)$, and $y_t=x_t^{\top }{\beta }_{z_t}+{\epsilon }_t$, in which the random error ${\epsilon }_t\sim N\left(0,{\sigma }^2\right)$, and $z_t$ is hidden state. Here, the initial hidden state $z_1$ is generated from $Mult\left(\pi \right)$, where $\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_K\right)$. For $t=2,\dots ,T$, the hidden state $z_t$ is generated from $Mult\left(A_j\right)$, where $A_j=\left(A_{1i},A_{2i},\dots A_{Ki}\right)$ and $A_{jk}=P\left(z_t=k\mid z_{t-1}=j\right)$. We set the number of hidden states $K=3$, $\sigma =0.4$, and $\left({\pi }_1,{\pi }_2,\dots ,{\pi }_K\right)={(0.6,0.3,0.1)}^{\top }$.

To assess the predictive performance, we use the samples in the last m time intervals as the testing set and the samples in the first $T-m$ time intervals as the training set. In addition, we use four criteria: (1) the mean absolute percentage error $\mathrm{MAPE}=\frac{100\%}{m}{\sum }_{t=1}^m\left|\frac{{\widehat{y}}_t-y_t}{y_t}\right|$, where $y_t$ is the true value and ${\widehat{y}}_t$ represents the predicted value; (2) the root mean square error $\mathrm{RMSE}=\sqrt{\frac{1}{m}{\sum }_{t=1}^m{\left(y_t-{\widehat{y}}_t\right)}^2}$; (3) the mean absolute error $\mathrm{MAE}=\frac{1}{m}{\sum }_{t=1}^m\left|y_t-{\widehat{y}}_t\right|$; and (4) $R^2=1-\frac{{\sum }_{t=1}^m{\left({\widehat{y}}_t-y_t\right)}^2}{{\sum }_{t=1}^m{\left(\overline{y}-y_t\right)}^2}$, where $\overline{y}$ represents the sample mean, ${\sum }_{t=1}^m{\left({\widehat{y}}_t-y_t\right)}^2$ is the error caused by the prediction, and ${\sum }_{t=1}^m{\left(\overline{y}-y_t\right)}^2$ is the error caused by the mean. The smaller the MAPE, RMSE and MAE values are, the better the performance of the method is. The larger the $R^2$ is, the better the performance of the method is. To evaluate the performance of the parameter estimation, we use two criteria: (1) the root mean square error loss $\mathrm{RMSE}=\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\left({\widehat{\theta }}_i-\theta \right)}^2}$, where n is the number of repeated experiments, ${\widehat{\theta }}_i$ is the estimated value of the parameter obtained in the ith experiment, and $\theta$ is the true parameter value; and (2) $Bias\left(\widehat{\theta }\right)=\frac{1}{n}{\sum }_{i=1}^n{\widehat{\theta }}_i-\theta$. The RMSE and Bias values closer to zero imply better performance for the method. We repeat 10 simulation examples and calculate the average values of the above metrics for each method.

4.1. Experiment 1

In experiment 1, we consider different dimensions p = 20, 30 and 40. In addition, the state transition matrix A is set as follows:

$$A=\left(\begin{array}{ccc}0.2\hfill & 0.3\hfill & 0.5\hfill \\ 0.1\hfill & 0.6\hfill & 0.3\hfill \\ 0.5\hfill & 0.4\hfill & 0.1\hfill \end{array}\right).$$

Due to $K=3$, we have three regression coefficients ${\beta }_1$, ${\beta }_2$, ${\beta }_3$. We set the coefficient as follows:

$$\beta ={\left({\beta }_1,{\beta }_2,{\beta }_3\right)}^{\top }={\left(\begin{array}{ccc}0.5& 1& 1.5\\ -2& -2& -1.5\\ 2& 1.5& 1\\ -1& -1.5& -2\\ 0& 0& 0\\ ⋮& ⋮& ⋮\\ 0& 0& 0\end{array}\right)}_{p\times 3}$$

where the first four rows are nonzero and other elements are zero. We set the number of the discrete time intervals $T=300$ and the sample size in the testing set $m=10$. In addition, the hyperparameters r, $\delta$ in the HDVBHMM method are set to 1. The results are shown in

Table 1. Average values of four metrics of all approaches with standard deviation in each parenthesis based on 10 simulations under $T=300$.

p	Method	Estimate Performance
		MAPE	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$
$p=20$	LSTM	$0.957$ $\left(1.109\right)$	$1.183$ $\left(0.132\right)$	$1.416$ $\left(0.316\right)$	$0.871$ $\left(0.057\right)$
	BP	$0.948$ $\left(1.018\right)$	$1.210$ $\left(0.175\right)$	$1.492$ $\left(0.436\right)$	$0.826$ $\left(0.117\right)$
	Random Forest	$1.215$ $\left(1.844\right)$	$1.430$ $\left(0.194\right)$	$2.081$ $\left(0.531\right)$	$0.741$ $\left(0.119\right)$
	HDVBHMM	$\mathbf{0.467}$ $\left(0.325\right)$	$\mathbf{1.008}$ $\left(0.152\right)$	$\mathbf{1.038}$ $\left(0.329\right)$	$\mathbf{0.887}$ $\left(0.082\right)$
$p=30$	LSTM	$0.949$ $\left(0.485\right)$	$1.354$ $\left(0.266\right)$	$1.898$ $\left(0.740\right)$	$0.789$ $\left(0.144\right)$
	BP	$1.312$ $\left(0.685\right)$	$1.524$ $\left(0.198\right)$	$2.358$ $\left(0.615\right)$	$0.659$ $\left(0.169\right)$
	Random Forest	$1.081$ $\left(0.641\right)$	$1.568$ $\left(0.223\right)$	$2.505$ $\left(0.717\right)$	$0.595$ $\left(0.252\right)$
	HDVBHMM	$\mathbf{0.876}$ $\left(0.919\right)$	$\mathbf{1.186}$ $\left(0.244\right)$	$\mathbf{1.461}$ $\left(0.629\right)$	$\mathbf{0.861}$ $\left(0.073\right)$
$p=40$	LSTM	$1.471$ $\left(1.121\right)$	$1.404$ $\left(2.471\right)$	$2.026$ $\left(0.716\right)$	$0.763$ $\left(0.131\right)$
	BP	$1.555$ $\left(1.218\right)$	$1.427$ $\left(0.156\right)$	$2.060$ $\left(0.412\right)$	$0.772$ $\left(0.117\right)$
	Random Forest	$1.210$ $\left(0.718\right)$	$1.457$ $\left(0.231\right)$	$2.173$ $\left(0.736\right)$	$0.754$ $\left(0.115\right)$
	HDVBHMM	$\mathbf{1.023}$ $\left(0.759\right)$	$\mathbf{1.155}$ $\left(0.264\right)$	$\mathbf{1.398}$ $\left(0.619\right)$	$\mathbf{0.822}$ $\left(0.202\right)$

In the Long-Term and Short-Term Memory methods, the learning rate is lr = 0.001, the number of training cycles (Epochs) is set to 50, and the size of the hidden layer is set to 10. The hidden layer of the BP method consists of 20 neurons, and the maximum number of iterations is set to 10,000. In the random forest regression model, the number of trees is set to 100. The bold results are the optimal ones among four methods.

and

Table 2. Average values of the RMSE and Bias of A and $\beta$ based on 10 replications in Experiment 1.

p	Parameter	Estimate Performance
		RMSE	Bias
$p=20$	$\beta$	$0.001$	$0.001$
	A	$0.002$	$0.001$
$p=30$	$\beta$	$0.002$	$0.001$
	A	$0.005$	$0.001$
$p=40$	$\beta$	$0.004$	$0.001$
	A	$0.011$	$0.001$

.

In Table 1, the smaller MAPE, RMSE, and MAE index values, the better the algorithm performance. The larger the $R^2$ index, the better the algorithm performance. Bold indicates the optimal result in each scenario. It is clear that our method is optimal in all cases (bold), especially for $p=20$, $p=30$, and $p=40$. In the small sample case, the prediction performance of the LSTM method decreases significantly as the dimensionality of the covariates increases. The prediction performance of the Random Forest and BP methods is not stable with increasing covariate dimensions. Although the performance of our method decreases as the covariate dimension increases, it is still significantly better than the other methods. Table 2 shows the RMSE and Bias of the estimated values of $\beta$ and A. From Table 2, we can see that the proposed method performs well. Two metrics are small when the covariate dimension is 20 and 30. When the dimension is increased to 40, the value of the RMSE index increases, but it is still within the acceptable range.

To better illustrate the performance of parameter estimation, Figure 1 shows box plots of the estimator values of A, ${\beta }_1$, ${\beta }_2$, ${\beta }_3$ under $p=30$, where the horizontal coordinate is the index of the variables and the vertical coordinate is the values of estimators. The corresponding figures on $p=20$ and $p=40$ are shown in Appendix A.2. For the estimators ${\beta }_1$, ${\beta }_2$ and ${\beta }_3$, we can see the first four elements are estimated close to the true value, and the remaining values are estimated clear to zero; This implies that the proposed method can achieve good variable selection performance. In addition, all elements of the state increment matrix A are estimated close to the true values, which also confirms the good performance of our method.

In addition, to further verify that the algorithm is sensitive to the choice of hyperparameters $r,\delta$, we conduct experiments on data with a covariate dimension of 30. Consider the following three experiments, the first with $r=0.5,\delta =0.5$; the second with $r=1.0,\delta =1.0$; and the third with $r=1.5,\delta =1.5$. The experimental results show that the estimation results are not sensitive to the choice of the two hyperparameters r and $\delta$. The images of the Gamma distributions for the three different hyperparameter settings are very similar in shape. This similarity may contribute to the reason why, for a certain range of variations in r and $\delta$ values, the model’s performance may not show sensitivity to these hyperparameters. We show the results in Appendix A.4.

4.2. Experiment 2

In experiment 2, we consider the higher dimension cases: p = 60, 90, 120. We set the same A as experiment 1 and the coefficient as follows:

$$\beta ={\left({\beta }_1,{\beta }_2,{\beta }_3\right)}^{\top }={\left(\begin{array}{ccc}0.5& 1& 1.5\\ -2& -2& -1.5\\ 2& 1.5& 1\\ -1& -1.5& -2\\ ⋮& ⋮& ⋮\\ 0& 0& 0\end{array}\right)}_{p\times 3}$$

where the first four rows are nonzero and other elements are zero. We set the number of discrete time intervals $T=600$ and the sample size in the testing set $m=10$. In addition, the hyperparameters r and $\delta$ in the HDVBHMM method are set to 1. The results are shown in

Table 3. Average values of four metrics of all approaches with standard deviation in each parenthesis based on 10 simulations under $T=600$.

p	Method	Estimate Performance
		MAPE	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$
$p=60$	LSTM	$1.608$ $\left(1.826\right)$	$1.524$ $\left(0.104\right)$	$2.332$ $\left(0.304\right)$	$0.722$ $\left(0.126\right)$
	BP	$1.511$ $\left(1.326\right)$	$1.513$ $\left(0.209\right)$	$2.328$ $\left(0.662\right)$	$0.725$ $\left(0.143\right)$
	Random Forest	$1.236$ $\left(1.748\right)$	$1.459$ $\left(0.175\right)$	$2.159$ $\left(0.525\right)$	$0.728$ $\left(0.151\right)$
	HDVBHMM	$\mathbf{0.851}$ $\left(0.871\right)$	$\mathbf{1.091}$ $\left(0.218\right)$	$\mathbf{1.235}$ $\left(0.489\right)$	$\mathbf{0.884}$ $\left(0.078\right)$
$p=90$	LSTM	$1.690$ $\left(2.046\right)$	$1.725$ $\left(0.152\right)$	$2.998$ $\left(0.542\right)$	$0.523$ $\left(0.282\right)$
	BP	$2.780$ $\left(5.010\right)$	$1.606$ $\left(0.225\right)$	$2.626$ $\left(0.706\right)$	$0.636$ $\left(0.239\right)$
	RF	$\mathbf{0.718}$ $\left(0.365\right)$	$1.374$ $\left(0.246\right)$	$1.942$ $\left(0.694\right)$	$0.797$ $\left(0.091\right)$
	HDVBHMM	$0.878$ $\left(0.887\right)$	$\mathbf{1.156}$ $\left(0.249\right)$	$\mathbf{1.392}$ $\left(0.617\right)$	$\mathbf{0.862}$ $\left(0.135\right)$
$p=120$	LSTM	$1.941$ $\left(1.839\right)$	$1.884$ $\left(0.372\right)$	$3.677$ $\left(1.389\right)$	$0.463$ $\left(0.323\right)$
	BP	$1.235$ $\left(1.023\right)$	$1.651$ $\left(0.274\right)$	$2.792$ $\left(0.854\right)$	$0.684$ $\left(0.181\right)$
	Random Forest	$\mathbf{0.832}$ $\left(0.679\right)$	$1.571$ $\left(0.193\right)$	$2.502$ $\left(0.621\right)$	$0.718$ $\left(0.154\right)$
	HDVBHMM	$0.910$ $\left(0.717\right)$	$\mathbf{1.321}$ $\left(0.253\right)$	$\mathbf{1.804}$ $\left(0.718\right)$	$\mathbf{0.763}$ $\left(0.393\right)$

In the Long-Term and Short-Term Memory methods, the learning rate is lr = 0.001, the number of training cycles (Epochs) is set to 50, and the size of the hidden layer is set to 10. The hidden layer of the BP method consists of 20 neurons, and the maximum number of iterations is set to 10,000. In the random forest regression model, the number of trees is set to 100. The bold results are the optimal ones among four methods.

and

Table 4. Average values of the RMSE and Bias of A and $\beta$ based on 10 replications in Experiment 2.

p	Parameter	Estimate Performance
		RMSE	Bias
60	$\beta$	0.001	0.001
	A	0.002	0.001
90	$\beta$	0.001	0.001
	A	0.005	0.001
120	$\beta$	0.015	0.007
	A	0.034	0.009

.

As can be seen in Table 4, when the covariate dimensions are increased and the sample size reaches 600, our method still performs well among the four methods. It should be noted that when the covariate dimension is 90 and 120, the MAPE metric of the random forest method is slightly smaller than our method. In addition, as the covariate dimension increases from p = 60 to P = 120, the performance of the LSTM method decreases significantly, which is the worst performance among the four methods. This shows that LSTM does not perform well on such small-sample high-dimensional datasets. As the dimensionality of the covariates increases, although the BP and Random Forest methods show better prediction performance than the LSTM method, they are also poorer than the prediction performance of the HDVBHMM method. Overall, our method outperforms the other three methods in terms of prediction performance as the dimensionality increases, suggesting that our method performs better on small-sample high-dimensional datasets.

Figure 2 shows box plots of the estimator values of A, ${\beta }_1$, ${\beta }_2$, ${\beta }_3$ under $p=90$. The corresponding figures on $p=60$ and $p=120$ are shown in Appendix A.3. From Figure 2, we can see that the regression coefficients ${\beta }_1$, ${\beta }_2$, and ${\beta }_3$ are accurately estimated. the first four elements are estimated close to the true value, and the remaining values are estimated clear to zero. It implies that the proposed method can successfully achieve variable screening even as the covariate dimension increases. In addition, all elements of the state increment matrix A are estimated close to the true values, which also confirms the good performance of the proposed method.

5. Application to Real Datasets

In this section, we focus on Beijing Multi-Site Air-Quality data, which include 6 major air pollutants and 6 related meteorological variables at multiple locations in Beijing. These air-quality measurements are created by the Beijing Municipal Environmental Monitoring Center. In addition, meteorological data at each air quality location are paired with the nearest weather station provided by the China Meteorological Administration. The data span from 1 March 2013 to 28 February 2017. In our study, we consider PM2.5 concentration as response variable, and PM10 concentration, SO₂ concentration, NO₂ concentration, CO concentration, O₃ concentration, Temperature (TEMP), Pressure (PRES), Dew point temperature (DEWP), Precipitation (RAIN), and Wind speed (WSPM) as covariates; that is, $p=10$. In order to study the performance on small sample datasets, we delete the missing values in the data and select the data samples in the first 200 time intervals from the Shunyi observation point in Beijing in 2017 for analysis. To assess the predictive performance, we use the first 140 samples as the training set, and the remaining 60 samples as the testing set. We compare the proposed method with the BP neural network, LSTM and Random Forest method similar to Section 4.

One of the main challenges in implementing the HMM is to determine the optimal number of hidden states. The Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) are two common model selection techniques, which select the best model by balancing the fitting accuracy and complexity of the model. In selecting the number of hidden states for a Hidden Markov Model, both AIC and BIC evaluate multiple models containing different numbers of states and select an optimal model that balances fitting accuracy and complexity. Multiple HMMs are trained separately using different numbers of hidden states, then the AIC or BIC values are calculated for each model, and finally the model with the smallest AIC or BIC value is selected [34]. Similar to the work of Dofadar et al. [34], we use AIC and BIC to select the number of the hidden states. The AIC equation used in this study is given by $AIC=2k-2L$, where k is the number of free parameters in the model and L is the log probability value. The formula for k used in this research is $k=n^2+2n-1$, where n is the current value of the hidden state. The BIC equation used in this study is expressed as $BIC=ln\left(T\right)k-2L$, where T is the total number of observations. To find the best number of hidden states, we calculate AIC and BIC values based on the different numbers of hidden states: 2, 3, 4, and 5. The results are shown in Figure 3. Figure 3 shows that when the number of hidden states is 3, the AIC and BIC values are the smallest, indicating that choosing the number of hidden states as 3 is the closest to the real model. Therefore, we set the the number of hidden states $K=3$.

Similar to Section 4, we calculate MAPE, RMSE, MAE and $R^2$ to evaluate the predictive performance. Since the time series data are positively skewed, MAE and MASE are the best evaluation metrics for evaluating the model performance [35]. The results are shown in

Table 5. Prediction Performance of Four Methods on the testing data.

Method	MAPE	RMSE	MAE	${\mathit{R}}^2$
Random Forest	1.043	4.039	16.318	0.941
BP	1.025	5.250	27.570	0.852
LSTM	0.462	4.924	24.249	0.876
HDVBHMM	$\mathbf{0.317}$	$\mathbf{2.249}$	$\mathbf{5.058}$	$\mathbf{0.993}$

In the Long-Term and Short-Term Memory methods, learning rate is lr = 0.001, the number of training cycles (Epochs) is set to 50, and the size of the hidden layer is set to 40. the BP method contains 12 neuron hidden layers and the maximum number of iterations is set to 10,000. The hyperparameters hyperparameters r, $\delta$ in the HDVBHMM method were set to 1.0. The bold results are the optimal ones among four methods.

. From Table 5, we can see that the MAPE and MAE of the proposed HDVBHMM method are smaller than ones of other methods, and the $R^2$ value of the proposed method is larger than one of other methods, indicating that the performance of the HDVBHMM method is better than other methods. Among the other three competing methods, the MAPE and MAE values of the Random Forest method are lowest among those of the three competing methods, but its MAPE and MAE are still much larger than ones of the proposed method. The BP method is the worst performing among four methods with MAE = 27.570 and MAPE = 1.025.

To better illustrate the predictive performance, Figure 4 shows the true data and predicted values via four methods on the testing set. From Figure 4, we can see that in the first 30 time points, the proposed method fits the true values very well. In the second set of 30 time points, as the prediction time period increases, the predicted values exhibit a slight error, but they are still better than those of other methods. Overall, the prediction accuracy of the proposed method is much better than ones of other methods in term of both short and long time periods.

The estimated values of $\beta$ corresponding to the three states are shown in

Table 6. Estimates of the regression coefficients $\beta$ for each hidden state.

State	PM10	SO2	NO2	CO	O3	TEMP	PRES	DEWP	RAIN	WSPM
State 1	−0.833	−0.349	−0.030	0.003	−0.090	−2.625	−0.017	−3.552	1.524	3.547
State 2	0.965	−0.230	−0.189	−0.003	0.004	−2.154	−0.010	0.745	−4.227	3.891
State 3	−0.303	1.080	1.462	0.000	−0.279	18.55	−0.123	−0.127	9.057	19.217

. From Table 6, we can see that PM10, SO₂, TEMP (temperature), DEWP (dew point temperature), RAIN (precipitation), and WSPM (wind speed) have the greatest influence on PM2.5 emissions in state 1. PM10, SO₂, TEMP, and DEWP are the four factors that have a negative effect on the presence of PM2.5 emissions in the area, and as these four factors increase, PM2.5 emissions will decrease; meanwhile RAIN and WSPM have a positive effect on the presence of PM2.5 in the area. Rainfall and high wind speed may have increased PM2.5 concentrations through physical effects (such as windblown dust). The prediction formula of the PM2.5 in State 1 is as follows:

$$\begin{array}{cc}\hfill {\mathrm{PM}2.5}_1=& -0.833\mathrm{PM}10-0.349{\mathrm{SO}}_2+0.003\mathrm{CO}-0.03{\mathrm{NO}}_2-0.09\mathrm{O}3-2.625\mathrm{TEMP}\hfill \\ \hfill & -0.017\mathrm{PRES}-3.552\mathrm{DEWP}+1.524\mathrm{RAIN}+3.547\mathrm{WSPM}.\hfill \end{array}$$

In addition, PM10, Sulfur Dioxide, Nitrogen Dioxide, TEMP (temperature), DEWP (dew point temperature), RAIN (precipitation), and WSPM (wind speed) have the largest effect on PM2.5 in State 2. The results showed that in state 2, some chemical reactions led to the depletion of gases such as SO₂ and NO₂, which reduced the production of PM2.5, and rainfall also reduced the production of PM2.5. The high wind speed led to an increase in PM2.5 concentration, probably because the wind speed increased the diffusion and transport of particulate matter. The prediction formula of the PM2.5 in this State is as follows:

$$\begin{array}{cc}\hfill {\mathrm{PM}2.5}_2=& 0.965\mathrm{PM}10-0.23{\mathrm{SO}}_2-0.003\mathrm{CO}-0.189{\mathrm{NO}}_2-0.04\mathrm{O}3-2.154\mathrm{TEMP}\hfill \\ \hfill & -0.01\mathrm{PRES}+0.745\mathrm{DEWP}-4.227\mathrm{RAIN}+3.891\mathrm{WSPM}.\hfill \end{array}$$

PM10, SO₂, NO₂, O₃, TEMP (temperature), PRES (pressure), DEWP (dew point temperature), RAIN (precipitation), and WSPM (wind speed) have the greatest impact on PM2.5 in state 3. It is worth noting that the increase in variables such as SO₂ and NO₂ leads to an increase in PM2.5 concentration. In addition, the significant positive coefficients for temperature indicate that higher temperatures promote the formation of PM2.5, which may be related to the acceleration of certain chemical reactions by high temperatures. The increase in SO₂ and NO₂ may promote the formation of secondary particulate matter, which in turn increases the PM2.5 concentration. Wind speed increases particulate dispersion, and rainfall may also promote the formation of secondary particulate matter from some soluble substances. The prediction formula of the PM2.5 in this State is as follows:

$$\begin{array}{cc}\hfill {\mathrm{PM}2.5}_3=& -0.303\mathrm{PM}10+1.08{\mathrm{SO}}_2+1.462{\mathrm{NO}}_2-0.279\mathrm{O}3+18.55\mathrm{TEMP}\hfill \\ \hfill & -0.123\mathrm{PRES}-0.127\mathrm{DEWP}+9.057\mathrm{RAIN}+19.217\mathrm{WSPM}.\hfill \end{array}$$

In summary, the regression coefficients for the three states reflect the effects of different environmental factors on PM2.5 concentrations. The positive and negative signs and magnitudes of these coefficients can provide scenarios on how to manage and predict PM2.5 concentrations by controlling these environmental factors under different environmental conditions. In particular, the fact that temperature, rainfall and wind speed have different effects on PM2.5 concentrations in different states suggests that PM2.5 management needs to take into account complex meteorological conditions and interactions between air pollutants.

6. Conclusions

In this paper, the variable selection for high-dimensional HMMs is studied based on the variational inference. We develop a Variational Bayesian method with the double-exponential shrinkage prior for variable selection. The proposed method can quickly and accurately estimate the posterior distributions and the unknown parameters. In the simulation studies and real data analysis, the proposed method outperforms the common methods in term of variable selection and prediction. In the Beijing Multi-Site Air-Quality analysis, we select the optimal number of the hidden stats based on the AIC and BIC methods, and fit the HMMs of the response variable PM2.5. In the current research work, we investigate variational inference for linear HMMs with high dimensional covariates; that is, the mean of the response variable is linear with respect to the high dimensional covariates. Many of the relationships between variables in practical applications may be not linear, so variational inference for nonlinear HMMs is worth studying. In addition, it is assumed that the variances in observations are the same in different hidden states in this study, but in practical applications, heteroskedasticity may be more in line with real-world data characteristics. For that reason, the heteroskedasticity issue for HMMs is also worth exploring deeply. Moreover, Ivan Gorynin’s work [36] verifies that the Pairwise Markov Model (PMM) outperforms the traditional HMM in terms of accuracy when the observed variable y is highly autocorrelated or when the hidden chain is not Markovian. Unlike the HMM, which assumes that the hidden chain z is Markovian, the PMM assumes that ($z,y$) is Markovian. Since hidden chains are not necessarily Markovian in the PMM, it is more general than the HMM. Parameter estimation of PMM models is done using Variational Bayesian methods in the work of Katherine Morales [37]. However, the effect of including the covariate x on the target variable y was not considered in their work. Therefore, as an extension of the proposed method, which replaces the HMM with the PMM, the inclusion of high-dimensional covariates in the PMM may yield more accurate predictions.