Particle Filtering Estimation of Regime Switching Factor Model and Its Application in Statistical Arbitrage Strategy

Abstract

Statistical factor models are widely applied across various domains of the financial industry, including risk management, portfolio selection, and statistical arbitrage strategies. However, conventional factor models often rely on unrealistic assumptions and fail to account for the fact that financial markets operate under multiple regimes. In this paper, we propose a regime-switching factor model estimated via a particle filtering algorithm, which is a Monte Carlo-based method well-suited for handling nonlinear and non-Gaussian systems. Our empirical results show that incorporating dynamic structure and a regime-switching mechanism significantly enhances the model’s ability to detect structure breaks and adapt to evolving market conditions. This leads to improved performance and reduced drawdowns in the equity statistical arbitrage strategies.

1. Introduction

Factor analysis plays a critical role in quantitative finance, with applications in asset pricing, risk management, portfolio selection, and statistical arbitrage strategies. Broadly speaking, factor models can be categorized into two types: explicit factor models, which fall under supervised learning, and statistical factor models, which are typically unsupervised. Compared to explicit factor models such as Barra risk model (Barra, 2004), statistical factor models offer two key advantages: they are numerically more stable in capturing the underlying covariance structure using fewer factors, and they are asset-class agnostic, making them broadly applicable across different markets.

However, financial markets are far from stationary. They undergo regime shifts that fundamentally alter market dynamics. For example, the transition from the unusually low-volatility environment of 2017 to the volatility spike in February 2018, the U.S. Federal Reserve’s abrupt rate hikes in 2022, or geopolitical shocks such as Russia’s invasion of Ukraine all represent regime-switching events. Standard statistical factor models, which assume globally static factor loadings, often fail to forecast covariances accurately during such transitions, the very moments when robust risk modeling is most needed.

The literature on statistical factor models has evolved to address such challenges. Early PCA-based factor models (Jolliffe, 2004) are limited in parametric structure and perform poorly when within group variation dominates between group variation (Hinton & Dayan, 1997; McLachlan & Peel, 2000). In the 1980s, maximum likelihood (ML) factor analysis estimated via the EM algorithm was introduced by (Rubin & Thayer, 1982). This was further generalized in the 1990s by (Ghahramani & Hinton, 1996), who proposed the mixture of factor analyzers (MFA). By combining clustering with local dimension reduction, MFA works better than classical ML factor analysis due to its ability of segmenting data into different market regimes, each with its own factor structure, improving on the “one-size-fits-all” ML factor models.

Despite these advancements, within each regime, MFA still uses a classical ML factor model which assumes i.i.d. hidden factors and Gaussian innovations. Dynamic factor models extend this framework by introducing a recursive structure for hidden factor returns, capturing their time dynamics more realistically. Earlier works such as (Bai & Wang, 2014; Forni et al., 2009; Geweke & Zhou, 1996; Kim & Halbert, 1999) typically assume a simple first-order autoregressive (AR(1)) structure for factor returns, while later developments extend the underlying process to ARMA and VARMA formulations (see Peña & Poncela, 2004, 2006; Urteaga et al., 2017; Varga & Szendrei, 2024). These dynamic factor models are capable of handling both stationary and nonstationary hidden factors.

In this paper, we focus on the AR(1)-based dynamic factor models with regime-switching factor loadings governed by a first order Markov chain. Such models, or, more generally, regime switching state space models, have been extensively studied in the literature (see Carvalho et al., 2010; Kim & Halbert, 1999; Whiteley et al., 2010). This framework not only models the hidden factor process but also captures the regime switching effects inherent in financial data. Common estimation approaches include Gibbs sampling, the EM algorithm, particle MCMC and particle filtering. We adopt the particle filtering framework for its computational efficiency and sequential structure, which aligns closely with real world trading scenarios.

Particle filtering (Arulampalam et al., 2002; Doucet & Johansen, 2009) is a class of Monte Carlo methods designed to perform recursive Bayesian inference when analytical solutions such as the Kalman filter or Baum–Welch algorithm are infeasible. Although it is well developed when model parameter values are known, parameter learning in more complex settings remains challenging. Specifically, when parameters evolve according to a stochastic process rather than being constant, such as in applications in finance, an adaptive parameter learning algorithm becomes critical. To learn the parameter values, one early approach by (Liu & West, 2001) incorporates parameters into the hidden state vector, but this often exacerbates the degeneracy issue in particle filtering. More computationally robust approaches have been introduced in (Djuric et al., 2012) and (Carvalho et al., 2010). Instead of combining parameter estimation to the learning algorithm they marginalize over parameters to reduce the estimation variance of their system. In this work, we implemented a particle filtering algorithm inspired by (Djuric et al., 2012) and (Carvalho et al., 2010) and applied it to estimate a regime switching dynamic factor model using real financial data.

The goal of this study is not merely methodological. By evaluating statistical factor models in the context of statistical arbitrage strategies, we directly test their practical usefulness for risk modeling. Following (Avellaneda & Lee, 2010), we adopt a simplified trading framework where performance differences are driven primarily by the quality of covariance forecasts. In this setup, a superior factor model should deliver more accurate residuals, adapt more quickly to regime shifts, and ultimately provide a stronger risk foundation for arbitrage strategies.

The rest of paper is organized as follows: Section 2 introduces the state space model and its estimation methods, including Kalman filter, EM algorithm, Particle filter, and the more recent development in parameter learning—the particle learning algorithm. Section 3 presents statistical factor models such as the MLE factor model, mixture of factor analyzers and the regime-switching dynamic factor model. We highlight the evolution of statistical factor models over recent decades and provides a simulation study demonstrating the effectiveness of particle learning algorithm for estimating the regime-switching dynamic factor model. Section 4 presents empirical studies applying the regime-switching dynamic factor model in the context of equity stat-arb strategy and compares its performance to the conventional MLE factor model. Finally, Section 5 concludes the paper.

2. State Space Model and Its Estimation

A state space model is a framework used to describe the evolution of a dynamical system over time, particularly when the system is only partially observed. It has numerous applications in finance, such as time series prediction (alpha generation), statistical factor analysis (risk modeling), and portfolio selection (see Kim & Halbert, 1999; Kolm & Ritter, 2014). In the most general form, a state space model can be represented by the following:

$$\begin{array}{cc}\hfill {\mathbf{x}}_t& =f_t\left({\mathbf{x}}_{t-1},{\mathbf{w}}_t\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{y}}_t& =g_t\left({\mathbf{x}}_t,{\mathbf{v}}_t\right)\hfill \end{array}$$

(2)

where ${\mathbf{x}}_t$ denotes the hidden states with dimension k, and ${\mathbf{y}}_t$ the observations with dimension s. The terms ${\mathbf{w}}_t$ and ${\mathbf{v}}_t$ represent the noise terms for the evolution Equation $\left(1\right)$ and the observation Equation $\left(2\right)$, with dimensions w and v, respectively. Throughout this work, we use bold lowercase to denote vectors, and bold uppercase to denote matrices. The functions $f_t$ and $g_t$ are general nonlinear functions defining the state transition and observation processes. When the parameter values of the above equations are known, inference about the hidden states can be efficiently performed using the recursive Bayesian algorithm (Arulampalam et al., 2002), which is summarized as Algorithm 1:

Algorithm 1 Recursive Bayesian Algorithm

Prediction: $$p\left({\mathbf{x}}_t\mid {\mathbf{y}}_{1:t-1}\right)=\int p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}\right)p\left({\mathbf{x}}_{t-1}\mid {\mathbf{y}}_{1:t-1}\right)d{\mathbf{x}}_{t-1}$$ (3) Update: $$p\left({\mathbf{x}}_t\mid {\mathbf{y}}_{1:t}\right)={\displaystyle \frac{p\left({\mathbf{y}}_t\mid {\mathbf{x}}_t\right)p\left({\mathbf{x}}_t\mid {\mathbf{y}}_{1:t-1}\right)}{p\left({\mathbf{y}}_t\mid {\mathbf{y}}_{1:t-1}\right)}}$$ (4)

Where the denominator in Equation $\left(4\right)$ is called the marginal likelihood (or evidence), which is another intergral needs to be solved (see Appendix A).

2.1. Kalman Filter and EM-Algorithm Estimation

When the dynamical system described by Equations $\left(1\right)$ and $\left(2\right)$ is linear, time-invariant, and finite-dimensional, functions $f_t$, and $g_t$ can be expressed in the following form:

$$\begin{array}{cc}\hfill {\mathbf{x}}_t& =\mathbf{B}{\mathbf{x}}_{t-1}+{\mathbf{w}}_t\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{y}}_t& =\mathbf{A}{\mathbf{x}}_t+{\mathbf{v}}_t\hfill \end{array}$$

(6)

and if we further assume ${\mathbf{w}}_t\sim N\left(0,\mathbf{Q}\right)$, ${\mathbf{v}}_t\sim N\left(0,\mathbf{R}\right)$, then the well-known Kalman filter can be derived to solve Equations $\left(5\right)$ and $\left(6\right)$ analytically as described in (Arulampalam et al., 2002) and in Algorithm A1 (see Appendix B). When parameters $\mathbf{A}$, $\mathbf{B}$ are unknown, the EM algorithm (Ghahramani & Hinton, 1996) can be employed for their estimation. In this framework, the Kalman filter is used in the E-step to compute the expected sufficient statistics of the hidden states, while the M-step updates the parameter estimates (see Appendix C).

2.2. Particle Filter

When the dynamical system $\left(1\right)$ and $\left(2\right)$ do not admit a linear-Gaussian representation as in $\left(5\right)$ and $\left(6\right)$, deriving an analytical solution for the recursive Bayesian inference becomes challenging. In such cases, particle filtering techniques has been a very popular approach for solving nonlinear dynamical system, including models like the stochastic volatility model (See Djuric et al., 2012).

The Sequential Importance Sampling (SIS) filter (Gordon et al., 1993) approximates the joint posterior density at time t using the importance sampling method:

$$p\left({\mathbf{x}}_{1:t}\mid {\mathbf{y}}_{1:t}\right)\approx \sum _{i=1}^N{W}_t^i\delta \left({\mathbf{x}}_{1:t}-{\mathbf{x}}_{1:t}^i\right)$$

(7)

where $\delta \left(·\right)$ is the Dirac delta function, and $W_t^i$ is the importance weight for particle i, given by the following:

$$W_t^i\propto {\displaystyle \frac{p\left({\mathbf{x}}_{1:t}\mid {\mathbf{y}}_{1:t}\right)}{q\left({\mathbf{x}}_{1:t}\mid {\mathbf{y}}_{1:t}\right)}}$$

(8)

With the factorization

$$q\left({\mathbf{x}}_{1:t}\mid {\mathbf{y}}_{1:t}\right)=q\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1},{\mathbf{y}}_t\right)q\left({\mathbf{x}}_{1:t-1}\mid {\mathbf{y}}_{1:t-1}\right)$$

(9)

sequential sampling of states becomes feasible, which forms the foundation of SIS particle filter (see Algorithm A4).

In practice, the SIS particle filter suffers from weight degeneracy: the variance of the weight $W_t^i$ will increase over time until only one particle dominates. To alleviate this issue, the Sequential Importance Resampling (SIR) filter resamples the particles based on their weights, replicating those particles with large weights and eliminating particles with small weights, thereby reducing variance (Doucet & Johansen, 2009).

Although the optimal importance density for a generic particle filter algorithm is $p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}^i,{\mathbf{y}}_t\right)$ (Doucet et al., 2000), the SIR particle filter typically uses $p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}^i\right)$ as an important density and applies resampling at each step (see Algorithm A5). While SIR filter improves stability, it still has weaknesses (Pitt & Shephard, 1999). First, if there is an outlier at time t, the weights $W_t^i$ will be unevenly distributed and the filter will become imprecise. Second, the predictive density $p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}^i\right)$ often fails to capture tail behavior accurately, causing some particles to drift into low-likelihood regions during step 2 in Algorithm A5.

To address these limitations, the auxiliary particle filter proposed by (Pitt & Shephard, 1999) introduces a lookahead step—also known as the auxiliary variable mechanism. This approach uses a predictive weighting scheme (see Algorithm A6 step 1) that estimates the likelihood of the upcoming observation given the predicted state, a process often referred to as lookahead weighting. When the observation equation is informative in a dynamical system, the auxiliary particle filter usually will reduce variance in particle weights and hence perform better.

2.3. Particle Filter with Parameter Learning

The particle filter algorithms discussed so far assume that model parameters are known. However, in many applications, model parameters are typically unknown. Learning the static parameter values—or more broadly—identifying the coefficient evolution process—is a critical aspect of system identification. The integration of parameter learning into the particle filtering framework has been extensively studied (Kantas et al., 2015).

One idea is to treat parameters as additional hidden state variables in the particle filtering framework, but this approach often leads to severe degeneracy issue. To address this issue, Liu and West (2001) proposed introducing an artificial evolutional model for the parameters and augment the hidden state space by including them.

Another idea is to integrate out (or marginalize out) the parameters whenever feasible (Djuric et al., 2012; Schon et al., 2005; Storvik, 2002). The benefits of the marginalized particle filter have been discussed in (Özkan et al., 2013).

Building on these ideas, Carvalho et al. (Carvalho et al., 2010) introduced particle learning approach, a novel particle filter with the embedded parameter learning step. The particle learning method marginalizes both the parameters and the hidden states by only tracking their sufficient statistics, which are updated later using recursive Bayesian algorithm and Kalman filter, respectively. It is also constructed under the auxiliary particle filter framework; the performance improvement has been demonstrated in (Carvalho et al., 2010; Lopes & Tsay, 2010).

In our paper, we develop a simplified version of the particle learning algorithm to estimate regime-switching dynamic factor model.

2.3.1. Particle Learning

Particle learning algorithm was introduced by Carvalho et al. in (Carvalho et al., 2010), extending the idea of auxiliary particle filter to incorporate parameter learning. While the overall filtering scheme is the same as auxiliary particle filter, the key innovation lies in tracking the “essential state” ${\mathbf{z}}_t$.

Specifically, ${\mathbf{z}}_t$ includes the following:

• the sufficient statistics of the parameter vector ${\mathbf{s}}_t$;
• the sufficient statistics of hidden states ${\mathbf{x}}_t$, ${\mathbf{s}}_t^x$; and
• the current value of the parameter vector, ${\mathit{\theta }}_t$;

  $${\mathbf{z}}_t=\left\{{\mathbf{s}}_t,{\mathbf{s}}_t^x,{\mathit{\theta }}_t\right\}.$$

The reason we include parameter value in the essential state is because for evaluating likelihood and drawing samples from predictive density we need to use parameter values. However, their values are not directly sampled at each step, which most likely causes degeneracy issues; instead, they are inferred offline from their associated sufficient statistics. This approach is also referred to as marginalization by simulation.

The derivation of this filter starts with the recursive relation:

$$p\left({\mathbf{z}}_t\mid {\mathbf{y}}_{1:t}\right)\propto \int p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1}\right)p\left({\mathbf{z}}_t\mid {\mathbf{z}}_{t-1},{\mathbf{y}}_t\right)p\left({\mathbf{z}}_{t-1}\mid {\mathbf{y}}_{1:t-1}\right)d{\mathbf{z}}_{t-1}$$

where

$$\begin{array}{c}p\left({\mathbf{z}}_t\mid {\mathbf{z}}_{t-1},{\mathbf{y}}_t\right)=\int p\left({\mathbf{s}}_t\mid {\mathbf{s}}_{t-1},{\mathbf{x}}_t,{\mathbf{y}}_t\right)p\left({\mathbf{s}}_t^x\mid {\mathbf{s}}_{t-1}^x,{\mathbf{x}}_t,{\mathbf{y}}_t\right)\\ ·p\left({\mathbf{x}}_t\mid {\mathbf{s}}_{t-1}^x,{\mathbf{y}}_t\right)d{\mathbf{x}}_t\end{array}$$

and

$$p\left({\mathbf{x}}_t\mid {\mathbf{s}}_{t-1}^x,{\mathbf{y}}_t\right)=\int p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1},{\mathbf{y}}_t\right)p\left({\mathbf{x}}_{t-1}\mid {\mathbf{s}}_{t-1}^x,{\mathbf{y}}_t\right)d{\mathbf{x}}_{t-1}$$

So, given the particles of ${\mathbf{s}}_{t-1}^x$, we can construct a Monte Carlo estimate for $p\left({\mathbf{x}}_t\mid {\mathbf{s}}_{t-1}^x,{\mathbf{y}}_t\right)$ by sampling from $p\left({\mathbf{x}}_{t-1}\mid {\left({\mathbf{s}}_{t-1}^x\right)}^i,{\mathbf{y}}_t\right)$ and $p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}^i,{\mathbf{y}}_t\right)$, respectively. After that we can update ${\mathbf{s}}_t$ and ${\mathbf{s}}_t^x$ deterministically. The particle learning method can be summarized as Algorithm 2:

Algorithm 2 Particle Learning

(Resample) Draw $k^i\sim Multi\left\{W_t^1,\dots ,W_t^N\right\}$, where $W_t^i\propto p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1}^i\right)$ (Propagate) Draw ${\mathbf{x}}_{t-1}^i\sim p\left({\mathbf{x}}_{t-1}\mid {\left({\mathbf{s}}_{t-1}^x\right)}^{k^i},{\mathit{\theta }}_{t-1}^{k^i},{\mathbf{y}}_t\right)$ (Propagate) Draw ${\mathbf{x}}_t^i\sim p\left({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1}^i,{\mathit{\theta }}_{t-1}^{k^i},{\mathbf{y}}_t\right)$ (Update) ${\mathbf{s}}_t^i=\mathcal{S}\left({\mathbf{s}}_{t-1}^{k^i},{\mathbf{x}}_t^i,{\mathbf{x}}_{t-1}^i,{\mathbf{y}}_t\right)$ (Sample) Draw ${\mathit{\theta }}_t^i\sim p\left({\mathit{\theta }}_t\mid {\mathbf{s}}_t^i\right)$ (Update) ${\left({\mathbf{s}}_t^x\right)}^i=\mathcal{K}\left({\left({\mathbf{s}}_{t-1}^x\right)}^{k^i},{\mathit{\theta }}_t^i,{\mathbf{y}}_t\right)$

Where $\mathcal{S}$ denotes the deterministic updating function for parameter sufficient statistics and $\mathcal{K}$ denotes Kalman filter for state sufficient statistics update.

3. Statistical Factor Analysis

Statistical factor analysis is a technique used to explain the correlation structure of multivariate observations through a lower-dimensional set of unobserved factors. The model can be expressed as follows:

$${\mathbf{y}}_t=\mathbf{B}{\mathbf{x}}_t+{\mathbf{v}}_t$$

(10)

where ${\mathbf{y}}_t$ is $\left(s\times 1\right)$ vector of observations, ${\mathbf{x}}_t$ is $\left(k\times 1\right)$ vector of statistical factors, and $\mathbf{B}$ is $\left(s\times k\right)$ factor loading matrix. ${\mathbf{v}}_t$ is the $\left(s\times 1\right)$ error term that follows certain distribution assumption. The hidden factors ${\mathbf{x}}_t$ are often assumed to follow an i.i.d. standard normal distribution, i.e., ${\mathbf{x}}_t\sim N\left(0,{\mathbf{I}}_k\right)$, and are assumed to be independent of the error term ${\mathbf{v}}_t$.

3.1. MLE Factor Analysis

The most commonly used approach for estimating the factor model in Equation $\left(10\right)$ is the Expectation-Maximization (EM) algorithm for maximum likelihood estimation (MLE) (Ghahramani & Hinton, 1996; McLachlan & Peel, 2000; Rubin & Thayer, 1982). The EM algorithm for the maximum likelihood estimation of the factor model is outlined below as Algorithm 3:

Algorithm 3 EM algorithm for Statistical Factor Analysis

E-Step: Given current fit of $\mathbf{B}$ and $\mathbf{R}$, calculate sufficient statistics: $$\begin{array}{cc}\hfill \gamma & ={\mathbf{B}}^T{\left(\mathbf{B}{\mathbf{B}}^T+\mathbf{R}\right)}^{-1}\hfill \end{array}$$ (11) $$\begin{array}{cc}\hfill E\left[{\mathbf{x}}_t\mid {\mathbf{y}}_t\right]& =\gamma {\mathbf{y}}_t\hfill \end{array}$$ (12) $$\begin{array}{cc}\hfill E\left[{\mathbf{x}}_t{\mathbf{x}}_t^T\mid {\mathbf{y}}_t\right]& =\mathbf{I}-\gamma \mathbf{B}+\gamma {\mathbf{y}}_t{\mathbf{y}}_t^T{\gamma }^T\hfill \end{array}$$ (13) M-Step: Update parameter estimation $$\begin{array}{cc}\hfill \widehat{\mathbf{B}}& =\left(\sum _{t=1}^T{\mathbf{y}}_{t}E\left[{\mathbf{x}}_t\mid {\mathbf{y}}_t\right]\right){\left(\sum _{t=1}^{T}E\left[{\mathbf{x}}_t{\mathbf{x}}_t^T\mid {\mathbf{y}}_t\right]\right)}^{-1}\hfill \end{array}$$ (14) $$\begin{array}{cc}\hfill \widehat{\mathbf{R}}& ={\displaystyle \frac{1}{T}}\mathrm{diag}\left(\sum _{t=1}^T\left({\mathbf{y}}_t{\mathbf{y}}_t^T-\widehat{\mathbf{B}}E\left[{\mathbf{x}}_t\mid {\mathbf{y}}_t\right]{\mathbf{x}}_t^T\right)\right)\hfill \end{array}$$ (15)

The EM algorithm starts with certain initial values for $\mathbf{B}$ and $\mathbf{R}$ and terminates when the rate of change in the log likelihood falls below some critical value. The log likelihood can be computed as follows:

$$\begin{array}{ccc}\hfill logL\left(\mathbf{B},\mathbf{R}\mid {\mathbf{y}}_{1:T}\right)& =& -{\displaystyle \frac{1}{2}}\left(\phantom{\sum _{t=1}^T}\right.T\left(slog\left(2\pi \right)+log\left(det\left(\mathbf{B}{\mathbf{B}}^T+\mathbf{R}\right)\right)\right)\hfill \\ & & +\sum _{t=1}^T{\mathbf{y}}_t^T{\left(\mathbf{B}{\mathbf{B}}^T+\mathbf{R}\right)}^{-1}{\mathbf{y}}_t\left.\phantom{\sum _{t=1}^T}\right)\hfill \end{array}$$

(16)

3.2. Mixture of Factor Analyzer

The factor model in Equation $\left(10\right)$ is a global linear model. However, in many real world applications—such as in financial engineering—it fails to reflect the fact that the observed data present multiple regimes whose behaviors differ materially. To address this limitation, the mixture of factor analyzer (MFA) model extends the MLE statistical factor model by combining local dimension reduction method with a Gaussian mixture model for clustering. Specifically, the MFA model can be described in the following form:

$${\mathbf{y}}_t=\mathbf{B}\left(i\right){\mathbf{x}}_t+{\mathbf{v}}_t\mathrm{with}\mathrm{prob}.{\pi }_i\left(i=1,\dots ,m\right)$$

(17)

where ${\sum }_{i=1}^m{\pi }_i=1,{\pi }_i\ge 0$, and the index i denotes the component (or regime), drawn from a categorical distribution with mixing probabilities $\left\{{\pi }_i\right\}$. $\mathbf{B}\left(i\right)$ is the factor loading matrix specific to component i. Following the formulation in (Ghahramani & Hinton, 1996), we assume the observation covariance $\mathbf{R}$ for each component are the same, and the number of mixture is fixed prior to the estimation step. The EM algorithm for estimating the parameters of the MFA model is described below in Algorithm 4:

Algorithm 4 EM algorithm for Mixture of Factor Analyzer

E-Step: Given current fit of ${\pi }_i\mathrm{for}i=1,\dots ,m$, $\mathbf{B}$ and $\mathbf{R}$, calculate, $$\begin{array}{ccc}\hfill {\tau }_{ij}& =& {\displaystyle \frac{{\pi }_{i}N\left({\mathbf{y}}_j,\mathbf{B}\left(i\right)\mathbf{B}{\left(i\right)}^T+\mathbf{R}\right)}{{\sum }_{i=1}^m{\pi }_{i}N\left({\mathbf{y}}_j,\mathbf{B}\left(i\right)\mathbf{B}{\left(i\right)}^T+\mathbf{R}\right)}}\hfill \end{array}$$ (18) $$\begin{array}{ccc}\hfill \gamma \left(i\right)& =& \mathbf{B}{\left(i\right)}^T{\left(\mathbf{B}\left(i\right)\mathbf{B}{\left(i\right)}^T+\mathbf{R}\right)}^{-1}\hfill \end{array}$$ (19) $$\begin{array}{ccc}\hfill \Sigma \left(i\right)& =& {\displaystyle \frac{1}{T}}\sum _{j=1}^T\left(\sqrt{{\tau }_{ij}}{\mathbf{y}}_j\right){\left(\sqrt{{\tau }_{ij}}{\mathbf{y}}_j\right)}^T\hfill \end{array}$$ (20) M-Step: $$\begin{array}{ccc}\hfill \widehat{\mathbf{B}}\left(i\right)& =& {\displaystyle \frac{\Sigma \left(i\right)\gamma {\left(i\right)}^T}{\frac{1}{T}{\sum }_{j=1}^T{\tau }_{ij}\left({\mathbf{I}}_s-\gamma \left(i\right)\mathbf{B}\left(i\right)\right)+\gamma \left(i\right)\Sigma \left(i\right)\gamma {\left(i\right)}^T}}\hfill \end{array}$$ (21) $$\begin{array}{ccc}\hfill \widehat{\mathbf{R}}& =& {\displaystyle \frac{1}{T}}\mathrm{diag}\left(\sum _{i=1}^m\Sigma \left(i\right)-\sum _{i=1}^m\widehat{\mathbf{B}}\left(i\right)\gamma \left(i\right)\Sigma \left(i\right)\right)\hfill \end{array}$$ (22) $$\begin{array}{ccc}\hfill {\widehat{\pi }}_i& =& {\displaystyle \frac{1}{T}}\sum _{j=1}^T{\tau }_{ij}\hfill \end{array}$$ (23)

Starting with initial values of ${\pi }_i$ for $i=1,\dots ,m$, $\mathbf{B}\left(i\right)$, and $\mathbf{R}$, we can iterate Algorithm 4 until convergence. The log-likelihood of the observed data under the MFA model is given by the following:

$$logL\left(\mathbf{\Psi }\right)=\sum _{j=1}^{T}log\left\{\sum _{i=1}^m{\pi }_{i}N\left({\mathbf{y}}_j,\mathbf{B}\left(i\right)\mathbf{B}{\left(i\right)}^T+\mathbf{R}\right)\right\}$$

where $\mathbf{\Psi }$ denotes the set of all model parameters.

3.3. Regime-Switching Dynamic Factor Model

Although the mixture of factor analyzer (MFA) model can capture data generated from multiple regimes, it still assumes the observations are i.i.d. and fails to account for the temporal persistence of market regimes. In practice, particularly in financial applications, market regimes tend to persist over time, with each regime exhibiting distinct statistical properties. Regime switching model equipped with Markov chain mechanism can better model the regime persistence and capture the abrupt regime shifts more efficiently than commonly used “low-pass filtering” approaches such as rolling window estimation. It fits better for financial time series, where capturing time dynamics is essential.

3.3.1. Methods

In this work, we represent the regime-switching dynamic factor model within a state space framework. The estimation of regime switching state space models has been well studied in the literature (Andrieu et al., 2010; Carter & Kohn, 1994; Kim & Halbert, 1999), with most approaches relying on the Markov Chain Monte Carlo (MCMC) algorithms. MCMC approaches are not only computationally very expensive, but also suffer slow convergence issue. Moreover, they are batch (offline) algorithms in nature which limits their use in real-time trading.

In this work, we will rely on the particle learning technique that is introduced in Section 2.3.1 as our estimation method. Computationally, it is significantly faster than MCMC algorithms and, being an online algorithm, it can be adopted in the real time trading environment. The convergence of particle filtering algorithm is well studied in the literature (Del Moral, 2004; Doucet et al., 2000).

Another key advantage of the particle filtering framework is its modular design, which makes it highly adaptable to complex model structures. Additional structure can be incorporated into the system relatively easier than MCMC framework, as long as it remains possible to draw samples from the importance density and evaluate the likelihood function upon receiving the new observations (see Djuric et al., 2012; Urteaga et al., 2017 for examples).

To introduce our model, we first define a discrete latent regime variable $r_t\in \left\{1,\dots ,K\right\}$ which captures the regime at time t. The essential state used in the particle learning algorithm is then extended as follows:

$${\mathbf{Z}}_t=\left\{r_t,{\mathbf{s}}_t,{\mathbf{s}}_t^x,{\mathit{\theta }}_t\right\},$$

where ${\mathbf{s}}_t$ and ${\mathbf{s}}_t^x$ denote the sufficient statistics for the parameters and the hidden states, respectively, and ${\mathit{\theta }}_t$ denotes the model parameters.

The model is specified as follows:

$$\begin{array}{cc}\hfill {\mathbf{x}}_t& =\mathbf{A}{\mathbf{x}}_t+{\mathbf{w}}_t\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathbf{y}}_t& =B\left(r_t\right){\mathbf{x}}_t+\sqrt{{\lambda }_t}{\mathbf{v}}_t\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\lambda }_t& \sim IG\left(v/2,v/2\right)\hfill \end{array}$$

(26)

where $IG\left(·\right)$ denotes the inverse Gamma distribution ${\mathbf{v}}_t\sim N\left(0_s,{\mathbf{R}}_s\right)$, ${\mathbf{w}}_t\sim N\left(0_k,{\mathbf{I}}_k\right)$. In this representation, we model heavy tailed innovation using the data augmentation scheme of (Carlin & Polson, 1991), where at each time t, we will follow the two step approach: (i) ${\lambda }_t\sim \mathrm{Inv}\text{-}\mathrm{Gamma}\left(v/2,v/2\right)$, (ii) ${\mathit{\eta }}_t={\sqrt{\lambda }}_t{\mathbf{v}}_t$, so that ${\mathit{\eta }}_t\sim t_v\left(0_s,{\mathbf{R}}_s\right)$ where ${\mathbf{R}}_s$ is a diagonal matrix.

If the degree of freedom is given, then at each time step t, our system is a conditionally Gaussian model. To derive the particle learning algorithm for this model (Equations (24)–(26)), we compute the predictive density $p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1},{\lambda }_t\right)$ by integrating out the latent regime variable $r_t$:

$$\begin{array}{ccc}\hfill p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1},{\lambda }_t\right)& =& \sum _{r_t=1}^{K}p\left({\mathbf{y}}_t\mid r_t,{\left({\mathbf{s}}_{t-1}^x,{\mathit{\theta }}_{t-1},{\lambda }_t\right)}^i\right)\hfill \\ & & ·p\left(r_t\mid r_{t-1},{\mathit{\theta }}_{t-1}\right)\hfill \end{array}$$

(27)

where $p\left({\mathbf{y}}_t\mid r_t,{\left({\mathbf{s}}_{t-1}^x,{\mathit{\theta }}_{t-1},{\lambda }_t\right)}^i\right)$ is the conditional likelihood, which is derived in Appendix G, and $p\left(r_t\mid r_{t-1},{\mathit{\theta }}_{t-1}\right)$ is given by the estimated transition matrix. To propagate $r_t$, we sample from the posterior distribution:

$$\begin{array}{ccc}\hfill p\left(r_t\mid {\left(r_{t-1},{\mathbf{s}}_{t-1}^x,{\mathit{\theta }}_{t-1},{\lambda }_t\right)}^i,{\mathbf{y}}_t\right)& \propto & p\left({\mathbf{y}}_t\mid r_t,{\left({\mathbf{s}}_{t-1}^x,{\mathit{\theta }}_{t-1},{\lambda }_t\right)}^i\right)\hfill \\ & & ·p\left(r_t\mid r_{t-1},{\mathit{\theta }}_{t-1}\right)\hfill \end{array}$$

(28)

The particle filtering algorithm for regime switching dynamic factor model is described below in Algorithm 5:

Algorithm 5 Particle Filter for Regime Switching Dynamic Factor Model

(Sampling) Draw ${\lambda }_t^i\sim IG\left(v/2,v/2\right)$ (${\lambda }_t^i=1$ for Gaussian model) (Resample) Draw $k^i\sim Multi\left\{W_t^1,\dots ,W_t^N\right\}$, with $W_t^i\propto p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1}^i,{\lambda }_t\right)$ where $$p\left({\mathbf{y}}_t\mid {\mathbf{z}}_{t-1}^i,{\lambda }_t\right)=\sum _{k=1}^{K}N\left({\mathbf{y}}_t;{\mathbf{m}}_y\left(k\right),{\mathbf{C}}_y\left(k\right)\right)p\left(r_t=k\mid r_{t-1}^i,{\mathit{\theta }}_{t-1}\right)$$ and ${\mathbf{m}}_y\left(k\right)=\mathbf{B}\left(k\right)\mathbf{A}{\mathbf{m}}_{t-1}$, ${\mathbf{C}}_y\left(k\right)=\mathbf{B}\left(k\right)\mathbf{A}{\mathbf{C}}_{t-1}{\mathbf{A}}^T\mathbf{B}{\left(k\right)}^T+\mathbf{B}\left(k\right)\mathbf{B}{\left(k\right)}^T+{\lambda }_t{\mathbf{R}}_s$. (Propagate) Draw $r_t^i\sim p\left(r_t\mid {\left({\mathbf{s}}_{t-1}^x,r_{t-1},{\mathit{\theta }}_{t-1}\right)}^{k^i},{\mathbf{y}}_t\right)$, which follows multinomial distribution with: $$p\left(r_t=k\mid {\mathbf{s}}_{t-1}^x,r_{t-1},{\mathit{\theta }}_{t-1},{\mathbf{y}}_t\right)\propto N\left({\mathbf{y}}_t;{\mathbf{m}}_y^{k^i}\left(k\right),{\mathbf{C}}_y^{k^i}\left(k\right)\right)p\left(r_t=k\mid r_{t-1}^{k^i},{\mathit{\theta }}_{t-1}^{k^i}\right)$$ (Propagate) Draw ${\mathbf{x}}_t^i\sim p\left({\mathbf{x}}_t\mid r_t^i,{\mathbf{x}}_{t-1}^{k^i},{\mathit{\theta }}_{t-1}^{k^i},{\mathbf{y}}_t\right)$ from normal distribution $N\left({\mathbf{m}}_x,{\mathbf{C}}_x\right)$ with: $$\begin{array}{ccc}\hfill {\mathbf{m}}_x& =& {\mathbf{C}}_x\left(\mathbf{B}{\left(r_t\right)}^T{\left({\lambda }_t{\mathbf{R}}_s\right)}^{-1}{\mathbf{y}}_t+\mathbf{A}{\mathbf{x}}_{t-1}\right)\hfill \end{array}$$ (29) $$\begin{array}{ccc}\hfill {\mathbf{C}}_x^{-1}& =& \mathbf{B}{\left(r_t\right)}^T{\left({\lambda }_t{\mathbf{R}}_s\right)}^{-1}\mathbf{B}\left(r_t\right)+{\mathbf{I}}_k\hfill \end{array}$$ (30) (Update) ${\mathbf{s}}_t^i=\mathbf{S}\left({\mathbf{s}}_{t-1}^{k^i},r_t^i,r_{t-1}^{k^i},{\mathbf{x}}_t^i,{\mathbf{x}}_{t-1}^{k^i},{\mathbf{y}}_t\right)$. (Update) ${\left({\mathbf{s}}_t^x\right)}^i=\mathbf{K}\left({\left({\mathbf{s}}_{t-1}^x\right)}^{k^i},{\mathbf{s}}_t^i,{\mathbf{y}}_t\right)$ and draw estimated states ${\widehat{\mathbf{x}}}_t^i\sim N\left({\mathbf{m}}_{x1}^i,{\mathbf{C}}_{x1}^i\right)$.

Where the deterministic updating formula $\mathbf{S}$ in step 5 is given by the following:

$$\begin{array}{ccc}\hfill v_t& =& {\phi }^2{v}_{t-1}+1\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill {\mathbf{g}}_t& =& {\mathbf{\Phi }}_{t-1}{\mathbf{x}}_t\hfill \end{array}$$

(32)

$$\begin{array}{ccc}\hfill {\zeta }_t^2& =& {\lambda }_t{\phi }^2+{\mathbf{x}}_t^T{\mathbf{g}}_t\hfill \end{array}$$

(33)

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{e}}}_t& =& {\mathbf{y}}_t-{\mathbf{b}}_{t-1}^T{\mathbf{x}}_t\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill {\mathbf{b}}_t& =& {\mathbf{b}}_{t-1}+{\displaystyle \frac{1}{{\zeta }_t^2}}{\mathbf{g}}_t{\widehat{\mathbf{e}}}_t^T{I}_{r_t=k}\hfill \end{array}$$

(35)

$$\begin{array}{ccc}\hfill {\mathbf{\Phi }}_t& =& {\displaystyle \frac{1}{{\phi }^2}}\left({\mathbf{\Phi }}_{t-1}-{\displaystyle \frac{1}{{\zeta }_t^2}}{\mathbf{g}}_t{\mathbf{g}}_t^T{I}_{r_t=k}\right)\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill {\mathbf{V}}_t& =& {\phi }^2\left({\mathbf{V}}_{t-1}+\sum _{k=1}^K{\displaystyle \frac{1}{{\zeta }_t^2}}{\widehat{\mathbf{e}}}_t{\widehat{\mathbf{e}}}_t^T{I}_{r_t=k}\right)\hfill \end{array}$$

(37)

$$\begin{array}{ccc}\hfill {\mathbf{g}}_t^x& =& {\mathbf{\Psi }}_{t-1}{\mathbf{x}}_{t-1}\hfill \end{array}$$

(38)

$$\begin{array}{ccc}\hfill {\sigma }_t^2& =& {\phi }^2+{\mathbf{x}}_{t-1}^T{\mathbf{g}}_t^x\hfill \end{array}$$

(39)

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{e}}}_t^x& =& {\mathbf{x}}_t-{\mathbf{a}}_{t-1}^T{\mathbf{x}}_{t-1}\hfill \end{array}$$

(40)

$$\begin{array}{ccc}\hfill {\mathbf{a}}_t& =& {\mathbf{a}}_{t-1}+{\displaystyle \frac{1}{{\sigma }_t^2}}{\mathbf{g}}_t^x{\widehat{\mathbf{e}}}_t^{xT}\hfill \end{array}$$

(41)

$$\begin{array}{ccc}\hfill {\mathbf{\Psi }}_t& =& {\displaystyle \frac{1}{{\phi }^2}}\left({\mathbf{\Psi }}_{t-1}-{\displaystyle \frac{1}{{\sigma }_t^2}}{\mathbf{g}}_t^x{\mathbf{g}}_t^{xT}\right)\hfill \end{array}$$

(42)

where $i=1,\dots ,s$. The sampling density in step 4 is derived in Appendix H. As discussed in Appendix G, we can assume each row of transition matrix $\mathbf{T}$, ${\mathbf{p}}_i$, follows a Dirichlet distribution $Dir\left({\mathbf{p}}_i\mid {\mathit{\alpha }}_i\right)$, for $i=1,\dots ,K$, and their sufficient statistics can be updated as follows:

$${\alpha }_{i,j,t}={\alpha }_{i,j,t-1}+I_{r_t=j,r_{t-1}=i},\mathrm{for}i,j=1,\dots ,K$$

(43)

For step 6, following the spirit of (Djuric et al., 2012), we compute parameter values directly from their sufficient statistics rather than sampling them. Compared with the particle learning method of (Carvalho et al., 2010), which samples parameter values from their posterior distributions, our analytical solution reduces Monte Carlo variance and improves computational efficiency without sacrificing model fidelity. Specifically, the parameter updates are as follows:

$$\begin{array}{ccc}\hfill \mathbf{B}\left(r_t\right)& =& {\mathbf{b}}_{k,t}\hfill \end{array}$$

(44)

$$\begin{array}{ccc}\hfill {\mathbf{R}}_s& =& {\mathbf{V}}_t/\left(v_t-2\right)\hfill \end{array}$$

(45)

$$\begin{array}{ccc}\hfill \mathbf{A}& =& {\mathbf{a}}_t\hfill \end{array}$$

(46)

$$\begin{array}{ccc}\hfill {\mathbf{T}}_{ij}& =& {\displaystyle \frac{{\alpha }_{i,j,t}-1}{{\alpha }_{0,t}-K}}\hfill \end{array}$$

(47)

Here, we use the posterior mode (MAP) as a transition probability estimate. In this step, $\mathbf{K}$ denotes the Kalman filter update, and the detailed recursion is presented in Algorithm A1 of Appendix B. We also extend Carvalho’s approach by incorporating a mixture of Gaussian innovations, which provides a closer alignment with the heavy-tailed and non-Gaussian features observed in empirical financial data.

3.3.2. Simulation Analysis

To validate our particle filtering estimation algorithm, we apply it to a controlled synthetic data environment. We generate synthetic scenarios based on a three-regime dynamic factor model involving 15 synthetic instruments. This setup allows us to test the algorithm’s ability to recover latent states and parameters under regime-switching dynamics in a controlled setting.

The “true” model parameters are specified as follows:

Factor loading matrices ($\mathbf{B}\left(r_t\right)$):

$$\mathbf{B}\left(r_t=1\right)=\left[\begin{array}{ccc}0.93& 0.316& 0.184\\ 0.205& 0.568& 0.596\\ ⋮& ⋮& ⋮\\ 0.81& 0.096& 0.219\end{array}\right]$$

$$\mathbf{B}\left(r_t=2\right)=\left[\begin{array}{ccc}-0.56& 1.011& 0.945\\ 2.821& -1.165& 1.486\\ ⋮& ⋮& ⋮\\ 0.152& 2.381& -0.153\end{array}\right]$$

$$\mathbf{B}\left(r_t=3\right)=\left[\begin{array}{ccc}-0.694& -0.654& -0.443\\ -0.15& -2.678& -0.268\\ ⋮& ⋮& ⋮\\ -0.917& -0.849& -0.811\end{array}\right]$$

The observation covariance matrices ($\mathbf{R}$):

$$\mathrm{Diag}\mathbf{R}=\left[\begin{array}{ccccc}0.49& 0.211& \dots & 0.518& 0.488\end{array}\right]$$

The state transition matrix ($\mathbf{A}$):

$$\mathbf{A}=\left[\begin{array}{ccc}0.553& 0& 0\\ 0& 0.477& 0\\ 0& 0& 0.312\end{array}\right]$$

and the regime transition matrix $\mathbf{T}$ is as follows:

$$\mathbf{T}=\left[\begin{array}{ccc}0.995& 0.0025& 0.0025\\ 0.0025& 0.995& 0.0025\\ 0.0025& 0.0025& 0.995\end{array}\right]$$

Since factor loadings and factor returns are not separately identifiable within our model structure, our validation analysis focuses on three key aspects:

• Regime detection: assessing whether the algorithm can correctly identify latent regime shifts (see Figure 1);
• Idiosyncratic risk estimation: evaluating the accuracy of instrument-specific variance estimates (see Figure 2);
• Cumulative expected returns’ tracking: examining how well the estimated expected returns align with the “true” cumulative return over time (see Figure 3).

To initialize the particle filtering algorithm, we begin with non-informative priors. For example, the mean of the factor loading matrix is set to a zero rectangular matrix under each regime

$${\mathbf{b}}_0\left(r_0\right)=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ ⋮& ⋮& ⋮\\ 0& 0& 0\end{array}\right],r_0\in \left\{0,1,2\right\}.$$

The prior left covariance of the loading matrix is initialized with high dispersion, specifically a diagonal matrix whose entries are set to 10, to ensure weak prior information and allow the data to drive the estimation. The same applies to the prior distribution chosen for the factor transition matrix $\mathbf{A}$, and the idiosyncratic risk distribution, ${\mathbf{R}}^{-1}\sim W_s\left(V_0^{-1},v_0\right)$. In each case, we begin with non-informative means and large prior dispersion, thereby minimizing prior influence and ensuring that the posterior estimates are predominantly shaped by the data.

As the following results demonstrate, even when initialized with non-informative priors, our algorithm converges accurately to the “true” parameters, thereby validating the effectiveness of Algorithm 5.

As shown in Figure 1, the algorithm accurately captures the evolution of hidden regimes from both regime detection plot and cumulative regimes comparison plot.

In terms of idiosyncratic risk (or error variance), the algorithm also converges to their unbiased estimates, as illustrated in Figure 2. The blue line represents the mean of the particle estimates, while the green lines denote the 25th and 75th quantiles.

The most critical metric in statistical factor analysis is the model’s ability to accurately track expected returns. In our synthetic experiments, the proposed model exhibits strong performance in the cumulative expected return, as illustrated in Figure 3.

4. Results in Statistical Arbitrage Strategy

In both academic and industry, equity statistical arbitrage strategy (equity stat-arb) usually means the idea of trading groups of stocks against either explicit or synthetic factors, which can be seen as the generalization of “pairs-trading” (see Avellaneda & Lee, 2010). In some cases, we would long an individual stock and short the factors, and in others we would short the stock and long the factors. Overall, due to the netting of long and short positions, our exposure to the factors will be small. One key step of the equity stat-arb strategy is the decomposition of assets’ returns into the systematic part and idiosyncratic part. When we use a more effective factor model, we can have a better decomposition and hence a better strategy performance.

4.1. Market Neutral Strategy

In this paper, we evaluate the quality of factor models within the context of an equity statistical arbitrage strategy. Inspired by the approach of (Khandani & Lo, 2008), in order to avoid the unnecessary complexity of signal generation and potential overfitting issues, we adopt a deliberately simple reversion signal which is the negative value of previous day’s residuals. To ensure market neutrality, we construct a dollar-neutral portfolio each day, with equal dollar amounts allocated to long and short positions.

To be specific, we compare the performance of trading strategies derived from both the regime-switching dynamic factor model and static MLE factor model. For estimating the regime-switching dynamic factor model, we employ Algorithm 5. The initialization follows the same procedure as in the simulation analysis (Section 3.3.2), with one refinement: to improve initial convergence, we use the estimates from a mixture of factor analyzer (MFA), trained on a “burn-in” sample from 3 January 2005 to 1 January 2011, as the initial mean of the factor loadings, ${\mathbf{b}}_0\left(r_0\right)$. Furthermore, the residualization step is implemented in a rolling window framework so that the residuals are always generated out-of-sample. For the estimation of the static MLE factor model, we employed Algorithm 3 as described in Section 3.1. This serves as our baseline approach, providing a standard maximum likelihood estimation of factor loadings and covariances under the assumption of a single, time-invariant regime.

For computational efficiency, we designed a relatively lean experiment by restricting the dataset to the daily prices of the Dow Jones 30 constituents. Since factor models do not require the same scale-invariance adjustments as PCA-based approaches, we use raw returns rather than normalized return for the estimation. In this setting, we employ a three-factor model and assume that the factor loadings evolve according to a two-regime Markov chain. For each experiment, we use rolling window walk forward time series cross-validation framework to run our backtesting. The training data sample size is 120 days (around 6 months) and the test data sample size is 5 days (around one week). The objective of this experiment is not to identify or interpret the economic meaning of market regimes. Instead, our goal is to demonstrate that when a regime shift occurs, regardless of its underlying cause, our system adapts more rapidly to the new distribution (or data-generating process). This adaptability enables it to deliver a superior risk model for statistical arbitrage strategies.

After the residualization step, we can generate our signals using a naive rule-based approach. If the previous day’s residual for asset i has $v_{i,t-1}>d$, where d is the threshold, then we will say that the asset i is over-priced at time $t-1$ and short this asset for the next time stamp t. For $v_{i,t-1}<-d$, we will perform the opposite operation. For simplicity, in our experiment, we set $d=0$. At each time step, we form a dollar-neutral portfolio by applying the mean-variance closed form solution to both the long and short sides:

$$w^{*}=\lambda {\Sigma }^{-1}\mu$$

(48)

where $\Sigma$ is the covariance matrix of returns, $\mu$ represents the rule-based prediction, and $\lambda$ is a scaling factor that controls overall risk.

4.2. Performance Comparison

We run this naive strategy using the MLE static factor model, the dynamic factor model without regime switching mechanism, and the regime switching dynamic factor model, respectively, and the strategy performance is as shown in Figure 4:

We can see from Figure 4 that the strategy achieves a higher Sharpe ratio and a lower maximum drawdown when using the regime-switching dynamic factor model compared to both the static MLE factor model and the simple dynamic factor model. Specifically, the improvement in Sharpe is approximately 63%, while the reduction in maximum drawdown is about 12%. We also observe that the simple dynamic factor model performs well, telling us the importance of modeling the dynamics of factor returns in equity arbitrage strategies. However, incorporating the regime-switching mechanism provides an additional advantage: it meaningfully reduces drawdowns during market regime shifts (most notably during the COVID period) and improves estimation accuracy in the aftermath of such events. Based on these findings, we conclude that dynamic factor models consistently outperform static factor models, and that incorporating a regime-switching mechanism enables the model to adapt much more quickly to abrupt regime shifts, such as those observed during COVID, than conventional approaches.

The associated historical effective sample size plot and regime detection plot are given in Figure 5:

In the above graphs, the effective sample size (ESS) is used to measure the stability of particle filtering algorithm in real data, and defined as:

$$\mathrm{ESS}={\displaystyle \frac{1}{{\sum }_{i=1}^N{\left(w_t^{\left(i\right)}\right)}^2}}$$

where $w_t^{\left(i\right)}$ are normalized weights.

From the ESS graph, we observe that the particle filter estimation is stable, as most of the time the ESS is larger than 2000 for a three-factor model. From the regimes detection, we can see there are more regime changes in the 2020 COVID period and early 2024. Indeed our regime switching model performs better during those periods compared with the simple dynamic factor model and static factor model.

It is also worth noting that although our model does not assume the factor return process is stationary, it is also informative to check the estimated transition matrix:

$$\widehat{A}=\left[\begin{array}{ccc}-0.00117863& 0.02577551& 0.03975952\\ -0.07506055& -0.04159962& 0.02756931\\ 0.01579488& 0.07781727& 0.01877453\end{array}\right]$$

The modulus of its eigenvalues are: 0.074, 0.046, 0.046. Since $|{\lambda }_i\left(A\right)|<1\mathrm{for}\mathrm{all}{\lambda }_i$, the implied factor dynamics are stationary.

Although determining the number of factors and regimes is beyond the scope of this work, it is still worthwhile to examine results under varying specifications. By checking different numbers of factors and regimes, we can assess whether our earlier findings remain robust across alternative setups. The relevant results are summarized in the

Table 1. Sharpe ratio across alternative setups.

	1 Factor	2 Factors	3 Factors	4 Factors
static factor model	0.6011	0.4524	0.5126	0.5710
dynamic factor model	0.5966	0.7080	0.6546	0.5869
2 regime factor model	0.8312	0.7453	0.8374	0.9512
3 regime factor model	0.9184	0.8102	0.8617	0.8266

and

Table 2. Maximum drawdown across alternative setups.

	1 Factor	2 Factors	3 Factors	4 Factors
static factor model	−8.59%	−9.88%	−7.46%	−7.52%
dynamic factor model	−13.76%	−12.47%	−12.16%	−12.04%
2 regime factor model	−7.62%	−7.52%	−6.57%	−7.33%
3 regime factor model	−7.93%	−7.15%	−8.19%	−9.37%

as follows:

From the results above, we observe that our findings remain consistent across varying specifications of factor numbers and regime counts. This robustness test provides further confidence that improvements from regime-switching dynamic factor model are not sensitive to any specific parameter choices.

5. Discussion

This paper investigates the estimation of a regime-switching dynamic factor model using a particle learning algorithm. Through simulation studies, we validate our estimation approach by evaluating regime detection accuracy, idiosyncratic risk estimation, and the model’s ability to track expected returns. The empirical study in the equity statistical arbitrage framework further demonstrates that the regime-switching dynamic factor model outperforms a conventional static MLE factor model in capturing underlying data dynamics.

The particle learning algorithm implemented in this paper builds on the framework of (Carvalho et al., 2010), with several modifications. First, we extend the innovation distribution to a mixture of Gaussian distributions. Second, we simplify the parameter learning step by only tracking the sufficient statistics, rather than using the marginalization via simulation. Lastly, we apply the estimated model to the empirical data and compare its performance against the conventional MLE factor model.

We are able to see the performance improvements in the equity statistical arbitrage strategy when using our model. Our model not only improves the Sharpe ratio but also minimizes the maximum drawdown. These results align with our initial motivation: It is well known that financial market operates under multiple regimes, and incorporating the regime switching mechanism that is governed by the hidden Markov model in our model representation allows us better model the structural shifts and enhances risk management—particularly during periods of elevated volatility and contagion risk.

While the current particle learning algorithm is based on the vanilla auxiliary particle filter, there remains a lot of room to improve the quality of our particle filtering algorithm and hence improve the tracking of hidden states (or hidden factors in this model). For instance, the ABC-based sequential Monte Carlo filter (Jasra et al., 2010) could further enhance the filtering accuracy. Additionally, incorporating more robust, heavy-tailed innovation distributions (Schoutens, 2005) may improve the robustness of our estimation, particularly in the presence of extreme market movements. Investigating more sophisticated specifications for the hidden factor process, including nonstationary factor models, would also be a promising direction for future research (Peña & Poncela, 2004, 2006).