Semiparametric Estimation and Application of Realized GARCH Model with Time-Varying Leverage Effect

Abstract

To describe the stylized features of volatility comprehensively, this paper embeds the time-varying leverage effect of volatility into the Realized Generalized AutoRegressive Conditional Heteroskedasticity (RG) model and proposes a new volatility model with a time-varying leverage effect. The Quasi-Maximum Likelihood-Kernel (QML-K) method is proposed to approximate the density function of returns and to estimate the parameters in the new model. Under some mild regularity conditions, the asymptotic properties of the resulting estimators are achieved. Simulation studies demonstrate that the proposed model yields better performances than traditional RG models under different situations. Finally, the empirical analysis shows better finite sample performance of the estimation method and the new model on real data compared with existing methods.

1. Introduction

The financial market is constantly changing with both returns and risks due to the existence of various unstable factors. In this context, volatility has become one of the most important tools to evaluate the risks caused by these factors. Therefore, volatility plays a crucial role in the research of financial markets and attracts widespread attention from many scholars. Thus, much literature has developed various time series models to measure volatility. One of the popular existing research areas in volatility modeling is the GARCH model proposed by Bollerslev [1]. However, the traditional GARCH family models were based on only low-frequency data, which means the models are not well suited for situations where volatility rapidly shifts to a new level. As Andersen et al. [2] pointed out, this is because a GARCH model is slow at ’catching up,’ and it may require more periods for the conditional variance (implied by the GARCH model) to attain its new level.

It is worth mentioning the Realized GARCH (RG) model proposed by Hansen et al. [3], which can more effectively capture market changes by utilizing high-frequency data. Numerous studies have extensively investigated the RG model and yielded significant results. Gerlach and Wang [4] proposed the realized range, which is observed at a 5 min frequency, as the realized measure, thus improving the out-of-sample predictive likelihood and the forecasting performance of Value at Risk (VaR) and Expected Shortfall (ES). Wang et al. [5] considered the two-sided Weibull distribution in tail risk forecasting for index and asset returns and confirmed the proposed model had better performance than the existing models. A recent study considers the use of the Itô process in the GARCH model to deal with high-frequency financial data, such as Song et al. [6], Fu et al. [7] and Fu et al. [8].

In addition, the RG model takes the existence of the leverage effect in the financial market into account. Black [9] pointed out that based on financial market information, the same degrees of positive and negative news lead to varying degrees of ups and downs in a company’s assets, which may affect stock prices and cause asymmetric fluctuations. This phenomenon is called the leverage effect. Cheung and NG [10], Poon and Taylor [11], Koutmos [12], etc., have conducted sufficient research on this effect and confirmed that it widely presents in mature financial markets. The applications of the leverage effect in high-frequency data have also been made. Wang and Mykland [13] explored the estimation method of the leverage effect in high-frequency data; Aït-Sahalia, Fan, Laeven et al. [14] focused on the leverage effect and proposed a kind of nonparametric estimation for stochastic leverage measures using high-frequency data; Chong and Todorov [15] proposed model-free (nonparametric) estimators of the volatility of volatility and leverage effect using high-frequency observations of short-dated options. The above literuare shows that the leverage effect plays an important role in volatility research. Considering this effect, Hansen et al. [3] introduced a leverage function in the RG model to better capture the leverage effect.

However, since the above studies only considered the constant leverage effect, few of them have discussed the time-varying leverage effect. Existing research showed that the time-varying leverage effect actually exists. Yu [16] developed a nonparametric approach to estimating leverage, which depends on jumps in stochastic volatility models, and found economically significant time-varying leverage that is more negative at higher variance levels. Wu and Wang [17] proposed the Realized Stochastic Volatility model with a Time-Varying Leverage effect (RSV-TVL) and demonstrated the effect of their model with empirical applications. Catania and Proietti [18] proposed a bivariate model that captures key characteristics of realized volatility and asset returns, including long-term memory, heavy-tailedness, and negative dependence, and demonstrated its out-of-sample forecasting performance outperforms the benchmark HAR-GARCH model.

Although GARCH-family models are widely used, the stochastic volatility (SV) framework provides distinct advantages, such as modeling latent volatility dynamics. The SV model conceptualizes volatility as an unobservable latent process driven by stochastic shocks, enabling it to capture complex market dynamics more effectively than purely deterministic specifications like GARCH. Unlike GARCH models that condition volatility solely on historical information, SV models incorporate exogenous stochastic innovations, enhancing their flexibility in describing financial time series. However, the latent nature of volatility in SV specifications introduces significant estimation challenges, particularly regarding state variable identification and parameter inference. Despite these computational complexities, SV models remain indispensable in modern financial econometrics due to their theoretical coherence and empirical accuracy. The SV framework has inspired extensive methodological developments: Tauchen and Pitts [19] and Taylor [20] pioneered the application of stochastic principles to financial volatility modeling; Chib, Nardari, and Shephard [21] advanced Bayesian estimation techniques for high-dimensional multivariate SV models with time-varying correlations; Jensen and Maheu [22] introduced a semiparametric Bayesian approach incorporating Markov chain Monte Carlo methods to address distributional uncertainty; Fernández-Villaverde, Guerrón-Quintana, and Rubio-Ramírez [23] developed computationally efficient particle filtering algorithms tailored for large-scale SV models. Recent innovations continue to expand the SV paradigm, as evidenced by contributions from Rømer [24], Yazdani, Hadizadeh, and Fakoor [25], Bolko, Christensen, Pakkanen et al. [26], and Chan [27], among others. Notwithstanding these advancements, SV models remain computationally intensive, particularly for parameter estimation and short-term forecasting. This computational burden renders them better suited for long-term dynamic volatility analysis than high-frequency applications. Given our focus on high-frequency trading risk management, we prioritize GARCH-family models for their tractability, closed-form estimation, and proven efficacy in operational settings.

Unlike SV models that require computationally intensive latent variable estimation, our proposed GARCH-family model is more easily calculated, making it more suitable for high-frequency risk management. In this paper, we introduce a time-varying leverage function with a linear spline representation to the RG model, thus constructing the Realized GARCH model with the Time-Varying Leverage effect (RG-TVL) model. Different from the leverage function in the RG model, the new function is sufficient to reflect the time-varying leverage effect. Besides, due to the complexity of the market data, the error terms usually do not follow a normal distribution but a kind of heavy tail distribution. Thus, the traditional QML method is not effective enough. A QML estimation method based on the Kernel density estimator (QML-K) is used to address this issue. The consistency and asymptotic normality of QML-K estimators are proved in this paper. Simulation studies and applications indicate that (1) The RG-TVL model has better performance in capturing the time-varying leverage effect than the RG model; (2) The QML-K method is more robust than the QML for the finite sample estimation. Thus, our paper has important theoretical significance and application value for financial risk prediction and macroeconomic market regulation.

The rest of this paper is organized as follows. In Section 2, we introduce the structure of the RG model with the time-varying effect. In Section 3, a semiparametric method is described. The convergence properties of the method and the related optimization algorithm are also proposed. In Section 4, we perform some simulation studies. In Section 5, empirical data are analyzed. In Section 6, we provide a conclusion and some comments.

2. Realized GARCH with Time-Varying Leverage Effect

2.1. The Model Construction

Hansen et al. [3] introduced the RG model, a novel framework that integrates a GARCH structure for returns with a joint modeling approach for realized volatility measures. By effectively incorporating high-frequency data, the model demonstrates superior performance compared with conventional low-frequency time series models. Moreover, the leverage function embedded in the model effectively captures the asymmetric volatility response observed in financial markets, thereby enhancing its ability to model the leverage effect.

The mathematical form of the RG($p,q$) model is as follows:

$$\begin{array}{cc}\hfill r_t=& \mu +h_t^{1/2}{z}_t,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill log{h}_t=& \alpha +\sum _{i=1}^p{\beta }_{i}log{h}_{t-i}+\sum _{j=1}^q{\gamma }_{j}log{x}_{t-j},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill log{x}_t=& \xi +\varphi log{h}_t+\tau \left(z_t\right)+u_t,\hfill \end{array}$$

(3)

where $r_t$ denotes the logarithmic return at time t, and $\mu$ and $h_t$ represent the conditional expectation and variance of $r_t$, respectively. The leverage effect is captured by the function $\tau \left(z_t\right)$, where $z_t$ is the error term of the logarithmic return. Additionally, $x_t$ stands for the realized measure. In the RG($p,q$) model, the leverage function takes the form $\tau \left(z_t\right)={\tau }_1{z}_t+{\tau }_2(z_t^2-1)$. Finally, $u_t$ corresponds to the error term of the realized measure. Furthermore, $z_t\sim i.i.d.(0,1)$ and $u_t\sim i.i.d.N(0,{\sigma }_u^2)$. In order to better consider the property, we assume a mild condition that $z_t$ follows a symmetric distribution. In addition, we require $-1<{\sum }_{i=1}^p{\beta }_i+\varphi {\sum }_{j=1}^q{\gamma }_j<1$ to ensure the stability of the model.

However, a global linear relationship between $\tau \left(z_t\right)$ and $z_t$ in the RG model may be too strict to describe the time variety of the leverage effect. To solve this problem, inspired by Yu [16], we insert a linear spline leverage function into the measure equation. The expression of the linear spline leverage function is as follows:

$$\begin{array}{c}\hfill \tau \left(z_t\right)={\tau }_1{z}_t+\sum _{i=1}^n{\tau }_{2i}(z_t^2-1)I_{(b_{i-1}\le z_t<b_i)},\end{array}$$

(4)

where $(b_0,b_1,\dots ,b_n)$ is a monotonically increasing sequence, n represents the number of intervals and there are $n+1$ nodes. I is the indicative function. Therefore, the leverage function discussed in this article is a generalization of Hansen et al. [3]. The form of RG-TVL($p,q$) model consists of Equations (1)–(4).

Remark 1.

In the proposed model, we need to determine n and $b_i$s. Yu [16] suggested that n needs to increase with the growth of the sample size; however, he also pointed out that the larger n indicates more parameters in the model and hence leads to higher computational cost. Essentially, increasing n results in a tradeoff where a smaller bias is achieved at the cost of a larger variance. The reason is that fewer effective observations are utilized to estimate ${\tau }_{2i}$. To control the computational costs, we assume $n=2$ for the model construction and asymptotic properties. The form of the leverage function is as follows:

$$\begin{array}{c}\hfill \tau \left(z_t\right)={\tau }_1{z}_t+{\tau }_{21}(z_t^2-1)I_{(z_t\le 0)}+{\tau }_{22}(z_t^2-1)I_{(z_t>0)}.\end{array}$$

(5)

The situation of $n=3$ has also been considered in the simulation study, which has no significant advantage compared. Since this does not affect the analysis of the estimators’ properties, we take $n=2$ as an example in the asymptotic properties and Appendix A sections.

According to Equation (1), since $h_t^{1/2}>0$, if $r_t-\mu \le 0$, then $z_t\le 0$, the leverage function in Equation (5) is $\tau \left(z_t\right)={\tau }_1{z}_t+{\tau }_{12}(z_t^2-1)$; otherwise, $z_t>0$, the leverage function is $\tau \left(z_t\right)={\tau }_1{z}_t+{\tau }_{22}(z_t^2-1)$. When ${\tau }_{21}\ne {\tau }_{22}$, the values of the leverage function will be different, which reflects the time variety of the leverage effect. Under the assumptions of Hansen et al. [3], $E\left[\tau \left(z_t\right)\right]=0$ holds, which also holds in our leverage function to make sure the equation is stable. Compared with the RG model, the RG-TVL model fully considers the different values that may exist in the coefficients of the leverage function under two situations. In addition, if ${\tau }_{21}={\tau }_{22}$, the model degenerates into an RG model, where the leverage effect parameters are the same across different periods and cannot reflect the time-variety nature of the leverage effect. When the realized measure is replaced by the lag volatility term, the model degenerates into a GARCH model, which ignores high-frequency data and the leverage effect.

2.2. Strict Stationarity

After proposing the model, we then discuss the sufficient and necessary conditions for the existence of a strict stationary solution of the RG-TVL ($p,q$) model. Considering an ARMA representation as follows:

$$\begin{array}{c}\hfill log{h}_t={\alpha }_1+\sum _{i=1}^p{\beta }_{i}log{h}_{t-i}+\varphi \sum _{j=1}^q{\gamma }_{j}log{h}_{t-j}+\sum _{j=1}^q{\gamma }_j{\eta }_{t-j},\end{array}$$

(6)

where ${\alpha }_1=\alpha +\xi {\sum }_{j=1}^q{\gamma }_j$ and ${\eta }_t=\tau \left(z_t\right)+u_t$. In Equation (6), the second and the third terms represent the AR terms, and the fourth term represents the MA term. Furthermore, when applying the ARMA representation, the absolute value of unit roots of AR terms should be less than 1 to ensure the stationarity of the model.

Denote $\mathit{\theta }$ as the parameter space of $\mathit{\theta }={({\mathit{\theta }}_1^{\prime },{\mathit{\theta }}_2^{\prime },{\sigma }_u^2)}^{\prime }$, where ${\mathit{\theta }}_1={(\alpha ,{\beta }_1,\dots ,{\beta }_p,{\gamma }_1,\dots ,{\gamma }_q)}^{\prime }$ and ${\mathit{\theta }}_2={(\xi ,\varphi ,{\tau }_1,{\tau }_{21},{\tau }_{22})}^{\prime }$. Without loss of generality, we assume $\Theta ={\Theta }_1\times {\Theta }_2\times {\Theta }_3$, where ${\Theta }_1,{\Theta }_2,{\Theta }_3$ represent the space of ${\mathit{\theta }}_1,{\mathit{\theta }}_2,{\sigma }_u^2$, respectively. For the convenience of analysis, let $m=p\wedge q,{\mathcal{A}}_{{\mathit{\theta }}_1}\left(s\right)={\sum }_{j=1}^q{\gamma }_j{s}^j$ and ${\mathcal{B}}_{\mathit{\theta }}\left(s\right)=1-{\sum }_{i=1}^m({\beta }_i+\varphi {\gamma }_i)s^i$. Then Equation (6) can be written with backshift operator notation as follows:

$$\begin{array}{c}\hfill {\mathcal{B}}_{\mathbf{\theta }}\left(B\right)log{h}_t={\alpha }_1+{\mathcal{A}}_{{\mathit{\theta }}_1}\left(B\right){\eta }_t,\end{array}$$

where B is the backshift operator. Thus, we have the following propositions to ensure the strict stationarity of RG-TVL ($p,q$) model, which follows from the discussion of the classical ARMA ($p,q$) time series Brockwell and Davis [28].

Proposition 1.

If ${\mathcal{A}}_{{\mathit{\theta }}_1}\left(s\right)$ and ${\mathcal{B}}_{\mathit{\theta }}\left(s\right)$ have no common roots, then the stationary solution $log{h}_t$ exists if and only if ${\mathcal{B}}_{\mathit{\theta }}\left(s\right)\ne 0$ for all $s\in \mathbb{C}$ such that $\left|s\right|\le 1$.

Then, let ${\mathcal{C}}_{{\mathit{\theta }}_1}\left(s\right)=1-{\sum }_{i=1}^p{\beta }_i{s}^i$. Thus, Equation (2) can be rewritten as

$$\begin{array}{c}\hfill {\mathcal{C}}_{{\mathbf{\theta }}_1}\left(B\right)log{h}_t=\alpha +{\mathcal{A}}_{{\mathit{\theta }}_1}\left(B\right)log\left(x_t\right)\end{array}$$

Proposition 2.

Denote ${log}^{+}\left(s\right)$ as maximum$(log(s),0)$. Suppose $log\left(x_t\right)$ is a stationary and ergodic process with $E[{log}^{+}\left(\right|log\left(x_t\right)\left|\right)]<\infty$. Meanwhile, assume ${\mathcal{A}}_{{\mathit{\theta }}_1}\left(x\right)$ and ${\mathcal{C}}_{{\mathit{\theta }}_1}\left(x\right)$ have no common roots. If ${\mathcal{C}}_{{\mathit{\theta }}_1}\left(x\right)\ne 0$ for all $x\in \mathbb{C}$ such that $\left|x\right|\le 1$, then the solution $log{h}_t$ of the RG-TVL ($p,q$) model has a linear representation in terms of $log\left(x_t\right)$.

Remark 2.

The condition $E[{log}^{+}\left(\right|log{x}_t\left|\right)]<\infty$ is a very weak condition. By Lemma 2.5.3 in Straumann [29], $E|log{x}_t{|}^d<\infty$ for some $d>0$ indicates that $E[{log}^{+}\left(\right|log{x}_t\left|\right)]<\infty$.

With the argument above, we can make sure that $log{h}_t$ is stationary and ergodic. Since ${\eta }_t$ are i.i.d. random variables, $\left\{(r_t,log{x}_t)\right\}$ are stationary and ergodic. This conclusion will be important in the estimation of the RG-TVL ($p,q$) model.

3. Semiparametric Estimation

Quasi-Maximum Likelihood (QML) estimation has been widely applied in fields of spatial econometrics, dynamic panel data models, nonlinear models, etc. The main idea is to construct an approximate likelihood function, which can still obtain a consistent and asymptotically normal estimator in the case of mild conditions. Pfaff [30] considered the quick consistency of the QML estimator and proposed its exponential convergence rate under general conditions; Lee and Hansen [31] applied the QML method to the GARCH(1,1) model, thus constructing the consistency and asymptotic normality; Since then, the QML method has been utilized in various econometric models; see Komunjer I [32], Lee L. F. [33], Fan J., Qi L., and Xiu D. [34], etc.

However, an important issue for estimation is that the distribution of $z_t$ is unknown in the RG-TVL model. The QML method pretends a standard normal distribution for the solution, which might lead to a certain degree of deviation. Fang and Han [35] and Wang et al. [36] performed a series of experiments that confirmed that inserting the Kernel density estimator in the QML method can effectively improve estimation performance. Referring to Wang et al. [36]’s method, we apply the QML-K method in our model.

3.1. QML-K Estimation

With the RG-TVL model, the likelihood function is as follows:

$$L_T\left(\mathit{\theta }\right|\mathbf{r},\mathbf{x})={\displaystyle \frac{1}{T}}\sum _{t=1}^T{l}_t\left(\mathit{\theta }\right),$$

where T is the sample size of the data, $\mathbf{r}$ and $\mathbf{x}$ represent the sequences of $r_t$ and $x_t$, respectively; $l_t\left(\mathit{\theta }\right)$ is as follows:

$$\begin{array}{c}\hfill l_t\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{2}}\left[-2log\left(f_z\left(z_t\right)\right)+log{h}_t+log\left({\sigma }_u^2\right)+{\displaystyle \frac{u_t^2}{{\sigma }_u^2}}\right],\end{array}$$

where $f_z\left(z\right)$ is the probability density function of $z_t$.

We apply the Quasi-Maximum Likelihood estimation to gain the estimators. Denote the true parameter as ${\mathit{\theta }}_0={({\mathit{\theta }}_{1,0}^{\prime },{\mathit{\theta }}_{2,0}^{\prime },{\sigma }_{u,0}^2)}^{\prime }$. Rewrtie the variance equation, replace $log\left(h_t\right)$ by $log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)$ and we have

$$\begin{array}{c}\hfill log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)=\alpha +\sum _{i=1}^p{\beta }_{i}log{\tilde{h}}_{t-i}\left({\mathit{\theta }}_1\right)+\sum _{j=1}^q{\gamma }_{j}log{x}_{t-j}.\end{array}$$

Here the function $log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)=log{h}_t$ when ${\mathit{\theta }}_1={\mathit{\theta }}_{1,0}$. Since the recursive way may not well define $log{h}_t\left({\mathit{\theta }}_1\right)$, the following is assumed:

Assumption 1.

${\mathcal{A}}_{{\mathit{\theta }}_1}\left(s\right)$ and ${\mathcal{C}}_{{\mathit{\theta }}_1}\left(s\right)$ do not have any common roots, and $|{\mathcal{C}}_{{\mathit{\theta }}_1}\left(s\right)|=0$ as $\left|s\right|>1$, for any ${\mathit{\theta }}_1\in {\Theta }_1$.

If $log{x}_t$ is stationary and ergodic with $E[{log}^{+}\left(\right|log\left(x_t\right)\left|\right)]<\infty$ and Assumption 1 holds, then we can define

$$\begin{array}{c}\hfill log{\tilde{h}}_t\left({\mathit{\theta }}_1\right):={\mathcal{C}}_{{\mathit{\theta }}_1}^{-1}\left(B\right)\left(\alpha +{\mathcal{A}}_{{\mathit{\theta }}_1}log{x}_t\right),{\mathit{\theta }}_1\in {\Theta }_1.\end{array}$$

(7)

Similarly, we can define ${\tilde{u}}_t({\mathit{\theta }}_1,{\mathit{\theta }}_2)$ as follows:

$$\begin{array}{c}\hfill {\tilde{u}}_t({\mathit{\theta }}_1,{\mathit{\theta }}_2)=log{x}_t-\xi -\varphi log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)-\tau \left({\displaystyle \frac{r_t-\mu }{{\tilde{h}}_t^{1/2}\left({\mathit{\theta }}_1\right)}}\right),\end{array}$$

and ${\tilde{u}}_t({\mathit{\theta }}_{1,0},{\mathit{\theta }}_{2,0})=u_t$. $log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)$ and $\tilde{u}({\mathit{\theta }}_1,{\mathit{\theta }}_2)$ will be used in the likelihood function. Replacing $h_t$ and $u_t$ by ${\tilde{h}}_t\left({\mathit{\theta }}_1\right)$ and ${\tilde{u}}_t({\mathit{\theta }}_1,{\mathit{\theta }}_2)$, we can obtain

$$\begin{array}{c}\hfill {\tilde{L}}_T\left(\mathit{\theta }\right|\mathbf{r},\mathbf{x})={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\tilde{l}}_t\left(\mathit{\theta }\right),\end{array}$$

where

$$\begin{array}{c}\hfill {\tilde{l}}_t\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{2}}\left[-2log\left(f_z\left(z_t\right)\right)+log{\tilde{h}}_t\left({\mathit{\theta }}_1\right)+log\left({\sigma }_u^2\right)+{\displaystyle \frac{{\tilde{u}}_t{({\mathit{\theta }}_1,{\mathit{\theta }}_2)}^2}{{\sigma }_u^2}}\right].\end{array}$$

Note that ${\tilde{h}}_t\left({\mathit{\theta }}_1\right)$ and $u_t$ will be used in the likelihood function. We apply an alternating iterative approach to estimate the two sequences. $log{\widehat{h}}_t$ can be recursively defined as follows:

$$\begin{array}{c}\hfill log{\widehat{h}}_t\left({\mathit{\theta }}_1\right)=\left\{\begin{array}{c}log{\widehat{h}}_0,t=0\hfill \\ \alpha +\sum _{i=1}^q{\beta }_{i}log{\widehat{h}}_{t-i}\left({\mathit{\theta }}_1\right)+\sum _{j=1}^q\gamma log{x}_{t-j},t>0\hfill \end{array}\right.,\end{array}$$

(8)

Thus, ${\widehat{u}}_t$ can be defined as follows:

$$\begin{array}{c}\hfill {\widehat{u}}_t({\mathit{\theta }}_1,{\mathit{\theta }}_2)=log{x}_t-\xi -\varphi log{\widehat{h}}_t\left({\mathit{\theta }}_1\right)-\tau \left({\displaystyle \frac{r_t-\mu }{{\widehat{h}}_t^{1/2}\left({\mathit{\theta }}_1\right)}}\right).\end{array}$$

Based on the conditional distribution of $r_t$ and $x_t$, the logarithmic likelihood function of the model can be described as

$$\begin{array}{c}\hfill {\check{L}}_T\left(\mathit{\theta }\right|\mathbf{r},\mathbf{x})={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\check{l}}_t\left(\mathit{\theta }\right),\end{array}$$

where

$$\begin{array}{c}\hfill {\check{l}}_t\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{2}}\left[-2log\left(f_z\left(z_t\right)\right)+log{\widehat{h}}_t\left({\mathit{\theta }}_1\right)+log\left({\sigma }_u^2\right)+{\displaystyle \frac{{\widehat{u}}_t{({\mathit{\theta }}_1,{\mathit{\theta }}_2)}^2}{{\sigma }_u^2}}\right].\end{array}$$

Since the distribution of $z_t$ is unknown, the probability density function of $z_t$ needs to be estimated. We apply the Kernel density method to estimate the probability density function of $z_t$. The form of the estimator is as follows:

$$\begin{array}{c}\hfill {\widehat{f}}_{z,T}\left(z_t\right)={\displaystyle \frac{1}{T{b}_T}}\sum _{i=1}^{T}K\left({\displaystyle \frac{z_t-{\widehat{z}}_i}{b_T}}\right),\end{array}$$

where ${\widehat{f}}_{z,T}\left(z_t\right)$ is the estimator of $f_z\left(z_t\right)$, $b_T$ is the bandwidth of Kernel estimation, T is the sample size, ${\widehat{z}}_t=(r_t-\mu )/{\widehat{h}}_t^{1/2}$ and $K(·)$ is a Gaussian Kernel function:

$$\begin{array}{c}\hfill K\left(s\right)={\displaystyle \frac{1}{\sqrt{2\pi }}}{e}^{-{\displaystyle \frac{s^2}{2}}}.\end{array}$$

The size of the bandwidth affects the accuracy of Kernel density estimation. In this section, we follow Silverman’s (1978) idea and select a fixed bandwidth suitable for Gaussian Kernel functions:

$$\begin{array}{c}\hfill b_T={\left({\displaystyle \frac{4}{3T}}\right)}^{{\displaystyle \frac{1}{5}}}{\widehat{\sigma }}_z\approx 1.06{\widehat{\sigma }}_z{T}^{-{\displaystyle \frac{1}{5}}},\end{array}$$

where ${\widehat{\sigma }}_z^2$ is the variance of ${\tilde{z}}_t$ and the estimator of $Var\left(z_t\right)$. By substituting ${\widehat{f}}_z\left(z_t\right)$ for $f_z\left(z_t\right)$ in ${\check{L}}_T$, we have

$$\begin{array}{c}\hfill {\widehat{L}}_T\left(\mathit{\theta }\right|\mathbf{r},\mathbf{x})={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\widehat{l}}_t\left(\mathit{\theta }\right),\end{array}$$

where

$$\begin{array}{c}\hfill {\widehat{l}}_t\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{2}}\left[-2log\left({\widehat{f}}_z\left(z_t\right)\right)+log{\widehat{h}}_t\left({\mathit{\theta }}_1\right)+log\left({\sigma }_u^2\right)+{\displaystyle \frac{{\widehat{u}}_t{({\mathit{\theta }}_1,{\mathit{\theta }}_2)}^2}{{\sigma }_u^2}}\right].\end{array}$$

We will refer to this method as the QML Kernel method, abbreviated as QML-K, in the following text. The estimator of QML-K is defined as follows:

$$\begin{array}{c}\hfill {\widehat{\mathit{\theta }}}_T=arg\underset{\mathit{\theta }}{max}{\widehat{L}}_T.\end{array}$$

3.2. Asymptotic Properties

In this subsection, we focus on the asymptotic properties of the QML-K estimator.

Assumption 2.

For any real number $({\tilde{\tau }}_1,{\tilde{\tau }}_{21},{\tilde{\tau }}_{22})\ne (0,0,0)$, it is obvious that $\tilde{\tau }\left(z_t\right)={\tilde{\tau }}_1{z}_t+{\tilde{\tau }}_{21}(z_t^2-1)I_{(z_t\le 0)}+{\tilde{\tau }}_{22}(z_t^2-1)I_{(z_t>0)}$ does not degenerate, for any $t\in \mathbb{Z}$.

Assumption 3.

${\mathcal{A}}_{{\mathit{\theta }}_1}\left(s\right)$ and ${\mathcal{B}}_{\mathit{\theta }}\left(s\right)$ have no common zeros, and  ${\mathcal{B}}_{\mathit{\theta }}\left(s\right)\ne 0$ for all $s\in \mathbb{C}$ such that $\left|s\right|\le 1$.

Assumption 4.

There exists $s>0$ such that $E[exp(s|log{x}_t\left|\right)]<\infty$.

Remark 3.

Assumption 2 makes sure the distribution of $z_t$ is not concentrated at two points, which means the RG-TVL($p,q$) model is identifiable. Assumption 3 shows the future-independence of the model. Assumption 4 on $|log{x}_t|$ indicates that the density function of $x_t$ around 0 does not explode too fast.

Theorem 1.

Suppose that $log{h}_t$ is a stationary RG-TVL ($p,q$) process defined by Equations (1)–(4) with parameters ${\mathit{\theta }}_0$. If ${\mathit{\theta }}_0\in \Theta$, where $\Theta$ is compact, and Assumptions 1–3 hold, then we have

$$\begin{array}{c}\hfill {\widehat{\mathit{\theta }}}_T\stackrel{a.s.}{⟶}{\mathit{\theta }}_0\mathrm{as}T\to \infty .\end{array}$$

Then, we consider the asymptotic normality. For ease of explanation, we use the following notation:

$$\begin{array}{c}\hfill E_0[·]=E[·|\mathit{\theta }={\mathit{\theta }}_0],{\nabla }_1=\partial /\partial {\mathit{\theta }}_1,{\nabla }_2=\partial /\partial {\mathit{\theta }}_2,\mathrm{and}{\nabla }_{12}=\partial /\partial {({\mathit{\theta }}_1^{\prime },{\mathit{\theta }}_2^{\prime })}^{\prime }.\end{array}$$

And define

$$\begin{array}{cc}\hfill & \mathbf{A}=E_0\left[{\nabla }_1(log{\tilde{h}}_t){\nabla }_1{(log{\tilde{h}}_t)}^{\prime }\right],\mathbf{F}=E[z_t^4-3]\mathbf{A},{\mathbf{B}}_{11}={\displaystyle \frac{2}{{\sigma }_{u,0}^2}}{E}_0\left[{\nabla }_1{\tilde{u}}_t{\nabla }_1{\tilde{u}}_t^{\prime }\right],\hfill \\ \hfill & {\mathbf{B}}_{21}={\displaystyle \frac{2}{{\sigma }_{u,0}^2}}{E}_0\left[{\nabla }_2{\tilde{u}}_t{\nabla }_1{\tilde{u}}_t^{\prime }\right],\mathrm{and}{\mathbf{B}}_{22}={\displaystyle \frac{2}{{\sigma }_{u,0}^2}}{E}_0\left[{\nabla }_2{\tilde{u}}_t{\nabla }_2{\tilde{u}}_t^{\prime }\right].\hfill \end{array}$$

Theorem 2.

Suppose that the assumptions in Theorem 1 and Assumption 4 are satisfied. If ${\mathit{\theta }}_0$ is an interior point of $\Theta ,E|z_t{|}^4<\infty$ and $E|u_t{|}^4<\infty$, then we have

$$\begin{array}{c}\hfill \sqrt{T{b}_T}({\widehat{\mathit{\theta }}}_T-{\mathit{\theta }}_0)\stackrel{d}{\to }N(0,\mathbf{V}),\end{array}$$

where

$$\begin{array}{c}\hfill \mathbf{V}=\left[\begin{array}{ccc}{V}_{11}& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right]+b_T{\mathbf{H}}^{-1}\mathbf{J}{\mathbf{H}}^{-1},\end{array}$$

$V_{11}$ is a bounded unknown variable that is discussed in Appendix A.

$$\begin{array}{c}\hfill \mathbf{J}=\left[\begin{array}{ccc}\mathbf{F}+2\mathbf{A}+2{\mathbf{B}}_{\mathbf{11}}& 2{\mathbf{B}}_{\mathbf{21}}& -2{\displaystyle \frac{E\left[u_t^3\right]}{{\sigma }_{u,0}^6}}{E}_0\left[{\nabla }_1{\tilde{u}}_t^{\prime }\right]\\ 2{\mathbf{B}}_{\mathbf{21}}& 2{\mathbf{B}}_{\mathbf{22}}& -2{\displaystyle \frac{E\left[u_t^3\right]}{{\sigma }_{u,0}^6}}{E}_0\left[{\nabla }_2{\tilde{u}}_t^{\prime }\right]\\ -2{\displaystyle \frac{E\left[u_t^3\right]}{{\sigma }_{u,0}^6}}{E}_0\left[{\nabla }_1{\tilde{u}}_t^{\prime }\right]& -2{\displaystyle \frac{E\left[u_t^3\right]}{{\sigma }_{u,0}^6}}{E}_0\left[{\nabla }_2{\tilde{u}}_t^{\prime }\right]& {\displaystyle \frac{E\left[u_t^4\right]-\left({\sigma }_{u,0}^4\right)}{{\sigma }_{u,0}^8}}\end{array}\right],\end{array}$$

and

$$\begin{array}{c}\hfill \mathbf{H}=\left[\begin{array}{ccc}\mathbf{A}+{\mathbf{B}}_{\mathbf{11}}& {\mathbf{B}}_{\mathbf{21}}^{\prime }& \mathbf{0}\\ {\mathbf{B}}_{\mathbf{21}}& {\mathbf{B}}_{22}& \mathbf{0}\\ {\mathbf{0}}^{\prime }& {\mathbf{0}}^{\prime }& {\displaystyle \frac{1}{{\sigma }_{u,0}^4}}\end{array}\right].\end{array}$$

Remark 4.

The asymptotic covariance matrix $\mathbf{V}$ is singular if and only if $u_t^2\equiv {\sigma }_{u,0}^2$, for all t.

The first term of $\mathbf{V}$ represents the variance of the Kernel estimator, and the second represents that of the QML estimator. Since the Kernel method only focuses on $z_t$, which only affects ${\mathit{\theta }}_1$ in Equation (2), other elements of the matrix are equal to 0 except $V_{11}$.

3.3. Optimization Algorithm

In this subsection, our aim is to search for a ${\widehat{\mathit{\theta }}}_T=argmax{\widehat{L}}_T\left(\mathit{\theta }\right)$. The penalty function method is selected to solve constrained optimization problems. The accuracy of this method is limited by the parameter selection of the augmented objective function; thus, improper parameter selection may lead to lower accuracy of the optimization method.

Considering that the model contains inequality constraints mentioned in Section 2, the interior point method is selected to solve an optimization problem with inequality constraints. Based on the idea of the method, we first need to construct an augmented objective function as follows:

$$\begin{array}{c}\hfill F\left(\mathsf{\theta }\right|\mathbf{r},\mathbf{x},\delta )=-{\widehat{L}}_T\left(\mathit{\theta }\right)+\delta \Psi \left(\mathit{\theta }\right),\end{array}$$

where $\delta >0$ represents obstacle factors to control the impact degree of $\Psi \left(\mathit{\theta }\right)$, $\Psi \left(\mathit{\theta }\right)>0$, and as $\mathit{\theta }$ approches the feasible domain boundary, $\Psi \left(\mathit{\theta }\right)$ tends towards infinity. A common construction of $\Psi \left(\mathit{\theta }\right)$ is as follows:

$$\begin{array}{c}\hfill \Psi \left(\mathit{\theta }\right)=-\sum _{i=1}^n{\left[p_i\left(\mathit{\theta }\right)\right]}^{-1},\end{array}$$

where $p_i\left(\mathit{\theta }\right)$ represents n restrictive conditions. In this paper we have $p_1\left(\mathit{\theta }\right)=-{\sum }_{i=1}^p{\beta }_i-\varphi {\sum }_{j=1}^q{\gamma }_j-1$ and $p_2\left(\mathit{\theta }\right)={\sum }_{i=1}^p{\beta }_i+\varphi {\sum }_{j=1}^q{\gamma }_j-1$. The parameter estimation results of the model can be obtained by solving the problem:

$$\begin{array}{c}\hfill \widehat{\mathit{\theta }}=\underset{\mathit{\theta }\in \mathcal{W}}{argmin}F(\mathit{\theta }\mid \mathbf{r},\mathbf{x}),\end{array}$$

where $\mathcal{W}=\{w\mid p_i\left(w\right)<0,i=1,2,3,\dots ,m\}$.

Denote ${\widehat{\mathit{\theta }}}_{T,k}$ as the estimator of parameters in the k-th iteration.

With the optimization algorithm (Algorithm 1), we can calculate the estimator of our model. There may be a more effective algorithm than we used in this paper; however, we do not focus on the optimization algorithm. More research and exploration can be conducted.

Algorithm 1: The interior point method

4. Simulation Study

In this section, we aim to rigorously verify and assess the fitting effectiveness of the RG-TVL model utilizing QML and QML-K. The simulation is conducted with different control variables: estimation methods, sample sizes, and distributions of $z_t$. By analyzing the simulations, we hope to gain deeper insights into the robustness and adaptability of the RG-TVL model across various scenarios and conditions, thus contributing to its better understanding and potential applications in various fields.

The evaluation criteria in the simulation study are bias and mean square error (MSE). In the simulation study and application, we take the RG-TVL(1,1) model as a sample. Additionally, since $\mu$ in Equation (6) has a negligible effect on parameter estimation, we assume it to be zero. A smaller absolute value of bias shows the higher the accuracy of parameter estimation, and a smaller MSE indicates stronger stability of parameter estimation. In this section, we mainly pay attention to MSE.

Under the restrictive condition in Section 2, some true parameters are set as follows:

$${\left(\alpha ,\beta ,\gamma ,\xi ,\varphi ,{\tau }_1,{\tau }_{21},{\tau }_{22},{\sigma }_u,h_0\right)}^{\prime }={\left(0.2,0.7,0.25,0.1,0.9,-0.5,0.5,0.2,0.1,1.0\right)}^{\prime }.$$

To better describe the degree of leverage effect, we use the correlation between $\tau \left(z_t\right)$ and $z_t$ as the indicator, which is called the leverage coefficient and noted as $\rho$. The value of $\rho$ is between −1 and 1, and it can reflect a leverage effect if it is negative. The value of $\rho$ close to −1 means a strong leverage effect. The formula for $\rho$ is as follows:

$$\begin{array}{cc}\hfill \rho & =Corr[\tau \left(z_t\right),z_t]={\displaystyle \frac{Cov[\tau \left(z_t\right),z_t]}{\sqrt{Var\left[\tau \left(z_t\right)\right]Var\left(z_t\right)}}}\hfill \\ \hfill & ={\displaystyle \frac{{\tau }_1+({\tau }_{21}-{\tau }_{22})c_1}{\sqrt{{\tau }_1^2+\frac{1}{2}({\tau }_{21}^2+{\tau }_{22}^2)c_2+2{\tau }_1({\tau }_{21}-{\tau }_{22})c_1}}},\hfill \end{array}$$

(9)

where $c_1={\int }_{-\infty }^0(z^3-z)f\left(z\right)dz$, $c_2={\int }_{-\infty }^{\infty }{(z^2-1)}^{2}f\left(z\right)dz$ and $f\left(z\right)$ represents the probability distribution function of $z_t$.

By Equation (9), if $z_t$ follows a standard normal distribution and a standardized $t\left(4\right)$ distribution, we have $\rho =-0.864$ and $\rho =-0.757$, respectively. Both values show a relatively strong leverage effect. The results are shown as follows:

In

Table 1. Results of simulation study with $z_t$ following standard normal distribution ($\times {10}^{-3}$).

Sample Size		100		500		1000
Method		QML	QML-K	QML	QML-K	QML	QML-K
$\alpha$	Bias	−71.218	−43.409	6.844	38.065	16.364	34.121
	MSE	522.667	410.000	0.705	2.272	0.534	1.634
$\beta$	Bias	24.160	33.363	0.311	−0.165	0.511	0.450
	MSE	31.386	39.980	0.092	0.071	0.038	0.030
$\gamma$	Bias	−5.476	−2.191	−0.729	0.042	−0.316	−0.057
	MSE	0.821	1.201	0.073	0.047	0.029	0.021
$\xi$	Bias	−35.051	−75.455	−22.681	−80.928	−31.988	−65.200
	MSE	39.221	62.144	3.872	10.434	2.532	6.419
$\varphi$	Bias	8.243	−6.166	3.689	2.562	0.281	−0.503
	MSE	3.549	7.490	0.307	0.267	0.156	0.109
${\tau }_1$	Bias	0.017	−10.706	−1.416	−9.984	−23.309	−47.293
	MSE	0.251	0.510	0.054	0.170	0.830	2.834
${\tau }_{21}$	Bias	4.605	36.424	7.761	50.227	22.236	45.592
	MSE	1.484	2.958	0.646	3.482	0.827	2.671
${\tau }_{22}$	Bias	4.606	57.931	4.318	30.541	14.481	28.862
	MSE	3.407	7.280	0.273	1.298	0.333	1.056
${\sigma }_u$	Bias	1.587	1.270	0.910	0.864	0.028	0.037
	MSE	0.506	0.297	0.012	0.011	0.008	0.007

, when the sample size is equal to 100, the MSEs of $\alpha$ and ${\sigma }_u$ using QML-K are lower than those using QML, while the other parameters are the opposite. When the sample size grows to 500, the MSEs of $\beta$, $\gamma$, $\varphi$, and ${\sigma }_u$ using QML-K are less than those using QML, while the other parameters are on the contrary; When the sample size reaches 1000, the MSEs of $\beta$, $\gamma$, $\varphi$, and ${\sigma }_u$ are less than those using QML, while the other parameters are in contrast.

In

Table 2. Results of simulation study with $z_t$ following standardized $t\left(4\right)$ distribution $(\times {10}^{-3})$.

Sample Size		100		500		1000
Method		QML	QML-K	QML	QML-K	QML	QML-K
$\alpha$	Bias	42.755	−2.496	38.524	33.582	25.831	27.543
	MSE	17.136	25.832	24.829	2.351	23.076	1.582
$\beta$	Bias	7.799	7.522	7.809	−0.266	7.531	0.486
	MSE	6.042	5.411	6.466	0.119	5.882	0.045
$\gamma$	Bias	−0.619	−0.616	−1.436	1.715	−2.993	−0.577
	MSE	0.964	0.973	0.967	0.101	0.972	0.040
$\xi$	Bias	−111.174	−30.199	−113.242	−64.578	−85.109	−54.789
	MSE	19.152	12.110	24.323	9.701	14.043	6.384
$\varphi$	Bias	−3.181	−2.437	0.802	−2.894	2.832	0.488
	MSE	0.808	1.259	1.599	0.432	1.079	0.140
${\tau }_1$	Bias	−80.346	−24.331	−15.001	−9.918	−54.902	−37.313
	MSE	7.269	2.762	0.270	0.161	3.455	2.703
${\tau }_{21}$	Bias	74.682	22.193	74.026	49.586	52.457	35.748
	MSE	6.335	2.443	6.121	3.555	3.151	2.451
${\tau }_{22}$	Bias	44.697	13.089	45.462	30.402	31.231	21.332
	MSE	2.261	0.930	2.325	1.376	1.133	0.880
${\sigma }_u$	Bias	94.784	95.310	−1.579	0.168	−1.140	0.368
	MSE	230.514	230.690	0.288	0.013	0.241	0.007

, when the sample size is equal to 100, the MSEs of $\beta$, $\xi$, ${\tau }_1$, ${\tau }_{21}$, and ${\tau }_{22}$ using QML-K are less than those using QML, while the other parameters are contrary to the former; when the sample size reaches 500 or 1000, the MSEs of all parameters using QML-K are less than those using QML.

Finally, we are interested in the impact of n in Equation (6) on the estimation. In Section 2, we mentioned that the increasing n may lead to more accuracy but more cost. Thus, we select the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) to balance cost and accuracy. In this part, we perform an experiment on which model to choose. Denote the situations $n=2$ and $n=3$ as Spline1 RG-TVL and Spline2 RG-TVL models, respectively. Yu [16] previously examined alternative node specifications, such as partitioning the interval into two or three segments with nodes at 0 or (−0.4, 0.4), motivated by the empirical properties of the Gaussian distribution. Thus, we set the leverage function in the Spline2 RG-TVL model as

$$\begin{array}{c}\hfill \tau \left(z_t\right)={\tau }_1{z}_t+(z_t^2-1)[{\tau }_{21}{I}_{(z_t\le -0.4)}+{\tau }_{22}{I}_{(-0.4<z_t\le 0.4)}+{\tau }_{23}{I}_{(z_t>0.4)}].\end{array}$$

Compared with Spline1 RG-TVL, this model adds ${\tau }_{23}$ in parameters. Then, we generate data from Spline2 RG-TVL and Spline3 RG-TVL models and apply the two models to those data to compare their effects. In this experiment, we set ${\tau }_{23}=0.7$, $z_t\sim N(0,1)$, and other parameters the same as above. We conducted 100 estimations and recorded the mean and standard deviation as follows:

It can be seen from the table that the AICs and BICs of Spline1 do not show a significant difference between Spline1 and Spline2 in general.

According to Table 1, Table 2 and

Table 3. Comparison results of Spline1 RG-TVL and Spline2 RG-TVL under 100 samples.

Model	Spline1		Spline2
Data	Spline1	Spline2	Spline1	Spline2
LL	−191.970	−215.087	−212.258	−191.480
AIC	401.939	448.174	444.515	402.960
BIC	425.386	471.620	470.567	429.012
Model	Spline1		Spline2
Data	Spline1	Spline2	Spline1	Spline2
LL	−391.596	−439.119	−432.489	−396.312
AIC	801.192	896.238	884.977	812.625
BIC	830.877	925.923	917.960	845.608

, we can draw conclusions:

(a) As the sample size rises, the MSEs of parameters tend to decrease using both methods and both distributions of $z_t$, which conforms to general statistical theory;

(b) MSEs of parameters for both QML and QML-K are relatively small;

(c) Under the standard normal distribution assumption, the MSEs using QML and QML-K do not show a sigificant difference, while under the other assumption, the MSEs using QML-K are remarkably less than those using QML, especially when the sample size is larger than 500;

(d) There is no evidence to suggest that the Spline2 RG-TVL models have better finite sample performance than the Spline1 model.

The conclusions drawn from (a) and (b) demonstrate that the RG-TVL model exhibits good performance when applied to finite sample scenarios, highlighting its efficacy and reliability within such constrained datasets. Meanwhile, the observations made in (c) imply that, in comparison to the QML method, the QML-K method emerges as a more robust one for estimation purposes. This suggests that QML-K may offer advantages in terms of stability and accuracy, particularly in scenarios where the assumptions underlying QML may be violated. (d) demonstrates that increasing the number of nodes may not significantly improve the estimation performance of the model.

5. Applications

5.1. Data Acquisition and Processing

In this section, we study the applications of the RG-TVL model in the Hang Seng Index (HSI) and Standard & Poor’s 500 Index (S&P 500). The former is a market capitalization-weighted stock index in Hong Kong, tracking the performance of the 50 largest and most liquid companies listed on the Hong Kong Stock Exchange. It serves as an important benchmark for the Hong Kong stock market. The latter is one of the most representative stock indices in the US stock market, consisting of approximately 500 large publicly traded companies and covering about 80% of the total market value of the US stock market. It covers multiple industries such as technology, finance, healthcare, and consumer goods. Due to the quality and diversity of its constituent stocks, it has become a “barometer” of global investors’ attention for the US stock market. The required data were collected from the Wind Financial Terminal.

The date ranges selected are from 16 May 2022 to 17 September 2024, with a total of 568 working days of data (HSI), and from 16 January 2024 to 21 March 2025, with a total of 297 working days of data (S&P 500). The daily logarithmic return rate formula and 5-min logarithmic return rate formulas are

$$\begin{array}{cc}\hfill r_t=& log{y}_t-log{y}_{t-1},\hfill \\ \hfill r_{t,i}=& log{y}_{t,i}-log{y}_{t,i-1},\hfill \end{array}$$

where $y_t$ represents the transaction price at day t and $y_{t,i}$ represents the ith transaction price at day t.

According to Andersen and Bollerslev [37], we use realized volatility (RV) as the realized measure inserted in the RG-TVL model. The form of RV is as follows:

$$\begin{array}{cc}\hfill {\mathrm{RV}}_t=& \sum _{i=1}^T{r}_{i,t}^2.\hfill \end{array}$$

Note that RV is always positive, which ensures the stability of the model. Corsi et al. [38] have proved that the variable has unbiased and stable characteristics.

Firstly, we conduct descriptive statistics on the data and draw a daily logarithmic return rate of HSI and S&P 500 as shown in Figure 1.

Figure 1 shows that the daily logarithmic return rates of HSI and S&P 500 both have a small degree of change over time. The two sets of data do not show a noteworthy trend of growth or decrease. However, the HSI indicates significant fluctuations in October 2022 and February 2024, while the S&P 500’s fluctuations are in April 2024, September 2024, and January 2025. The RV changes in HSI and S&P 500 are plotted as shown in Figure 2.

From Figure 2, we can obtain that the values of RV are relatively small and had significant fluctuations, which correspond to the changes in daily logarithmic rates of return, indicating that RV can effectively reflect the changes in returns. The descriptive statistics of the daily logarithmic return rate and RV of HSI and S&P500 are shown in

Table 4. Descriptive statistics of HSI and S&P 500.

	HSI		S&P500
	r	RV	r	RV
Mean	−3.437 $\times {10}^{-4}$	1.258 $\times {10}^{-4}$	−4.821 $\times {10}^{-5}$	4.169 $\times {10}^{-5}$
Max	4.550 $\times {10}^{-2}$	7.939 $\times {10}^{-4}$	1.655 $\times {10}^{-2}$	2.894 $\times {10}^{-4}$
Min	−4.600 $\times {10}^{-2}$	2.279 $\times {10}^{-5}$	−2.945 $\times {10}^{-2}$	6.007 $\times {10}^{-6}$
Std	1.170 $\times {10}^{-2}$	8.424 $\times {10}^{-5}$	6.600 $\times {10}^{-3}$	3.977 $\times {10}^{-5}$
Skewness	0.319	3.500	−0.850	2.607
Kurtosis	4.088	21.098	4.548	12.238
25% Quantile	−8.400 $\times {10}^{-3}$	7.831 $\times {10}^{-5}$	−3.028 $\times {10}^{-3}$	1.680 $\times {10}^{-5}$
Median	−1.100 $\times {10}^{-3}$	1.044 $\times {10}^{-4}$	5.051 $\times {10}^{-4}$	2.787 $\times {10}^{-5}$
75% Quantile	6.900 $\times {10}^{-3}$	1.443 $\times {10}^{-4}$	4.401 $\times {10}^{-3}$	5.188 $\times {10}^{-5}$

.

Table 4 indicates that, for the HSI, the maximum daily logarithmic return is 4.550 $\times {10}^{-2}$, while the minimum is −4.600 $\times {10}^{-2}$, indicating substantial variability in returns. The mean is close to zero, with a positive skewness, suggesting a right-skewed distribution. The kurtosis exceeds 3, reflecting leptokurtic behavior. As a proxy for integrated volatility, the realized volatility (RV) is strictly positive and exhibits a right-skewed distribution with significantly higher kurtosis (>3), consistent with a pronounced leptokurtic and heavy-tailed distribution. For the S&P 500, the daily logarithmic returns range from a maximum of 1.655 $\times {10}^{-2}$ to a minimum of −2.945 × 10⁻², demonstrating considerable dispersion. The near-zero mean and negative skewness indicate a left-skewed distribution, while the kurtosis above 3 confirms leptokurtosis. The RV is strictly positive and displays a right-skewed, highly leptokurtic distribution (kurtosis $> 3$), characteristic of a heavy-tailed process. Then, perform normality tests on the samples separately and create Q–Q diagrams as follows:

From Figure 3, it is evident that the daily logarithmic return rate of both indices does not exhibit a straight line, suggesting that it does not follow a normal distribution. To further validate this assumption, the Jarque–Bera test was conducted on the return rate. The results indicate a p-value of 0.001, which further confirms that the return rate does not conform to normal distributions.

5.2. In-Sample Estimation

Next, we compare the performance of two estimation methods in two models (RG and RG-TVL) within the sample. The comparison indicators of the fitting effect are the log-Likelihood function (LL) and the AIC criterion. The estimated results are shown in

Table 5. Results of in-sample estimation.

	RG		RG-TVL
	QML	QML-K	QML	QML-K
$\alpha$	0.300	0.218	0.206	0.114
$\beta$	0.730	0.729	0.739	0.737
$\gamma$	0.301	0.296	0.281	0.277
$\xi$	−2.120	−1.902	−1.827	−1.573
$\varphi$	0.776	0.788	0.808	0.824
${\tau }_1$	−0.046	−0.043	−0.009	−0.001
${\tau }_2$	0.022	0.019
${\tau }_{21}$			0.077	0.070
${\tau }_{22}$			−0.023	−0.023
${\sigma }_u$	0.373	0.373	0.368	0.368
$\rho$	−0.890	−0.902	−0.850	−0.821
LL	−1506.503	−1510.676	−1512.973	−1517.231
AIC	−2995.007	−3003.352	−3005.946	−3014.462
BIC	−2949.586	−2957.931	−2962.525	−2971.041
$\alpha$	−3.570	−2.897	−2.521	−2.780
$\beta$	0.300	0.358	0.279	0.278
$\gamma$	0.339	0.350	0.461	0.438
$\xi$	−11.367	−5.306	1.725	2.381
$\varphi$	−0.093	0.502	1.195	1.258
${\tau }_1$	−0.082	−0.167	−0.024	−0.025
${\tau }_2$	0.014	−0.012
${\tau }_{21}$			0.031	0.031
${\tau }_{22}$			−0.016	−0.016
${\sigma }_u$	0.868	0.739	0.566	0.565
$\rho$	−0.971	−0.995	−0.964	−0.963
LL	721.701	785.719	828.333	836.232
AIC	−1427.403	−1555.437	−1636.665	−1652.463
BIC	−1397.853	−1529.419	−1525.887	−1615.526

.

From Table 5, we can obtain that the estimated results of $\rho$ are all negative and further away from 0, indicating a significant leverage effect in the data. There is a significant difference between ${\tau }_{21}$ and ${\tau }_{22}$, reflecting the time variety of the leverage effect. According to the numerical values of AIC and BIC, the order from low to high is RG-TVL-QML-K, RG-TVL-QML RG-QML-K, and RG-QML.

5.3. Out-of-Sample Prediction

The selected period for out-of-sample prediction is a total of 15 working days. We choose MSE, mean absolute error (MAE), and root mean square error (RMSE) to be the standards to measure the effectiveness of out-of-sample prediction. According to the form of the model, the predictive model is as follows:

$$\begin{array}{cc}\hfill log{\widehat{h}}_{T+i}=& \widehat{\alpha }+\widehat{\beta }log{\widehat{h}}_{T+i-1}+\widehat{\gamma }log{\widehat{x}}_{T+i-1},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill log{\widehat{x}}_{T+i}=& \widehat{\xi }+\widehat{\varphi }log{\widehat{h}}_{T+i}+\widehat{\tau }\left(z_{T+i}\right)+{\widehat{u}}_{T+i},\hfill \end{array}$$

(11)

where $\widehat{\mathit{\theta }}={(\widehat{\alpha },\widehat{\beta },\widehat{\gamma },\widehat{\xi },\widehat{\varphi },{\widehat{\tau }}_1,{\widehat{\tau }}_{21},{\widehat{\tau }}_{22},{\widehat{\sigma }}_u)}^{\prime }$ can be obtained by the result of in-sample estimation; $z_i$ can be obtained from standard normal random numbers; $u_i$ can be obtained from normal random numbers with an expectation of 0 and a variance of ${\sigma }_u$. The multi-step prediction results can be obtained by repeatedly calculating Equations (10) and (11), thus obtaining the predictor of $x_i$.

The results of the prediction are shown in

Table 6. Results of out-of-sample prediction.

Measure	RG		RG-TVL
	QML	QML-K	QML	QML-K
MSE $(\times {10}^{-9})$	5.721	5.113	4.061	3.949
MAE $(\times {10}^{-5})$	4.618	4.276	4.082	4.063
RMSE $(\times {10}^{-4})$	2.929	2.769	2.468	2.434
	RG		RG-TVL
	QML	QML-K	QML	QML-K
MSE ($\times {10}^{-6}$)	1.263	1.254	1.252	1.250
MAE ($\times {10}^{-4}$)	6.520	6.476	6.455	6.442
RMSE ($\times {10}^{-3}$)	1.124	1.120	1.119	1.118

.

With the results above, the differences between predicted values are relatively small, which demonstrates that both RG and RG-TVL models can well predict the volatility by QML and QML-K.

Overall, the results indicate that the RG-TVL model consistently outperforms the standard RG specification across all error metrics (MSE, MAE, RMSE), highlighting its superior volatility forecasting capability. Furthermore, the QML-K estimator demonstrates enhanced precision compared with QML, reinforcing the importance of incorporating Kernel-based adjustments for robust parameter estimation in volatility modeling.

6. Conclusions and Discussion

We start from the perspective of the time-varying leverage effect and consider the asymmetric characteristics of volatility in the current capital market. Referring to Yu [16], a linear spline function is introduced into the RG model to reflect the time variety of the leverage effect, thus constructing the RG-TVL model. The QML-K method is applied to the RG-TVL model to estimate the parameters. Some asymptotic properties of the estimator have been proved. An experiment with different variables is conducted to explore the fitting effect of the RG-TVL model. Finally, we validate the predictive performance of the model with representative stock index data from Hong Kong.

The complexity and dynamism inherent in the time-varying leverage effect pose significant challenges for accurate measurement and prediction, making it a notoriously elusive phenomenon to capture. Our method offers a reliable and innovative approach to tackling this intricate problem. It is designed specifically to account for the temporal variations in the leverage effect, leveraging advanced statistical and econometric techniques that enable it to adapt dynamically to changes in market conditions. By offering an adaptable solution to the challenging task of capturing the time-varying leverage effect, our method represents a significant step forward in the field of financial risk management.