CQEformer: A Causal and Query-Enhanced Transformer Variant for Time-Series Forecasting

Abstract

Structural breaks and volatility clustering are fundamental challenges in time series analysis. We propose CQEformer, an encoder-only Transformer variant for time-series modeling that addresses these challenges via two complementary innovations. First, the causal residual embedding (CRE) ensures temporal causality and improves local adaptivity to abrupt structural changes. Second, the query-enhanced multi-head self-attention (QEAttention) incorporates multi-order moment statistics and entropy to guide attention dynamically toward high-volatility regions while preserving global dependence structures. For parameter optimization, we derive analytical gradients for all components and update them using the Adam stochastic optimization algorithm. Empirical evaluations on financial time series datasets and the public Traffic dataset show that CQEformer consistently outperforms established baselines, including LSTM, GRU, TCN, and the standard Transformer. Time-window sensitivity analyses demonstrate the robustness of the framework, while ablation studies further confirm that the proposed modules are complementary and contribute to improved forecasting performance across different volatility regimes.

1. Introduction

Time series arising in complex systems, such as financial markets, exhibit key statistical properties in asset price time series, including volatility clustering, heteroskedasticity, periodicity, and nonstationarity [1]. These characteristics increase the difficulty of accurate modeling and forecasting, which is critical in practical scenarios such as high-frequency trading, quantitative stock selection, and asset allocation.

Traditional approaches, including Autoregressive Integrated Moving Average [2], Vector Autoregression [3], and Generalized Autoregressive Conditional Heteroskedasticity [4], offer interpretability and theoretical completeness and can model linear correlations and conditional heteroskedasticity to some extent. However, they fail to adequately model nonlinear and high-dimensional dependencies in financial markets and lack robustness due to sensitivity to specification, distributional assumptions, and regime shifts [5,6,7].

Deep learning methods have demonstrated strong capabilities in capturing nonlinear patterns and high-dimensional representations. Recurrent architectures such as Recurrent Neural Network (RNN), Long Short-Term Memory network (LSTM), Gated Recurrent Unit (GRU), and their variants have been applied to stock price forecasting. For instance, Nguyen et al. [8] proposed the SR-SV model, combining RNN with stochastic volatility models (SV), achieving strong out-of-sample predictive performance and interpretability. Sang and Li [9] enhanced LSTM structures with attention mechanisms in AMV-LSTM, improving stability and predictive accuracy, while Ming-Che Lee integrated GRU with attention to predict significant price fluctuations [10]. Nevertheless, these methods face limitations in modeling long-term dependencies and enabling efficient parallel training.

The introduction of the Transformer architecture has provided a promising alternative [11]. Its self-attention mechanism excels at capturing long-range dependencies and has inspired variants such as Informer [12], FEDformer [13], PatchTST [14], TimesNet [15], iTransformer [16], Pathformer [17], Twinsformer [18], and Timer-XL [19], demonstrating strong performance in general-purpose time series forecasting. Yet these models do not fully account for financial data’s unique properties, such as local anomalies, tail risks, and volatility clustering.

To address these domain-specific challenges, some works have customized Transformer architectures for financial applications. Ding et al. [20] proposed a Transformer enhanced with multi-scale Gaussian priors, orthogonal regularization, and a trading-gap segmenter, achieving strong results on NASDAQ and Chinese A-share markets. Zhang et al. [21] combined refined small-sample feature engineering with multiple attention mechanisms, attaining high predictive accuracy and improved realized returns. Nonetheless, domain-specific Transformer variants remain limited and lack systematic development, and as a result, leaving complex statistical characteristics and risk features underexploited.

To confront the key challenges of financial time series prediction—dynamic volatility, structural breaks, and nonstationarity—we propose CQEformer, an Encoder-Only Transformer variant that integrates Causal Convolution Residual Embedding with a Statistical-Prior-Enhanced Query Mechanism. This architecture is specifically designed to capture both local temporal patterns and global structural dynamics in complex, high-noise financial data, with two core innovations detailed as follows:

• Causal convolutional residual embedding in the input layer, emphasizing spatiotemporal locality and enforcing temporal causality. By focusing on residual features between consecutive time steps, the model improves sensitivity to sudden market shocks, structural breaks, and local anomalies, thereby enhancing short-term predictive accuracy and stability.
• Query-Enhanced multi-head self-attention mechanism, dynamically adjusting the Query matrix using higher-order statistical moments and entropy in time and frequency domains. This guides attention to regions exhibiting volatility clustering and sudden structural shifts, improving modeling of tail risks and nonlinear market behaviors.

Unlike recent frequency-aware or decomposition-based Transformers, CQEformer explicitly combines CRE for local temporal patterns with QEAttention, which injects multi-order time– and frequency–domain priors into attention logits. This design enables more effective amplification of tail and high-volatility signals and facilitates capturing both local and global dynamics, complementing existing approaches.

In Section 4, we provide comprehensive evaluations, including multi-model comparison across three real-world datasets and further analyses—time-window sensitivity and ablation studies—conducted on the CSI 300 dataset. The results show that CQEformer consistently outperforms baseline models, effectively captures complex temporal dynamics, and maintains robustness across varying market conditions.

2. Background

2.1. Problem Setup

Consider the problem of time series forecasting: Given a multivariate time series $\mathit{X}\in {\mathbb{R}}^{L\times D}$, where L denotes the number of time steps and D is the feature dimension. Let $\mathit{y}\in {\mathbb{R}}^{L^{\prime }}$ denote the target label, the objective is to learn a mapping f: ${\mathbb{R}}^{L\times D}\to {\mathbb{R}}^{L^{\prime }}$ to predict the future values $\widehat{\mathit{y}}\in {\mathbb{R}}^{L^{\prime }}$ over the next $L^{\prime }$ steps, as follows:

$$\widehat{\mathit{y}}=f\left(\mathit{X}\right).$$

(1)

2.2. Encoder-Only Transformer

An encoder-only Transformer typically consists of four main components: an input embedding layer that maps raw features to a high-dimensional space, positional encoding to inject sequence order information, a stack of encoder layers—each containing multi-head self-attention (MSA) and feed-forward network (FFN) with residual connections (Res) and layer normalization (LayerNorm)—to extract contextual representations, and an output layer to produce the final predictions. The architecture is shown in Figure 1.

• Input Embedding and Positional Encoding

For a standardized multivariate time series $\mathit{X}\in {\mathbb{R}}^{L\times D}$, it is first projected into a high-dimensional space ${\mathbb{R}}^{L\times H}$, where H is the embedding dimension. The standard Transformer applies a fully connected layer to obtain ${\mathit{X}}_e$, and then

$${\mathit{X}}_e=\mathit{X}·{\mathit{W}}_e+{\mathbf{1}}_L·{\mathit{b}}_e^{\mathrm{T}},$$

(2)

where ${\mathit{W}}_e\in {\mathbb{R}}^{D\times H}$, ${\mathit{b}}_e\in {\mathbb{R}}^H$ are learnable, ${\mathbf{1}}_L$ denotes a L-dimensional all-ones vector. The Transformer is permutation-equivariant and cannot capture order [22], a positional encoding $\mathit{PE}\in {\mathbb{R}}^{L\times H}$ is therefore introduced, with elements defined as

$${\mathit{PE}}_{ij}=\left\{\begin{array}{cc}sin\left({\displaystyle \frac{i-1}{{10000}^{\frac{j-1}{H}}}}\right),\hfill & j=2k-1,\hfill \\ cos\left({\displaystyle \frac{i-1}{{10000}^{\frac{j-1}{H}}}}\right),\hfill & j=2k,\hfill \end{array}\right.$$

(3)

where $i=1,2,\cdots ,L$, $k=1,2,\cdots ,⌊H/2⌋$. Then $\mathit{PE}$ is added to the ${\mathit{X}}_e$ to obtain ${\mathit{X}}_{in}$,

$${\mathit{X}}_{in}={\mathit{X}}_e+\mathit{PE}.$$

(4)

• Multi-Head Self-Attention

$\mathrm{MSA}(·)$ is defined as the multi-head self-attention, calculated as follows. With N heads (H divisible by N), let $S=H/N$ denote head’s dimension, this layer input ${\mathit{X}}_{in}$ is mapped into N sets of query, key, and value matrices $({\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i)$, $i=1,2,\cdots ,N$,

$$\left\{\begin{array}{c}{\mathit{Q}}^i={\mathit{X}}_{in}·{\mathit{W}}_{Q^i}+{\mathbf{1}}_L·{\mathit{b}}_{Q^i},\hfill \\ {\mathit{K}}^i={\mathit{X}}_{in}·{\mathit{W}}_{K^i}+{\mathbf{1}}_L·{\mathit{b}}_{K^i},\hfill \\ {\mathit{V}}^i={\mathit{X}}_{in}·{\mathit{W}}_{V^i}+{\mathbf{1}}_L·{\mathit{b}}_{V^i},\hfill \end{array}\right.$$

(5)

where ${\mathit{W}}_{Q^i},{\mathit{W}}_{K^i},{\mathit{W}}_{V^i}\in {\mathbb{R}}^{H\times S}$, ${\mathit{b}}_{Q^i},{\mathit{b}}_{K^i},{\mathit{b}}_{V^i}\in {\mathbb{R}}^S$. The attention for one head is computed as

$${\mathit{Head}}^i=\mathrm{Attention}\left({\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i\right):=\mathrm{Softmax}\left({\displaystyle \frac{{\mathit{Q}}^i·{\left({\mathit{K}}^i\right)}^{\mathrm{T}}}{\sqrt{S}}}\right)·{\mathit{V}}^i,$$

(6)

where Softmax is applied row-wise, so that the elements of each row sum to 1. Let $\mathit{Head}=\mathrm{Concat}\left({\mathit{Head}}^1,{\mathit{Head}}^2,\cdots ,{\mathit{Head}}^N\right)\in {\mathbb{R}}^{L\times H}$, the multi-head self-attention is computed as

$${\mathit{X}}_{msa}=\mathit{Head}·{\mathit{W}}_o+{\mathbf{1}}_L·{\mathit{b}}_o^{\mathrm{T}},$$

(7)

where ${\mathit{W}}_o\in {\mathbb{R}}^{H\times H}$, and ${\mathit{b}}_o\in {\mathbb{R}}^H$.

• First Res & LayerNorm

After multi-head self-attention, a residual connection and a layer normalization are applied, Let ${\mathit{\mu }}_{ln}$ and ${\mathit{\sigma }}_{ln}$ denote the row-wise mean and standard deviation, and the normalized output is ${\mathit{X}}_{ln}$,

$${\mathit{X}}_{re}=\mathrm{Res}({\mathit{X}}_{in},\mathrm{MSA}):={\mathit{X}}_{in}+\mathrm{MSA}\left({\mathit{X}}_{in}\right).$$

(8)

$${\mathit{X}}_{ln}=\mathrm{LayerNorm}\left({\mathit{X}}_{re}\right):=\left({\mathbf{1}}_L·{\mathit{\gamma }}_{ln}\right)\circ \left(\left({\mathit{X}}_{re}-{\mathit{\mu }}_{ln}·{\mathbf{1}}_H^{\mathrm{T}}\right)\oslash \left({\mathit{\sigma }}_{ln}·{\mathbf{1}}_H^{\mathrm{T}}\right)\right)+{\mathbf{1}}_L·{\mathit{\beta }}_{ln},$$

(9)

where $\mathrm{MSA}\left({\mathit{X}}_{in}\right)={\mathit{X}}_{msa}$, ${\mathit{\gamma }}_{ln},{\mathit{\beta }}_{ln}\in {\mathbb{R}}^H$ are learnable, ∘ and ⊘ denote element-wise multiplication and division.

• Feed-Forward Network

$D_f$ denotes this network’s hidden dimension, calculation follows:

$${\mathit{X}}_f=\mathrm{FFN}\left({\mathit{X}}_{ln}\right):=\mathrm{ReLU}\left({\mathit{X}}_{ln}·{\mathit{W}}_{f_1}+{\mathbf{1}}_L·{\mathit{b}}_{f_1}^{\mathrm{T}}\right)·{\mathit{W}}_{f_2}+{\mathbf{1}}_L·{\mathit{b}}_{f_2}^{\mathrm{T}},$$

(10)

where ${\mathit{W}}_{f_1}\in {\mathbb{R}}^{H\times D_f}$, ${\mathit{W}}_{f_2}\in {\mathbb{R}}^{D_f\times H}$, ${\mathit{b}}_{f_1}\in {\mathbb{R}}^{D_f}$, ${\mathit{b}}_{f_2}\in {\mathbb{R}}^H$.

• Second Res & LayerNorm

Residual connection and layer normalization are applied again on ${\mathit{X}}_{ln}$ as in Equations (8) and (9), yielding ${\mathit{X}}_{en}$,

$${\mathit{X}}_{en}=\mathrm{LayerNorm}({\mathit{X}}_{ln}+{\mathit{X}}_f),$$

(11)

and this LayerNorm contains two learnable parameters ${\mathit{\gamma }}_{en},{\mathit{\beta }}_{en}\in {\mathbb{R}}^H$.

• Output Layer

$$\widehat{\mathit{y}}={\mathit{W}}_y·{\mathit{x}}_L+{\mathit{b}}_y,$$

(12)

where ${\mathit{x}}_L$ denotes the transpose of the L-th row vector of ${\mathit{X}}_{en}$, ${\mathit{W}}_y\in {\mathbb{R}}^{L^{\prime }\times H}$, ${\mathit{b}}_y\in {\mathbb{R}}^{L^{\prime }}$.

3. CQEformer (Causal Query-Enhanced Transformer)

Time series with rapidly changing volatility and structural shifts pose challenges for standard Transformers. To address this, we propose CQEformer, which integrates a causal residual embedding (CRE) module and query-enhanced multi-Head self-attention (QEAttention) mechanism—a novel Attention that injects multi-order time- and frequency–domain priors into attention to focus on high-volatility regimes. CQEformer’s architecture is shown in Figure 2.

3.1. Causal Residual Embedding

We introduce a causal convolutional residual embedding in the input layer, which leverages residual connections with causal convolutions. This design preserves temporal causality and local spatio-temporal structure, enabling the model to capture local fluctuations and respond to sudden shocks. By combining convolutional feature extraction with residual pathways, the embedding remains robust to abrupt changes while retaining the original sequence information.

Causal convolution was introduced in WaveNet to ensure temporal causality in audio modeling [23]. For a standardized multivariate time series $\mathit{X}$, we define $\mathit{C}=\mathrm{CausalConv}\left(\mathit{X}\right)$, where CausalConv denotes a causal convolution, each element ${\mathit{C}}_{ij}$ is computed as follows:

$${\mathit{C}}_{ij}=\sum _{d=1}^D\sum _{k=1}^K{\mathbb{I}}_{i-k+1\ge 0}·{\mathit{X}}_{i^{\prime },d}·{\left({\mathit{W}}_c\right)}_{k,d,j}+{\left({\mathit{b}}_c\right)}_j,$$

(13)

where K is the size of the causal convolution kernel, $K\in {\mathbb{N}}^{+}$, $\mathbb{I}$ is the indicator function, $i^{\prime }=max(0,i-k+1)$, ${\mathit{W}}_c\in {\mathbb{R}}^{K\times D\times H}$, ${\mathit{b}}_c\in {\mathbb{R}}^H$. In our experiments, we adopt a small kernel size $K=3$ to capture local abrupt changes while maintaining a certain temporal context, which yields the best performance among tested values $K=3,5,7,9,11$.

An efficient module, residual connection, was first proposed in ResNet [24], and have been widely adopted. Based on this two structures, we propose causal convolution residual embedding, defined as follows:

$${\mathit{X}}_e=(\mathit{X}+\mathrm{CausalConv}\left(\mathit{X}\right))·{\mathit{W}}_e+{\mathbf{1}}_L·{\mathit{b}}_e^{\mathrm{T}},$$

(14)

where ${\mathit{W}}_e\in {\mathbb{R}}^{D\times H}$, ${\mathit{b}}_e\in {\mathbb{R}}^H$.

3.2. Query-Enhanced Multi-Head Self-Attention Mechanism

Traditional Transformer models struggle to capture volatility that changes rapidly across the entire sequence. To address this, we propose QEAttention, a query-enhanced multi-head self-attention mechanism that projects multi-order statistical moments and entropy from time and frequency domains into head space and uses them as scaling to dynamically reshape attention logits—effectively amplifying and propagating volatility signals across the full attention map, improving model’s ability to capture risk and extreme events.

Given the input matrix ${\mathit{X}}_{in}$, we first compute statistical prior information, including multi-order moments and entropy features in both the time and frequency domains. These features are then used by the QEAttention mechanism to obtain the attention scores.

The time-domain statistics comprise the first-order raw moment ${\mathit{\mu }}_t$, the second-order central moment ${\mathit{\sigma }}_t$, the normalized third-order central moment ${\mathit{\tau }}_t$, and the entropy ${\mathit{ϵ}}_t$, which is computed from its probability matrices ${\mathit{P}}_t$, via a feature-wise Softmax, as follows,

$${\mathit{P}}_t=\mathrm{Softmax}\left({\mathit{X}}_{in}\right),$$

(15)

so, these statistics are calculated as follows:

$${\mathit{\mu }}_t={\displaystyle \frac{1}{H}}{\mathit{X}}_{in}·{\mathbf{1}}_H,$$

(16)

$${\mathit{\sigma }}_t={\left({\displaystyle \frac{1}{H}}{\left({\mathit{X}}_{in}-{\mathit{\mu }}_t·{\mathbf{1}}_H^{\mathrm{T}}\right)}^{\circ 2}·{\mathbf{1}}_H\right)}^{\circ 0.5},$$

(17)

$${\mathit{\tau }}_t={\displaystyle \frac{1}{H}}{\left(\left({\mathit{X}}_{in}-{\mathit{\mu }}_t·{\mathbf{1}}_H^{\mathrm{T}}\right)\oslash \left({\mathit{\sigma }}_t·{\mathbf{1}}_H^{\mathrm{T}}\right)\right)}^{\circ 3}·{\mathbf{1}}_H,$$

(18)

$${\mathit{ϵ}}_t=-\left({\mathit{P}}_t\circ log{\mathit{P}}_t\right)·{\mathbf{1}}_H,$$

(19)

where $\circ i$ denotes the element-wise i-th power.

The frequency–domain statistics—namely the first-order raw moments ${\mathit{\mu }}_s$, the second-order central moments ${\mathit{\sigma }}_s$, the normalized third-order central moments ${\mathit{\tau }}_s$, and entropy ${\mathit{ϵ}}_s$—are computed from the power spectral density (PSD) of the input, which is obtained by applying the Fast Fourier Transform (FFT) along the temporal dimension and taking the squared magnitude of the resulting complex coefficients. Specifically, for ${\mathit{X}}_{in}\in {\mathbb{R}}^{L\times H}$, we compute

$${\mathit{X}}_{in}^{\prime }=\mathrm{FFT}\left({\mathit{X}}_{in}\right):=\Omega ·{\mathit{X}}_{in},{\Omega }_{\mathit{kj}}=\exp \left(-2\pi i·{\displaystyle \frac{(k-1)(j-1)}{L}}\right),$$

(20)

where $k=1,\dots ,L$ indexes the frequency components, and $j=1,\dots ,L$ indexes the time steps. And we compute PSD as follows:

$${\mathit{X}}_{ps}={\left|{\mathit{X}}_{in}^{\prime }\right|}^{\circ 2},$$

(21)

where $|·|$ denotes element-wise modulus.

From ${\mathit{X}}_{ps}$, we derive frequency–domain statistics ${\mathit{\mu }}_s$, ${\mathit{\sigma }}_s$, ${\mathit{\tau }}_s$, ${\mathit{ϵ}}_s$. ${\mathit{ϵ}}_s$ is computed using the probability matrix ${\mathit{P}}_s$, which is obtained via L1 normalization to preserve energy distribution, as follows:

$${\mathit{P}}_s={\mathit{X}}_{ps}\oslash \left(\parallel {\mathit{X}}_{ps}{\parallel }_{1,\mathrm{row}}·{\mathbf{1}}_H^{\mathrm{T}}\right),$$

(22)

where $\parallel {\mathit{X}}_{ps}{\parallel }_{1,\mathrm{row}}$ denotes the vector of row-wise L1 norms of ${\mathit{X}}_{ps}$.

The frequency–domain statistics of ${\mathit{X}}_{in}$ are then calculated as described follows:

$${\mathit{\mu }}_s={\displaystyle \frac{1}{H}}{\mathit{X}}_{ps}·{\mathbf{1}}_H,$$

(23)

$${\mathit{\sigma }}_s={\left({\displaystyle \frac{1}{H}}{\left({\mathit{X}}_{ps}-{\mathit{\mu }}_s·{\mathbf{1}}_H^{\mathrm{T}}\right)}^{\circ 2}·{\mathbf{1}}_H\right)}^{\circ 0.5},$$

(24)

$${\mathit{\tau }}_s={\displaystyle \frac{1}{H}}{\left(\left({\mathit{X}}_{ps}-{\mathit{\mu }}_s·{\mathbf{1}}_H^{\mathrm{T}}\right)\oslash \left({\mathit{\sigma }}_s·{\mathbf{1}}_H^{\mathrm{T}}\right)\right)}^{\circ 3}·{\mathbf{1}}_H,$$

(25)

$${\mathit{ϵ}}_s=-\left({\mathit{P}}_s\circ log{\mathit{P}}_s\right)·{\mathbf{1}}_H.$$

(26)

Next, we concatenate the statistics along the feature dimension and apply LayerNorm to obtain the statistical prior matrix $\mathit{\Pi }$:

$$\begin{array}{c}\hfill \mathit{\Pi }=\mathrm{LayerNorm}\left(\mathrm{Concat}\left({\mathit{\mu }}_t,{\mathit{\sigma }}_t,{\mathit{\tau }}_t,{\mathit{ϵ}}_t,{\mathit{\mu }}_s,{\mathit{\sigma }}_s,{\mathit{\tau }}_s,{\mathit{ϵ}}_s\right)\right),\end{array}$$

(27)

and this LayerNorm contains two learnable parameters ${\mathit{\gamma }}_{\mathit{\Pi }},{\mathit{\beta }}_{\mathit{\Pi }}\in {\mathbb{R}}^8$. Then, we project $\mathit{\Pi }$ into head space and apply Tanh to get the statistical prior weight matrix $\Psi$:

$$\begin{array}{c}\hfill \Psi =\mathrm{Tan}\mathrm{h}\left(\mathit{\Pi }·{\mathit{W}}_{\Psi }+{\mathbf{1}}_L·{\mathit{b}}_{\Psi }^{\mathrm{T}}\right),\end{array}$$

(28)

where ${\mathit{W}}_{\Psi }\in {\mathbb{R}}^{8\times N}$, ${\mathit{b}}_{\Psi }\in {\mathbb{R}}^N$. ${\mathit{X}}_{in}$ is projected into N sets of query, key, and value matrices $({\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i)$, as defined in Equation (5), containing learnable ${\mathit{W}}_{Q^i},{\mathit{W}}_{K^i},{\mathit{W}}_{V^i},{\mathit{b}}_{Q^i},{\mathit{b}}_{K^i},{\mathit{b}}_{V^i}$. The QEAttention for head i produces ${\mathit{Head}}^i\in {\mathbb{R}}^{L\times S}$, as follows:

$$\begin{array}{cc}\hfill {\mathit{Head}}^i& =\mathrm{QEAttention}\left({\Psi }^i,{\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i\right)\hfill \\ \hfill & :=\mathrm{Softmax}\left({\displaystyle \frac{\left(\left({\Psi }^i·{\mathbf{1}}_S^{\mathrm{T}}\right)\circ {\mathit{Q}}^i\right)·{\left({\mathit{K}}^i\right)}^{\mathrm{T}}}{\sqrt{S}}}\right)·{\mathit{V}}^i,\hfill \end{array}$$

(29)

where ${\Psi }^i$ represents the i-th column vector of $\Psi$.

Finally, let $\mathit{Head}=\mathrm{Concat}\left({\mathit{Head}}^1,{\mathit{Head}}^2,\cdots ,{\mathit{Head}}^N\right)$, and the query-enhanced multi-head self-attention is completed through a linear transformation (containing learnable ${\mathit{W}}_o\in {\mathbb{R}}^{H\times H},{\mathit{b}}_o\in {\mathbb{R}}^H$) in the same way as described in Equation (7), yielding ${\mathit{X}}_{msa}$.

3.3. Forward Propagation Algorithm

After introducing the two improvements, the complete forward propagation algorithm of CQEformer is summarized in Algorithm 1.

3.4. Parameter Optimization

This section details the parameter optimization process of CQEformer during training. We first derive the gradients for all CQEformer modules. Then, we present the parameter update scheme based on the Adam optimizer.

3.4.1. Gradient Derivation

In traditional encoder-only Transformer for time series forecasting (with a single encoder), there are $14+6N$ learnable parameter groups (N denotes the number of attention heads), each represented by a vector or matrix. These include weight matrices in the set $\left\{{\mathit{W}}_e,{\mathit{W}}_{Q^i},{\mathit{W}}_{K^i},{\mathit{W}}_{V^i},{\mathit{W}}_o,{\mathit{W}}_{f_1},{\mathit{W}}_{f_2},{\mathit{W}}_y|i=1,2,\cdots ,N\right\}$, bias vectors in the set $\left\{{\mathit{b}}_e,{\mathit{b}}_{Q^i},{\mathit{b}}_{K^i},{\mathit{b}}_{V^i},{\mathit{b}}_o,{\mathit{b}}_{f_1},{\mathit{b}}_{f_2},{\mathit{b}}_y|i=1,2,\cdots ,N\right\}$ and four vectors ${\mathit{\gamma }}_{ln},{\mathit{\gamma }}_{en},{\mathit{\beta }}_{ln},{\mathit{\beta }}_{en}$. For CQEformer, a total of $20+6N$ parameter groups must be updated, which includes the six additional ${\mathit{W}}_c,{\mathit{W}}_{\Psi },{\mathit{b}}_c,{\mathit{b}}_{\Psi },{\mathit{\gamma }}_{\mathit{\Pi }},{\mathit{\beta }}_{\mathit{\Pi }}$ from the CRE and QEAttention modules.

Algorithm 1 CQEformer Forecasting Procedure

Require: Time series $\mathit{X}\in {\mathbb{R}}^{L\times H}$, forecast horizon $L^{\prime }$, model dimension H, encoder numbers E, head numbers N Ensure: Forecast $\widehat{\mathit{y}}\in {\mathbb{R}}^{L^{\prime }}$ 1: Step 1: Input Layer $${\mathit{X}}_{in}=\left(\mathit{X}+\mathrm{CausalConv}\left(\mathit{X}\right)\right)·{\mathit{W}}_e+{\mathbf{1}}_L·{\mathit{b}}_e^{\mathrm{T}}+\mathit{PE}$$ 2: Step 2: CQEformer Encoder Layer 3: for each encoder layer $(i=1,\dots ,E)$ do 4:     Compute statistical prior information $\mathit{\Pi }$: $$\mathit{\Pi }=\mathrm{LayerNorm}\left(\mathrm{Concat}\left({\mathit{\mu }}_t,{\mathit{\sigma }}_t,{\mathit{\tau }}_t,{\mathit{ϵ}}_t,{\mathit{\mu }}_s,{\mathit{\sigma }}_s,{\mathit{\tau }}_s,{\mathit{ϵ}}_s\right)\right)\in {\mathbb{R}}^{L\times 8}$$ 5:     Compute statistical prior weight matrix $\Psi$: $$\Psi =\mathrm{Tan}\mathrm{h}\left(\mathit{\Pi }·{\mathit{W}}_{\Psi }+{\mathbf{1}}_L·{\mathit{b}}_{\Psi }^{\mathrm{T}}\right)$$ 6:     Compute QEAttention mechanism: 7:     for each head $(i=1,\dots ,N)$ do $${\mathit{Q}}^i={\mathit{X}}_{in}·{\mathit{W}}_{Q^i}+{\mathbf{1}}_L·{\mathit{b}}_{Q^i}^{\mathrm{T}},{\mathit{K}}^i={\mathit{X}}_{in}·{\mathit{W}}_{K^i}+{\mathbf{1}}_L·{\mathit{b}}_{K^i}^{\mathrm{T}},{\mathit{V}}^i={\mathit{X}}_{in}·{\mathit{W}}_{V^i}+{\mathbf{1}}_L·{\mathit{b}}_{V^i}^{\mathrm{T}},$$ $${\mathit{Head}}^i=\mathrm{Softmax}\left({\displaystyle \frac{\left(\left({\Psi }^i·{\mathbf{1}}_S^{\mathrm{T}}\right)\circ {\mathit{Q}}^i\right)·{\left({\mathit{K}}^i\right)}^{\mathrm{T}}}{\sqrt{S}}}\right)·{\mathit{V}}^i$$ $${\mathit{X}}_{msa}=\mathrm{Concat}\left({\mathit{Head}}^1,{\mathit{Head}}^2,\cdots ,{\mathit{Head}}^N\right)·{\mathit{W}}_o+{\mathbf{1}}_L·{\mathit{b}}_o^{\mathrm{T}}$$ 8:     end for 9:     Apply first residual connection and layer normalization: $${\mathit{X}}_{ln}=\mathrm{LayerNorm}\left({\mathit{X}}_{in}+{\mathit{X}}_{msa}\right)$$ 10:     Apply feedforward network, second residual connection and layer normalization: $${\mathit{X}}_{en}=\mathrm{LayerNorm}\left({\mathit{X}}_{ln}+\mathrm{FFN}\left({\mathit{X}}_{ln}\right)\right)$$ 11: end for 12: Step 3: Output Layer $$\widehat{\mathit{y}}={\mathit{W}}_y·{\mathit{x}}_L+{\mathit{b}}_y$$ 13: return  $\widehat{\mathit{y}}$

The Mean Squared Error (MSE) is used as the loss function, as Equation (30). For any parameter $\mathit{X}$, let ${\delta }_{\mathit{X}}$ represent the gradient of $\mathcal{L}(\mathit{y},\widehat{\mathit{y}})$ with respect to $\mathit{X}$. Then ${\delta }_{\widehat{\mathit{y}}}$ can be computed as Equation (31):

$$\mathcal{L}(\mathit{y},\widehat{\mathit{y}})={\displaystyle \frac{1}{L^{\prime }}}{∥\mathit{y}-\widehat{\mathit{y}}∥}_2^2,$$

(30)

$${\delta }_{\widehat{\mathit{y}}}={\displaystyle \frac{2}{L^{\prime }}}(y-\widehat{y}).$$

(31)

In output layer, ${\delta }_{{\mathit{W}}_y}$, ${\delta }_{{\mathit{b}}_y}$ and ${\delta }_{{\mathit{X}}_{en}}$ need to be computed as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_y}={\delta }_{\widehat{\mathit{y}}}·x_y^{\mathrm{T}},\hfill \\ {\delta }_{{\mathit{b}}_y}={\delta }_{\widehat{\mathit{y}}},\hfill \\ {\delta }_{{\mathit{X}}_{en}}=\left(0;0;\cdots ;0;{\delta }_{\widehat{\mathit{y}}}^{\mathrm{T}}·{\mathit{W}}_y\right).\hfill \end{array}\right.$$

(32)

There are 12 learnable parameter groups in a single encoder. First, we present the derivation of gradients for a general LayerNorm. For $\mathit{X}\in {\mathbb{R}}^{L\times H}$, let $\mathit{Y}=\mathrm{LayerNorm}\left(\mathit{X}\right)$, we define centering operator $\mathit{M}=\mathit{I}-{\displaystyle \frac{1}{H}}{\mathbf{1}}_H·{\mathbf{1}}_H^{\mathrm{T}}$ ($\mathit{I}$ is the identity matrix) and normalized matrix $\tilde{\mathit{X}}=(\mathit{X}·\mathit{M})\oslash (\mathit{\sigma }·{\mathbf{1}}_H^{\mathrm{T}})$. These gradients in LayerNorm are given as follows:

$$\left\{\begin{array}{c}{\delta }_{\mathit{\gamma }}={\mathbf{1}}_L^{\mathrm{T}}·({\delta }_{\mathit{Y}}\circ \tilde{\mathit{X}}),\hfill \\ {\delta }_{\mathit{\beta }}={\mathbf{1}}_L^{\mathrm{T}}·{\delta }_{\mathit{Y}},\hfill \\ {\delta }_{\mathit{X}}={\delta }_{\mathit{Y}}\circ ({\mathbf{1}}_L·\mathit{\gamma })\oslash {(\mathit{\sigma }·\mathbf{1})}_L·\mathit{M}.\hfill \end{array}\right.$$

(33)

Using the method outlined in Equation (33), for the backpropagation of second Res & LayerNorm, we can obtain ${\delta }_{{\mathit{\gamma }}_{en}}$, ${\delta }_{{\mathit{\beta }}_{en}}$ and ${\delta }_{\mathit{Z}}$, where $\mathit{Z}={\mathit{X}}_{ln}+\mathrm{FFN}\left({\mathit{X}}_{ln}\right)$. Additional gradients ${\delta }_{{\mathit{W}}_{f_1}}$, ${\delta }_{{\mathit{W}}_{f_2}}$, ${\delta }_{{\mathit{b}}_{f_1}}$, ${\delta }_{{\mathit{b}}_{f_2}}$ and ${\delta }_{{\mathit{X}}_{ln}}$ can be computed as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_{f_1}}={\mathit{X}}_{ln}^{\mathrm{T}}·\left({\delta }_{\mathit{Z}}·{\mathit{W}}_{f_2}^{\mathrm{T}}\circ {\mathbb{I}}_{(\mathit{T}>0)}\right),\hfill \\ {\delta }_{{\mathit{b}}_{f_1}}={\left({\delta }_{\mathit{Z}}^{\mathrm{T}}·{\mathit{W}}_{f_2}^{\mathrm{T}}\circ {\mathbb{I}}_{(\mathit{T}>0)}\right)}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \\ {\delta }_{{\mathit{W}}_{f_2}}={\left(\mathrm{ReLU}\left(\mathit{T}\right)\right)}^{\mathrm{T}}·{\delta }_{\mathit{Z}},\hfill \\ {\delta }_{{\mathit{b}}_{f_2}}={\delta }_{\mathit{Z}}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \\ {\delta }_{{\mathit{X}}_{ln}}={\delta }_{\mathit{Z}}+{\delta }_{\mathit{Z}}·{\mathit{W}}_{f_1}^{\mathrm{T}},\hfill \end{array}\right.$$

(34)

where $\mathit{T}={\mathit{X}}_{ln}·{\mathit{W}}_{f_1}+{\mathbf{1}}_L·{\mathit{b}}_{f_1}^{\mathrm{T}}$ and ${\mathbb{I}}_{(\mathit{T}>0)}$ denotes a matrix of the same size as $\mathit{T}$, where each element is 1 when ${\mathit{T}}_{ij}>0$, and 0 otherwise. Similarly, we can also obtain the gradients ${\delta }_{{\mathit{\gamma }}_{ln}}$, ${\delta }_{{\mathit{\beta }}_{ln}}$, ${\delta }_{{\mathit{X}}_{msa}}$ and ${\delta }_{{\mathit{X}}_{in}}^{\left(ln\right)}$ in first Res & LayerNorm, among which ${\delta }_{{\mathit{X}}_{in}}^{\left(ln\right)}$ denotes a part of the gradient of the loss with respect to ${\mathit{X}}_{in}$ that originates from the LayerNorm.

Next, the computation methods of the gradients in QEAttention will be provided. The gradients in the final linear transformation therein are computed as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_o}={\mathit{Head}}^{\mathrm{T}}·{\delta }_{{\mathit{X}}_{msa}},\hfill \\ {\delta }_{{\mathit{b}}_o}={\delta }_{{\mathit{X}}_{msa}}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \\ {\delta }_{\mathit{Head}}={\delta }_{{\mathit{X}}_{msa}}·{\mathit{W}}_o^{\mathrm{T}},\hfill \end{array}\right.$$

(35)

and we can further obtain ${\delta }_{{\mathit{Head}}^i}$ since ${\delta }_{\mathit{Head}}=\mathrm{Concat}({\delta }_{{\mathit{Head}}^1},{\delta }_{{\mathit{Head}}^2},\cdots ,{\delta }_{{\mathit{Head}}^N})$.

For Equation (29), let ${\mathit{T}}^i=\left({\Psi }^i·{\mathbf{1}}_S^{\mathrm{T}}\circ {\mathit{Q}}^i\right)·{\left({\mathit{K}}^i\right)}^{\mathrm{T}}$ and $\mathit{A}=\mathrm{Softmax}\left({\displaystyle \frac{{\mathit{T}}^i}{\sqrt{S}}}\right)$, we can compute ${\delta }_{{\mathit{T}}^i}={\displaystyle \frac{{\mathit{T}}^i}{\sqrt{S}}}\mathit{A}\circ \left({\delta }_{{\mathit{Head}}^i}·{\left({\mathit{V}}^i\right)}^{\mathrm{T}}-\mathit{A}\circ ({\delta }_{{\mathit{Head}}^i}·{\left({\mathit{V}}^i\right)}^{\mathrm{T}})·{\mathbf{1}}_S·{\mathbf{1}}_S^{\mathrm{T}}\right)$, and then we can compute the following:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{Q}}^i}=({\delta }_{{\mathit{T}}^i}·{\mathit{K}}^i)\circ ({\Psi }^i·{\mathbf{1}}_S^{\mathrm{T}}),\hfill \\ {\delta }_{{\mathit{K}}^i}={\delta }_{{\mathit{T}}^i}^{\mathrm{T}}·({\delta }_{{\mathit{T}}^i}·{\mathit{K}}^i),\hfill \\ {\delta }_{{\mathit{V}}^i}=\mathit{A}·{\delta }_{{\mathit{Head}}^i},\hfill \\ {\delta }_{{\Psi }^i}=({\delta }_{{\mathit{T}}^i}·{\mathit{K}}^i)\circ {\mathit{Q}}^i·{\mathbf{1}}_S,\hfill \end{array}\right.$$

(36)

further we obtain ${\delta }_{\Psi }=({\delta }_{{\Psi }^1},{\delta }_{{\Psi }^2},\cdots ,{\delta }_{{\Psi }^N})$. Thus, ${\delta }_{{\mathit{W}}_{\Psi }},{\delta }_{{\mathit{b}}_{\Psi }}$, ${\delta }_{\mathit{\Pi }}$ can be computed as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_{\Psi }}={\mathit{\Pi }}^{\mathrm{T}}·\left(\left({\mathbf{1}}_L·{\mathbf{1}}_H^{\mathrm{T}}-{\Psi }^{\circ 2}\right)\circ {\delta }_{\Psi }\right),\hfill \\ {\delta }_{{\mathit{b}}_{\Psi }}={\left(\left({\mathbf{1}}_L·{\mathbf{1}}_H^{\mathrm{T}}-{\Psi }^{\circ 2}\right)\circ {\delta }_{\Psi }\right)}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \\ {\delta }_{\mathit{\Pi }}=\left(\left({\mathbf{1}}_L·{\mathbf{1}}_H^{\mathrm{T}}-{\Psi }^{\circ 2}\right)\circ {\delta }_{\Psi }\right)·{\mathit{W}}_{\Psi }^{\mathrm{T}},\hfill \end{array}\right.$$

(37)

${\delta }_{{\mathit{\gamma }}_{\mathit{\Pi }}},{\delta }_{{\mathit{\beta }}_{\mathit{\Pi }}},{\delta }_{\mathit{\Lambda }}$ can be computed based on ${\delta }_{\mathit{\Pi }}$ using the method as Equation (33), where $\mathit{\Lambda }=\left({\mathit{\mu }}_t,{\mathit{\sigma }}_t,{\mathit{\tau }}_t,{\mathit{ϵ}}_t,{\mathit{\mu }}_s,{\mathit{\sigma }}_s,{\mathit{\tau }}_s,{\mathit{ϵ}}_s\right)$, and then ${\delta }_{{\mathit{\mu }}_t},{\delta }_{{\mathit{\sigma }}_t},{\delta }_{{\mathit{\tau }}_t},{\delta }_{{\mathit{ϵ}}_t},{\delta }_{{\mathit{\mu }}_s},{\delta }_{{\mathit{\sigma }}_s},{\delta }_{{\mathit{\tau }}_s},{\delta }_{{\mathit{ϵ}}_s}$ can be obtained.

Let ${\delta }_{{\mathit{X}}_{in}}^{\left(qkv\right)}$ denote the partial gradients of ${\mathit{X}}_{in}$ that originate from the generation process of ${\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i$. Next, we present the form of the gradients in the generation process of ${\mathit{Q}}^i,{\mathit{K}}^i,{\mathit{V}}^i$, as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_{{\mathit{Q}}^i}}={\mathit{X}}_{in}^{\mathrm{T}}·{\delta }_{{\mathit{Q}}^i},\hfill \\ {\delta }_{{\mathit{b}}_{{\mathit{Q}}^i}}={\delta }_{{\mathit{Q}}^i}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \end{array}\right.\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_{{\mathit{K}}^i}}={\mathit{X}}_{in}^{\mathrm{T}}·{\delta }_{{\mathit{K}}^i},\hfill \\ {\delta }_{{\mathit{b}}_{{\mathit{K}}^i}}={\delta }_{{\mathit{K}}^i}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \end{array}\right.\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_{{\mathit{V}}^i}}={\mathit{X}}_{in}^{\mathrm{T}}·{\delta }_{{\mathit{V}}^i},\hfill \\ {\delta }_{{\mathit{b}}_{{\mathit{V}}^i}}={\delta }_{{\mathit{V}}^i}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \end{array}\right.$$

(38)

$${\delta }_{{\mathit{X}}_{in}}^{\left(qkv\right)}=\sum _{i=1}^N({\delta }_{{\mathit{W}}_{{\mathit{Q}}^i}}·{\mathit{W}}_{{\mathit{Q}}^i}^{\mathrm{T}}+{\delta }_{{\mathit{W}}_{{\mathit{K}}^i}}·{\mathit{W}}_{{\mathit{K}}^i}^{\mathrm{T}}+{\delta }_{{\mathit{W}}_{{\mathit{V}}^i}}·{\mathit{W}}_{{\mathit{V}}^i}^{\mathrm{T}}).$$

(39)

To obtain the gradients of the last four parameter groups ${\mathit{W}}_c,{\mathit{W}}_e,{\mathit{b}}_c,{\mathit{b}}_e$ whose gradients originate from the loss with respect to the encoder’s input ${\mathit{X}}_{in}$, we first need to compute ${\delta }_{{\mathit{X}}_{in}}$ that integrates ${\delta }_{{\mathit{X}}_{in}}^{\left(ln\right)},{\delta }_{{\mathit{X}}_{in}}^{\left(qkv\right)}$, and ${\delta }_{{\mathit{X}}_{in}}^{\left(\mathit{\Lambda }\right)}$; the latter refers to the partial gradients of ${\mathit{X}}_{in}$ originating from the generation process of $\mathit{\Lambda }$. To do this, we first present the general method—one applicable to both temporal and frequency domains—for obtaining the gradient of input, given the precomputed gradients of statistical moments and entropy.

For a matrix $\mathit{X}\in {\mathbb{R}}^{L\times H}$ from time or frequency domain, given these statistics’ gradients ${\delta }_{\mathit{\mu }},{\delta }_{\mathit{\sigma }},{\delta }_{\mathit{\tau }},{\delta }_{\mathit{ϵ}}$, we can obtain the partial gradients of $\mathit{X}$ originating from the calculation of these statistics as follows:

$$\left\{\begin{array}{c}{\delta }_{\mathit{X}}^{\left(\mathit{\mu }\right)}={\displaystyle \frac{1}{H}}{\delta }_{\mathit{\mu }}·{\mathbf{1}}_H^{\mathrm{T}},\hfill \\ {\delta }_{\mathit{X}}^{\left(\mathit{\sigma }\right)}={\displaystyle \frac{1}{H}}\left(\left({\delta }_{\mathit{\sigma }}\oslash \mathit{\sigma }\right)·{\mathbf{1}}_H^{\mathrm{T}}\right)\circ \left(\mathit{X}·\mathit{M}\right),\hfill \\ {\delta }_{\mathit{X}}^{\left(\mathit{\tau }\right)}={\displaystyle \frac{3}{H}}\left({\delta }_{\mathit{\tau }}·{\mathbf{1}}_H^{\mathrm{T}}\right)\circ {\mathit{S}}^{\circ 2}\oslash \left(\mathit{\sigma }·{\mathbf{1}}_H^{\mathrm{T}}\right)·\left(\mathit{M}-\mathit{S}·{\mathbf{1}}_H\oslash \mathit{\sigma }·{\mathbf{1}}_H^{\mathrm{T}}\circ \mathit{X}·{\mathit{M}}^2\right),\hfill \\ {\delta }_{\mathit{X}}^{\left(\mathit{ϵ}\right)}=-{\left({\displaystyle \frac{\partial \mathit{P}}{\partial \mathit{X}}}\right)}^{\mathrm{T}}·\left({\delta }_{\mathit{ϵ}}·{\mathbf{1}}_H^{\mathrm{T}}\right)\circ \left(\log \left(\mathit{P}\right)+{\mathbf{1}}_L·{\mathbf{1}}_H^{\mathrm{T}}\right),\hfill \end{array}\right.$$

(40)

where $\mathit{S}=\left(\mathit{X}·\mathit{M}\right)\oslash \left(\mathit{\sigma }·{\mathbf{1}}_H^{\mathrm{T}}\right)$, $\mathit{P}$ is the probability matrix, namely ${\mathit{P}}_t$ in time domain and ${\mathit{P}}_s$ in frequency, ${\displaystyle \frac{\partial {\mathit{P}}_t}{\partial \mathit{X}}}=\mathit{P}-\mathit{P}\circ \left(\mathit{P}·{\mathbf{1}}_H·{\mathbf{1}}_H^{\mathrm{T}}\right)$ in time domain, and ${\displaystyle \frac{\partial {\mathit{P}}_s}{\partial \mathit{X}}}=\left(\mathit{X}·{\mathbf{1}}_H·{\mathbf{1}}_H^{\mathrm{T}}\right)\circ \left({\mathbf{1}}_L·{\mathbf{1}}_H^{\mathrm{T}}-{\mathit{P}}_s·{\mathbf{1}}_H·{\mathbf{1}}_H^{\mathrm{T}}\right)$.

Based on Equation (40), we can compute ${\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\mu }}_{\mathit{t}}\right)}$, ${\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\sigma }}_{\mathit{t}}\right)}$, ${\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\tau }}_{\mathit{t}}\right)}$, ${\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{ϵ}}_{\mathit{t}}\right)}$, ${\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\mu }}_{\mathit{s}}\right)}$, ${\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\sigma }}_{\mathit{s}}\right)}$, ${\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\tau }}_{\mathit{s}}\right)}$, ${\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{ϵ}}_{\mathit{s}}\right)}$, and further we can obtain ${\delta }_{{\mathit{X}}_{in}}^{\left(t\right)},{\delta }_{{\mathit{X}}_{in}}^{\left(s\right)}$, which are the partial gradients of ${\mathit{X}}_{in}$ originating from both time and frequency domains, as follows:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{X}}_{in}}^{\left(t\right)}={\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\mu }}_{\mathit{t}}\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\sigma }}_{\mathit{t}}\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{\tau }}_{\mathit{t}}\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left({\mathit{ϵ}}_{\mathit{t}}\right)},\hfill \\ {\delta }_{{\mathit{X}}_{in}}^{\left(s\right)}=2{\Omega }^{\mathrm{T}}·\Omega ·{\mathit{X}}_{in}\circ \left({\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\mu }}_{\mathit{s}}\right)}+{\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\sigma }}_{\mathit{s}}\right)}+{\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{\tau }}_{\mathit{s}}\right)}+{\delta }_{{\mathit{X}}_{ps}}^{\left({\mathit{ϵ}}_{\mathit{s}}\right)}\right).\hfill \end{array}\right.$$

(41)

Thus, we can obtain ${\delta }_{{\mathit{X}}_{in}}^{\mathit{\Lambda }}={\delta }_{{\mathit{X}}_{in}}^{\left(t\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left(s\right)}$ and then ${\delta }_{{\mathit{X}}_{in}}={\delta }_{{\mathit{X}}_{in}}^{\left(ln\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left(qkv\right)}+{\delta }_{{\mathit{X}}_{in}}^{\left(\mathit{\Lambda }\right)}$. The gradients of the final four parameter groups, ${\delta }_{{\mathit{W}}_c},{\delta }_{{\mathit{W}}_e},{\delta }_{{\mathit{b}}_c},{\delta }_{{\mathit{b}}_e}$, are given below, where ${\delta }_{{\mathit{W}}_c}$ and ${\delta }_{{\mathit{b}}_c}$ are defined element-wise:

$$\left\{\begin{array}{c}{\delta }_{{\mathit{W}}_e}={\left(\mathit{X}+\mathrm{CausalConv}\left(X\right)\right)}^{\mathrm{T}}·{\mathit{X}}_e,\hfill \\ {\delta }_{{\mathit{b}}_e}={\left({\mathit{X}}_e\right)}^{\mathrm{T}}·{\mathbf{1}}_L,\hfill \end{array}\right.\left\{\begin{array}{c}{\delta }_{{\left({\mathit{W}}_c\right)}_{k,d,j}}=\sum _{i=1}^L({\mathbb{I}}_{i-k+1\ge 1}·{\left({\delta }_c\right)}_{i,j}·{\mathit{X}}_{i^{\prime }}),\hfill \\ {\delta }_{{\left({\mathit{b}}_c\right)}_j}=\sum _{i=1}^L\left({\left({\delta }_c\right)}_{i,j}\right).\hfill \end{array}\right.$$

(42)

3.4.2. Parameter Update

Given the gradients derived above, CQEformer’s parameters are updated using the Adam optimizer  [25]. For parameter $\mathit{\Theta }$, given the gradient of the loss with respect to $\mathit{\Theta }$ at t step, ${\delta }_{\mathit{\Theta }}^{\left(t\right)}$, the calculation method is as follows:

$${\mathit{M}}^{\left(t\right)}={\beta }_1{\mathit{M}}^{(t-1)}+(1-{\beta }_1){\delta }_{\mathit{\Theta }}^{\left(t\right)},$$

(43)

$${\mathit{V}}^{\left(t\right)}={\beta }_2{\mathit{V}}^{(t-1)}+(1-{\beta }_2){\left({\delta }_{\mathit{\Theta }}^{\left(t\right)}\right)}^{\circ 2},$$

(44)

$$\widehat{{\mathit{M}}^{\left(t\right)}}={\displaystyle \frac{{\mathit{M}}^{\left(t\right)}}{1-{\beta }_1^t}},\widehat{{\mathit{V}}^{\left(t\right)}}={\displaystyle \frac{{\mathit{V}}^{\left(t\right)}}{1-{\beta }_2^t}},$$

(45)

$${\mathit{W}}^{(t+1)}={\mathit{W}}^{\left(t\right)}-\alpha {\displaystyle \frac{\widehat{{\mathit{M}}^{\left(t\right)}}}{\sqrt{\widehat{{\mathit{V}}^{\left(t\right)}}}+ϵ}},$$

(46)

where ${\beta }_1,{\beta }_2$ are the exponential decay rates, $\alpha$ is the learning rate, and $ϵ$ is a very small constant added to avoid division by zero. Utilizing this method presented in Equation (46), all parameters in CQEformer can be updated in a similar manner. The ReduceLROnPlateau scheduler is employed to dynamically decrease the learning rate once the validation loss stops improving, ensuring smoother convergence and enhanced training stability.

4. Experiments

To evaluate CQEformer’s performance, we performed three experiments—multi-model comparison, time-window sensitivity analysis, and ablation studies—primarily on financial time series, which are representative of datasets exhibiting structural breaks and volatility clustering. In addition, we conducted supplementary experiments on the public Traffic dataset to assess cross-domain generalization under nonstationary conditions.

4.1. Data, Indicators and Standardization

The dataset consists of twenty years of historical transaction records for the CSI 300 Index from 13 June 2006, to 12 June 2025. Since the Bank of Communications was listed in 2007, we additionally use its nineteen years of historical stock transaction data from 13 June 2007 to 12 June 2025. Both datasets are retrieved from AKShare  [26]. Ten raw indicators are considered (Open, Close, High, Low, Volume, Turnover, Amplitude, Change, ChangeAmount, TurnoverRate) and 17 technical indicators are further constructed (SMA(10), EMA(12), EMA(26), DEMA(10), TEMA(10), WMA(10), MACD, Signal, Histogram, RSI(14), ROC(12), MOM(10), CCI(20), WILLR(14), CMO(14), K(14), D). The definitions for these indicators are provided in

Table 1. Technical Indicator Definitions and Calculation Methods.

Indicator Name	Formula
Simple Moving Average	${\mathrm{SMA}}_t\left(n\right)={\displaystyle \frac{1}{n}}{\sum }_{i=0}^{n-1}{P}_{t-i}$
Exponential Moving Average	${\mathrm{EMA}}_t\left(n\right)=\alpha ·P_t+(1-\alpha )·{\mathrm{EMA}}_{t-1}\left(n\right)$
Double Exponential MA	$\mathrm{DEMA}\left(n\right)=2\mathrm{EMA}\left(n\right)-{\mathrm{EMA}}^{\left(2\right)}\left(n\right)$
Triple Exponential MA	$\mathrm{TEMA}\left(n\right)=3\mathrm{EMA}\left(n\right)-3{\mathrm{EMA}}^{\left(2\right)}\left(n\right)+{\mathrm{EMA}}^{\left(3\right)}\left(n\right)$
Weighted Moving Average	$\mathrm{WMA}\left(n\right)={\displaystyle \frac{{\sum }_{i=0}^{n-1}(i+1)P_{t-i}}{{\sum }_{i=0}^{n-1}(i+1)}}$
MACD	${\mathrm{MACD}}_t={\mathrm{EMA}}_t\left(12\right)-{\mathrm{EMA}}_t\left(26\right)$
MACD Signal Line	${\mathrm{Signal}}_t={\mathrm{EMA}}_t$
MACD Histogram	${\mathrm{Histogram}}_t={\mathrm{MACD}}_t-{\mathrm{Signal}}_t$
Relative Strength Index	${\mathrm{RSI}}_t\left(n\right)=100-{\displaystyle \frac{100{\mathrm{SumLoss}}_t\left(n\right)}{{\mathrm{SumGain}}_t\left(n\right)}}$
Rate of Change	${\mathrm{ROC}}_t\left(n\right)={\displaystyle \frac{P_t-P_{t-n}}{P_{t-n}}}\times 100\%$
Momentum	${\mathrm{MOM}}_t\left(n\right)=P_t-P_{t-n}$
Commodity Channel Index	${\mathrm{CCI}}_t\left(n\right)={\displaystyle \frac{T{P}_t-\mathrm{SMA}{\left[TP\right]}_t\left(n\right)}{0.015{\mathrm{MD}}_t\left(n\right)}}$
Williams R	${\mathrm{WILLR}}_t\left(n\right)={\displaystyle \frac{P_t-{max}_{i=0}^{n-1}{\mathrm{High}}_{t-i}}{{max}_{i=0}^{n-1}{\mathrm{High}}_{t-i}-{min}_{i=0}^{n-1}{\mathrm{Low}}_{t-i}}}\times 100$
Chande Momentum Oscillator	${\mathrm{CMO}}_t\left(n\right)={\displaystyle \frac{{\mathrm{SumGain}}_t\left(n\right)-{\mathrm{SumLoss}}_t\left(n\right)}{{\mathrm{SumGain}}_t\left(n\right)+{\mathrm{SumLoss}}_t\left(n\right)}}\times 100$
Stochastic K	${\mathrm{K}}_t\left(n\right)={\displaystyle \frac{P_t-{\mathrm{LowestLow}}_t\left(n\right)}{{\mathrm{HighestHigh}}_t\left(n\right)-{\mathrm{LowestLow}}_t\left(n\right)}}\times 100$
Stochastic D	${\mathrm{D}}_t\left(m\right)=\mathrm{SMA}\left[{\mathrm{K}}_t\left(14\right)\right]\left(m\right)$

Symbol Explanation: $P_t$: price at time t; n: window size; $\alpha ={\displaystyle \frac{2}{n+1}}$ (smoothing factor); ${\mathrm{EMA}}_0={\mathrm{SMA}}_0$; ${\mathrm{EMA}}^{\left(k\right)}$: k-times applied EMA; ${\mathrm{SumGain}}_t\left(n\right)={\sum }_{i=0}^{n-1}max(P_{t-i}-P_{t-i-1},0)$; ${\mathrm{SumLoss}}_t\left(n\right)={\sum }_{i=0}^{n-1}max(P_{t-i-1}-P_{t-i},0)$; ${\mathrm{TP}}_t={\displaystyle \frac{{\mathrm{High}}_t+{\mathrm{Low}}_t+P_t}{3}}$; ${\mathrm{MD}}_t\left(n\right)={\displaystyle \frac{1}{n}}{\sum }_{i=0}^{n-1}\left|{\mathrm{TP}}_{t-i}-\mathrm{SMA}{\left[\mathrm{TP}\right]}_t\left(n\right)\right|$.

. To further assess the generalization capability of CQEformer beyond the financial domain, we also conduct experiments on the public Traffic dataset (1 July 2017 –30 June 2018) [27], which captures hourly road occupancy rates with evident structural shifts and volatility clustering patterns.

Prior to experimentation, all features are range-normalized. Specifically, for the j-th column feature ${\mathit{X}}_j={[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_n]}^{\mathrm{T}}$ of the input matrix (where n is the number of samples), the normalized feature is denoted as ${\mathit{X}}_j^{\prime }={[{\mathit{x}}_1^{\prime },{\mathit{x}}_2^{\prime },\cdots ,{\mathit{x}}_n^{\prime }]}^{\mathrm{T}}$, calculated as

$${\mathit{x}}_i^{\prime }={\displaystyle \frac{{\mathit{x}}_i-min\left({\mathit{X}}_j\right)}{max\left({\mathit{X}}_j\right)-min\left({\mathit{X}}_j\right)}}.$$

(47)

4.2. Experimental Setup

Both multivariate time series datasets are partitioned into samples via a sliding window approach (step size = 1, task-specific window size) and split into 80%, 10%, and 10% for training, validation, and testing sets. Random seed is 2025. Training uses Adam (lr = ${10}^{-4}$, batch size = 32) with a ReduceLROnPlateau scheduler (patience = 5, factor = 0.3, mode = rel, threshold = $5\times {10}^{-5}$) for 50 epochs. Model hyperparameters include a hidden layer dimension of 256 (FFN = 512), dropout = 0.1, encoder depth = 1, and 8 attention heads.

After denormalization, model performance is evaluated using four metrics: mean squared error (MSE), mean absolute percentage error (MAPE), coefficient of determination (${\mathrm{R}}^2$), and training time. Let $\overline{\mathit{y}}$ denote the mean of elements in $\mathit{y}$, MSE, MAPE, and ${\mathrm{R}}^2$ are defined as follows:

$$\mathrm{MSE}={\displaystyle \frac{1}{n}}{\parallel \mathit{y}-\widehat{\mathit{y}}\parallel }_2^2,\mathrm{MAPE}={\displaystyle \frac{1}{n}}{∥{\displaystyle \frac{\mathit{y}-\widehat{\mathit{y}}}{\mathit{y}}}∥}_1,{\mathrm{R}}^2=1-{\displaystyle \frac{\parallel \mathit{y}-\widehat{\mathit{y}}{\parallel }_2^2}{\parallel \mathit{y}-\overline{\mathit{y}}{\parallel }_2^2}}.$$

4.3. Multi-Model Comparison Experiment

In this experiment, for two financial datasets, the time window size is fixed at 60 (approximately one quarter). We choose representative baseline models in the field of time-series prediction, including LSTM, GRU, TCN, and Transformer. These models are evaluated alongside our proposed CQEformer on all datasets, aim to validate the prediction accuracy of CQEformer and further highlight its advantages over existing approaches. The empirical results are shown in Figure 3 and

Table 2. Multi-indicator prediction effect comparison of multi-models on the test set.

Data	Model	MSE	MAPE	${\mathbf{R}}^2$	Training Time
CSI 300	LSTM	5979.9843	1.7470	0.8967	38.50
	GRU	6791.7403	1.9428	0.8827	37.71
	TCN	5156.9241	1.3713	0.9109	42.26
	Transformer	9720.6929	2.2986	0.8320	68.85
	CQEformer	3839.0661	1.2252	0.9337	106.56
Bank of Communications	LSTM	0.0219	1.6819	0.9582	40.46
	GRU	0.0290	2.0708	0.9446	36.56
	TCN	0.0192	1.5426	0.9633	39.11
	Transformer	0.0430	2.5738	0.9179	64.35
	CQEformer	0.0163	1.3648	0.9688	102.27

.

In the multi-model comparison experiment, CQEformer achieves the best overall performance in trend fitting accuracy, local fluctuation capture, and adaptation to extreme market conditions. This superiority is visually illustrated in Figure 3 and quantitatively supported in Table 2. Across the two datasets, CQEformer reduces the MSE by an average of 38.90% compared to all baseline models. Benefiting from the causal residual embedding mechanism, the model quickly responds to sudden changes, effectively captures long-term fluctuation patterns, and demonstrates substantially higher robustness compared with the baseline models.

In contrast, the Transformer, although effective in trend fitting, exhibits a lagged response and low accuracy in capturing instantaneous fluctuations. LSTM and GRU, while computationally efficient and stable in trend tracking, lack the ability to capture fine-grained details. TCN, which excels in local feature extraction, is overly sensitive to sudden changes, leading to overreactions and reduced long-term prediction robustness.

Overall, CQEformer achieves a good balance between training time and predictive accuracy. It demonstrates clear advantages in high-precision forecasting scenarios and offers substantial potential for practical applications.

In addition to financial datasets, we also evaluated CQEformer on the public Traffic dataset to examine its cross-domain adaptability. Although traffic flow data differ from financial series in semantics, they similarly exhibit structural shifts and volatility clustering caused by external disturbances. In this experiment, the time-window length was set to 48, corresponding to approximately two days of data, to capture short-term fluctuation patterns. CQEformer achieved the best overall performance (${\mathrm{R}}^2=0.9299$, MSE = $4.35\times {10}^{-5}$), reducing the average MSE of baseline models by approximately 27.29%, confirming its robustness and transferability to non-financial time series.

4.4. Time Window Sensitivity Experiment

Time window length is a key hyperparameter in time-series prediction tasks, directly affecting the model’s ability to capture historical information. To evaluate its impact for forecasting and to further verify the robustness of the proposed model, we conduct a time window sensitivity analysis by varying the window size from 10 to 100 with a step of 10, and compare the prediction performance of traditional encoder-only temporal Transformer and CQEformer.

Figure 4 summarizes the time-window sensitivity experiment. As the window size changes, CQEformer exhibits consistently low prediction error and strong stability, reducing MSE variance by 94.04% compared to the traditional Transformer, demonstrating robust adaptability to variations in time window size.

4.5. Ablation Experiment

For the ablation study, we use the CSI 300 dataset with a time window of 60 and compare four variants: Encoder-Only Transformer (baseline), CREformer (causal residual embedding), QEformer (QEAttention), and CQEformer. The aim is to assess each module’s contribution, their combined effects, and validate the model design. Figure 5 and

Table 3. Multi-indicator performance of ablation variants.

Model	MSE	MAPE	${\mathbf{R}}^2$	Training Time
Transformer	9720.6929	2.2986	0.8320	66.63
CREformer	5946.8689	1.6355	0.8972	73.53
QEformer	6516.6724	1.8091	0.8874	96.61
CQEformer	3839.0661	1.2252	0.9337	106.12

summarize each module’s effect on the CSI 300 dataset (window size 60).

CREformer reduces MSE by 38.82% vs. Transformer, increases ${\mathrm{R}}^2$ by 7.84%, and decreases MAPE by 28.85%, confirming its role in capturing temporal dependencies. QEformer also outperforms the baseline with MSE reduced by 32.96%, ${\mathrm{R}}^2$ increased by 6.66%, and MAPE decreased by 21.30%, demonstrating the module’s independent value. CQEformer achieves the best results: MSE down by 60.51%, ${\mathrm{R}}^2$ up by 12.22%, and MAPE down by 46.70%; relative to CREformer and QEformer, MSE is further reduced by 35.44% and 41.09%. Although CQEformer requires the longest training time, the substantial improvement in prediction accuracy justifies the extra cost. And CQEformer converges quickly and stably, generalizes well, and outperforms all ablation variants as shown in Figure 6.

To explore the fitting ability of ablation variants under different market risk regions, we design the experiment as follows. Using historical CSI 300 closing prices, we calculate the 20-day rolling volatility as a measure of market risk:

$${\mathrm{Risk}}_t=\mathbb{E}\left[{(r_{t-i}-{\overline{r}}_t)}^2\right],$$

(48)

where $r_{t-i}$ is the return on day $t-i$ ($i=0,\cdots ,19$), and ${\overline{r}}_t={\displaystyle \frac{1}{20}}{\sum }_{i=0}^{19}{r}_{t-i}$.

Test data are divided into three risk regions (stable, mildly volatile, highly volatile) based on the 33rd and 66th percentiles of full-series volatility: Low ($\mathrm{Risk}\le 0.0077$), Medium ($0.0077<\mathrm{Risk}\le 0.0097$), and High ($\mathrm{Risk}>0.0097$) (Figure 7).

Using the risk region classification, we compare the MSE, MAPE, and ${\mathrm{R}}^2$ of the ablation variants, Figure 8 and

Table 4. Fitting performance of ablation variants across volatility regions.

Volatility (Average)	Model	MSE	MAPE	${\mathbf{R}}^2$
Low (0.0064)	Transformer	6680.7169	2.0489	0.8552
	CREformer	3488.8279	1.3186	0.9244
	QEformer	4586.1880	1.7105	0.9006
	CQEformer	2228.9627	0.9931	0.9517
Medium (0.0088)	Transformer	8523.8722	2.1798	0.8122
	CREformer	5165.9218	1.6306	0.8862
	QEformer	5061.9404	1.6978	0.8885
	CQEformer	3233.8762	1.2110	0.9288
High (0.0155)	Transformer	13840.4316	2.6567	0.7718
	CREformer	9097.2318	1.9489	0.8500
	QEformer	9805.1429	2.0127	0.8383
	CQEformer	5993.2861	1.4650	0.9012

show the prediction performance of these four models across low, medium, and high volatility regions. As volatility increases, all models exhibit higher prediction errors and lower accuracy. For CQEformer, MSE rises by 168.88% from low to high volatility, MAPE increases by 47.52%, and ${\mathrm{R}}^2$ drops by 5.31%, indicating that higher market volatility substantially increases prediction difficulty.

Across all regions, CQEformer achieves the best performance. Compared with Transformer, its MSE decreases by 66.64%, 62.06%, and 56.70% in low, medium, and high volatility regions, respectively, demonstrating strong risk adaptability. While the improvement is slightly limited in high-volatility scenarios, CQEformer still effectively handles sudden changes, reflecting its structural robustness to high-noise and nonlinear price sequences. CREformer and QEformer perform similarly, both clearly outperforming Transformer, highlighting the effectiveness of the two enhanced modules in volatility modeling.

5. Conclusions

This study proposes CQEformer, an encoder-only model addressing the limitations of standard Transformers in nonstationary time series with structural breaks and volatility clustering, through two core innovations: causal residual embedding (CRE) and query-enhanced multi-head self-attention (QEAttention).

Key findings validate its effectiveness. In comparative experiments on the CSI 300 Index and Bank of Communications datasets, CQEformer outperforms baselines (LSTM, GRU, TCN, standard Transformer) across core metrics, achieving an average MSE reduction of 38.90%. Ablation studies confirm the modular synergy: CRE and QEAttention individually reduce MSE by 38.82% and 32.96%, respectively, while their combination yields a 60.51% reduction, highlighting the rationale of the dual-module design. CQEformer also demonstrates strong robustness to temporal window variations, outperforming the standard Transformer in adaptability. Evaluation on the public Traffic dataset further verifies its cross-domain generalization.

Theoretically, CQEformer advances nonstationary time-series prediction by integrating causal structure awareness with statistical prior guidance, effectively balancing local detail capture and global context preservation. Practically, its superior performance across financial and traffic domains underscores its potential for high-precision forecasting under dynamic, volatile conditions, while its interpretable framework aids understanding of complex temporal dynamics. Future work will focus on developing lightweight architectures, enhancing cross-domain generalization, improving long-term forecasting, and integrating multi-source information including external indicators.