A Pretrained Spatio-Temporal Hypergraph Transformer for Multi-Stock Trend Forecasting

Abstract

Predicting stock trends has garnered extensive attention from investors and researchers due to its potential to optimize stock investment returns. The fluctuation of stock prices is complex and influenced by multiple factors, presenting two major challenges: the first challenge lies in the the temporal dependence of individual stocks and the spatial correlation among multiple stocks. The second challenge emerges from having insufficient historical data availability for newly listed stocks. To address these challenges, this paper proposes a spatio-temporal hypergraph transformer (STHformer). The proposed model employs a temporal encoder with an aggregation module to capture temporal patterns, utilizes self-attention to dynamically generate hyperedges, and selects cross-attention to implement hypergraph-associated convolution. Furthermore, pretraining based on reconstruction of masked sequences is implemented. This framework enhances the model’s cold-start capability, making it more adaptable to newly listed stocks with insufficient training data. Experimental results show that the proposed model, after pretraining on data from over two thousand stocks, performed well on datasets from the stock markets of the United States and China.

1. Introduction

Stock price prediction, aimed at forecasting future trends in the stock market through the analysis of historical data, is a crucial technique for guiding profitable investment decisions. However, due to the complex interplay of macroeconomic factors, capital flows, significant events, and other variables, stock price series diverge from typical stationary sequences, exhibiting high volatility and chaos. Consequently, accurately predicting stock prices and utilizing the information to formulate intricate investment strategies remains an open question.

Researchers have made extensive efforts to enhance the accuracy of stock price predictions. Most traditional stock price prediction models utilize fundamental statistical methods and machine learning techniques to learn potential patterns from stock price data, such as support vector machines (SVMs) [1], random forests [2], and XGBoost [3]. However, these methods typically rely on analyzing static patterns in historical sequences and thus fail to capture the nonlinear characteristics of stock price series. With the advancement of deep learning, deep neural networks have been widely employed to extract temporal features from stock time series. Researchers have explored and proposed models like long short-term memory (LSTM) [4], graph neural networks (GNNs) [5,6], and Transformers [7,8,9] to better capture the complex patterns and dynamic changes in stock price series.

Existing models often focus primarily on utilizing historical information of a single stock to predict its future trends, overlooking the natural correlations among stocks resulting from industry chains and company attributes. The future price of a stock is intuitively and insightfully influenced by other stocks. If these useful pieces of information can be effectively extracted, predictions would become more accurate. A few studies have attempted to utilize graph networks to capture the information across different stocks. However, these approaches are often oversimplified or require extensive manual efforts. For example, the studies by Feng et al. [10] and Kim et al. [11] on stock correlations were based on predefined industry relationships. On the one hand, traditional graph networks can only model pairwise relationships between stocks, failing to model the real relationships of multiple stocks in the market, while hypergraph networks with hyperedge structures can capture the high-order relationships among stocks. On the other hand, predefined static architectures lack the capacity to learn dynamic inter-stock correlations. Therefore, how to effectively leverage the dynamic relationships among stocks to enhance prediction performance remains a challenge.

In financial markets, new stocks are constantly being listed, and index datasets are also regularly updated, such as the China securities index 300 (CSI300) and national association of securities dealers automated quotations 100 (NASDAQ100). Typically, models rely on existing historical data and market features for training and prediction. However, when dealing with newly listed stocks, these stocks often lack sufficient historical data, which limits the predictive power of the model. Effectively predicting the performance of these “new stocks” has become a pressing problem. Inspired by the recent significant progress in pretraining methods in the natural language processing (NLP) and computer vision (CV) domains [12], various pretrained time series analysis approaches have been proposed [13,14]. The adoption of pretraining paradigms enables deep models to learn prior knowledge from large-scale unlabeled data, further enhancing the performance of various downstream tasks. Currently, there has been limited work on applying pretraining methods to stock prediction. Some researchers have pretrained the encoder of their model by using contrastive learning based on stock price co-movement [15]. However, the behavior of different stocks may not always adhere to similar patterns, especially when dealing with newly listed stocks.

To sum up, research on stock price trend prediction using stock relationships faces two major challenges:

• Traditional methods rely on simple correlations between stock pairs, without considering the complex high-order relationships among multiple stocks. This leads to an inability to capture the complex dynamic relationships in the stock market and results in inefficient utilization of available data.
• Newly listed stocks usually lack historical data, which poses a cold-start problem for traditional models that heavily rely on historical sequence modeling. Pretraining methods, while promising in other fields, face challenges in stock prediction due to the heterogeneous behavior of different stocks. This highlights the need for more adaptive pretraining frameworks tailored to financial data.

To address the aforementioned challenges, inspired by self-attention mechanisms and hypergraph learning technology, we propose a spatio-temporal hypergraph transformer (STHformer). Firstly, a temporal encoder is utilized to capture the essential representation of the sequence, while the temporal aggregation module enhances the model’s overall representation of temporal characteristics. Secondly, we develop a hybrid attention hypergraph module to extract relationships among stocks. Specifically, it generates hyperedges among stocks through self-attention, and then performs hypergraph-related convolution through an improved cross-attention. We formulate and prove Theorem 1 to establish that our proposed architecture is equivalent to a novel hypergraph convolution framework. This framework uniquely enables the modeling of dynamic higher-order relationships among stocks Finally, we adopt a self-supervised masked sequence reconstruction task to pretrain the model, enabling it to adapt to evolving index datasets. Our contributions are summarized as follows:

• We propose a spatio-temporal hypergraph transformer for stock trend prediction. The network considers the temporal correlation of each stock through a temporal encoder, and the spatial correlation between stocks through a hybrid attention hypergraph module, which consists of self-attention and cross-attention mechanisms. The self-attention dynamically generates hyperedges, while the cross-attention performs convolution operations on the hypergraph associations to model complex and high-order stock relationships.
• We design an efficient pretraining and fine-tuning framework, employing a pretraining task that reconstructs masked time series to enhance the model’s generalization ability and reduce its dependency on historical data, thereby making it applicable to newly listed stocks with limited available samples.
• Experiments on the CSI300 and NASDAQ100 datasets demonstrated the effectiveness of STHformer, with superior predictive performance and long-term profitability compared to several state-of-the-art solutions.

2. Related Work

2.1. Spatial-Temporal Models

Time series prediction is a crucial task that has attracted significant attention in recent years, due to its extensive applications. To capture temporal dependencies, Xu et al. [16] devised a Tensorized LSTM with adaptive shared memory capabilities, specifically tailored to grasp long-term temporal dependencies. Meanwhile, Liu et al. [17] developed a prediction framework based on temporal convolutional networks (TCNs), to extract unique and valuable features from time series. Recently, Transformer-based approaches have emerged as a dominant paradigm, due to their ability to model long-range dependencies through self-attention mechanisms. Liu et al. [18] employed a transposed Transformer architecture to capture temporal multivariate correlations. For the problem of long-term time series forecasting, Transformer architectures have achieved significant results. Their effectiveness primarily originates from the self-attention mechanism’s ability to model distant temporal relationships. Zhou et al. [19] proposed the Informer model, which employs a sparse attention matrix to significantly reduce computational complexity, and adopts a generative decoder for prediction. Ren et al. [20] introduced the temporal diffusion Transformer (TDT), which leverages the physical principles of diffusion processes to model the intrinsic evolution of time series, improving prediction performance by incorporating multi-scale periodicity and energy-minimization constraints.

However, the aforementioned models consider each series in isolation, neglecting the complex relationships among them and failing to harness these connections to enhance prediction accuracy. In response to this problem, researchers have explored the use of graph convolutional networks (GCNs) to capture the mutual influence among series by treating time series as nodes in a graph and the relationships between series as edges. Relevant studies have already demonstrated the effectiveness of GCNs in time series forecasting. For instance, Wu et al. [21] improved the graph convolution module and incorporated a graph learning module to learn the relationships among variables from data. To achieve faster training speeds with fewer parameters, Yu et al. [22] proposed a method to grasp the essential temporal features and the most valuable spatial features through graph convolution and gated temporal convolution. Du et al. [23] utilized a LSTM Network with Attention Layer to extract temporal features and self-attention mechanisms to capture spatial relationships. Zhang et al. [24] constructed a spatio-temporal forecasting model leveraging a multi-head attention model, where scaled dot-product attention with positional encoding was used to capture temporal features, and a graph attention network with a multi-head attention mechanism was employed to capture spatial features.

However, graph networks can only capture the spatio-temporal dependencies between node pairs and are unable to handle high-order correlations among time series. To address the aforementioned issue, Wang et al. [25] employed the K-nearest neighbors method to dynamically generate hypergraph structures, which were then utilized in a spatio-temporal hypergraph network. Shang et al. [26] introduced the framework of multi-scale hypergraphs to explore the interactions between temporal patterns at different scales and thereby further improve prediction accuracy.

Numerous research endeavors have demonstrated that designing spatio-temporal deep neural networks that concurrently consider temporal dependencies and spatial relationships can enhance the performance of time series forecasting models. The model proposed in this paper is capable of extracting crucial temporal features, to reduce computation time, while also mining dynamic spatial relationships.

2.2. Self-Supervised Pretraining in Time Series Modeling

In recent years, pretrained models have witnessed significant advancements. The success of autoencoder-based approaches underscores the robust representational power of the masked reconstruction pretraining framework. Nie et al. [27] proposed a method for predicting masked subsequence-level patches, aiming to capture local semantic information, while minimizing memory usage. They employed masked modeling as an auxiliary task to enhance the prediction and classification performance of their Transformer-based methods. By bridging masked modeling with manifold learning, Dong et al. [14] simplified the reconstruction task by combining corrupted yet complementary temporal variations across multiple masked subsequences. Zhou et al. [13] evaluated their approach by freezing the self-attention and feedforward layers in a pretrained Transformer and fine-tuning it across all major types of tasks involving time series analysis. The results demonstrated that the model achieved comparable or state-of-the-art performance across all temporal analysis tasks. Furthermore, Xu et al. [15] introduced a self-supervised method based on contrastive learning, which pretrains a multi-scale encoder through the task of detecting co-movements in stock prices, ultimately utilizing a GRU for the final prediction.

2.3. Stock Price Forecasting

Researchers have investigated various methods using deep neural networks for stock prediction over recent years. Gupta et al. [28] and Hu et al. [29] employed recurrent neural networks to model individual stock price movements. Alternatively, Yao et al. [30] utilized convolutional neural networks to predict short-term trends. To improve the accuracy of stock prediction, various approaches have been explored. Qiu et al. [31] introduced a self-attention mechanism, Ma et al. [32] explored multitask learning approaches, while Li et al. [33] proposed market-gating techniques and Lin et al. [34] introduced a method based on deep reinforcement learning. Later studies achieved remarkable results by taking into account the relationships among stocks. For instance, Sawhney et al. [35] proposed a spatial-temporal hypergraph attention network for stock ranking (STHAN-SR) to rank stocks based on profit. It captures the complex relationships between stocks through hypergraph and temporal attention mechanisms. Huynh et al. [36] utilized a hypergraph to capture non-pairwise correlations with temporal generative filters tailored to individual patterns for each stock. Su et al. [37] employed adaptive hypergraph generation and adaptive spatial-temporal convolution to model the extrinsic and intrinsic relationships of stocks, and for the first time, took noise into account by utilizing noise-aware attention to enhance prediction accuracy.

However, existing stock prediction methods fail to provide accurate predictions for newly listed stocks. The model proposed in this paper is designed to address this issue.

3. Preliminary

Problem Formulation

• Stock Trend Forecasting: Assuming that there are N stocks in the stock market, a set of historical sequence data at day t is represented by $X^t=[x_1^t,x_2^t,\dots ,x_N^t]\in R^{N\times T\times F}$, where T is the length of the sequence, and F is the dimension of the original feature, such as open, high, low, close and volume. Each stock i has historical sequence data $x_i^t\in R^{1\times T\times F}$ at day t. If the closing price of stock $x_i^{t+h}$ is higher than the opening price of $x_i^t$, the stock $x_i$ is labeled with “up” ($y_i^{t+h}=1$), otherwise it is labeled with “down” ($y_i^{t+h}=0$), where h is a specified horizon ahead of the current timestamp. The objective of stock trend forecasting is to establish a mapping relationship, which is defined as follows:

  $$\left[X^t\right]\stackrel{f}{⟶}{\widehat{Y}}^{t+h}$$

  (1)

  where $X^t=[x_1^t,x_2^t,\dots ,x_N^t]$ is the input stock sequence data and ${\widehat{Y}}^{t+h}=[{\widehat{y}}_1^{t+h}$,${\widehat{y}}_2^{t+h},\dots ,{\widehat{y}}_N^{t+h}]$ is the predicted trend for all stocks at day t+h.
• Hypergraph: A hypergraph $G=(V,E,W)$ is constructed to model the interdependence among stocks, where the hyperedges encode higher-order relationships between them. The collection of vertices is denoted by V, while the set of hyperedges is represented by E. Each hyperedge $\epsilon$ is associated with a positive weight w, and these weights are collectively organized in a diagonal matrix $W\in R^{\left|E\right|\times \left|E\right|}$.

4. Model

The overall architecture of our framework, illustrated in Figure 1, is referred to as a pretrained spatio-temporal hypergraph transformer. The essential vectors in the process of the computation are summarized in

Table 1. Nomenclature.

Symbols	Notation
$\overline{E}$	The temporal embedding
$\tilde{E}$	The aggregated temporal embedding
H	The incidence matrix
$Q_H$	The query matrix generated by the incidence matrix
$K_H$	The key matrix generated by the incidence matrix
V	The value matrix generated by the aggregated temporal embedding
$D_e$	The edge degree matrix
$D_v$	The node degree matrix
$X^l$	The hidden embedding of the l-th layer in hypergraph convolution.

. This includes the following four key components:

• Temporal Encoder: This module employs a Transformer encoder to extract the temporal correlations in each stock’s daily data.
• Temporal Aggregation Module: This module integrates information across various time points to produce a localized embedding. This embedding captures all significant signals along the temporal axis, while maintaining the intricate local details of the stock’s behavior over time
• Hybrid Attention Hypergraph Module: This module is designed to explore the spatial relationships between stocks. It comprises two attention layers: the self-attention-based hyperedge construction layer dynamically generates hyperedges, while the cross-attention-based hypergraph convolution layer performs hypergraph convolution on stock nodes based on the generated hyperedge matrix.
• Self-Supervised Pretraining Scheme: This strategy involves reconstructing the original time series from the masked time series, with the objective of learning the valuable low-level information from data in an unsupervised manner.

4.1. Temporal Encoder

In view of the demonstrated efficacy of attention mechanisms in processing time series data, we utilize a Transformer encoder to develop the temporal encoder. The historical sequence of stock i on the trading day t is denoted as $x_i^t\in R^{1\times T\times F}$. It is mapped to the embedding space of dimension U via a learnable linear projection $W_m\in R^{F\times U}$, and sinusoidal position encoding $W_{pos}\in R^{T\times U}$ is incorporated into the embedded input to represent the sequential information in the time series:

$$e_i=x_i^t{W}_m+W_{pos}$$

(2)

The data from the feature mapping layer are then fed to the Transformer encoder. The multi-head attention transform these data into a query matrix $Q_i^h\in R^{T\times d_k}$, key matrix $K_i^h\in R^{T\times d_k}$, and value matrix $V_i^h\in R^{T\times U}$:

$$\begin{array}{c}{Q}_i^h=e_i{W}_q^h\hfill \\ K_i^h=e_i{W}_k^h\hfill \\ V_i^h=e_i{W}_v^h\hfill \end{array}$$

(3)

where $W_q^h$, $W_k^h$, $W_v^h$ denote the parameter matrix; h denotes the h-th head. Subsequently, a scaled production is used for obtaining the attention output $O_i^h\in R^{T\times U}$:

$$O_i^h=softmax({\displaystyle \frac{Q_i^h{(K_i^h)}^T}{\sqrt{d_k}}})V_i^h$$

(4)

Finally, the outputs from multiple heads are concatenated and multiplied by a weight matrix $W_o$ to obtain the final output, and feed-forward layers, $FFN(•)$, are utilized to aggregate contexts with residual connections:

$$\begin{array}{c}{a}_i=concat(O_i^1,\dots ,O_i^h)W_o\hfill \\ \overline{e_i}=FFN(e_i+a_i{e}_i)+e_i+a_i{e}_i\hfill \end{array}$$

(5)

where $\overline{e_i}\in R^{T\times U}$ is the final representation of the stock i.

4.2. Temporal Aggregation Module

Existing works typically generate one or more embedding representations for each stock after modeling stock correlations, resulting in increased computational costs. In contrast, our method queries all historical time embeddings for each stock using their final time embedding to produce a comprehensive stock embedding before correlation modeling. This strategy enhances computational efficiency, while preventing the loss of temporal information. To aggregate the obtained representation encoding, we employ a temporal attention mechanism along the time axis:

$$\begin{array}{c}{\mu }_i^t={\displaystyle \frac{exp({\overline{e}}_i^t{W}_{\mu }{\overline{e}}_i^{\tau })}{{\sum }_{j\in [1,\tau ]}exp({\overline{e}}_i^j{W}_{\mu }{\overline{e}}_i^{\tau })}}\hfill \\ {\tilde{e}}_i={\displaystyle \sum _{t\in [1,\tau ]}}{\mu }_i^t{\overline{e}}_i^t\hfill \end{array}$$

(6)

where the latest temporal embedding ${\overline{e}}_i^{\tau }$ is used to query the overall temporal embeddings, and the attention score ${\mu }_i^t$ is calculated through the transformation matrix $W_{\mu }$. The temporal embedding of each stock ${\overline{e}}_i$ is updated to ${\tilde{e}}_i\in R^{1\times U}$ based on the normalized attention scores.

4.3. Hybrid Attention Hypergraph Module

The interconnections between stocks can be regarded as a type of spatial relationship. Traditional graph-based models are unable to capture high-order spatial relationships. Moreover, traditional hypergraph-based models have static hypergraph structures, making them unable to capture dynamic spatial relationships. This paper employs a hybrid attention hypergraph module to capture the time-varying spatial relationships among nodes, without any prior knowledge like industry information. This module consists of a dynamic hyperedge generation layer and a hypergraph convolutional layer, as shown in Figure 2. In the modeling of the spatial relationship, each node represents a stock, while the hyperedges capture higher-order relationships among multiple stocks, not just pairs. For example, if several stocks in the same industry react to news together, our model captures this group behavior. The incidence matrix is built using self-attention, which maps nodes to edges. Finally, the hypergraph convolutional layer is used to perform cross-attention with the incidence matrix and multiply the node features to look at how each stock relates to the group and updates the stock information based on the group’s behavior. The details of each layer will be described in the following subsections.

4.3.1. Self-Attention-Based Hyperedge Construction Layer

To accurately capture the time-varying hypergraph structures embedded in time series data, we introduce a dynamic hyperedge construction layer based on a self-attention mechanism.

Specifically, we first feed the aggregated temporal embedding $\tilde{E}=[{\tilde{e}}_1,{\tilde{e}}_2,\dots ,{\tilde{e}}_N]$ into a multi-head attention layer, which can capture information from different representation subspaces of the input data by applying multiple attention mechanisms in parallel. This is denoted as $MHA()$. The process of extracting the embedding $M=[m_1,m_2,\dots ,m_N]\in R^{N\times U}$ is as follows:

$$\begin{array}{c}{Q}_s=\tilde{E}{W}_s^q,K_s=\tilde{E}{W}_s^k,V_s=\tilde{E}{W}_s^v\hfill \\ M=MHA(Q_s,K_s,V_s)\hfill \end{array}$$

(7)

where $W_s^q$, $W_s^k$, $W_s^v$ denote the parameter matrix.

Subsequently, the embedding M is fed into a fully connected layer with a ReLU activation function to generate hyperedges and construct a dynamic incidence matrix:

$$H=ReLU(W_{f}M+b_f)$$

(8)

where $W_f$ and $b_f$ are the learnable parameters, $H\in R^{N\times U}$ represents the dynamic incidence matrix embedded in stock data, N denotes the number of stocks, and U represents the number of hyperedges. This matrix captures the complex relationships among multiple stocks by representing how each stock is connected to various hyperedges. In this paper, we set the number of hyperedges as equal to the number of the hidden size U.

Specifically, the incidence matrix enables the modeling of higher-order associations among N stocks via each hyperedge. This approach goes beyond simple pairwise relationships and captures more intricate interactions within the stock market.

4.3.2. Cross-Attention-Based Hypergraph Convolution Layer

In this study, we use a cross-attention-based hypergraph convolution layer to extract the high-order relational embedding of stocks. Specifically, this layer employs a cross-attention-based architecture to aggregate and update node information based on the incidence matrix obtained above. We revise the cross-attention mechanism to compute the hypergraph convolution, which can be formulated as follows:

$$Q_H=H{W}_Q,K_H=H{W}_K,V=\tilde{E}\Theta$$

(9)

$$\widehat{E}=softmax(D_v^{-\frac{1}{2}}{Q}_H{D}_e^{-1}{K_H}^T{D}_v^{-\frac{1}{2}})V$$

(10)

where $Q_H,K_H$ are the query matrix and the key matrix generated from the incidence matrix, and V is the node feature obtained form the aggregated embedding. $W_Q,W_K,\Theta \in R^{U\times U}$ are the learnable parameters, $\widehat{E}$ is the aggregated stock features through self-queries of the incidence matrix, $D_v\in R^{N\times N}$ represents the node degree matrix, and $D_e\in R^{U\times U}$ represents the edge degree matrix, both of which are diagonal matrices.

Specifically, the incidence matrix captures complex interactions among multiple stocks via hyperedges, and these relationships are embedded into $\widehat{E}$ through the aggregation process. During the process of self-querying, $D_e$ and $D_v$ adaptively adjust the association weights among these stocks and balance the information flow. The calculation process for their diagonal elements is as follows:

$$d_{i,i}^e=\sum _{j=1}^U{h}_{j,i},d_{i,i}^v=\sum _{j=1}^N{h}_{i,j}$$

(11)

where $h_{i,j}$ is the element of the incidence matrix H. $d_{i,i}^e$ represents the sum of node connections for each hyperedge, indicating how many stocks contribute to a hyperedge. $d_{i,i}^v$ indicates the sum of hyperedge connections for each stock, indicating how many hyperedges a stock is involved in.

The cross-attention convolutional layer aggregates node features through hyperedges, which inherently captures the shared characteristics among a group of stocks. For instance, if the fluctuation of a particular stock is correlated with other stocks within the same hyperedge, the model will incorporate this relationship, thereby better understanding the interdependencies among stocks. Although the hybrid attention mechanism differs in design from conventional hypergraph convolution operations, it can still be fundamentally regarded as a variant of hypergraph convolution. Compared to traditional hypergraph convolution, this approach not only preserves the capability to model high-order relationships but also improves the model’s flexibility and expressive power. This enhancement allows the model to capture complex relationships among stocks in evolving market conditions. The capability of the cross-attention convolutional layer to model high-order relationships can be formally explained by the following theorem.

Theorem 1.

Our cross-attention convolution layer in Equation (10) can be regarded as following special hypergraph convolution:

$$X^{l+1}=\sigma (softmax(D_v^{-\frac{1}{2}}HW{D}_e^{-1}{H}^T{D}_v^{-\frac{1}{2}})X^l\Theta )$$

(12)

Proof of Theorem 1.

By bringing Equation (9) into Equation (10), the expression of cross-attention is obtained as follows:

$$X^{l+1}=\sigma (softmax(D_v^{-\frac{1}{2}}H{W}_Q{D}_e^{-1}{(H{W}_K)}^T{D}_v^{-\frac{1}{2}})X^l\Theta )$$

(13)

Through the commutative law of matrix transposition and utilizing the distinctive properties of the identity matrix $D_e$, the operational sequence can be exchanged to obtain

$$X^{l+1}=\sigma (softmax(D_v^{-\frac{1}{2}}H{W}_Q{W_K}^T{D}_e^{-1}{H}^T{D}_v^{-\frac{1}{2}})X^l\Theta )$$

(14)

Consequently, we define $W=W_Q{W_K}^T$, which yields the formulation presented in Equation (12). W is considered as the normalized edge weight matrix of the hypergraph. □

Based on Theorem 1, the cross-attention-based hypergraph convolution layer is equivalent to a special hypergraph convolution operation, and W is the learnable weight matrix, which dynamically adjusts the importance of hyperedges, ensuring that the module can flexibly adapt to different stock relationships. For example, during iterations, the model may enhance the associations of stocks with larger recent fluctuations. The cross-attention-based hypergraph convolution layer completes the information aggregation from stocks to stock groups, and the feature updating from stock groups to stocks, by multiplying the weight matrix, incidence matrix self-query, and the temporal embeddings of stocks. Meanwhile, the attention matrix is adaptively scaled using the node degree matrix $D_v$ and the hyperedge degree matrix $D_e$, ensuring a balanced weight distribution among different stocks and stock groups. The $softmax$ function maps the weights to a probability distribution, highlighting important associations among stocks. Through these mechanisms, the cross-attention layer is capable of utilizing the hypergraph structure to model the complex high-order relationships among stocks.

4.4. Self-Supervised Pretraining Scheme

A masked time series modeling task was adopted for self-supervised pretraining of the model. Given $x_i\in R^{T\times F}$ as a mini-batch of a stock sample, we can generate a masked series for each variate by randomly masking a portion of the time steps along the time dimension, formalized as

$$\overline{x_i}=Mas{k}_r(x_i)$$

(15)

where $r\in [0,1]$ denotes the masked portion, and the values of the masked time points are replaced by zeros. These masked sequences are used as input to the model, after passing through the encoder and decoder, we can obtain the reconstructed original time series, namely ${\widehat{x}}_i$.

Following the masked modeling paradigm, the model is supervised by a reconstruction loss:

$$Loss=\sum _{i=1}^N\left|\right|{\widehat{x}}_i-x_i{\left|\right|}_2^2$$

(16)

The pretraining process of the proposed STHformer framework is shown in Algorithm 1.

Algorithm 1 The pretraining algorithm of STHformer

Input: the masked stock data ${\overline{x}}_i$, the original sequence of stock data $x_i$, training epochs E,   model parameters $\theta$ Output: ${\theta }^{*}$   Initialize model parameters $\theta$;   for $e=1,2,\dots ,E$ do     for $i=1,2,\dots ,N$ do       Calculate the temporal embedding ${\overline{e}}_i$ of stock i according to Equation (5);       Calculate the integration representation ${\tilde{e}}_i$ according to Equation (6);       Calculate the relational representation ${\widehat{e}}_i$ according to Equation (10);       Output the reconstructed sequence of stock $\widehat{x_i}$;     end for     Calculate the loss function L according to Equation (16) and update parameters $\theta$;   end for   return ${\theta }^{*}$

4.5. Fine-Tuning Scheme Tailored for Stock Price Classification

Finally, the comprehensive stock embedding is fed into a predictor to output the probabilities of future trends of each stock. The calculation process is as follows:

$${\widehat{Y}}^{t+h}=softmax(W\widehat{E}+b)$$

(17)

where the $Y^{t+h}\in R^{N\times C}$ is the probability matrix of stock node, C is the number of classes, $W\in R^{C\times D}$ is the weight matrix, and $b\in R^C$ is the bias vector.

The cross-entropy loss function is utilized to guide the model’s learning and optimization of parameters:

$$\begin{array}{cc}L\hfill & ={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{L}_i\hfill \\ \hfill & =-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}[y_i^{t+h}log({\widehat{y}}_i^{t+h})+(1-y_i^{t+h})log(1-{\widehat{y}}_i^{t+h})]\hfill \end{array}$$

(18)

where $y_i^{t+h}$ and ${\widehat{y}}_i^{t+h}$ denote the ground truth and predicted trend of stock i on day $t+h$.

The comprehensive fine-tuning process of the proposed STHformer framework is shown in Algorithm 2.

Algorithm 2 The fine-tuning algorithm of STHformer

Input: stock data $x_i^t$, ground truth trend label of stock data $y_i^{t+h}$, training epochs E, model   parameters $\theta$ Output: ${\theta }^{*}$   Initialize model parameters $\theta$;   for $e=1,2,\dots ,E$ do     for $i=1,2,\dots ,N$ do       Calculate the temporal embedding ${\overline{e}}_i$ of stock i according to Equation (5);       Calculate the integration representation ${\tilde{e}}_i$ according to Equation (6);       Calculate the relational representation ${\widehat{e}}_i$ according to Equation (10);       Output the predicted trend of stock $\widehat{y_i^{t+h}}$ according to Equation (17);     end for     Calculate the loss function L according to Equation (18) and update parameters $\theta$;   end for   return ${\theta }^{*}$

5. Experiments and Analysis

5.1. Experimental Settings

5.1.1. Dataset

We used 2000 stocks from China Securities Index 2000 (CSI2000), covering the time period from 2012 to 2018, to pretrain our model. We evaluated the effectiveness of our approach using constituent stocks of two prominent indices: the CSI300 Index, and the NASDAQ100 Index. The historical stock price data we used for the Chinese market came from www.baostock.com (accessed on 4 June 2024) and the data for NASDAQ100 came from Yahoo https://finance.yahoo.com (accessed on 4 June 2024). We used six price attributes for each stock: opening price, closing price, highest price, lowest price, previous day’s closing price, and trading volume. The details of the processed datasets are shown in

Table 2. Sample division for fine-tuning.

Dataset		Time Interval	Length	Num of Samples
CSI300	Train set	1 April 2012–1 April 2018	1459	310,680
	Valid set	2 April 2018–1 April 2020	487	127,055
	Test set	2 April 2020–1 May 2024	1026	292,171
NASDAQ100	Train set	1 April 2012–1 April 2018	1459	125,474
	Valid set	2 April 2018–1 April 2020	487	41,882
	Test set	2 April 2020–1 May 2024	1026	88,236

.

5.1.2. Evaluation Metrics

To measure the classification performance in the experiments, we adopted accuracy (ACC), recall (REC), and F1-score (F1). Their specific definitions are outlined below:

• Accuracy:

  $$ACC={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

  (19)

  where TP stands for true positive, TN for true negative, FP for false positive, and FN for false negative.
• Recall:

  $$REC={\displaystyle \frac{TP}{TP+FN}}$$

  (20)
• F1-score:

  $$F1=2\times {\displaystyle \frac{PRE\times REC}{PRE+REC}}$$

  (21)

To fully assess the investment performance of the methods in the real-world stock market, we utilized three widely used profitability metrics, including the annualized rate of return (AR), Sharpe ratio (SR) and maximum drawdown (MDD):

• AR: The annualized return measures the geometric mean growth rate of an investment portfolio over a one-year period, reflecting its compounded profitability. In our experiment, we computed AR based on 252 trading days per year.

  $$AR={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{r}_n}{N}}\times 252\times 100\%$$

  (22)

  where N denotes the total number of trading days in the evaluation period, and $r_n$ is the daily return on day n.
• SR: Sharpe ratio quantifies a portfolio’s risk-adjusted performance by comparing its excess return to its return volatility. Mathematically, it is defined as

  $$SR={\displaystyle \frac{u(R)}{\sigma (R)}}\times \sqrt{252}$$

  (23)

  where $R=r_1,r_2,\dots ,r_n$ is the sequence of portfolio returns. To ensure comparability across time horizons, we annualized the ratio as follows:
• MDD: Maximum drawdown measures the short-term loss in cumulative portfolio value.

  $$MDD=\underset{0<i<j<n}{max}{\displaystyle \frac{C{R}_i-C{R}_j}{C{R}_i}}$$

  (24)

  where $C{R}_i$ and $C{R}_j$ represent the cumulative return at moment i and moment j, respectively.

5.1.3. Compared Methods

• Transformer [8]: Transformer uses a self-attention mechanism along the time axis to present global information in the input data.
• GATs [6]: A graph-based baseline, which utilizes graph attention networks to model the relationships between different stocks.
• PatchTST [27]: A Transformer-based model which utilizes channel independence to handle multivariate time series and employs patch technology to extract local semantic information from the time series.
• StockMixer [38]: A MLP-based model, which consists of indicator mixing, time mixing, and stock mixing to capture the complex correlations in stock data.
• DTSMLA [23]: A multi-level attention prediction model, which considers the influence of the market on stock through a gating mechanism and uses a multi-task process to predict price changes.
• MASTER [33]: A Transformer-based model, which introduces a gating mechanism to integrate market information and mines the cross-time stock correlation with learning-based methods.

5.1.4. Backtesting Settings

To evaluate the profitability of our method in different markets, a backtest experiment was conducted using a trading strategy. The backtest experiment was based on two assumptions: (1) All buy and sell orders could be executed at the daily closing price. (2) Trading activity had no market impact. The details of backtest experiment are as follows:

• Cost: A trading cost distribution of 0.1% was applied to CSI300 and NASDAQ100.
• Trading Period: The backtesting period spanned from 1 April 2020 to 1 May 2024.
• Select Stocks: The strategy selected the top 10% stocks with the highest predicted probability of price appreciation for trading day t.
• Initial Funding and Position Distribution: We assumed an initial capital of 10 million units of the local currency. Portfolio weights were allocated equally among all selected stocks, maintaining uniform position sizing.
• Trading Strategy: The strategy established equal-weighted long positions in all selected stocks at the closing price of day t, maintaining these positions for a fixed 15-trading-day horizon before liquidating at the closing price on day $t+15$. In the next trading period, the strategy was repeated. It is noteworthy that, due to the recent overall bear market of the CSI300 Index, we introduced a stop-loss mechanism in our trading. This mechanism terminated the current buy–sell transaction when the maximum drawdown within the period exceeded the threshold $\beta$, which was set to 0.1 in our experiment.

5.2. Hyperparameter Settings

In this paper, the time window T was set as 60, and the batch size of baselines and our model was equal to the daily stock quantity. To select the optimal hyperparameters, we employed grid search to tune our approach as well as the other comparative methods, specifically evaluating their performance based on the validation set. This included tuning the learning rate in {$2\times {10}^{-4},1\times {10}^{-4},5\times {10}^{-5}$} to identify optimal model configurations.

• Transformer [8]: We tuned the number of hidden layers within {32, 64, 128} for encoding layers and tested the number of parallel attention heads within {2, 4, 8}.
• GATS [6]: We tuned the number of hidden units within {32, 64, 128} and the number of layers within {1, 2, 3}, respectively.
• PatchTST [27]: We followed the original settings in the works, tuning the numbers of hidden layer width within {32, 64, 128} and tuning the patch length within {5, 10, 15}, respectively.
• StockMixer [38]: We followed the original settings in the works, tuning the number of market dimensions within {10, 20, 30} and tuning the multi-scale factor within {1, 2, 3, 4}, respectively.
• DTSMLA [23]: We followed the original settings in the work and tuned the number of attention heads within {2, 4, 8}.
• MASTER [33]: We maintained the original architecture of the model, tuning the number of attention heads in the intra-stock layer within {2, 4, 8} and tuning the number of attention heads in the inter-stock layer within {2, 4, 8}, respectively.

5.3. Performance Comparison

5.3.1. Predictive Performance

The experimental results presented in

Table 3. Comparison of predictive performance and backtest performance on the CSI300 dataset and NASDAQ100 dataset. All methods were evaluated through 10 independent trials with randomized initializations. The classification metrics are presented in the form of means (standard deviation).

Dataset	Model	ACC	REC	F1	AR	MDD	SR
CSI300	Transformer	0.5071 (0.0023)	0.5053 (0.0019)	0.4802 (0.0022)	0.096	0.346	0.503
	GATs	0.5021 (0.0039)	0.5602 (0.0040)	0.5293 (0.0035)	0.109	0.353	0.671
	PatchTST	0.5024 (0.0018)	0.5573 (0.0019)	0.5384 (0.0022)	0.122	0.296	0.552
	StockMixer	0.5039 (0.0022)	0.5337 (0.0023)	0.5258 (0.0025)	0.144	0.221	0.805
	DTSMLA	0.5074 (0.0031)	0.5664 (0.0032)	0.5425 (0.0031)	0.157	0.271	0.780
	MASTER	0.5092 (0.0026)	0.5943 (0.0024)	0.5324 (0.0027)	0.147	0.238	0.837
	STHformer (Ours)	0.5153 (0.0028)	0.6281 (0.0029)	0.5724 (0.0029)	0.264	0.166	0.966
NASDAQ100	Transformer	0.5142 (0.0014)	0.5214 (0.0014)	0.5392 (0.0016)	0.163	0.302	0.833
	GATs	0.5134 (0.0022)	0.5472 (0.0021)	0.5334 (0.0024)	0.196	0.339	0.843
	PatchTST	0.5113 (0.0015)	0.5414 (0.0017)	0.5478 (0.0018)	0.192	0.253	0.886
	StockMixer	0.5198 (0.0019)	0.6279 (0.0022)	0.5613 (0.0019)	0.213	0.283	0.941
	DTSMLA	0.5182 (0.0026)	0.6043 (0.0029)	0.5726 (0.0028)	0.241	0.366	0.865
	MASTER	0.5211 (0.0021)	0.5633 (0.0018)	0.5624 (0.0023)	0.253	0.333	1.001
	STHformer (Ours)	0.5281 (0.0025)	0.6906 (0.0028)	0.6176 (0.0029)	0.345	0.250	1.352

Optimal values are indicated in bold.

demonstrate STHformer’s consistent superiority across all evaluation metrics on both benchmark datasets. The comprehensive comparisons reveal that our proposed method achieved statistically significant improvements over all baseline approaches.

On the CSI300 dataset, PatchTST and GATs outperformed the Transformer. Some advanced methods like MASTER and DTSMLA showed a better performance to simpler models. STHformer demonstrated statistically improvements across all evaluation metrics. Specifically, STHformer achieved substantial gains over DTSMLA, including at least 5.5% improvements in F1, a 10.9% increase in REC, and 1.5% enhancement in ACC. On the NASDAQ100 Dataset, our model achieved substantial enhancements of at least 10.1%, 7.8%, and 9.9% in F1-score, compared to StockMixer, DTSMLA, and MASTER.

Additionally, we calculated the F1 score for all stocks on each day and conducted a Wilcoxon signed-rank test across the time dimension to further validate the performance improvements of our proposed model. We computed the paired differences in F1 scores between STHformer and the other models for each day, ranking these differences by their absolute values, while ignoring the zeros. The test statistic was then calculated based on the sum of the ranks of the positive and negative differences.The results of the Wilcoxon test show that the positive rank sum was much larger than the negative rank sum. We used the smaller negative rank sum as the test statistic to calculate the relative statistic and found that its absolute value was far greater than 3, indicating that the difference between the positive and negative rank sums was significant. The p-value derived from the test statistic indicated the probability of observing such differences by chance. Furthermore, the effect size was calculated from the standardized statistic and the square root of the sample size. For the CSI 300 dataset, all effect sizes were greater than 0.1, while for the NASDAQ100 dataset, all effect sizes were greater than 0.4. This suggests that the observed difference was not only statistically significant but also practically meaningful.

Table 4. Results of Wilcoxon signed-rank test on all datasets.

Compared Models	Wilcoxon Signed-Rank Test
	$\mathbf{\alpha }$= 0.05  p-Value
Dataset	CSI300	NSDAQ100
STHformer vs. Transformer	0.0000 **	0.0000 **
STHformer vs. GATs	0.0000 **	0.0000 **
STHformer vs. PatchTST	0.0000 **	0.0000 **
STHformer vs. StockMixer	0.0000 **	0.0000 **
STHformer vs. DTSMLA	0.0000 **	0.0000 **
STHformer vs. MASTER	0.0177	0.0000 **

0.0000 ** indicates that the p-value is less than 0.0001.

demonstrates STHformer’s statistically significant outperformance across the CSI300 and NASDAQ100 datasets at a significance level of $\alpha$ = 0.05. The obtained p-values were much lower than 0.05 and provide strong evidence that the observed improvements reflect genuine model capabilities rather than random chance. This confirms that the advantages of our method are statistically significant.

5.3.2. Backtesting Performance

Figure 3 presents the cumulative returns of all methods under identical trading conditions. STHformer demonstrated superior financial performance for both the Chinese and American markets, generating significantly higher terminal wealth. This outperformance across distinct markets confirms the model’s enhanced stock selection capability.

5.4. Model Stability Performance

To test the model’s stability under different conditions, we selected the backtest results during the Russia–Ukraine war period for analysis. Figure 4 presents the cumulative return for all models during the Russia–Ukraine war period. The event period, marked by fluctuating market conditions, presented a challenging environment for the models. The different models exhibited varying levels of adaptability to the market volatility induced by the war. STHformer showed a stronger cumulative return, while the others demonstrated more moderate responses. It is worth noting that, on the CSI300 dataset, most models executed a stop-loss strategy due to the decline reaching the stop-loss threshold, while STHformer had a smaller decline, staying below the threshold, and thus actually carry out trades.

5.5. Ablation Study

We conducted systematic ablation experiments to evaluate the contribution of each key component in our framework. The following variant models were compared:

• STHformer w/o HA & Pre: The hybrid attention module and pretraining process in STHformer was removed and all other components were retained.
• STHformer w/o HA: The hybrid attention module in STHformer was removed and all other components were retained.
• STHformer w/o Pre: The pretraining process in STHformer was removed and all other components were retained.

The results of the ablation experiments in

Table 5. Comparisons of ablation experiment results on CSI300 dataset and NASDAQ100 dataset.

Dataset	Model	ACC	REC	F1	AR	MDD	SR
CSI300	STHformer w/o HA&Pre	0.5014 (0.0023)	0.5304 (0.0027)	0.5032 (0.0029)	0.106	0.332	0.565
	STHformer w/o HA	0.5054 (0.0022)	0.5607 (0.0025)	0.5342 (0.0027)	0.203	0.267	0.864
	STHformer w/o Pre	0.5114 (0.0030)	0.5512 (0.0032)	0.5423 (0.0033)	0.228	0.294	0.912
	STHformer (Ours)	0.5153 (0.0028)	0.6281 (0.0029)	0.5724 (0.0029)	0.264	0.166	0.966
NASDAQ100	STHformer w/o HA&Pre	0.5071 (0.0019)	0.5284 (0.0022)	0.5139 (0.0022)	0.175	0.280	0.820
	STHformer w/o HA	0.5182 (0.0016)	0.5688 (0.0019)	0.5633 (0.0020)	0.202	0.279	0.924
	STHformer w/o Pre	0.5243 (0.0028)	0.6021 (0.0031)	0.5806 (0.0033)	0.257	0.248	1.064
	STHformer (Ours)	0.5281 (0.0025)	0.6906 (0.0028)	0.6176 (0.0029)	0.345	0.250	1.352

Optimal values are indicated in bold.

demonstrate the individual contributions of STHformer’s components, which will be analyzed from the following perspectives:

5.5.1. Effectiveness of Hybrid Attention Module

The hybrid attention module played a crucial role in capturing the dynamic high-order relationships among stocks. By focusing on spatial interactions, this module enabled the model to better represent complex financial data. Compared to STHformer w/o HA, STHformer achieved notable improvements in F1 and AR, by 7.15% and 30.15%, on the CSI300 dataset. When applied to the NASDAQ100, STHformer showed a significant increase in F1 and AR, by 9.64% and 70.79%, respectively. This indicates that the hybrid attention module effectively captured dynamic high-order relationships among stocks, thereby enhancing the model’s ability to represent complex financial data. The performance degradation without this module suggests that it is essential for capturing the complex interactions that are critical for accurate predictions.

5.5.2. Effectiveness of Pretraining Strategy

Pretraining is a key component of enhancing the model’s representation learning and robustness, allowing it to better capture complex patterns in data. The pretrained model learned generalizable features from large datasets, which enhanced its ability to adapt to different datasets and tasks. It also enabled the model to obtain strong cold-start capabilities, so as to achieve better prediction performance on newly listed stocks with insufficient historical data. Compared to STHformer w/o Pre, STHformer demonstrated a notable increase in F1 and AR, by 5.55% and 15.80%, on the CSI300 dataset. When applied to the NASDAQ100 dataset, STHformer showed improvements in F1 and AR of 6.37% and 34.24%, respectively. This highlights the importance of pretraining for improving the model’s generalization ability on different datasets and for improving the model’s ability to cold-start.

5.6. Parameter Analysis

• Number of hidden sizes: Figure 5a shows the comparison of F1 scores for the STHformer model with different hidden layer sizes. The experimental results indicate that the model achieved the best predictive performance when the hidden layer size was set to $2^7$. It is worth noting that as the hidden layer size was increased further, the predictive performance of STHformer tended to stabilize. Based on this analysis, this chapter adopted $2^7$ as the optimal hidden layer configuration for the model.
• Number of heads: Figure 5b demonstrates the predictive performance comparison of the STHformer model with different numbers of attention heads in the hyperedge generation layer. The experimental results show that the STHformer model achieved the best predictive performance when the number of heads was set to $2^1$, significantly outperforming configurations with more attention heads. Based on the results, this chapter adopted $2^1$ attention heads in the hyperedge generation layer as the optimal hyperparameter configuration for the model.

5.7. Complexity Analysis

In this paper, the time complexity of the model was represented by the time required to complete a single training epoch, while the space complexity was indicated by the number of model parameters. As shown in

Table 6. Complexity analysis of the models.

Model	Model Parameters	One Epoch Training Time
Transformer	562,882	55 s
GATs	184,834	53 s
PatchTST	152,450	50 s
StockMixer	17,462	133 s
DTSMLA	322,125	61 s
MASTER	726,658	51 s
STHformer (finetune)	232,706	71 s
STHformer (pretrain)	278,888	102 s

, although the training time of STHformer was slightly longer than DTSMLA and MASTER, its parameter scale was at a relatively low level. Considering the overall performance improvement, this increase in training time was reasonable and acceptable. It is worth noting that although the Transformer architecture shares some structural similarities with STHformer, there are significant differences in the specific design and parameterization. For instance, the Transformer employs multiple layers, leading to a substantial overall parameter count. In contrast, STHformer achieved better performance with a more optimized parameter set, thereby demonstrating a more efficient utilization of resources. The combination of low parameter counts and reasonable training time makes STHformer highly scalable for real-world applications. Although STHformer requires a longer training time during the pretraining stage, it is important to note that pretraining is usually performed only once. The main computational load in practical applications comes from the fine-tuning stage, which can be updated multiple times as new data became available. In this regard, STHformer (finetune) exhibited both low parameter counts and training time, making it highly scalable and suitable for real-world scenarios where frequent updates are necessary.

6. Conclusions

This paper proposes a spatio-temporal hypergraph transformer (STHformer) for multi-stock trend forecasting. Firstly, a temporal encoder is employed to learn the sequences’ internal representation. Secondly, the final time embedding is used to query all historical time embeddings for each stock, to produce a comprehensive stock embedding. Thirdly, treating different stocks as distinct positions, a hybrid attention hypergraph module captures the high-order spatial correlations, characterizing dynamic relationships. Finally, a pretraining strategy utilizing masked autoencoding enhances the overall adaptability. Extensive experiments conducted on the CSI300 and NASDAQ100 datasets demonstrated that STHformer achieved superior performance compared to representative stock prediction methods, with improvements of at least 5.53% in F1 score and at least 15.4% in Sharpe ratio. In our future work, we plan to incorporate multitask learning into the pretraining process, aiming to enhance the model’s understanding and predictive capabilities for time series data by jointly optimizing the loss functions across different tasks. Moreover, in this paper, our model focused on daily data and did not consider intraday data. Therefore, we plan to incorporate intraday data to enhance the performance. Finally, we will apply the model to other financial datasets, such as futures and exchange rates.