Crash Diagnosis and Price Rebound Prediction in NYSE Composite Index Based on Visibility Graph and Time-Evolving Stock Correlation Network

Abstract

This study proposes a framework to diagnose stock market crashes and predict the subsequent price rebounds. Based on the observation of anomalous changes in stock correlation networks during market crashes, we extend the log-periodic power-law model with a metric that is proposed to measure network anomalies. To calculate this metric, we design a prediction-guided anomaly detection algorithm based on the extreme value theory. Finally, we proposed a hybrid indicator to predict price rebounds of the stock index by combining the network anomaly metric and the visibility graph-based log-periodic power-law model. Experiments are conducted based on the New York Stock Exchange Composite Index from 4 January 1991 to 7 May 2021. It is shown that our proposed method outperforms the benchmark log-periodic power-law model on detecting the 12 major crashes and predicting the subsequent price rebounds by reducing the false alarm rate. This study sheds light on combining stock network analysis and financial time series modeling and highlights that anomalous changes of a stock network can be important criteria for detecting crashes and predicting recoveries of the stock market.

1. Introduction

A stock market crash is one of the most significant systemic risks of the modern financial system, causing significant losses for investors. A recent example is the March 2020 stock market crash triggered by COVID-19 [1], during which the New York Stock Exchange (NYSE) Composite Index plunged roughly 35% within a month. Meanwhile, rebounds of the stock market after crashes usually signal the recovery of investors’ confidence or the taking effect of bailout policies. Therefore, it is critical for investors and policy makers to detect stock market crashes and predict price rebounds.

In the past decades, different methods have been proposed to diagnose the stock market crashes and predict rebounds. One of the most representative methods is the log-periodic power-law (LPPL) model [2]. Originally, the LPPL model was proposed to predict the bursting point of financial bubbles. Yan et al. [3] adopt the LPPL model to study stock market crashes by considering them as the “mirror images” of financial bubbles, which are also known as “negative bubbles”. The fundamental insight of modeling financial crashes with the LPPL model is to capture a particular pattern of the price time series, which can be described as “the faster-than-exponential decline accompanied by accelerating oscillations” [4]. The LPPL model and its extensions are successfully applied in diagnosing negative bubbles of many types of assets, such as crude oil [5] and cryptocurrency [6].

Despite previous achievements, all the existing LPPL-based models only focus on the price time series itself. However, it may not be sufficient to use only the price time series of the stock index when diagnosing stock market crashes. The price time series of the stock index can describe the overall fluctuation of the stock market, but it ignores the complex interactions among multiple assets.

Network analysis provides a novel tool for characterizing the complex interactions and co-movements in the stock market [7,8,9]. In fact, it has been discovered that stock market crashes and recoveries are accompanied by drastic changes in the topological structure of stock correlation networks, which can be captured by some network statistics such as assortativity [10], modularity [11], von Neumann entropy [12], the structural entropy [13,14] and the graph motif entropy [15]. Although the above literature qualitatively analyzes the dynamical changes of stock correlation networks during stock market crashes and rebounds, there is still a lack of work that applies such phenomenon to quantitatively diagnose stock market crashes and predict price rebounds.

This paper contributes to the literature by incorporating the analysis of a time-evolving stock correlation network into the LPPL model. The contribution of this paper has three folds. First, extending the LPPL model, we propose to characterize stock market crashes by two distinct characteristics: (1) faster-than-exponential decline of the stock index price; (2) abnormal changes in the market structure (i.e., the topology of the stock correlation network). Second, we design a prediction-guided anomaly detection method based on the extreme value theory (EVT) in order to define and detect “anomalies” for Characteristic (2). The intuition is that a properly trained predictor can forecast most of the “normal” situations well, with significant deviations when “abnormal” situations occur. The “normal” and “abnormal” deviations are distinguished based on EVT. Third, we propose a framework by combining Characteristic (1) and (2), where Characteristic (1) is captured by a visibility graph (VG)-based representation of the LPPL model proposed by Yan et al. [16]. A hybrid rebound indicator is calculated, which is a linear combination of Yan’s VG-based indicator and the anomalies of the stock correlation network. Specifically, we normalize the anomalies to 0 to 1 based on EVT and treat them as confidence levels (i.e., weights) of the VG-based rebound indicator.

Experiments are conducted based on the data of the New York Stock Exchange (NYSE) Composite Index. We predict the subsequent price rebounds of the 12 major crashes in the U.S. stock market from 4 January 1991 to 7 May 2021. Experimental results demonstrate that our proposed prediction-guided anomaly detection algorithm is well capable of identifying abnormal changes in stock correlation networks during market crashes and recoveries. Furthermore, our proposed hybrid indicator outperforms Yan’s VG-based indicator by reducing the false alarm rate. These findings imply that incorporating the analysis of the time-evolving stock correlation network into the modeling of the stock index time series is a promising direction for diagnosing and predicting financial markets.

The rest of this paper is organized as follows. Section 2 describes the data. Section 3 introduces our proposed framework for stock market crash diagnosis and rebound prediction. Section 4 presents the experimental results. Section 5 discusses the main findings, implications and limitations of this study. Section 6 concludes our work and provides future research directions.

2. Data Description and Labeling

2.1. Data Description

This paper uses the daily closing price of the NYSE Composite Index, which is collected from the Yahoo! financial database (http://finance.yahoo.com) (accessed on 25 November 2021). The time period is from 2 January 1986 to 7 May 2021, with the data from 2 January 1986 to 3 January 1994 as the training set and the rest of the data as the testing set. As shown in

Table 1. Major stock market crashes in the U.S. from January 1986 to May 2021.

Name	Date
Black Monday	19 October 1987
Friday the 13th mini-crash	13 October 1989
Early 1990s recession	3 July 1990
1997 Asian financial crisis	2 July 1997
Russian financial crisis	17 August 1998
Dot-com bubble	10 March 2000
September 11 attacks	11 September 2001
Stock market downturn of 2002	19 March 2002
Financial crisis of 2007–2008	31 October 2007
2009 Icelandic financial crisis	20 January 2009
European sovereign debt crisis	27 April 2010
August 2011 stock markets fall	1 August 2011
2015–2016 stock market selloff	18 August 2015
2018 cryptocurrency crash	20 September 2018
2020 stock market crash	24 February 2020

, there are 15 major crashes in the U.S. stock market, including 3 in the training set and 12 in the testing set. The list of stock market crashes in the U.S. before 2008 is provided in [15], while the crashes after 2008 are manually collected from Wikipedia (https://en.wikipedia.org/w/index.php?title=List_of_stock_market_crashes_and_bear_markets) (accessed on 25 November 2021). Furthermore, we select 199 stocks out of the 347 stocks in the NYSE dataset [11], whose data are available for the entire time period from 1986 to 2021. Their price time series are used to construct the time-evolving stock correlation network.

2.2. Labeling Rebounds of Price Time Series

The rebounds of the financial market are labeled based on the price time series of the stock index. Following the definition in the existing literature [16,17], we define the rebound as the time point at which a stock index turns from a downtrend to an uptrend after a stock market crash. This paper labels the trend of the stock index based on a recently proposed method [18]. The basic idea is that the price time series is considered to change from an upward trend to a downward trend when the price falls by more than w compared with the local peak. Similarly, when the price rises above w compared with the local trough, the price time series is considered to change from a downward trend to an upward trend. Here, w is a predetermined threshold for the proportion of price increases and decreases. The detailed procedure is illustrated in Appendix A.

For each stock market crash in Table 1, we detect the first turning point when the trend of the price time series changes from a downtrend to an uptrend. This turning point is defined as the time point of the price rebound after the corresponding stock market crashes. Here we choose $w=0.15$, which is an empirical value proposed along with the trend labeling method [18]. In other words, once the price rises by more than 15% from the local trough after a financial crash, it is regarded as a switch from a downtrend to an uptrend, and the local trough is labeled as a rebound. The labeling results are further manually adjusted according to the historical records. Figure 1 shows the time series of the daily closing price of the NYSE Composite Index, as well as the labeled price rebounds and the corresponding stock market crashes.

3. Methodology

Our proposed framework predicts market rebounds by observing and quantifying a specific phenomenon, which is a faster-than-exponential decline in the stock index price time series accompanied by anomalous changes in the topology of the correlation network of constituent stocks. Therefore, our proposed framework consists of the following four steps: (1) quantifying faster-than-exponential decline in stock index price; (2) constructing time-evolving stock correlation network; (3) detecting anomalous topological changes of stock correlation networks; (4) calculating a rebound alarm index. In this section, we first introduce our proposed framework and then describe each of these four steps.

3.1. Our Proposed Framework

Our proposed framework is illustrated in detail in Figure 2. We first measure the faster-than-exponential decline of the price time series of the stock index based on Yan’s VG-based LPPL model [16] (step a). The detailed procedures are described in Section 3.2.

The time series of the stock index can describe the overall fluctuation of the financial market, but it ignores the complex interactions between different assets. In our proposed framework, we exploit information of the market structure by investigating the time-evolving topology of the stock correlation network (step b and c). In step b, we first select the constituents of the stock index, then calculate the matrices of Pearson correlation coefficients based on the logarithmic return of the constituents through a sliding window, and finally extract the most important correlations in the matrix to form a stock correlation network. In step c, topological measurements are extracted from the time-evolving stock correlation network. Anomaly detection is performed on the extracted topological measurements to identify the emergence and disappearance of anomalous network topology. Section 3.3–Section 3.5 describe the implementation of step b, c and step d, respectively.

3.2. Quantifying Faster-Than-Exponential Decline in Stock Index Price

We adopt the method based on the visibility graph [16] to quantitatively describe the faster-than-exponential decline of the stock index prices. This subsection first briefly introduces the basic idea of the visibility graph and then describes the quantification of the faster-than-exponential decline of the stock index price based on the VG.

The visibility graph (VG) is first presented by Lacasa et al. [19] as an algorithm for converting time series into complex networks. Its basic idea is to map each time point in a discrete time series into a node in a network. The edges of the network are established based on the visibility criteria. The detailed procedure of constructing the visibility graph is introduced as follows.

For each pair of data points in the time series noted as $\left(t_i,y_i\right)$ and $\left(t_j,y_j\right)$ where $i<j$, the VG algorithm first connects them with a straight line. If all the data points between i and j are below the straight line, $\left(t_i,y_i\right)$ and $\left(t_j,y_j\right)$ are considered “visible” to each other, and an edge is created between them in the visibility graph. A more formal expression is: for a time series $\left[\left(t_1,y_1\right),\left(t_2,y_2\right),\cdots ,\left(t_N,y_N\right)\right]$ of length N, the adjacency matrix W of its visibility graph is defined as  

$${\omega }_{ij}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{y}_k<y_i+\frac{t_k-t_i}{t_j-t_i}\left(y_j-y_i\right),\forall i<k<j\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(1)

where ${\omega }_{ij}$ denotes the weight of the edge $(i,j)$ in the visibility graph.

Yan et al. [16] first represent the log periodic power law (LPPL) model with the visibility graph. Originally, the LPPL model characterizes the financial bubbles by the faster-than-exponential growth of the stock market prices. Yan et al. construct the visibility graph based on the logarithmic prices of stock indices. Since the exponential growth is a straight line in the logarithmic-linear scale, faster-than-exponential growth is represented as a convex curve. Therefore, as the price time series goes up super-exponentially, the degree of the last node of its VG increases. Furthermore, if two nodes $\left(t_i,y_i\right)$ and $\left(t_j,y_j\right)$ are “visible” to each other, the growth rate from $t_i$ to $t_j$ can be roughly considered as “faster-than-exponential”.

Since our goal is to characterize the financial crashes, we adopt the absolute invisibility graph, which is exactly the opposite of the visibility graph. Its adjacency matrix $W={\left\{{\omega }_{ij}\right\}}_{i,j=1,\cdots ,N}$ is defined as

$${\omega }_{ij}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{y}_k>y_i+\frac{t_k-t_i}{t_j-t_i}\left(y_j-y_i\right),\forall i<k<j\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right..$$

(2)

Similar to the visibility graph, if two nodes $\left(t_i,y_i\right)$ and $\left(t_j,y_j\right)$ are “absolutely invisible” to each other, we can roughly conclude a faster-than-exponential decline from $t_i$ to $t_j$. For each data point $\left(t_i,y_i\right)$ in a time series, we obtain $T_{\mathrm{VG}}-1$ historical data before $t_i$ through a sliding window of length $T_{\mathrm{VG}}$ to construct the absolute invisibility graph. The magnitude of the faster-than-exponential decline at time point $\left(t_i,y_i\right)$ is defined as

$$I_{\mathrm{VG}}\left(i\right)=\frac{1}{T_{\mathrm{VG}}}\sum _{j=i-T_{\mathrm{VG}}}^{i-1}{\mathbf{1}}_{\left[y_i<y_j\right]}·{\mathbf{1}}_{\left[y_k>y_i+\frac{t_k-t_i}{t_j-t_i}\left(y_j-y_i\right)\right]},$$

(3)

where ${\mathbf{1}}_{\left[·\right]}$ is the indicator function.

3.3. Constructing the Time-Evolving Stock Correlation Network

This paper studies the time-evolving nature of the correlation network, which is formed by the constituents of a stock index. Our basic assumption is that the topology of the stock correlation network changes significantly during financial crashes, and that the recovery of the network topology implies the recovery of the financial market.

The most straightforward way of studying a time-evolving network is to look at snapshots of the network taken at different time points. By stacking the snapshots in temporal order, a time-evolving network is denoted as $\mathcal{G}=\left(G_1,\dots ,G_T\right)$. Each $G_t=\left\{{\mathbf{N}}_t,{\mathbf{E}}_t\right\}$ is a snapshot recorded as time t, where ${\mathbf{N}}_t$ is the set of nodes and ${\mathbf{E}}_t$ is the set of edges. Since this paper considers the time-evolving correlations among the same set of stocks, all the snapshots in $\mathcal{G}$ share the same set of nodes, that is, ${\mathbf{N}}_1={\mathbf{N}}_2=\cdots ={\mathbf{N}}_T=\mathbf{N}$.

In this paper, each snapshot $G^t$ is obtained from the correlation matrix $C^t$ by a filtering method. The construction procedure is composed of four steps: (1) dividing time windows; (2) determining the constituents of the stock index; (3) calculating correlation matrices; (4) extracting single layer networks from the correlation matrices. The detailed procedure is illustrated as follows.

3.3.1. Dividing Time Windows

In terms of dividing time windows, three parameters need to be determined: (1) the length $T_{\mathrm{tw}}$ of the time window, (2) the step size $\Delta$ between two consecutive windows. The choice of time window length $T_{\mathrm{tw}}$ is a trade-off between over-smoothed and too noisy data [20]. In order to capture the dynamics of stock market correlations, we need to choose the smallest possible window length. On the other hand, the time window needs to be long enough to avoid the Epps effect [21]. Here, we choose a commonly used empirical value $T_{\mathrm{tw}}=25$ [15], meaning that each time window contains 25 trading days. To have a continuous tracking of the stock correlations, the step size is set as $\Delta =1$, shifting the time window 1 day forward at each step.

3.3.2. Determining Constituents of the Stock Index

In this step, we determine the name list of the stocks to track, that is, the set of nodes $\mathbf{N}$. Notice that the constituents of the stock index are constantly adjusted over time. To ensure continuous and stable tracking of the market structure, we select the constituents of the stock index whose historical data are available during the entire experimental period. In this paper, we select $N=199$ stocks from the NYSE dataset [11,22], whose daily closing prices are available from 2 January 1986 to 7 May 2021.

3.3.3. Calculating Correlation Matrices

For stock i in $M^t$, its logarithm return at time t is defined as

$$r_i\left(t\right)=ln{p}_i\left(t\right)-ln{p}_i(t-1),$$

(4)

where $p_i\left(t\right)$ is the adjusted closure price of stock i at time t. Then the Pearson correlation coefficient between stock i and stock j is calculated as

$$c_{ij}\left(t\right)=\frac{\mathbb{E}[{\mathbf{r}}_i\left(t\right)\odot {\mathbf{r}}_j\left(t\right)]-\mathbb{E}\left[{\mathbf{r}}_i\left(t\right)\right]\mathbb{E}\left[{\mathbf{r}}_j\left(t\right)\right]}{\sqrt{\left(\mathbb{E}\left[{\mathbf{r}}_i^2\left(t\right)\right]-{\left(\mathbb{E}\left[{\mathbf{r}}_i\left(t\right)\right]\right)}^2\right)\left(\mathbb{E}\left[{\mathbf{r}}_j^2\left(t\right)\right]-{\left(\mathbb{E}\left[{\mathbf{r}}_j\left(t\right)\right]\right)}^2\right)}},$$

(5)

where $\mathbb{E}(·)$ represents the sample mean, ⊙ denotes element-wise multiplication of vectors and

$${\mathbf{r}}_i\left(t\right)=\left[r_i(t-T_{\mathrm{tw}}+1),r_i(t-T_{\mathrm{tw}}+2),\cdots ,r_i\left(t\right)\right]$$

(6)

is the logarithm return series of stock i within the time window. Thus a $N\times N$ correlation matrix $C\left(t\right)$ is obtained.

3.3.4. Extracting Single-Layer Networks from Correlation Matrices

The threshold-based method [23] is applied to extract the strong correlations and construct the network. For the correlation matrix $C\left(t\right)$, we choose a threshold $\rho \left(t\right)$ and only keep $c_{ij}\left(t\right)>\rho \left(t\right)$ as the edges of the network $G^t$. This paper sets $\rho \left(t\right)$ as the 85th percentile of all elements in $C\left(t\right)$. The connection criterion of network $G^t$ is formally defined as

$$g_{ij}^t=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{c}_{ij}\left(t\right)>\rho \left(t\right)\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(7)

where $g_{ij}^t$ denotes the weight of the edge $(i,j)$ in the network $G^t$.

3.3.5. Calculating Singular Value Decomposition Entropy

We further measure the topology of each layer $M^t$ by the singular value decomposition (SVD) entropy, which has been applied to the analysis of financial market networks [24,25,26,27]. The definition of the SVD entropy is introduced below. It is worth noting that the topological characteristic is not limited to the SVD entropy, any proper network statistic, such as the von Neumann entropy [28] or the graph motif entropy [15], is applicable to our proposed framework.

The SVD entropy is based on the singular value decomposition of the $N\times N$ adjacent matrix A of the network G,

$$A=U\Sigma V^T,$$

(8)

where $\Sigma$ is a diagonal matrix of singular values,

$$\Sigma =diag\left({\sigma }_1,\cdots ,{\sigma }_N\right),$$

(9)

The SVD entropy is defined as

$${\mathrm{Ent}}_t=-\sum _i{\overline{\sigma }}_{i}ln\left({\overline{\sigma }}_i\right),$$

(10)

where ${\overline{\sigma }}_i$ is the normalized singular value defined as

$${\overline{\sigma }}_i=\frac{{\sigma }_i}{{\sum }_j{\sigma }_j}.$$

(11)

By calculating the topological characteristics, we transform the time-evolving network into a time series, which is much easier to interpret. Based on the time series of the SVD entropy, we identify and measure anomalous changes in the topology of the stock temporal network in the following subsections.

3.4. Prediction-Guided Anomaly Detection Based on Extreme Value Theory

In this subsection, we detect the anomalous value of each of the topological indicators based on the extreme value theory (EVT) and time series prediction. Our intuition is that we first train advanced time series forecasting algorithms only based on “normal” values. We assume that such predictors are able to capture the “normal” dynamics properly, while being completely unaware of abnormal changes in the dynamics. Based on the commonly adopted normal distribution assumption of the forecasting error, the residuals between the predicted and true value should be small and normally distributed for “normal” data, while the residuals of “abnormal” data should be large and their distribution can be portrayed by EVT. We determine the threshold between the “normal” and “abnormal” residuals based on the training dataset, as well as the parameters of the distribution of the extreme values.

Subsequently, we make predictions and calculate the residual for each day in the testing dataset. If the residual exceeds the threshold, we treat it as an “abnormal” value and calculate its “anomaly score” based on the extreme value distribution. Inspired by [29], our designed method can be divided into an initialization step and an execution step, whose detailed procedures are described below.

Initialization Step

In the initialization step, we first generate the training set that only contains “normal” data. The detailed procedure is described in Appendix B. The training set is generated to train a predictor $F(·)$. Without loss of generality, we assume that the predictor makes single-step predictions based on data from the previous d days as

$$I_t=F(I_{t-d},\cdots ,I_{t-1})+ϵ\left(t\right),$$

(12)

where $ϵ\left(t\right)$ is a normally distributed error term.

To ensure that the predictor only learns the “normal” dynamics of the time series, we generate a training set that only contains the “normal” values. We first split time series $\mathbf{I}$ into different segments based on the date of the financial crashes. For example, the first segment is the “normal period” from 1 January 1986 to 15 October 1987, which is followed by the financial crash from 19 October 1987 to 4 December 1987. We then extract training samples from the “normal periods” using a sliding window of length $d+1$, where the data of the first d days are the inputs to the predictor, and the last datum is the expected output. Finally, we obtain the set of the training inputs ${\mathbf{S}}_x$ and its corresponding target output set ${\mathbf{S}}_y$ based on which the predictor $F(·)$ is trained.

After training the predictor, for each day t in the first N days, we make the prediction based on the previous d days as

$${\widehat{I}}_t=F(I_{t-d},\cdots ,I_{t-1}).$$

(13)

Since we are looking for extremely small values, the residual is calculated as $X_t={\widehat{I}}_t-I_t$. Notice that here we consider both the “normal” periods and the financial crashes, so the set of the residuals should contain the extreme values corresponding to the financial crashes. Therefore, we analyze the tail distribution of the residual based on the EVT.

According to the EVT, the extreme values always follow the same type of distribution, regardless of the initial distribution of the data. It can be regarded as a theorem for the maximum values, which is similar to the central limit theorem for the mean values [30]. A mathematical formulation is provided by the Pickands–Balkema–de Haan theorem [31,32], which can be written as:

$${\overline{F}}_t\left(x\right)=\mathbb{P}(X-\tau >x\mid X>\tau )\sim {\left(1+\frac{\xi x}{\sigma }\right)}^{-\frac{1}{\xi }}.$$

(14)

This theorem shows that, for a random variable $X_t$, the excess over a sufficiently large threshold $\tau$ tends to follow a generalized Pareto distribution (GPD) with parameters $\xi$ and $\sigma$ [29].

A practical implication of EVT is that extreme and non-extreme events follow different distributions because they are often generated by different driving forces [33]. This is the theoretical basis for our use of EVT to identify abnormal changes in network topology. We argue that normal and abnormal residuals are caused by different driving forces, thus we aim to find the outliers that follow GPD. Based on the idea of EVT, we select the most appropriate threshold $\tau$ that allows GPD to fit the distribution of $X-\tau$ properly. This means that any value greater than $\tau$ can be regarded as an extreme value. Therefore, we consider a residual above the threshold $\tau$ as an anomalous value. The detailed procedure for obtaining the optimal value of $\tau$ is described in Appendix C.

3.5. Execution Step and Hybrid Rebound Indicator

The detailed procedure of the execution step is shown in Algorithm A4. It is designed to deal with the streaming data. For each day t, we calculate the residual $X_t=F(I_{t-d},\cdots ,I_{t-1})-I_t$. If the residual exceeds the threshold $\tau$, we raise an alarm while calculating the corresponding alarm index by

$$I_{\mathrm{Alm}}\left(t\right)=\mathrm{Alm}\left(X_t\right)=1-{\left[1+\frac{\xi (X_t-\tau )}{\sigma }\right]}^{-\frac{1}{\xi }}.$$

(15)

Notice that our alarm index is actually the CDF of the generalized Pareto distribution $F_{\xi ,\sigma }(X_t-\tau )$.

Considering the development of the financial market, the internal dynamics of the topological changes of the stock correlation network may also be changing. Therefore, we need to constantly update the predictor based on the new data. Because the predictor is not expected to learn any information about the abnormal changes of the network, we only include the “normal” data into the training set. After every K new samples are added into the training set, the predictor is retrained to ensure that it keeps tracking the latest dynamics of the system.

We finally propose an indicator to characterize the phenomenon of “faster-than-exponential decline in the stock index price accompanied by anomalous changes in the market structure”. The alarm index defined in Equation (15) has a value range of $(0,1)$, and its magnitude indicates the extent to which we believe the network structure is anomalous. Therefore, it can be considered as a “confidence level”, which is used as a mask to multiply the indicator defined in Equation (3). Considering that the anomalous changes in the stock correlation network are not necessarily perfectly synchronized with the plunge in the stock index, we make a moving average smoothing of $I_{\mathrm{Alm}}\left(t\right)$ with sliding window length $T_{\mathrm{Alm}}$. Notice that $T_{\mathrm{Alm}}$ should be a small integer. Based on the intuition that information from two weeks ago is hardly useful for forecasting, here we take $T_{\mathrm{Alm}}<10$. Since our proposed rebound indicator considers both the time series and the network, it is named as a hybrid indicator, whose formal definition is given as

$$I_{\mathrm{Hybrid}}\left(t\right)=I_{\mathrm{VG}}\left(t\right)·\left(\frac{1}{T_{\mathrm{Alm}}}\sum _{i=0}^{T_{\mathrm{Alm}}-1}{I}_{\mathrm{Alm}}(t-i)\right).$$

(16)

4. Experimental Results

In this section, we first qualitatively compare our proposed hybrid indicator with the baseline method [16] to give a general idea of how the analysis of the time-evolving stock correlation network helps to improve the prediction performance. We further quantitatively analyze the predictive power of our proposed framework based on the commonly adopted error diagram method. Experimental results demonstrate the effectiveness and robustness of our proposed method.

4.1. Qualitative Observation

In this subsection, we construct three indicators: (1) the VG-based indicator $I_{\mathrm{VG}}\left(t\right)$; (2) the alarm index $I_{\mathrm{Alm}}\left(t\right)$; (3) the hybrid indicator $I_{Hybrid}\left(t\right)$. The look-back scope of $I_{\mathrm{VG}}\left(t\right)$ is set as $T_{\mathrm{VG}}=262$, which is the same as the original paper [16]. The prediction algorithm for calculating $I_{\mathrm{Alm}}\left(t\right)$ is the Prophet forecasting model [34], whose open-source implementation is available at https://github.com/facebook/prophet (accessed on 25 November 2021). We use the data from 2 January 1986 to 3 January 1994 as the training set, and use the rest of the data as the testing set. The free parameter $T_{\mathrm{Alm}}$ is set as $T_{\mathrm{Alm}}=4$.

Figure 3 demonstrates the result of the prediction-guided anomaly detection procedure. The solid blue line in the figure indicates the SVD entropy of the stock correlation network for each day. The green dashed line indicates the alarm thresholds obtained based on the one-day ahead prediction, and each red dot indicates that an alarm is issued on that day. An alarm for the financial crisis will be raised on day t, if the SVD entropy on day t falls below the predicted alarm threshold on that day.

Figure 4 shows the price time series of the NYSE Composite Index as well as the three indicators. For the alarm index $I_{\mathrm{Alm}}\left(t\right)\in (0,1)$, a higher value indicates the higher confidence that the network structure on day t is anomalous. Thus, we can see that most of the financial crashes are accompanied by an anomalous change in the topology of the stock correlation network. However, an abnormal network topology does not necessarily imply the occurrence of a financial crisis. Similarly, a high $I_{\mathrm{Alm}}\left(t\right)$ usually implies that the stock index is close to a local trough, but there also exists a large number of false alarms. Our approach effectively suppresses the false alarms by considering both the faster-than-exponential decline in the stock index (i.e., $I_{\mathrm{VG}}$) and the anomalous changes in the topology of the stock correlation network (i.e., $I_{\mathrm{Alm}}$).

4.2. Error Diagram

In this subsection, the predictive power of our proposed framework is evaluated based on the widely-adopted error diagram [4,16]. We also compare the short-term and long-term prediction performance of our proposed framework with the VG-based baseline method.

Error diagrams are often used to decide whether an indicator has predictive power for events that are difficult to predict, such as earthquakes [35,36,37] and financial extremes [3,16,38]. Just like the ROC (receiver operating characteristic) curve, the error diagram demonstrates the prediction performance of a certain indicator under different determination criteria (e.g., thresholds). Its x-axis is the “alarm ratio” $R_{\mathrm{Alarm}}$, which is defined as the total number of alarms divided by the length of the testing period. The y-axis is the ratio of missed events $R_{\mathrm{Miss}}$, which is defined as the number of missed events divided by the total number of events in the testing period. Therefore, the prediction performance of a random guess is represented by the straight line $y=1-x$ in the error diagram. Any prediction method that is better than a random guess follows a curve below the anti-diagonal $y=1-x$. Furthermore, the p-value of the hypothesis that the prediction indicator is better than a random guess is $p=A/A_{unit}=A$, where A is the area under a curve (AUC) of the error diagram and $A_{unit}=1$ [16].

The error diagram of predicting financial rebounds of a stock index is created in the following way [4]:

• Determine the forward-looking period a, that is, if a rebound occurs within a days after an alarm being raised, we consider the rebound has successfully predicted the alarm.
• Count the number of rebounds in the testing set according to the definition in Section 2.
• Sort the values of the rebound indicator time series in a decreasing order and save them in ${\mathbf{I}}_{sort}$. The largest value in ${\mathbf{I}}_{sort}$ is the first threshold.
• We check the rebound indicator of each day during the testing period. If the rebound indicator on day t exceeds the predetermined threshold, an alarm is raised. If a rebound occurs during day t to day $t+a$, we consider the alarm successfully predicts the rebound.
• We compare the successful predictions of the current threshold and the previous threshold. If there is no new successful prediction, the threshold is moved down to the next value in ${\mathbf{I}}_{sort}$.
• If new predictions are made based on a threshold, we count the missed rebounds and calculate the ratio of missed events as

  $$R_{\mathrm{Miss}}=\frac{\mathrm{Number}\mathrm{of}\mathrm{missed}\mathrm{rebounds}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{rebounds}}$$

  (17)

The alarm ratio is calculated as

$$R_{\mathrm{Alarm}}=\frac{\mathrm{Number}\mathrm{of}\mathrm{alarms}}{T-a},$$

(18)

where T is the length of the testing period. We further plot $\left(R_{\mathrm{Alarm}},R_{\mathrm{Miss}}\right)$ in the error diagram.

7.

Steps 4 to 6 are continuously repeated until all the rebounds are successfully predicted.

Figure 5 compares the short-term and long-term prediction performance of our proposed method (i.e., the hybrid indicator) with the VG-based indicator. We use the same color to represent the same forward-looking period a. Solid lines with circular icons correspond to the VG-based method, while dashed lines with triangular icons represent the results for the hybrid indicator. It can be observed that our proposed hybrid indicator outperforms the benchmark VG-based indicator for any forward-looking period.

Table 2. The p-values of the predictions for all the forward-looking periods.

Forward-Looking Period a	VG-Based Indicator	Our Proposed Hybrid Indicator
2	0.026930	0.016197
3	0.024044	0.013115
5	0.024051	0.012719
10	0.016664	0.011346
15	0.015487	0.008927
25	0.015352	0.007980

shows the p-values of the predictions for all the forward-looking periods $a=2,3,5,10,15,25$. It can be observed that, for each forward-looking period a, the p-value of the VG-based indicator is smaller than $0.03$. This indicates that the VG-based indicator is superior to a random guess on a significance level $3\%$. In other words, the predictability of the VG-based indicator is significant on a significance level $3\%$, which matches the result in the previous paper [16]. Similarly, our proposed hybrid indicator is superior to a random guess at a $2\%$ significance level. We can conclude that the predictability of our proposed hybrid indicator is significant at a significance level of $2\%$. We can further observe that the p-value of our proposed indicator is smaller than the p-value of the VG-based indicator for each a, indicating that our proposed method constantly outperforms the benchmark method.

We finally test the robustness of our proposed method. We choose $T_{\mathrm{Alm}}$ from $\{2,3,\cdots ,$$10\}$ and calculate the nine corresponding p-values for each forward-looking period a. The box plot in Figure 6 shows the statistics of the p-values for different $T_{\mathrm{Alm}}$. In this figure, each box represents the nine p-values for a forward-looking period a. The green dots are the mean, while the orange solid lines are medians. The lower and upper edges of each box are the first and third quartiles, which are denoted as ${\mathrm{Q}}_1$ and ${\mathrm{Q}}_3$, respectively. The top and bottom boundaries are ${\mathrm{Q}}_3+1.5\mathrm{IQR}$ and ${\mathrm{Q}}_1-1.5\mathrm{IQR}$, where $\mathrm{IQR}={\mathrm{Q}}_3-{\mathrm{Q}}_1$ is the interquartile range. Points out of the upper and lower boundaries are outliers. For the convenience of comparison, we label the p-value of VG-based indicator with the blue dashed line for each a. It can be observed that the p-values of our method are constantly smaller than the VG-based benchmark, indicating that the performance of our proposed framework is consistently better. We can conclude that our proposed framework is robust to the choice of the free parameter $T_{\mathrm{Alm}}$.

5. Discussion

The findings of this study indicate that incorporating the analysis of a time-evolving network can improve the performance of traditional time series models. Qualitative observations show the rapid decline of SVD entropy during financial crashes and suggest that the network-based indicator helps with reducing the false alarm rate. Quantitative comparisons further confirm that the predictive power is improved by combining the network-based indicator.

According to the qualitative observations, the SVD entropy of the stock correlation network of the NYSE Composite Index decreases significantly during stock market crashes. This is consistent with the existing studies of various stock indices based on different definitions of network entropy [13,14,15,39]. For example, Zhang et al. [15] have observed decreases in the von Neumann entropy and the graph motif entropy of the NYSE network during financial crashes. Furthermore, it is also observed that the structural entropy declines rapidly during crisis periods in the stock correlation network of FTSE100 and NIKKEI225 [13]. Since “entropy” can be interpreted as “diversity”, such observations suggest that the “structural diversity” of the stock market commonly declines during financial crashes, and the recovery of structural diversity reflects the recovery of the market. We may further interpret this econophysics phenomenon from the perspective of behavioral finance. The structural diversity of the stock market may represent investors’ perceptions in the heterogeneity of different firms. During financial crashes, spillover effects and investors’ herding behavior may erase the differences between “good” and “bad” firms, and thus can eliminate the structural diversity of the stock market.

In addition, it can be observed that the network-based indicator helps with reducing the false alarm rate of the time series-based indicator. This finding can shed light on a better understanding of the stock market crashes. Traditionally, research on stock market crashes focuses on modeling the price time series of stock indices [16,17,40,41,42,43]. In particular, the benchmark indicator of this study (i.e., the VG-based LPPL indicator) [16] captures the faster-than-exponential decline in stock index price. However, our experimental results show that detection and prediction based on the time series alone can contain a number of false alarms. In other words, a rapid decline in stock index prices does not necessarily mean a financial crash. An intuitive explanation is that the plunge of highly weighted sectors or companies may drive down the stock index, but as long as the plunge is not propagated through the stock network, it will not trigger the crash of the whole market. By combining network-based and time series-based indicators, this study contributes to the LPPL-related literature and highlights that anomalous topological changes of the stock correlation network can be important criteria for confirming the occurrence and predicting the recovery of stock market crashes.

Quantitative comparisons confirm that the incorporation of SVD entropy improves the predictive power of the time series-based model. This finding agrees with the existing literature on the predictive power of SVD entropy for stock market dynamics [24,25,26,27]. For a number of representative stock indices such as the Dow Jones Industrial Average [24], the Shenzhen Component Index [25] and the Shanghai Component Index [27], it has been found that the SVD entropy of the constituent stock network has a predictive ability to the dynamics of the stock index. However, the existing literature mainly focuses on examining the predictive power of SVD entropy using the Granger causality test, without developing methods or models to practically leverage such predictive ability. This study contributes to this research gap by proposing a framework that jointly considers the SVD entropy and the stock index price. It is demonstrated that the predictive power of the SVD entropy can be practically applied.

It should be acknowledged that this work has several limitations. First, this study transforms the time-evolving stock correlation network into the time series of a network statistic (i.e., the SVD entropy) and therefore ignores the finer topology in the network. In particular, our proposed framework is not able to recognize the potential micro- and meso-level structural changes. This limitation also hinders a more in-depth observation of the topological changes during and after financial crashes. As a consequence, a number of critical research questions are left unanswered. For example, it is still unclear if similar micro- and meso-level patterns can be observed among multiple financial crashes throughout history. If so, developing models and methods based on such patterns to detect crashes and predict rebounds can be another interesting research topic. Another limitation of this study is that an in-depth analysis of the mechanism is not performed. For example, this study focuses on the diagnosis of major financial crashes. However, we have also observed rapid declines in stock indices accompanied by anomalous network changes during some periods that are not regarded as major financial crashes. Furthermore, the topological changes in the stock correlation network have become increasingly drastic from 1968 to the present. The causes of these phenomena are still unclear.

This study has several practical implications for professionals in a variety of fields. For both institutional and individual investors, our proposed rebound indicator provides a referential signal for market timing strategies. It can be observed that weak signals start to appear when a stock market crash occurs. As the stock market crash progresses, the rebound indicator gradually gets stronger and clusters around certain dates. When the rebound indicator reaches the peak and starts to decline, a change of regime is more likely to occur. Therefore, in the aftermath of stock market crashes, investors can start applying long strategies after observing the clustering and peaking of the rebound indicator. For policymakers, this study provides useful information that can help with the detection and management of market risks. When abnormal changes in the stock network are detected, policymakers can be alerted to conduct further in-depth research to determine whether a systemic risk exists. Furthermore, this study recommends introducing policies to stabilize key firms and sectors once a systemic risk occurs, since the contagion of risks may destroy the market structure and trigger herding behavior of investors. From a broader perspective, this study also has practical implications for machine learning researchers and algorithm developers. Our findings highlight the importance of incorporating stock network analysis into traditional time series models. Novel algorithms can be developed along this direction to better diagnose and forecast extreme financial events.

6. Conclusions

This study proposes a framework to diagnose stock market crashes and predict the subsequent price rebounds by jointly modeling plunges in the stock index price and abnormal changes in the stock correlation network. Experiments based on the NYSE Composite Index show that our proposed framework outperforms the benchmark VG-based LPPL model. In line with the existing literature, we observe rapid declines of the stock network’s SVD entropy during market crashes. It suggests that the elimination of structural diversity in the stock market can be an important characteristic of financial crashes. In addition, it is observed that we reduce the false alarm rate of the LPPL-based indicator by incorporating the network anomaly-based indicator. This finding can shed light on bridging the gap between stock network analysis and financial time series modeling, which are often considered as two relatively independent research directions. From the perspective of market participants (e.g., policymakers and investors), this study can provide referential signals for risk management and market timing strategies.

The main limitation of this paper is the conversion of the time-evolving network into a time series, thus losing the micro- and meso-level topological patterns. Future research can benefit from using representation learning methods (e.g., graph neural network and tensor decomposition) that can directly process dynamic networks. Techniques for interpretable machine learning can also be applied to identify specific topological patterns in the course of market crashes and recoveries. Moreover, future research can be conducted to reveal the mechanism of abnormal topological changes in stock correlation networks during market crashes. Finally, practitioners can integrate our proposed rebound indicator into their trading strategies to control risk and make profits in the aftermath of stock market crashes.