Macroeconomic Predictions Using Payments Data and Machine Learning

Abstract

This paper assesses the usefulness of comprehensive payments data for macroeconomic predictions in Canada. Specifically, we evaluate which type of payments data are useful, when they are useful, why they are useful, and whether machine learning (ML) models enhance their predictive value. We find payments data with a factor model can help improve accuracy up to 25% in predicting GDP, retail, and wholesale sales; and nonlinear ML models can further improve the accuracy up to 20%. Furthermore, we find the retail payments data are more useful than the data from the wholesale system; and they add more value during crisis and at the nowcasting horizon due to the timeliness. The contribution of the payments data and ML models is small and linear during low and normal economic growth periods. However, their contribution is large, asymmetrical, and nonlinear during crises such as COVID-19. Moreover, we propose a cross-validation approach to mitigate overfitting and use tools to overcome interpretability in the ML models to improve their effectiveness for policy use.

1. Introduction

Payment systems are often metaphorically referred to as the plumbing system of the economy, operating discreetly in the background and typically garnering attention only when they are not functioning as intended. However, with the increasing digitalization, the rise of instant payment systems, and the potential for digital currencies, we propose a new perspective: modern payment systems can be referred to as the circulatory system of the economy. Therefore, the data generated by the continuous flow of transactions in these systems can serve as valuable resources for central banks, facilitating real-time assessments of economic health, particularly in times of crisis.

Timely prediction of the economy’s short-term dynamics—known as nowcasting—is a vital input into every economic agent’s decision-making process, including central banks. However, it is difficult for several reasons. For instance, many different data series are needed to adequately describe the state of the economy, but many of these, particularly official national account statistics, are released with significant lags [1,2]. This problem is especially difficult during times of crisis, such as the 2008 global financial crisis (GFC) and the COVID-19 pandemic, primarily because of the large and nonlinear economic impact of crises and the unconventional policy responses needed for their mitigation [3,4].

In the past, econometricians have used new data and developed new techniques for nowcasting [5,6,7,8,9,10,11,12,13]. In a similar spirit, we combine both new data and ML approaches to create a nowcast of the Canadian economy. First, we use comprehensive and timely settlement data from Canada’s retail and large-value electronic payments systems; and asses which types of data are useful, when they are useful, and why they are useful, using a factor model. We then test whether ML can improve the prediction accuracy using the following five models: elastic net, support vector machines, random forest, gradient boosting, and artificial neural network [14]. We use these parametric and non-parametric ML models because they are popular among time series forecasters and preferred in macroeconomic prediction problems [7,15,16].

The payments data have been used for macroeconomic predictions in the past [9,17,18,19,20]. Canadian payments data are a particularly good candidate because they record transactions processed in various payment instruments. Thus, they capture a broad range of Canadian consumers, firms, and government economic activities. Furthermore, these data are gathered electronically and hence are immediately available, and they are free of measurement or sampling errors [21]. However, traditionally, researchers have used data from a few selected payment instruments for nowcasting [9,20]. One issue with this approach is that the use and importance of particular payment instruments may rise or fall for economic and non-economic reasons. For example, in Canada, the proportion of electronic means of payment is increasing, and the use of cash is declining, primarily due to ease of accessibility driven by technological advancements [22]. Using data from one or two payment streams in isolation might not capture the full economic picture; therefore, in [23], the authors use most of the stream settled in Canada’s retail payment system for macroeconomic nowcasting at the onset of COVID-19. In this paper, however, we also include settlement data from Canada’s wholesale payments system and cover the wider COVID-19 period, allowing us to assess which types of data are more useful and when they are useful. Furthermore, this paper compares the performance of the nowcasting models in both normal and crisis periods for the target variables available at both real-time vintages and the latest vintages. Additionally, this paper employs sophisticated econometric and ML models for analysis.

To exploit non-traditional and large-scale data, researchers have recently begun using nonlinear ML models for nowcasting [23,24,25,26]. These articles suggest that ML models complement traditional econometric tools and are useful in extracting economic value from non-traditional data sources. Furthermore, they show that, in nowcasting, ML models often outperform traditional modeling approaches, such as ordinary least squares and dynamic factor models. The flexible ML models, such as gradient boosting, employed in this paper, could prove useful in efficiently handling a wide variety of payments data and effectively managing collinearity and nonlinearity in such data [27,28]. This is beneficial because some of the payment streams used here are strongly correlated with each other. Such ML models can also help capture sudden, large, and possibly nonlinear effects of economic crises and the impact of unconventional policies designed to alleviate them [29,30,31]. This is important because different crises have reflected differently in payment streams, suggesting a tangled and possibly nonlinear relationship between some payments streams and macroeconomic targets. Moreover, ML models are beneficial when the emphasis is on improving prediction accuracy [27,32,33].

Our results suggest that timely retail and wholesale payments system data in a factor model can help improve prediction accuracy up to 25% in nowcasting GDP, retail trade sales (RTS), and wholesale trade sales (WTS). We use a linear benchmark model with timely market and survey indices and lagged macroeconomic indicators. Specifically, consumer price index (CPI), unemployment (UNE), Canadian financial stress indicator (CFSI), and the Conference Board’s consumer confidence index (CBCI). UNE incorporates the effects of public sector hiring, and CPI is useful since we are using nominal predictors [9]. CFSI is a composite measure of systemic financial market stress for Canada [34]. CBCC is based on a survey of Canadian households and has been shown to be useful in predicting household spending in Canada [35].

Moreover, in the presence of payments data and ML models—especially nonlinear gradient boosting regression (GBR)—compared to the dynamic factor models, can reduce prediction errors by as much as 20% at the nowcasting horizon. Out-of-sample performance gains using payments data and ML models are relatively greater (15 to 20%) during the COVID-19 crisis than in the pre-COVID-19 normal economic growth period; these gains are significantly higher only at the nowcasting horizon. Also, the models using payments data perform better against the latest vintages compared to the real-time vintages. These results suggest that the timeliness of payments data and the ability to capture nonlinear interactions of ML models are, primarily, helpful.

However, the use of ML models leads to a few challenges, such as overfitting and interpretability, that could reduce the effectiveness of these models for policy use. The literature is evolving to address such challenges [12,31,36,37]. Likewise, we mitigate the nowcasting ML models problem of overfitting; that is, due to the flexibility of these models, it is easy to overfit them on in-sample data, which could reduce their out-of-sample performance [38,39,40]. In addition, we address the difficulty in interpreting these models, which is important to understand their predictions—especially if they are used to support policy decisions [16,24,32,41].

To alleviate the classic ML issue of overfitting, we devise an improved cross-validation strategy tailored to macroeconomic nowcasting models. In cases where the out-of-sample test set has an economic crisis, but the validation set does not. The traditional k-fold or leave-p-out cross-validation [14,38] could be challenging for the following reasons: (a) the standard k-fold splitting breaks the order (serial correlation) of the series; (b) the distribution of test and validation sets could differ; (c) the model tuned predominantly on normal periods might not perform well on the out-of-sample crisis period. To overcome this, similar to [42], we use a randomized expanding window approach with k-fold cross-validation but without changing the order of the data (see Figure 4). Since we have the COVID-19 crisis period in the test set, using random sampling helps to include a few samples from the GFC period in the validation set. Consequently, the distribution of validation and test sets are somewhat similar (see Figure A4 in Appendix C), which could assist in selecting a model that performs well in both normal and crisis periods.

Next, we address the interpretability issue by using the Shapley additive explanations (SHAP) methodology [43,44], based on Shapley values from the coalition game theory [45,46]. To utilize this approach, we need to consider each nowcasting exercise as a game. Shapley values can then be used to fairly distribute the payout (i.e., the model prediction) among the players (i.e., the predictors) of the game. The SHAP provides a way to interpret any ML model predictions in terms of the marginal contribution of each predictor toward the final prediction. Therefore, it proves valuable for determining feature importance.

Similar Shapley-value-based approaches were employed in the recent articles by [31,37] for macroeconomic predictions. They note that although such methods are based on game theory, they do not provide any optimal statistical criterion, and asymptotics for such approaches are not available yet. To overcome such challenges, for instance, in the recent paper by [12], the authors propose ML-based mixed data sampling and develop the asymptotics in the context of linear regularized regressions. However, the literature is still evolving in this direction, and much progress needs to be made to use such asymptotics analysis for nonlinear ML approaches such as gradient boosting, which in many cases, including ours, outperform regularized linear models [25,28,31].

We observe improved model performance when the proposed randomized expanding window approach with k-fold cross-validation is used for ML model tuning. The average RMSE across k-folds is 10–15% smaller using the proposed approach compared to the traditional expanding window k-fold cross-validation approach. Further, the Shapely value-based nowcasting model interpretations reveal that, in general, many payment streams are important along with other traditional predictors in nowcasting GDP, RTS, and WTS; however, the data from the retail system are more useful than the data from the wholesale system. Moreover, during the COVID-19 crisis period, the contribution of the retail payments data are much higher than wholesale data and other benchmark predictors. Our analysis also suggests that the contribution of payments data in terms of Shapley values is small and linear during periods of low and normal growth. However, during periods of strong negative or positive growth, the payments data contribution is asymmetrical and nonlinear.

In summary, this paper demonstrates that combining timely payments data, nonlinear ML methods, tailored cross-validation approaches, and interpretability tools can provide policymakers with sophisticated models to accurately estimate key macroeconomic indicators in near real time. Consequently, this paper adds to the growing literature on the use of non-traditional data and nonlinear ML models for macroeconomic predictions; it provides additional econometric tools to mitigate challenges in such models to improve their effectiveness for policy use.

We proceed as follows: Section 2 describes the payments data and discusses the adjustments performed on these data for macroeconomic predictions. Subsequently, in Section 3, we briefly overview various methods employed for nowcasting and discuss challenges associated with using ML models for predictions. This is followed by a discussion of our results in Section 4. Finally, in Section 5, we set forth our conclusions. Several appendices provide further details on the payments data and the nowcasting methodology employed.

2. Payments Systems Data

The vast majority of non-cash transactions require settlements to extinguish the debt from the buyer to the seller. In modern economies, this is accomplished via centralized payments systems. The data coming from such systems are potentially useful because they are (a) timely, i.e., available immediately after the end of each period, (b) available at high-frequency, i.e., at the transaction or day levels, (c) precise, i.e., carry no sampling or measurement error, and (d) comprehensive, i.e., capture a broad range of financial activities across the country [9,21,23,47].

In Canada, the ACSS and LVTS are used to settle most transactions. Note that, as of September 2021, the LVTS system was replaced by a new system called Lynx [48]. Our data consist of all settled transactions in both the ACSS and LVTS payments systems. The ACSS settles the majority of retail and small-value payment items on a net basis. In 2019, the ACSS handled an average of 33 million transactions per business day, with an average daily total value of CAD 29 billion. The ACSS processes 22 payment streams. Broadly, these streams can be categorized into two groups: (1) electronic streams, which include, e.g., automated funds transfer (AFT), point-of-sale (POS) payments, and government direct deposit (GDD); (2) paper streams, which incorporate encoded paper, paper remittances, and government paper items. Note that the ACSS supports 99% of the daily transaction volume and 13% of daily value processed by Canadian payment systems.

In the ACSS, electronic means of payment have become more common than paper items due to their usability. This change is driven primarily by technological advancements leading to the inception and adoption of new payment instruments. However, economic crises such as the GFC and the COVID-19 crisis also influence payment flows. Historically, the encoded paper stream has the highest-value shares in the ACSS, followed by AFT credit. The POS payments stream has the largest volume of shares, followed by the encoded paper stream (see [23] for the breakdown of shares of payment streams in the ACSS).

The LVTS facilitates the transfer of large-value payments between Canadian financial institutions on a gross basis. In 2019, the LVTS handled an average of approximately 40,000 transactions per business day, with an average daily value of CAD 189 billion. The LVTS provides each participant with two options called tranches, T1 and T2, to exchange payments. Each tranche (henceforth also referred to as a stream) differs based on how individual payments are collateralized. Payments in the LVTS comprise foreign exchange payments, payments for settlement of Canadian-dollar-denominated securities, payments related to the final settlement of ACSS and Government of Canada transactions, as well as the Bank of Canada’s own and its clients’ payments (see [49] for further details on payment types settled in the LVTS). In the LVTS, payment value and volume are mostly processed through T2. Note that, historically, T2 has processed roughly 75% of the value and 98.7% of the volume of payments, and T1 has processed roughly 25% of the value and 1.3% of the volume.

2.1. Adjustments to Payments Data

In the past, driven by technological advancements, some payment instruments from the ACSS were discontinued or merged into others, and several new payment instruments were created. For example, starting in 2012, a new stream was created to process the Government of Canada’s direct deposit payments. This addition caused a sudden drop in the value and volume of payments in the AFT credit stream, where they were originally processed. To overcome the effects of such sudden changes and to obtain a better representation of payment flow, we merged several streams belonging to similar categories and settled related payments. Furthermore, to overcome the effects of consumers’ payment choices, i.e., when they switch payment method, we include the sum of all payment instruments in the ACSS “Allstream” as a separate series. This should help develop an overall picture from the ACSS and mitigate the effects of unused streams. Note that, for nowcasting, we are interested in capturing whether spending (or earning) has slowed (or stopped), rather than a switched payment method.

After these adjustments, we are left with seven streams from the ACSS, which comprise transactions settled in all ACSS payment instruments and two streams from the LVTS, which are listed in

Table 1. ACSS and LVTS payment streams used in this study ^a.

Stream (ID)	Short Description
AFT credit (C) b	Government direct deposit (GDD): payrolls and account transfers
AFT debit (D)	Pre-authorized debit (PAD): automated bill and mortgage payments
Encoded paper (E) c	Paper bills of exchange: cheques, bank drafts, and paper PAD
Shared ABM (N)	Debit card payments to withdraw cash at shared ABM network
POS payments (P) d	Point-of-sale (POS) payments using debit card
Corporate payments (X) e	Exchange of corporate-to-corporate and bill payments
Allstream (All) f	The sum of all payment streams settled in the ACSS
LVTS-T1 (T1) g	Time-critical payments and payments to the Bank of Canada
LVTS-T2 (T2) h	Security settlement, foreign exchange, and other obligations

• ^a The first six payment streams are representative of 20 payment instruments processed separately in the ACSS. There are a few additional payment instruments. However, they are not available for the entire period considered in this paper. Therefore, they are excluded from this study. The excluded streams are ICP regional image payments and ICP regional image payments return. Note: Excluded streams collectively account for only 0.001% of the total value settled in the system. For further details on individual ACSS streams, see Appendix A.
• ^b Stream C is the sum of AFT credit and Government direct deposit streams. We combine them because, starting in April 2012, Government direct deposit was separated from the AFT credit stream and processed independently.
• ^c Stream E is the sum of multiple streams settled separately in the ACSS. It combines encoded paper (E), largevalue encoded paper (L), image captured payments (O), Canada Savings Bonds (B), Receiver General warrants (G), and Treasury bills and bonds (H). It subtracts image-captured returns (S), unqualified (U), and computer rejects (Z) streams. We combine all of them because, over time, many of these streams were separated from the encoded paper stream and process similar types of payments.
• ^d The value and volume of stream P are obtained by summing online payments (J) and POS payments (P) streams and subtracting online returns (K) and POS refunds (Q) streams.
• ^e Stream X is the sum of paper remittances (F), EDI payments (X), and EDI remittances (Y). This stream is composed of all corporate-to-corporate payments and corporate bill payments and remittances.
• ^f Allstream is the sum of all payment streams processed in the ACSS.
• ^g We exclude payments from the Bank of Canada in stream T1.
• ^h The LVTS processes payment values equivalent to the annual GDP every five days, and the majority of the value and volume settled in the LVTS is processed in stream T2.

along with a short description. For nowcasting, we use both the monthly gross dollar amount, i.e., value, and number of transactions, i.e., volume, settled in the payment instruments; this yields a total of 18 series.

Like other macroeconomic time series, payments data have a strong seasonal component. We adjust all series (both value and volume) for seasonality using the X-13 ARIMA tool [50]. Such adjustments are performed because official macro indicators are released with similar adjustments. Specifically, we perform recursive seasonal adjustments in pseudo-real time using the data available up to the nowcasting horizon at each time step. Year-over-year (YOY) growth rates of the seasonality-adjusted payments series are used to predict the similarly adjusted YOY growth rates of macroeconomic indicators. Using growth rates (instead of levels) helps to induce (approximate) stationarity in both the target and predictors).

Our dataset does not include some payment instruments not settled through the ACSS or LVTS, such as credit card and also e-transfer payments. In 2019 credit card payments accounted for 6.2% of the value and 31.1% of the volume of total retail payments in Canada, likewise e-transfer payments accounted for 1.5% of the value and 2.5% of the volume [22]. However, the authors of [9] concluded that credit card payment data for Canada do not add significant value in nowcasting GDP and retail sales. It’s important to note that in their analysis they utilized a limited sample size when examining credit card data. As such, the results might vary if a larger sample size were employed.

Further, our dataset does not include on-us transactions, where both sender and receiver have an account with the same financial institution; such transactions do not need to be settled in a payment system. However, their shares are small and may not materially influence our analysis. Note that, on-us payments amount to roughly 20% more than those settled in the ACSS. The value of on-us transactions differs by stream, for instance, in encoded paper, it is about 25%, and in POS payments, it is 16% [22].

2.2. Payments Data for Macroeconomic Nowcasting

The crux of the nowcasting problem is that most official estimates of macro indicators are released with a substantial delay. For instance, in Canada, GDP, RTS, and WTS are released with a delay of six to eight weeks. In addition, they undergo multiple revisions, sometimes years later, highlighting the uncertainty of their measurement. Moreover, during a rapid crisis such as COVID-19, macroeconomic predictions are difficult because of the large and unprecedented economic impact. This can undermine the use of lagged data for nowcasting. Therefore, it is valuable to use more timely available information, in this case, payments systems data.

Payments data capture numerous types of transactions from both sides of macroeconomic accounts. For example, consumer income and expenditure, business-to-business payments, and Government of Canada spending. This variety and timeliness and the lack of sampling and measurement error in the payments dataset make it a rich economic information source.

For nowcasting exercises, we use Canada’s monthly GDP, RTS, and WTS at the latest available vintages (i.e., after revisions) and real-time vintages (i.e., first release) as target variables. We select these indicators because GDP is crucial for policymakers, and since we are using payments data, we think payments data have value in predicting RTS and WTS. All these indicators are released in Canada with a substantial lag and are available monthly for all historical releases. This variation allows us to test the robustness of our models. (Note: the latest vintages of seasonally adjusted monthly GDP, RTS, and WTS are obtained from Statistics Canada Tables 36-10-0434-01, 20-10-0008-01, and 20-10-0074-01, respectively. Similarly, historical releases of GDP, RTS, and WTS are obtained from Tables 36-10-0491-01, 20-10-0054-01, and 20-10-0019-01, respectively.)

YOY growth rates of the latest monthly GDP are plotted with encoded paper and AFT credit values in Figure 1 (top). Similarly, RTS’s YOY growth rates are plotted with POS payments and shared ABM values in Figure 1 (middle). The YOY growth rates of WTS are plotted with corporate payments and LVTS-T2 values in Figure 1 (bottom). To obtain a sense of the importance of payments data during a crisis, we highlight the growth rates of all variables during the 2008 GFC period (in gray) and the COVID-19 period (in blue).

During the GFC period, the decline and rebound in these payment streams’ growth rates go hand-in-hand with macroeconomic indicators. Similarly, during the COVID-19 crisis, we observe a sudden drop in most payment streams and in the macro variables. For instance, GDP and encoded paper, RTS, and POS payments, along with WTS and corporate payments, show similar movements during both crisis periods. This is a good indication of the economic value associated with these payment streams during such times.

During the COVID-19 period, however, we observe a complicated relationship between the macro indicators and some payment streams. This may be the result of the difference in the nature of the two crises. In 2008, it was a severe worldwide economic crisis; in 2019, however, there was a rebalancing of portfolios. Therefore, the nature of the 2008 GFC was consistent with a drop in payment flows, while the 2019 COVID-19 crisis led to increased flows in several payment streams. For instance, the value of payments through the AFT credit stream (which includes GDD payments) did not drop significantly at the onset of the COVID-19 crisis. On the contrary, starting in April 2020, the value of payments processed through the AFT credit stream increased due to the flow of government social payments to those directly affected by the pandemic (Figure 1, top). Similarly, we note that the value of payments through the LVTS-T2 stream surged significantly at the onset of COVID-19, showing an opposite behavior to that of macro indicators during the same period. Such behavior is not seen during the GFC period, where both WTS and LVTS-T2 growth rates drop (Figure 1, bottom). Similar behavior is observed in LVTS-T1, where payment value rose dramatically at the onset of COVID-19 due to extraordinary measures taken by the Bank of Canada under its quantitative easing policy [51]. Such complex behavior could be challenging to capture using traditional linear models.

3. Methodology

This section briefly describes the nowcasting models employed. First, we discuss ordinary least squares (OLS) and the dynamic factor model (DFM). This is followed by a brief discussion of the ML models.

Consider a set $X=\{{\mathbf{x}}^1,{\mathbf{x}}^2,\cdots ,{\mathbf{x}}^M\}$ of M predictors (also called features or independent variables) and a target $\mathbf{y}$ (dependent variable), each with N data (sample) points. For example, predictors could be monthly aggregated values settled in each of the payments streams, and the target could be monthly GDP (both recorded at the end of each month). This can be represented as a dataset $(X,\mathbf{y})$ where X is of size $N\times M$ and $\mathbf{y}$ is a vector of size $N\times 1$. Let us denote $\widehat{\mathbf{y}}$ as the predicted target, which can be estimated, for example, using payments data and OLS model as

$$\begin{array}{cc}\hfill \widehat{\mathbf{y}}(X,\mathbf{w})=& X\mathbf{w},\end{array}$$

(1)

where $\mathbf{w}$ is a vector of unknown coefficients (betas or weights) of size $M\times 1$. In OLS, the objective is to minimize the residual sum of squares between the observed $\mathbf{y}$ and the predicted $\widehat{\mathbf{y}}$ target variables. Such linear models have proven to be valuable and straightforward for prediction, and they are commonly used due to their simplicity and interpretability. However OLS can model only linear relationships in the parameters $\mathbf{w}$. Although the linearity assumption makes them easy to interpret on a modular level, it generally does not perform well on wide, large, and complex datasets [14]. Moreover, multicollinearity in the predictors, while not reducing the predictive power, could lead to reduced precision of the estimated coefficients.

The DFMs are a powerful approach to capturing the common dynamics of a set of predictors in a relatively small number of latent factors. This can act as a dimension-reduction technique by estimating a small set of dynamic factors from a large set of observed variables. DFM is a frequently selected model for macroeconomic nowcasting and forecasting [1,52]. Similar to [53], we estimate the factors using the model of [54], which effectively handles a large number of predictors. The basic representation of the model is

$$\begin{array}{cc}\hfill X_t& =\Lambda f_t+{ϵ}_t\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_t& =A_1{f}_{t-1}+\cdots +A_p{f}_{t-p}+u_t,\hfill \end{array}$$

(3)

where $X_t$ is a set of predictors at time t, $f_t$ is the unobserved factor at t, $\Lambda$ is the vector of factor loadings, ${ϵ}_t$ is the idiosyncratic disturbance at t, $A_i$ are matrices of autoregression coefficients, and $u_t$ is the factor disturbance at t. DFMs are successfully applied for economic monitoring and predictions around the world [52,55,56,57] including nowcasting Canada’s GDP [53]. Therefore, we employ this model to nowcast various Canadian macro indicators using payments data.

3.1. Machine Learning Models for Nowcasting

To exploit non-traditional data, researchers have recently begun utilizing ML models for economic nowcasting [23,25,26]. ML models have been shown to handle wide- and large-scale data and can manage collinearity. Further, they have been shown to capture nonlinear interactions between the predictors and the target [24,27,30,37,58].

We use some of the recently popularized parametric and non-parametric ML approaches, such as elastic net [59], support vector machines [60], random forest [61], gradient boosting [62], and feedforward artificial neural networks [63]. For each model, there are many variations proposed in the literature. However, we focus on the simpler version of each model to test their feasibility for macroeconomic nowcasting in Canada. In the remainder of this section, we provide a high-level description of these models. For further details on them, refer to Appendix B and various textbooks [14,37,63,64].

The elastic net (ENT) is a regularized linear regression model. Here, the objective is similar to that of the OLS with the addition of $L_1$ and $L_2$ penalties depending on how large the sum of the parameters $\mathbf{w}$ can become. In an ENT regression, the combination of $L_1$ and $L_2$ penalties allows for learning a sparse model while encouraging grouping effects, stabilizing regularization paths, and removing limitations on the number of selected variables [59]. (Note: a regression model that uses only the $L_1$ penalty is a Lasso regression, and a model that uses only the $L_2$ penalty is a Ridge regression [14,59].)

Support vector regression (SVR) is based on support vector machines, where the task is to find a hyperplane that separates the entire training dataset into, for example, two groups, by using a small subset of training points called support vectors. In SVR, the main objective is to determine a decision boundary at a distance from the support vectors such that the data points closest to the hyperplane are within that boundary line. This gives us the flexibility to define how much error is acceptable in our model [60,65].

Another popular approach is random forest regression (RFR), a decision-tree-based ensemble learning method built using a forest of many regression trees. This is a non-parametric approach that addresses the multicollinearity problem slightly differently from parametric approaches such as ENT. Random forest is a bagging (bootstrap aggregation) approach; that is, each tree is independently built from a subset of the training dataset. Each sample randomly selects a subset of features from the available set of features, helping in decorrelation. The final prediction is performed by averaging the predictions of all regression trees [61,66].

Similar to RFR, gradient boosting regression (GBR) is a tree-based, non-parametric ensemble learning approach. However, unlike RFR, GBR is based on a boosting in which a sequence of weak learners (decision trees) are built upon a repeatedly modified version of the training dataset. In this approach, the base learners are sequentially improved by repeatedly applying the same base learner with the target’s residuals as the outcome of interest [62].

The feedforward artificial neural network (ANN) with hidden layers consists of multiple layers of artificial neurons sandwiched between input and output layers. In this approach, the weighted sum of the first layers is typically passed through a nonlinear activation function resulting in a nonlinear function of the inputs. Then, the outputs are sent to the next layer, and the process continues until the last layer. Once we obtain the final output from the network, we measure how good that output is compared to the target’s actual value using an objective function, for example, mean squared error. Given these results, we go back and adjust the weights and biases of the network. Typically, we need a large training dataset to achieve good performance using ANN [63,67].

Note that there are many advanced versions of both tree-based methods, such as LightGBM [68], and ANN-based methods, such as long short-term memory (LSTM) [69], proposed in the literature. However, in our application, the performance of these models did not surpass that of GBR. This could be attributed to our limited data sample. However, with a larger number of data points, the performance of these advanced approaches may improve.

3.2. Machine Learning Model Cross-Validation

Overfitting is commonly attributed to the use of ML models—especially nonlinear models. ML models have many parameters that can be optimized to improve prediction accuracy (commonly known as hyperparameter tuning). Therefore, it is straightforward to tune the model to perform well on a specific dataset, for example, an in-sample training set. However, such models generally fail to perform well when applied to unseen data [14].

This problem can be alleviated using k-fold cross-validation techniques. In the standard approach, the training sample is randomly split into k-folds, then for each iteration, the $k-1$ folds are used for in-sample training, and the kth fold is used for out-of-sample testing [14]. Such procedure effectively avoids overfitting. However, random splitting of the training sample breaks the order of series (autocorrelation) and could lead to the use of future data points for past predictions, giving an unfair advantage to the model. For these reasons, it may not practical to use it in the same way in nowcasting models [38].

This challenge can be mitigated using an expanding window approach for cross-validation (Figure 2). Here, the end portion of the training set, often called a validation set, is kept aside for model tuning and cross-validation. Note that for each iteration of the expanding window, the training sample is increased by one period and the model prediction is performed on the next period from the validation sets [38]. Consequently, the model parameters can be chosen based on model performance on the validation sets (see Appendix C for additional details). This approach is useful for nowcasting during normal economic periods. However, in cases where the test sample includes an economic crisis but the validation sample does not, traditional expanding window cross-validation could be challenging because (a) the distribution of test and validation samples are quite different and (b) the model is predominantly tuned for normal periods and therefore may not perform well for out-of-sample crisis periods.

We implement a slightly altered version of the expanding window approach tailored to macroeconomic nowcasting models (see Figure 3). We randomly sample n-points (one for each iteration) from the validation superset (highlighted in gray), beginning just before the GFC and continuing to the end of the training set, and use them as a validation sample. For each iteration of expanding window validation, only data points that come before the chosen point are used for training—preserving the order of data and temporal dependency between observations (see Figure 4).

In the current exercises, since we have the COVID-19 period in the test set, using a random sampling strategy leads us to include a few sample points from the GFC in each validation subset. Therefore, the proposed approach helps make the distribution of validation and test sets somewhat similar (see Figure A4 in Appendix C) and assists in selecting a model that can perform well for both normal and crisis periods. Furthermore, our approach removes the restriction on the number of validation sets we can sample that could be binding in traditional cross-validation approaches. (Note: in our approach, due to random sampling, some observations may be selected more than once in the validation subsets and some may never be selected. This could lead to overfitting if too many validation sets are sampled.)

Further, instead of using all payment streams in each model or manually selecting a few payment streams for the given macro indicator, we use a data-driven approach for feature selection. We treat the number of payment steams p as similar to a model parameter and use the expanding window cross-validation approach to optimally select the best p streams for each target variable based on their performance on the training and validation sets. Further details on cross-validation and model selection are discussed in Appendix D. Note that the ENT model is designed to handle a large number of predictors. However, in our case, the out-of-sample performance did not improve when we used ENT with all predictors. Similarly, RFR can also accommodate a large set of predictors by constructing trees with different subsets of features, but using RFR with all predictors did not outperform the selected approach in terms of prediction performance.

3.3. Machine Learning Model Interpretability

Another problem commonly attributed to the use of ML model is the loss of interpretability due to their complex nature. However, interpretability is essential for many applications—including macroeconomic prediction [16,32,37].

Some ML models employed here, such as ENT, SVR, and the simple tree-based learning models, are inherently interpretable, but only to a certain extent [59,61]. Tree-based ensemble approaches such as GBR and RFR can also be interpreted—to some extent—using impurity or permutation-based feature importance methods [61,70]. However, they are mostly used for global (entire sample) interpretations. Furthermore, each of those methods has different interpretability approaches, making them difficult to compare. To address these challenges, we use the Shapley value-based model agnostic approach SHAP developed in [43]. The SHAP can be used for both the local (each prediction instance) and global interpretations. Moreover, the SHAP can be used for dependence plots which could be valuable to obtain further insights.

In SHAP, the Shapley value method from coalitional game theory [45,46] is used to fairly distribute the “payout” (=the prediction) among the “players” (=the predictors) [44]. In nowcasting, SHAP can be used to fairly distribute the ML model prediction among the set of predictors $X_t$ at each time horizon t for local model interpretation. Further, using Shapley values for each instance t, we can compute the global interpretation of the ML models in the form of feature importance for the entire training or testing datasets.

Ref. [43] propose two approaches based on the type of underlying process used to compute the Shapley values: (1) KernelSHAP, a kernel-based estimation approach, which can be used for many ML models, such as ENT, ANN, and tree-based models; (2) TreeSHAP, a computationally efficient approach for Shapley value estimation used only for tree-based ML models, such as decision trees, random forests, and gradient boosting models.

SHAP approaches are reliable because they are developed based on the Shapley value, which has game-theoretical foundations. However, the time required to estimate Shapley values could increase exponentially with the number of predictors. This is not a significant concern for our application because we have comparatively fewer predictors and smaller sample sizes. The KernelSHAP method also suffers from collinearity in the predictors, which could represent a concern for our work, given that several predictors are correlated. These problems can be mitigated to an extent using TreeSHAP, but only for tree-based models. Another challenge with SHAP approaches is that it is possible to create intentionally misleading interpretations to hide the bias. Furthermore, in some cases, the outcomes can easily misinterpret and lead to ambiguous conclusions. Therefore, SHAP should be used with caution [70,71,72]. Note that the Shapley value-based interpretation approach does not provide causal inference or any optimal statistical criterion. They only explain the marginal contribution of each feature towards prediction. The Shapley value-based approaches could also be used to select the best subset of predictors. However, in the context of model interpretation, the focus is only on computing the marginal contribution of all predictors [43,70]. Further details on Shapley values and SHAP with a test example are given in Appendix E.

3.4. Case Specifications and Model Training

As a benchmark (or base case), we employ a linear regression model using OLS. Here, we use the first available lagged target along with the latest CPI, unemployment, CFSI, and CBCI. The CFSI is computed using data from the following seven market segments: equity market, Government of Canada bond market, foreign exchange market, money market, bank loans market, corporate bonds market, and housing market and CBCI is based on the survey of Canadian households, which provides a measure of consumer optimism on current economic conditions, and both are available immediately and carry comprehensive and useful information about the macro indicators. Along with CPI and UNE (available with a one-to-two week delay), these predictors form a strong benchmark to assess information gain using payments systems data.

In the main case of interest, along with the predictors specified in the base case, we use the payments data listed in Table 1. Here, we first use DFM to assess the marginal contribution of payments data when used in a econometric model. Next, we test the usefulness of the various ML models discussed above in Section 3. Finally, we compare the performance of the ML models with the benchmark case and DFM. We use RMSE as the key performance indicator for out-of-sample model evaluation.

For all cases, using a procedure similar to [1,9], we perform nowcasting at three monthly time horizons, extending from the start of the month of interest (t) until the month before the official release ($t+2$). As we advance in time, we include new predictors when they become available. For example, GDP nowcasting at time horizon t, that is, on the first day of the month of interest, we use the latest available benchmark variables and the monthly aggregated payments data available at $t-1$. Model $\mathcal{F}$ can be specified as (Note: GDP is released with a two-month lag, CPI and UNE are released with a one-to-two week lag, and CFSI, CBCC, and payments data are available the day after the end of the period)

$$\begin{array}{cc}\hfill {\widehat{GDP}}_t=& \mathcal{F}({GDP}_{t-3},{CPI}_{t-2},{UNE}_{t-2},{CFSI}_{t-1},{CBCC}_{t-1},{Payments}_{t-1}).\end{array}$$

(4)

Similarly, at the next nowcasting horizons $t+1$ and $t+2$, using the latest available predictors, the models can be specified as follows. (Note: at the $t+2$ nowcasting horizon (on the first day of the month in which the target month’s macro indicators are released), we have $t+1$ months of payments data. However, we do not include this because we are interested mainly in assessing the usefulness of t month’s payment data to predict t month’s macro variables. Furthermore, note that the left-hand sides in Equations (4)–(6) represent the same month’s target but they are estimated at different time horizons.)

$$\begin{array}{cc}\hfill {\widehat{GDP}}_{t+1}=& \mathcal{F}({GDP}_{t-2},{CPI}_{t-1},{UNE}_{t-1},{CFSI}_t,{CBCC}_t,{Payments}_t),\end{array}$$

(5)

$$\begin{array}{cc}\hfill {\widehat{GDP}}_{t+2}=& \mathcal{F}({GDP}_{t-1},{CPI}_t,{UNE}_t,{CFSI}_t,{CBCC}_t,{Payments}_t).\end{array}$$

(6)

We train the nowcasting models using the expanding window approach as schematically outlined in Figure 4. First, we divide the dataset into two subsets: a training set (in-sample) for model training and a testing set (out-of-sample) for predictions. The OLS and DFM models are directly trained on the training set and used for predictions on the test set. The ML models, which require extensive hyperparameter tuning and cross-validation, are trained using the following procedure:

• From the training sample, we select two dates covering the wider range of training data as a validation superset (Figure 2) and randomly choose a set of n sample points as a validation set (where $n=24$ points, it is the same size as the test sample). (Note: we choose a start date just before the GFC period and an end date just before the test set, then select n random data points between these two dates as a validation set. This helps to include a few data points from the crisis period in each fold of the validation subset, at the same time avoiding use of a large cross-validation sample.)
• Thereafter, for each sample date in the validation subset, we select all the sample points before that date for training and use the sample date for prediction (Figure 4).
• Next, for each model, we specify the grid for selected hyperparameters. Then, for each value of a specified parameter, we iterate over the validation subset and compute RMSE.
• Steps 2 and 3 are repeated k times for the same set of hyperparameters but with a different validation subset randomly sampled from the validation superset ($k=5$ fold cross-validation).
• Next, we select the best set of model parameters, i.e., the parameters with the lowest average validation RMSE (averaged over k folds) as the final model.
• Finally, the chosen model is used for predictions on the test set by utilizing the standard expanding window approach over the training and test set (Figure 4).

4. Results and Discussion

The payments data used for nowcasting exercises range from Mar 2004 to Dec 2020. The in-sample training period is Mar 2005–Dec 2018 ($N=166$ sample points). The out-of-sample testing period is Jan 2019–Dec 2020 ($N=24$). Our training set includes the 2008 GFC period, and the test set combines a normal economic growth period (Jan 2019–Feb 2020) and part of the ongoing COVID-19 crisis period (Mar 2020–Dec 2020). This allows us to examine model performance during both normal and crisis periods.

Year-over-year GDP, RTS, and WTS growth rate nowcasting performance for the various cases outlined in Section 3.4 are discussed below.

Table 2. Out-of-sample RMSE comparisons for seasonally adjusted YOY growth rate of macro variables at time horizon t on the first day of the month of interest (top panel), $t+1$ on the first day after the month of interest (middle panel), and $t+2$ on the first day, two months after the month of interest (bottom panel) ^a.

Target b	Benchmark c	Main DFM d	Main ML e	RMSE Reduction (%) f
GDP	4.58	3.95	3.70	19
RTS	7.88	7.40	7.38	7
WTS	6.34	5.81	5.74	10
GDP	3.97	2.98	2.43 *	39
RTS	8.47	6.36	5.44 *	36
WTS	7.17	6.18	4.28 *	41
GDP	2.84	2.63	2.18	23
RTS	7.60	6.15	5.55	25
WTS	6.24	5.76	4.72	24

• ^a In-sample training period, Mar 2005–Dec 2018, and out-of-sample testing period, Jan 2019–Dec 2020.
• ^b GDP—gross domestic product; RTS—retail trade sales; WTS—wholesale trade sales. Note: we use the latest available values of these targets. We also perform similar exercises by using target variables at first release (real-time vintages). These results are presented in Appendix G.
• ^c As a benchmark, we use OLS with CPI, UNE, CFSI, CBCC, and the first available lagged target variable (i.e., the second lag at nowcasting horizon t).
• ^d For the main DFM case, we use payments data along with the predictors in the benchmark case. Similar to the model employed in [53], we use the DFM model with two factors and one lag in the VAR driving the dynamics of those factors. Idiosyncratic components are assumed to follow an AR(1) process. Note: including additional factors does not improve model performance.
• ^e We use GBR because it consistently performs better than other ML models (see Table A1 in Appendix B). We select model parameters using the cross-validation procedure outlined in Appendix C and Appendix D. For example, the selected model for GDP nowcasting at t + 1: learning_rate is 0.1, max_depth is 1, and n_estimators is 1000 (see Appendix B for further details).
• ^f Percentage reduction in RMSE over the benchmark model using ML on the main case. * denote statistical significance at the 10% level, for the Diebold–Mariano test using the benchmark.

compares nowcasting performance—in terms of out-of-sample RMSE—of the DFM and ML model, specifically gradient boosting model which consistently performed better across different target variables and time-horizons compared to the other models (see

Table A1. Out-of-sample RMSE comparisons of DFM with ML models for seasonally adjusted YOY growth rate of macro variables at the horizons $t+1$ (top panel), and $t+2$ (bottom panel) for the main case ^a.

Target b	DFM c	ENT d	SVR d	RFR d	GBR d	ANN d	% Reduction e
GDP	1.00	0.96	1.41	1.11	0.81 f	0.82	19
RTS	1.00	0.89	1.27	1.07	0.85	1.02	15
WTS	1.00	0.96	1.14	0.82	0.69	0.51	31
GDP	1.00	0.87	1.62	1.14	0.82	0.85	18
RTS	1.00	0.87	1.36	1.15	0.90	0.97	11
WTS	1.00	0.89	1.19	0.91	0.81	0.70	19

• ^a In-sample training period, Mar 2005–Dec 2018, ($p=166$) and out-of-sample testing period, Jan 2019–Dec 2020, ($p=24$). All RMSEs are normalized with respect to DFM. The performance gain using ML models for time horizon t are much smaller; however, the GBR model performed better compared to the other ML models.
• ^b RTS—retail trade sales; WTS—wholesale trade sales. Note: we use the latest available values of targets for these exercises.
• ^c For DFM, we use payments data along with the predictors in the benchmark case. We use the DFM model with two factors and one lag in the VAR driving the dynamics of those factors. Idiosyncratic components are assumed to follow an AR(1) process.
• ^d We use elastic net (ENT), support vector regression (SVR), random forest regression (RFR), gradient boosting regression (GBR), and ANN. For these ML models, we select the model parameters and number of payment predictors based on target variables using the cross-validation procedure outlined in Section 3. Further details on these models are provided in Appendix B. Model selection and cross-validation procedures are detailed in Appendix C and Appendix D.
• ^e Percentage reduction in RMSE over DFM for the GBR model.
• ^f The models with out-of-sample prediction RMSE less than DFM (<1) are highlighted (bold) and the best model is also underlined

in Appendix B) on the main case against the benchmark models at the time horizons t, $t+1$, and $t+2$.

Our results suggest that the payments systems data in a factor model can provide notable reductions in nowcasting RMSEs for all three macro variables considered. (Note: in this case, we use the DFM model with two factors. Including more factors does not improve the results.) Specifically, we observe a 20–$25\%$ reduction in RMSE over the benchmark case in nowcasting GDP, RTS, and WTS at time horizon $t+1$, i.e., when we use the same month’s payments data as the target variable. Comparatively, the information gained using payments data is smaller at prediction horizon t, that is, when we use the first lag of payments data, and $t+2$, that is, when the values of all benchmark indicators are available at t along with the first lag of the target variables. These results suggest that payments data provide the most prediction value when the given month’s payments data are used immediately to predict the same month’s macro variables (at $t+1$ time horizon, i.e., on the first day of the next month).

Next, we use payments data in the ML models (see Table 2). Overall, in predicting GDP, RTS, and WTS at all three-time horizons, the GBR model performs better than DFM (up to $20\%$) and other ML models (see Table A1 in Appendix B). Specifically, we observe a 35–$40\%$ reduction in RMSE over the benchmark model at time horizon $t+1$. The main case predictions at this time horizon are statistically significant for the Diebold–Mariano test using the benchmark. These results suggest that flexible ML models, such as GBR, can handle multiple predictors efficiently and are useful in increasing prediction accuracy. Moreover the proposed randomized expanding window CV approach selects models that perform slightly better (3 to 12%) than the standard expanding window CV approach in nowcasting GDP, RTS, and WTS.

We separately test our model’s out-of-sample performance during a normal economic period (Jan 2019–Feb 2020) and the COVID-19 period (Mar 2020–Oct 2020) of the test sample (see

Table A2. Out-of-sample RMSE comparisons for seasonally adjusted YOY growth rates of GDP, RTS, and WTS at nowcasting horizon $t+1$ using the gradient boosting model ^a.

Targets	Pre-COVID-19 Test Set b	COVID-19 Test Set c
GDP	16	34
RTS	14	35
WTS	27	37

^a At time horizon $t+1$, we use current, i.e., t month’s payments data, to predict the same month’s macro variables on the first day of the subsequent month. ^b For the pre-COVID-19 test set (or normal period): in-sample training period, Mar 2005–Dec 2018; out-of-sample testing period, Jan 2019–Feb 2020. Those numbers show the percentage gain over benchmark cases for the same period. We use OLS with CPI, UNE, CFSI, CBCC, and the first available lagged target variable for the benchmark. ^c For the COVID-19 test set (or crisis period): in-sample training period, Mar 2005–Feb 2020; out-of-sample testing period, Mar 2020–Dec 2020. These numbers show the percentage gain over benchmark cases for the same period.

in Appendix F). We observe a higher gain using payments data during the time of crisis (up to 35% RMSE reduction) compared to the normal period (15–25% reduction in RMSE). These results show that the payments data are useful during normal periods, and their significance surges during crises, substantially improving our model’s performance during those periods.

Incorporating payments data in ML models provides downturn and recovery indications (during crisis periods) much better than the benchmark model in both in-sample and out-of-sample periods. We conjecture that this is due to the new and timely information provided by the payments data and ML model flexibility, allowing this data to provide better predictions during crisis periods.Therefore, these results suggest that the timeliness of payments data and the ability to capture nonlinear interactions of ML models is, primarily, helpful.

Finally, we compare the GDP nowcasting performance of our model with the real-time vintages (first releases) and the latest vintages (see

Table A3. Out-of-sample RMSE comparisons for the seasonally adjusted YOY growth rate of GDP at nowcasting horizons t, $t+1$, and $t+2$ using the gradient boosting model ^a.

Nowcasting Horizon b	Latest Vintages c	Real-Time Vintages d
t	3.73	3.88
$t+1$	2.61	2.92
$t+2$	2.66	2.68

^a In-sample training period, Mar 2005–Dec 2018, and out-of-sample testing period, Jan 2019–Dec 2020. ^b Nowcasting horizons: t is on the first day of the month of interest (top panel), $t+1$ is on the first day after the month of interest (middle panel), and $t+2$ is on the first day, two months after the month of interest (bottom panel). ^c We use the latest available monthly levels of seasonally adjusted GDP from Statistics Canada Table 36-10-0434-01. ^d We use the historical real-time vintages (available as of Mar 2020) of seasonally adjusted monthly GDP from Statistics Canada Table 36-10-0491-01.

in Appendix G). Comparatively, the models using payments data perform better against the latest vintages. This makes sense, given that the latest vintages are more accurate (due to multiple revisions) than the real-time ones. Therefore, these results further emphasize the importance of timeliness of payments data for effectively provisioning accurate estimates of key macro indicators in near real-time, which is important to monitor the economy—especially during the crisis periods such as the COVID-19 pandemic.

Model Interpretation and Payments Data Contribution

We now discuss the Shapley value-based interpretation of ML model predictions using the SHAP. We primarily focus on nowcasting GDP at time horizon $t+1$ using the tuned GBR model. However, we discuss a few key results for nowcasting RTS and WTS using a GBR, at the end of this section.

For demonstration, we use the entire sample (Mar 2005–Dec 2020) for training. In Figure 5, (left), we plot the SHAP global feature importance obtained by averaging the absolute Shapley values for each predictor across the training set (in-sample). This plot shows, on average, how much each feature influences the model prediction. These features are ranked according to their average influence (from high to low). For example, in the case of in-sample training data, both the encoded paper value and the GDP lag have a strong influence. However, the encoded paper stream is the strongest predictor (on average, it changes the GDP growth rate by $0.6$ points). This is followed by the unemployment lag feature, LVTS-T2 value, and the sum of all the ACSS streams (Allstreams value). Note that for the tree-based models like GBR, we can also use impurity or permutation-based global feature importance [61]. In our case, the permutation approach gives similar results as SHAP for the top three major predictors and matches eight out of the top ten highest contributors but slightly in a different order. Moreover, all three approaches rank the same three predictors in the top five list, and the encode paper stream remains the most prominent feature in all approaches (see Figure A7).

In Figure 5, (right), we show the global feature importance plot for the COVID-19 period with high negative growth rates (Mar–Dec 2020). During this crisis period, the influence of the encoded paper stream increases substantially along with the Allstream, AFT debit, and POS payments. GDP lag, the second most important feature for the entire training sample, loses its prediction power during the COVID-19 crisis period. A similar contribution from several payment streams is observed during the GFC period. These results suggest that, although lagged macro indicators influence GDP growth rates during normal periods, they do not add much value during crisis periods due to the delayed signal. During such periods, payments data become much more valuable and contribute well to macroeconomic prediction.

Next, using the SHAP “force” plots, we compute local feature importance—that is, the usefulness of each feature for a chosen sample point in the training set. Such insights could be important for nowcasting exercises because, during each step of the expanding window approach (i.e., when we advance by one month), the force plots could provide additional insights into each month’s predictions by highlighting marginal contributions of individual predictors.

For instance, in Figure 6, we plot the Shapley values as forces for Feb (top), Mar (middle), and Apr (bottom) 2020. Here, each Shapley value is an arrow that forces an increase (higher in red) or decrease (lower in blue) in the prediction from the baseline (the average of all predictions). The length of these arrows indicates the magnitude of the Shapley value for that feature. These forces balance at the model prediction of that instance shown as $f\left(x\right)$. For Feb 2020, just before the pandemic began affecting Canada’s economy, most payment predictors are positive (red) and pushing GDP growth higher. However, for Mar 2020 (the first month of the COVID-19 crisis), most payment streams show a negative signal (blue). Similarly, for Apr 2020, all predictors show strong negative contributions, pushing the model prediction to the lowest value.

Figure 7 shows the force plots for each instance in the entire training sample rotated and stacked together vertically. We observe red clusters of predictors with positive values (most predictors contributing positively to GDP prediction during those months in our period, for instance, in the top plot in Figure 6) during positive economic growth periods and blue clusters with negative signals during crisis periods, such as the GFC and COVID-19 crisis (most predictors contributing negatively to GDP prediction during those months in our period, for instance, in the bottom plot in Figure 6). Such clustered signals could prove valuable in tracking crises in real time.

In Figure 8, we show dependence plots for encoded paper value (left) and Allstream value (right). These plots capture the relationship between the feature values on the x-axis and the corresponding Shapley values on the y-axis. Observe that the small and negative values of encoded paper growth rates provide higher contributions in Shapley values compared to the positive growth rates. However, both positive and negative growth rates of Allstreams value contribute similarly (or symmetrically). The encoded paper plot (left) suggests that the contribution of this stream, in terms of Shapley values, is small and linear during periods of normal growth. However, during periods of strong negative or positive growth (>=|10|), the contribution of this stream is asymmetrical and nonlinear.

Similar behavior is observed in nowcasting models for RTS and WTS using payments data and GBR model. In Figure 9, we show the dependence plots of RTS with POS payments value (left) and WTS with Allstream value (right). We also show how these payment streams are influenced by CFSI. These plots suggest that at high-stress levels—that is, at high values of CFSI (shown in red) and negative values of payment growth rates; the signal from these payment streams is strong and their contribution is high. However, for low levels of stress (blue) and high positive payment growth rates, the contributions from these streams are positive but small. This confirms the asymmetrical and potentially nonlinear relationship between these streams and the corresponding macro variables.

Finally, in Figure 10, we plot the SHAP global feature importance for the entire training set at time horizon $t+1$ for RTS (left) and WTS (right), respectively. These plots suggest, in the case of RTS, that the Allstream and POS payments values highly influence the model prediction. This makes sense, given that POS payments are commonly used for retail trade. In the case of WTS, the Allstream value has its strongest impact on the model prediction along with the encoded paper and corporate payments value streams, highlighting the importance of these streams in predicting WTS. These plots suggest that, in general, aggregated payments streams (Allstream and encoded paper) are crucial predictors in nowcasting GDP, RTS, and WTS.

5. Conclusions

We use comprehensive and timely payment systems data with DFM and ML models for macroeconomic nowcasting. We find the payments data provide information about the economy in real-time and help reduce dependence on variables released with a significant delay. ML provides a set of advanced econometric tools to effectively process various payment streams and capture the sudden, large, and nonlinear effects of a crisis. To improve the effectiveness of ML models for prediction, we use a Shapley value-based approach for model interpretability and devise a specialized cross-validation strategy to avoid nowcasting model overfitting.

Our results suggest that payments system data and ML models can lower nowcast errors significantly over benchmark models in nowcasting GDP, RTS, and WTS. The most significant improvements are observed when we use the same month’s payments data as the target variables, suggesting the timeliness of the data is driving the prediction gains. Our model’s out-of-sample performance is relatively higher during the COVID-19 period compared to the pre-COVID-19 period. We also notice that the ML model performance changes slightly for different nowcasting cases. However, the gradient-boosting regression model shows consistently good performance. The importance of payments data (especially the encoded paper stream) increases during crisis periods. Nonetheless, some payment streams strongly influence the model even during normal periods.

Additionally, the proposed cross-validation technique can help reduce overfitting and improve accuracy in macroeconomic prediction models. We also demonstrate the usefulness of the Shapley value-based SHAP approach in gaining insights into ML model predictions at each nowcasting step and for the entire training sample. Further, we show how the dependence plots could help understand the relationship between the predictors’ values and their influence on the target. Such insights could be valuable in macroeconomic monitoring and prediction, especially during crises. In conclusion, this paper substantiates the use of payments data and ML models for macroeconomic prediction and provides a set of econometric tools to overcome associated challenges.