Comparing Statistical and Machine Learning Methods for Time Series Forecasting in Data-Driven Logistics—A Simulation Study

Abstract

Many planning and decision activities in logistics and supply chain management are based on forecasts of multiple time dependent factors. Therefore, the quality of planning depends on the quality of the forecasts. We compare different state-of-the-art forecasting methods in terms of forecasting performance. Differently from most existing research in logistics, we do not perform this in a case-dependent way but consider a broad set of simulated time series to give more general recommendations. We therefore simulate various linear and nonlinear time series that reflect different situations. Our simulation results showed that the machine learning methods, especially Random Forests, performed particularly well in complex scenarios, with the differentiated time series training significantly improving the robustness of the model. In addition, the time series approaches proved to be competitive in low noise scenarios.

1. Introduction

Forecasting methods are essential for efficient planning in various logistics domains such as warehousing, transport, and supply chain management. They enable companies to anticipate and plan for future demand, capacity needs, and supply chain requirements. Thereby, different logistics applications require different forecasts due to their unique characteristics. In the transport domain, for example, accurate transportation forecasting enables logistics companies to optimize their transportation networks, reduce transportation costs, and enhance delivery reliability [1,2,3,4,5]. Precise forecasting allows warehouse managers to optimize space use, reduce stock-out risk, and improve overall efficiency [6,7]. In supply chain management, accurate forecasts are, for example, used to optimize resource use across the entire supply chain [8,9,10]. The above references show that the use of forecasting techniques such as time series models and machine learning methods has become increasingly popular in logistics in recent years. However, there is still a lack of consensus on which method is more effective, especially as most methods of comparison in logistics solely rely on comparing the performance on a few data sets [7,11]. In fact, differently from other fields (e.g., [12,13]), there do not exist rigorous benchmark studies in data-driven logistics to the best of our knowledge. In our opinion, the key reason for this is that, outside of specific examples (e.g., [14,15]), there is a lack of freely accessible and well-characterized data sets for benchmarking (e.g., [16,17]) in the logistics research domain. This hampers the analysis of domain-specific pros and cons of method choices or the formulation of general recommendations. To overcome this and to be in line with recent recommendations [18], we therefore focus on simulating data from various statistical time series models that reflect potential logistic scenarios.

Time series models have been used in forecasting for several decades and are widely used in logistics for sales or demand forecasting, see, e.g., [9,19] and the references cited therein. These models are based on historical data and use statistical techniques to identify patterns and trends in the data, which can then be used to make predictions about future demand. Some commonly used time series models in logistics include (seasonal) autoregressive integrated moving averages (ARIMA) and exponential smoothing models. For example, ref. [20] developed an ARIMA multistage supply chain model that is based on time series models. Another example is Prophet [21], a forecasting tool for time series analysis developed by Facebook, which includes additive modeling with components such as seasonality, holidays, and trend flexibility. Ref. [22] examined ARIMA and Prophet models for predicting supermarket sales. The Prophet models showed superior predictive performance in terms of lower errors. Ref. [23] investigated the performance of double exponential smoothing for inventory forecasting.

More recently, machine learning (ML) methods have become increasingly popular for demand forecasting in logistics due to their ability to handle large and complex data sets. There are many literature reviews [24,25,26,27,28] that discuss the use of machine learning techniques in forecasting for supply chain management, including an overview of the various techniques used and their advantages and limitations. However, our comment regarding a lack of neutral benchmarking studies still applies.

Several studies have shown that ML methods such as neural networks, support vector regression, and Random Forests can outperform traditional time series models for specific demand forecasting problems. For example, a study by [11] compared the prediction power of more than ten different forecasting models, including classical methods such as ARIMA and ML techniques such as long short-term memory (LSTM) and convolution neural networks, using a single data set containing the sales history of furniture in a retail store. The results showed that the LSTM outperformed the other models in terms of prediction performance. Another study by [29] also compared the forecasting power of ARIMA and neural networks using a single commodity prices data set. Again the neural network performed better than the ARIMA model. Similar results were obtained in [30,31]. However, other studies have found mixed results, with some suggesting that time series models perform better than ML methods. For instance, ref. [32] compared the forecasting accuracy of ARIMA and neural network models in predicting wind speed for short time intervals. The results showed that the performance of both can be very similar, indicating that a more simple and interpretable forecasting model could be used to administrate energy sources. A comparison of the daily hotel demand forecasting performance of SARIMAX, GARCH, and neural networks also showed that both time series approaches outperformed the neural networks [33]. In the latter examples, one reason may also be the difficulty in tuning complex machine learning procedures. That is one reason why we focus on out-of-the-box machine learning methods in our study.

The comparison of the forecasting performance of ML methods and time series models in logistics has significant implications for businesses seeking to improve their forecasting accuracy. By identifying the most effective forecasting methods, businesses can make better-informed decisions about production, inventory management, and resource allocation. Thus, this work aims to provide a comprehensive comparison of the forecasting performance of time series models and ML methods. Differently from the above-mentioned works that merely focus on single use cases, this task needs more variation in the data sets under study. To this end, we compare various forecasting methods in terms of out-of-the-box forecasting performance on a broad set of simulated time series. We thereby simulate various linear and nonlinear time series that are of importance for logistics and study the one-step forecast performance of different statistical learning methods.

This work is structured as follows: Section 2 presents the different forecasting methods used. More precisely, the (seasonal) ARIMA and TBATS models are presented. In addition, the machine learning approaches (Random Forest and XGBoost) are described in more detail. Section 3 presents the simulation design and framework, while Section 4 summarizes the main simulation results. In Section 5, an illustrative real-world data example is analyzed before the manuscript concludes with a discussion of our findings and an outlook for future research (Section 6).

2. Methods

In this section, we explain the one-step forecasting methods under investigation. There are various strategies for modeling and forecasting time series. Traditional time series models, including moving averages and exponential smoothing, follow a linear approach in which the predictions of future values are linear functions of past observations. Due to their relative simplicity in terms of understanding and implementation, linear models have found application in many forecasting problems [34,35,36]. To overcome the limitations of linear models and account for certain nonlinear patterns observed in real-world problems, several classes of nonlinear models have been proposed in the literature. Examples cover the threshold autoregressive model (TAR) [37] or the generalized autoregressive conditional heteroscedastic model (GARCH) [38]. Although some improvements have been noted, the utility of their application to general prediction problems is limited [39]: since these models were developed for specific nonlinear patterns, they are often unable to model other types of nonlinearities. Here, machine learning methods have been proposed as an alternative for time series forecasting [40,41]. Since it is impossible to cover the entire spectrum of machine learning models and time series methods in our simulation study, we limit ourselves to a selection of what we consider the most common algorithms in data-driven logistics. To evaluate the performance, we compare these methods with a naive approach, where the last observation of the time series is used as a prediction. The time series (Section 2.1) and machine learning methods (Section 2.2) under study are explained in more detail in the next two subsections.

2.1. Time Series Methods

We focus on three different time series models: ARIMA, SARIMA, and TBATS. The first two models are among the most popular models in traditional time series forecasting [42,43] and are often used as benchmark models for comparison with machine learning algorithms [44,45,46]. In addition, TBATS models combine several techniques such as exponential smoothing and Fourier terms, making it particularly adept at handling complex patterns, including multiple seasonalities and nonlinear behaviors [43]. This combination of traditional and advanced methods ensures that we cover a range of forecasting techniques commonly applied in the data-driven logistics domain.

2.1.1. ARIMA

The autoregressive integrated moving average (ARIMA) [47] model is a generalized model of the autoregressive moving average (ARMA) model and builds a composite model of the time series [48]. Denoted as ARIMA(p, d, q), $p,q,d\in \mathbf{N}$, the model is characterized by three key components:

• AR (Autoregression): Represents the regression of the time series on its own past values, capturing dependencies through lagged observations. The number of lagged observations included in the models is given by p.
• I (Integrated): The differencing order (d) indicates the number of times the time series is differenced to achieve stationarity. This transformation involves subtracting the current observation from its d-th lag, which is crucial for stabilizing the mean and addressing trends.
• MA (Moving Average): Incorporates a moving average model to account for dependencies between observations and the residual errors of the lagged observations (q).

In general, a time series ${\left\{x_t\right\}}_t$ generated from an ARIMA(p, d, q) model has the form:

$${\Phi }_p\left(B\right){(1-B)}^d{x}_t={\Theta }_q\left(B\right){\epsilon }_t,$$

where $p,d,q\in \mathbb{N}$ and B is the backshift operator defined as $B:\mathbb{R}\to \mathbb{R}$ with $x_t\mapsto x_{t-1}$. The AR component is described by the polynomial ${\Phi }_p\left(B\right)=1-{\varphi }_{1}B-{\varphi }_2{B}^2-\dots -{\varphi }_p{B}^p$, where ${\varphi }_1,\dots ,{\varphi }_p\in \mathbb{R}$. The MA component is represented by the polynomial ${\Theta }_q\left(B\right)=1+{\theta }_{1}B+{\theta }_2{B}^2+\dots +{\theta }_q{B}^q$, where ${\theta }_1,\dots ,{\theta }_q\in \mathbb{R}$. Residual errors at time t, denoted as ${\epsilon }_t$, are assumed to follow a white noise process with zero mean and constant variance.

2.1.2. SARIMA

With seasonal time series data, short-term non-seasonal components likely contribute to the model. Therefore, we need to estimate a seasonal ARIMA model incorporating non-seasonal and seasonal factors into a multiplicative model [48]. The general form of a seasonal ARIMA model is denoted as SARIMA$(p,d,q){(P,D,Q)}_m$, where p is the non-seasonal AR order, d is the non-seasonal differencing, q is the non-seasonal MA order, and P, D, and Q are the similar parameters for the seasonal part. The parameter mm represents the number of time steps in one full seasonal cycle, also known as the period length.

2.1.3. TBATS

For time series data exhibiting complex and diverse seasonal patterns, TBATS (Trigonometric Seasonal Exponential Smoothing) is a robust modeling approach. Introduced as an extension of exponential smoothing methods, TBATS accounts for different seasonalities through a combination of trigonometric functions and exponential smoothing [49]. The model is particularly effective in handling multiple seasonal cycles, making it suitable for data sets with intricate temporal structures.

The general form of a TBATS model consists of several components as described below:

• T (Trend): Captures the overall trend in the time series using an exponential smoothing mechanism.
• B (Box–Cox Transformation): Applies the Box–Cox transformation [50] to stabilize variance and ensure the homogeneity of variances.
• A (ARIMA Errors): Incorporates ARIMA errors to capture any remaining non-seasonal dependencies.
• S (Seasonal): Utilizes trigonometric functions to model multiple seasonal components, accommodating various seasonal patterns.

2.2. Machine Learning Methods

Machine learning methods are increasingly being used to address time series prediction problems. In fact, there exist too many approaches to consider in a comparison study like ours. We therefore restricted ourselves to a class that has already been successfully used for predictions in the logistics context [1,51,52,53,54]: tree-based ensemble learners. We thereby focus on two models, each studied with and without differencing: Random Forest and XGBoost on trees which are briefly introduced below. These methods are particularly well-suited to time series forecasting due to their flexibility in capturing both linear and nonlinear patterns, as well as their robustness to overfitting and ability to handle large data sets with complex structures.

2.2.1. XGBoost

Gradient boosting is an ensemble machine learning technique often used in classification and regression problems, and is particularly popular in predictive scenarios [55]. As an ensemble technique, gradient boosting combines the results of several weak learners, referred to as base learners, with the aim of building a model that generally performs better than the conventional single machine learning models. Typically, gradient boosting utilizes decision trees as base learners. Like other boosting methods, the core idea of gradient boosting is that, during the learning procedure, new models are built and fitted consecutively and not independently to provide better predictions of the output variable. Thereby, new base learners are constructed with the aim of minimizing a loss function associated with the whole ensemble. Instances that are not predicted correctly in previous steps and score higher errors are correlated with larger weight values so that the model can focus on them and learn from its mistakes.

XGBoost stands for Extreme Gradient Boosting and is a specific implementation of gradient boosting [56]. It incorporates randomization and regularization techniques to reduce overfitting while increasing training speed. Moreover, it computes second-order gradients of the loss function, which provides more information about the gradient’s direction, making it easier to minimize the loss function.

In general, the hyperparameters for XGBoost can be divided into two categories [56]: General boosting parameters, including the number of iterations and the learning rate, which controls how much information from a new tree will be used in the boosting step. Second, in base learner dependent parameters. When trees are used as base learners, the additional hyperparameters are used to control the complexity of the individual trees. Examples include limiting the maximum tree depth or specifying a minimum number of samples in each leaf [57]. There also exist other boosting variants [58,59,60], but we concentrate on XGBoost as it has emerged as one of the key machine learning models for prediction and was also referred to as ‘the Queen of Machine Learning’ [61] in this context. XGBoost models have also been used for time series forecasting, e.g., [62,63]. For example, in [64], the potential of XGBoost in retail for predicting store sales was investigated while ref. [1] studied this for predicting the travel time of NYC cabs.

2.2.2. Random Forest

A Random Forest [65] is a machine learning method based on building ensembles of decision trees. It was developed to address predictive shortcomings of traditional Classification and Regression Trees (CARTs) [66]. Random Forests consist of a large number of weak decision tree learners, which are grown in parallel to reduce the bias and variance of the model at the same time [65]. For training a Random Forest, bootstrap samples are drawn from the training data set. Each bootstrap sample is then used to grow a(n unpruned) tree. Instead of using all available features in this step, only a small and fixed number of randomly sampled $m_{try}$ features are selected as split candidates. A split is chosen by the CART-split criterion for regression, i.e., by minimizing the sum of squared errors in both child nodes. Instead of the CART-split criterion, many other distances, such as the least absolute deviations of the mean (L1-norm), can also be used. These steps are repeated until B such trees are grown, and new data are predicted by taking the mean of all B tree predictions. The most important hyperparameters for the Random Forest [67] are as follows:

• B is the number of grown trees. Note that this parameter is usually not tuned since it is known that more trees are better.
• The cardinality of the sample of features at every node is $m_{try}$.
• The minimum number of observations that each terminal node should contain (stopping criteria).

Though there exist other variants of bagged tree-based ensembles [68,69], we concentrate on the Random Forest as it is the best known method that is often seen as the machine learning benchmark procedure, e.g., [70]. In addition, Random Forests have also been frequently used for time series forecasting [1,71]. For example, in [72], a Random Forest approach was used to model real-time delivery time forecasts in online retailing while ref. [73] applied Random Forest to predict product demand for grocery items.

While machine learning methods are quite en vogue, we should not neglect the advantages of time series methods in terms of interpretability. Here, time series approaches enable a clearer understanding of the factors influencing the predictions.

3. Simulation Set-Up

In our simulation study, we compare the one-step forecast prediction performance of the methods described in Section 2. All simulations were conducted in the statistical computing software R [74]. We use the forecast package [75] for all time series approaches under consideration. For the machine learning methods, we used the ranger [67] and xgboost [76] packages for Random Forest and XGBoost, respectively. The concrete simulation settings and data generating processes (DGPs) are described below.

3.1. Data Generating Processes

We consider twelve DGPs in total—an autoregressive model (AR), two bilinear models (BLs), two nonlinear autoregressive models (NARs), a nonlinear moving average model (NMA), two sign autoregressive models (SARs), two smooth transition autoregressive models (STARs) and two TAR models. They are summarized in

Table 1. Data generating processes (DGPs) used in the simulation study. The error terms ${\epsilon }_t$ are i.i.d $\mathcal{N}(0,1)$.

Model Type	Variant(s)	Data Generating Process
Autoregressive	AR	$x_t=0.5x_{t-1}+0.45x_{t-2}+{\epsilon }_t$
Bilinear	BL 1	$x_t=0.7x_{t-1}·{\epsilon }_{t-2}+{\epsilon }_t$
	BL2	$x_t=0.4x_{t-1}-0.3x_{t-2}+0.5x_{t-2}·{\epsilon }_{t-1}+{\epsilon }_t$
Nonlinear Autoregressive	NAR 1	$x_t=\frac{0.7|x_{t-1}|}{|x_{t-1}|+2}+\epsilon$
	NAR2	$x_t=\frac{0.7|x_{t-1}|}{|x_{t-1}|+2}+\frac{0.35|x_{t-2}|}{|x_{t-2}|+2}+\epsilon$
Nonlinear Moving Average	NMA	$x_t={\epsilon }_t-0.3{\epsilon }_{t-1}+0.2{\epsilon }_{t-2}+0.4{\epsilon }_{t-1}{\epsilon }_{t-2}-0.25{\epsilon }_{t-2}^2$
Sign Autoregressive	SAR 1	$x_t=\mathrm{sign}\left(x_{t-1}\right)+{\epsilon }_t$,
	SAR 2	$x_t=\mathrm{sign}(x_{t-1}+x_{t-2})+{\epsilon }_t$,
Smooth Transition Autoregressive	STAR 1	$x_t=0.8{\epsilon }_t-{\displaystyle \frac{0.8{\epsilon }_{t-1}}{1+exp(-10x_{t-1})}}+{\epsilon }_t$,
	STAR 2	$x_t=0.3x_t+0.6x_{t-2}+{\displaystyle \frac{0.1-0.9x_{t-1}+0.8x_{t-2}}{1+exp(-10x_{t-1})}}+{\epsilon }_t$,
Threshold Autoregressive	TAR 1	$x_t=\left\{\begin{array}{cc}0.9x_{t-1}+{\epsilon }_t& \mathrm{if}|x_{t-1}|\le 1\\ -0.3x_{t-1}-{\epsilon }_t& \mathrm{if}|x_{t-1}|>1\end{array}\right.$
	TAR 2	$x_t=\left\{\begin{array}{cc}0.9x_{t-1}+0.05x_{t-2}+{\epsilon }_t& \mathrm{if}|x_{t-1}|\le 1\\ -0.3x_{t-1}+0.65x_{t-2}-{\epsilon }_t& \mathrm{if}|x_{t-1}|>1.\end{array}\right.$

, where the error terms ${\epsilon }_t$ are independent and identically distributed with a standard normal distribution.

Similar models have been used to evaluate time series forecasts [45] and are of importance in data-driven logistics. In particular, autoregressive models (AR, NAR1, and NAR2) are well suited to capturing the temporal persistence and trends often observed in historical logistics demand data, such as warehouse throughput or vehicle routing sequences [77]. Bilinear models (BL1 and BL2) reflect the complex interactions within logistics systems. For example, the interaction between past demand and various external factors such as weather conditions, production schedules, or transportation disruptions can have a substantial impact on future demand patterns. Bilinear models have been shown to capture such intricate interactions effectively, making them suitable for complex logistics environments where multiple variables influence each other simultaneously [78]. The nonlinear moving average (NMA) model is apt for situations where interdependencies exist between multiple factors influencing logistics outcomes, such as supply chain delays, inventory dynamics, or market fluctuations. This model accounts for the nonlinear relationships between past error terms, which can be influenced by the aggregation of multiple small factors. Sign autoregressive models (SAR1, SAR2) are useful in logistics systems where certain events, such as strikes, weather events, or sudden demand shifts, cause abrupt directional changes in future demand. These models can capture such threshold effects, providing valuable insights into logistics system resilience and responsiveness. Smooth transition autoregressive models (STAR1, STAR2) are particularly relevant for logistics systems that experience gradual transitions in demand patterns due to external factors like economic cycles, regulatory changes, or long-term supply chain restructuring. These models can help to predict how demand or supply dynamics may evolve over time as conditions change smoothly. Threshold autoregressive models (TAR1 and TAR2) are well suited to logistics settings where distinct operational regimes exist, such as different levels of demand or supply based on specific conditions like inventory thresholds or transportation capacity limits. By modeling regime-switching behavior, TAR models can provide insights into logistics processes that exhibit different behaviors under different operational conditions. This diverse set of DGPs depicts many aspects of the multi-layered nature of logistics data, which includes persistence, interactions, complicated dependencies, directional influences, smooth transitions, and different regimes. In the absence of comprehensive benchmark problems, this set-up allows us to evaluate the adaptability of forecasting methods in dynamic logistics scenarios.

3.2. Additional Complexities

To add additional complexity to the analysis, we have incorporated settings with a jump process and a random walk [48] into each DGP. The jump process captures sudden, abrupt changes in the system’s behavior, representing regime shifts that may occur in logistics due to unforeseeable events such as supply chain disruptions, equipment failures, or market shocks [79,80]. For example, sudden demand surges during a pandemic or temporary halts in operations due to extreme weather events are real-world analogs of such jumps. The random walk, by contrast, models persistent, stochastic variations that add noise to the data, reflecting phenomena such as cumulative forecasting errors, drifting demand trends, or inaccuracies in inventory measurements [48,81]. These complexities are particularly relevant to logistics scenarios where external factors introduce substantial uncertainty and variability. Our study considers four different scenarios: (1) the DGP without additional complexity, (2) the DGP superposed with the jump process, (3) the DGP superposed with random noise, and (4) the DGP superposed with both the jump process and random noise. The jumps are modeled using a compound Poisson process ${\left\{p_t\right\}}_t$ [79]. The original DGP ${\left\{x_t\right\}}_t$ is then superposed by $p_t$ as follows:

$$x_t^{*}=x_t+p_t,$$

where $x_t^{*}$ denotes the resulting DGP, and the compound Poisson process is given by

$$p_t=\sum _{i=1}^{N_t}{Z}_i,$$

where $N_t$ follows a Poisson distribution with parameter $\lambda$ and $Z_i\sim \mathcal{N}(0,{\sigma }_p^2).$ For the jump experiments, we set ${\sigma }_p^2$ to 1. A larger ${\sigma }_p^2$ results in larger jumps in magnitude, while the mean over positive and negative jumps remains zero. The parameter $\lambda$ is set to ${\displaystyle \frac{n}{10}}$, where n denotes the length of the generated time series. This means that, on average, a jump is expected to occur after every $\lambda$ period. Superposing the DGP with the compound Poisson process results in a mean shift by the actual jump size that occurred at each jump event. As mentioned before, the noise is modeled by a random walk ${\left\{w_t\right\}}_t$ with

$$w_t=w_{t-1}+e_t,$$

where $e_t\sim \mathcal{N}(0,{\sigma }_{rw}^2)$. In our study, we choose ${\sigma }_{rw}^2$ in such a way that we obtain a setting with medium noise, i.e., a signal-to-noise ratio (SNR) of four. The SNR [82] is a measure that characterizes the strength of the signal relative to the background noise. A higher SNR indicates a clearer and more discernible signal amidst the noise. By including the random walk, we achieve a resulting DGP that is globally nonstationary due to the random walk overlay.

3.3. Additional Queueing Models

Beyond these 48 simulation models, we include the M/M/1 and M/M/2 queueing models [83] in our study. Queueing models are commonly used in logistics, operations research, and industrial engineering to study the behavior of waiting lines or queues [84,85,86,87]. Both models have numerous real-world applications, such as in call centers [88], healthcare facilities [89], and transportation systems [90]. The M/M/1 model is a classic queueing model that assumes a single queue and one server. It is a stochastic model, where customer arrivals are assumed to follow a Poisson process, and service times are exponentially distributed. The M/M/1 model can be used to analyze the expected waiting time, the number of customers in the queue, and the expected server utilization. The M/M/2 model is a variation of the M/M/1 model that assumes two parallel servers. According to [87], we set the arrival rate to four and the service rate to two. We focus on the complete queueing model, including both the arrival process and service process, to capture the full system behavior.

3.4. Number of Different Settings

For each setting, we generate time series of length n from the respective DGPs with $n\in \{100,500,1000\}.$ In total, this results in 150 (=12 (time series DGPs) $\times 4$ (further complexity)$+2$ queueing models) $\times 3$ (lengths)) different simulation settings for each forecasting method.

3.5. Data Preprocessing

To forecast time series using a machine learning algorithm, we use the sliding window approach [91]. In this method, a fixed-sized window is moved over the time series data, where the data within each window are used as input for model training at each step. One key advantage of the sliding window approach is that it allows the machine learning algorithm to capture the temporal dependencies and patterns in the data. The window size is an important parameter [92]; if it is too small, it may not capture enough information, whereas, if it is too large, it may introduce noise and reduce the model’s accuracy. In this study, we evaluate window sizes of 2, 4, 8, and 16, examining their impact on forecasting performance for different time series lengths (100, 500, and 1000). We focus on one-step ahead forecasting at each time step, using both the original time series and the differentiated time series as input. Differencing is essential as it enhances stationarity and prevents the model from forecasting beyond the observed range, which trees cannot handle effectively.

3.6. Choice of Parameters

In this study, we applied different strategies for parameter selection based on the nature of the models. For machine learning models, we used default hyperparameter settings as recommended in the literature [56,66,67]. This decision was made to focus on their baseline performance and ensure consistency across comparisons while also reducing computational runtime. Specifically, each ensemble learner was configured with 500 trees, the inner bootstrap sample is equal to $m_{try}=\lfloor \frac{p}{3}\rfloor$, where p denotes the number of features, and the number of sample points in the bagging step is equal to the sample size. Each terminal node should at least contain five observations. For XGBoost, we employed a learning rate of 0.3 and a maximum tree depth of six. In contrast, to estimate the parameters of the time series approaches, we use the algorithms implemented in the R-package forecast. This was necessary to tailor these models to the specific properties of the data, such as trend and seasonality, as their performance heavily depends on optimized parameters.

3.7. Evaluation Measure

Since the mean square error (MSE) and the mean absolute percentage error (MAPE) are widely used in the forecasting of time series in logistics [9], we use them as evaluation measures, which are calculated over 1000 repeated forecasting steps. The MSE measures the model’s accuracy, expressed as the average squared difference between the observed and predicted values. Simultaneously, the MAPE, calculated as the average percentage difference between the observed and predicted values, offers insights into the model’s relative performance.

4. Results

In this section, we describe the results of the simulation study. In particular, we present the MSE of the different forecasting algorithms under various simulation configurations. The analysis of the MAPE results can be found in Appendix A. We start with the performance of the methods for queueing models.

4.1. Predictive Power in Queueing Models

The influence of the different sliding window sizes and the differencing is shown in Figure 1 and Figure 2. Generally, differencing improves the prediction power of both ML approaches in both settings. Especially for the Random Forest, the MSE decreases by one-fifth after differencing. The lengths of the time series only have a minor influence on the MSE. The Random Forest with differentiated data outperformed the other methods for all lengths. Comparing the effects of sliding window sizes, we find slight differences in performance. Random Forests have smaller MSE values with smaller sliding windows in both settings, while larger window sizes slightly improve performance in the other approaches.

The predictive power of the time series and naive approaches are given in Figure 3. Note that both the ARIMA and SARIMA models have identical MSE values. In both cases, the time series approach performs better than the naive approach. However, the difference in performance is smaller for M/M/2. Again, the influence of the time series length is marginal. While all time series approaches perform similarly in the M/M/1 setting, the TBATS method has slightly smaller values in the M/M/2 setting.

In both scenarios, the Random Forest approach with differenced data consistently showed the smallest MSE. However, the differences between this method and the time series approaches were not great.

4.2. Predictive Power in the Different Time Series Settings

In the following, we analyze the performance of the methods for the DGPs described in Table 1. When comparing the influence of sliding window size and differencing on the performance of Random Forest across all settings (Figure 4), we observed that non-differencing resulted in smaller MSE values except for the AR setting.

In the AR setting, differencing slightly outperformed non-differencing. However, it should be noted that, as the length of the time series increases, the differences between the two approaches become negligible. In all settings, the MSE values slightly decrease with an increase in time series length. The sliding window size has a small influence on the prediction power and shows similar behavior across different time series lengths.

Similar observations can be made for XGBoost, see Figure 5.

The sliding window’s size and the time series length have a small effect on the performance quality. For all DGPs, the MSE values decrease slightly with increasing time series length, except for BL1. Here, the MSE values first increase. The XGBoost approaches generally have slightly larger MSE values than the Random Forest approaches.

Figure 6 shows the MSE values for the time series approaches. The performance of the time series approaches is comparable to that of the Random Forest. All methods have very similar MSE values. The time series length has only a minor impact on the predictive power, except for the BL1 setting. As observed for the XGBoost approaches, MSE values in this setting first increase and then decrease with increasing time series length.

Additional results can be found in the Appendix A Figure A1 therein, for example, shows that the naive approach exhibits the largest MSE values compared to all methods. Thereby the performance of the naive approach is dependent on the DGP and the length of the time series. For BL2, longer time series lengths generally lead to better performance, but for NAR1 the performance may slightly decrease. For the AR, BL1, and NMA models, the MSE values typically decrease initially and then slightly increase as the time series length increases. Conversely, NAR2, SAR1, SAR2, STAR1, STAR2, TAR1, and TAR2 tend to show the opposite trend.

4.3. Influence of the Additional Complexities on the Predictive Power

Based on the findings of the previous sections, we focus on the simulation results obtained with a sliding window size of 8, as the choice of this size is due to the consistent performance observed with different sizes. Details of the results with other window sizes can be found in Appendix A, but a moderate size of 8 balances computational efficiency and information incorporation. Below, we first consider the influence of an additional jump process before discussing the white noise results.

The influence of the jumping process can be seen in Figure 7. All MSE values increase monotonically with increasing sample size, indicating that the jump process significantly impacts predictive performance. Note that, as the time series length increases, the Random Forest approach with differentiated data outperforms all other approaches. Using the differenced data significantly improves the MSE values for both ML approaches, particularly for increasing time series length. The predictive performance of the time series approaches is similar for all DGPs and slightly better than that of the naive approach.

Figure 8 summarizes the prediction results for all methods and all DGPs superposed by a random walk. Here, the time series length has only a minor influence on the prediction performance of the data overlaid with a random walk. For the AR and BL2 settings, the MSE values increase slightly when the time series length is increased from 100 to 500. For all other DGPs, the MSE values decrease slightly, except for the naive approach. The naive approach has the highest MSE values for all settings, followed by XGBoost, except for BL2. Here, both approaches have similar values. The performance of the other methods depends on the respective setting.

For the settings AR, BL2, SAR1, and SAR2, Random Forest with differenced data again shows the smallest MSE values, while the time series approaches show slightly larger values. Note that the XGBoosts with differentiated data perform better in these settings than the Random Forests with non-differentiated data. In the BL1, NAR1, NAR2, NMA, and STAR2 settings, only minor differences in the performance of the Random Forests and time series approaches can be observed. When comparing the two XGBoost approaches in these settings, the differencing reduces the MSE. The ML approaches show larger MSE values in the STAR1, TAR1, and TAR2 settings than the time series approaches, with Random Forests performing better than the XGBoost method.

The influence of both complexities, the random walk and the Poisson process, on the prediction performance is shown in Figure A6 in Appendix A. Similarly to the case where a composite Poisson process is superposed on the data, we observe an increase in MSE values with increasing time series length for all settings. In particular, for time series lengths of 500, we obtain MSE values of more than 2000.

4.4. Summarizing All Results

To evaluate the prediction performance across the spectrum of simulation settings, we calculate the median rank for each prediction method in

Table 2. Median performance rank of forecasting methods across different simulation settings and different time series lengths. Rankings are based on MSE values, with rank 1 indicating the method with the lowest MSE.

DGP		RF	RF Diff	XGBoost	XGBoost Diff	ARIMA	SARIMA	TBATS	Naive
Queueing Models		7	1	7	5	2.5	3.5	3	6
DGPS	no add. Compl.	1	6	5	7	3	3	3	8
from	Jumps	7	1	7	5	3	3	3	6
Table 1	Random Walks	5	1	7	6	3	3	3	8
with	Both	7	1	7	6	3	3	3	5

. The ranking is based on the MSE values, with rank 1 indicating the method with the lowest MSE. Each entry in the table represents the median rank of a particular prediction method in all settings of a particular DGP model described in Section 2. Furthermore, the results for the ranking take into account the performance of machine learning algorithms with a sliding window size of 8.

The results in Table 2 provide useful insights into the relative predictive performance of the different methods in different simulation scenarios. In particular, Random Forest with differentiated inputs proves to be the best performing method, achieving the lowest median value across different complexities, including scenarios with jumps, random walks, or a combination of both. While XGBoost is competitive, it tends to have a slightly higher median value under these conditions. Traditional time series methods such as ARIMA, SARIMA, and TBATS consistently show a robust and similar performance.

5. Real-World Data Example

As explained at the onset, there is a lack of freely available and good documented data sets in logistics research. We therefore use a rather simple real-world data example for illustration. The data set contains daily demand orders from a Brazilian logistics company [93] and was sourced from the UCI Machine Learning Repository [94]. Covering a span of 60 consecutive days, the data set consists of three time series that capture orders for products A, B, and C. Figure 9 shows the corresponding time series in which specific shocks in the data can be identified.

This observation puts us in a similar setting to the simulation study where the DGP was overlaid with a Poisson process. Given this context, it is of interest to evaluate whether the robust performance of (differentiated) machine learning algorithms observed in the simulation study is also apparent in this data set.

The machine learning algorithms adhere to the hyperparameters outlined in Section 3, with a sliding window size of eight, as informed by insights from our simulation study. We use the first 50 observations to train all methods and the last ten observations to test the performance via time series crossvalidation ([43], Chapter 5.10). The MSE and MAPE are again used as evaluation measures. The summarized results are presented in

Table 3. Mean MAPE and MSE of the methods considered in Section 2 using daily demand order data set.

	MAPE			MSE
Method	Prod. A	Prod. B	Prod. C	Prod. A	Prod. B	Prod. C
Random Forest	24.30	35.05	30.79	22.39	262.41	695.70
Random Forest Diff	6.67	21.80	15.84	4.91	197.23	1.97
XGBoost	25.06	41.62	19.51	22.34	376.62	147.20
XGBoost Diff	10.70	37.98	27.15	13.10	841.56	41.00
(S)ARIMA	28.57	49.30	33.56	29.48	1142.14	655.88
TBATS	28.37	36.17	33.56	43.14	446.18	663.78
Naive	33.18	30.71	30.59	25.10	194.21	82.03

. Note that the results of SARIMA and ARIMA are identical due to the absence of seasonality and are therefore combined into one method.

The results show that the performance of the forecasting methods is different in the various product categories. In general, the machine learning algorithms deliver consistently better results than the traditional time series methods. This is in line with our simulation study, where ML methods showed better performances when additional complexities were present. Random Forest with differencing performed best for all three time series and evaluation measures, again confirming the results obtained in the simulation study for such settings. It should be noted that the introduction of differencing is beneficial for Random Forest in all predictions. For XGBoost, however, performance on product A improves significantly when differenced data is used, but in the other two time series differencing leads to worse forecasting performance.

6. Summary, Discussion, and Outlook

6.1. Summary with Highlights

The main objective of this simulation study was to perform a one-step comparative analysis of prediction accuracy and evaluate the performance of tree-based machine learning and time series approaches that are typically used in data-driven logistics. Through a comprehensive investigation of different data generating processes, queueing models, and additional complexities, we aimed to determine each method’s inherent strengths and limitations. Our analysis included conventional time series methods, including (seasonal) ARIMA models and TBATS, as well as machine learning methods such as Random Forest and XGBoost. In addition, we investigated the impact of data differencing on the performance of the two latter algorithms. The key findings from our study are as follows:

• The out-of-the-box Random Forest emerged as the ML benchmark method.
• Training on differentiated time series can significantly improve the ML resilience.
• ML models are more robust with respect to additional (nonlinear) complexity, settings in which they outperformed statistical time series approaches.
• In all other settings, the time series approaches were at least competitive or even performed better.

6.2. Detailed Discussion and Outlook

In our study, the Random Forest approach performed consistently better in all simulation settings than the XGBoost approaches. It is worth noting that no hyperparameter tuning was made in our study. Random Forests are known to be robust to hyperparameter settings and often perform well with default values [95,96]. This robustness can be a crucial factor contributing to their superior performance compared to XGBoost. Applying techniques such as Bayesian Optimization or more simple grid or random search for hyperparameter tuning could change this observation and should be investigated in future studies. Regarding the effect of data differencing on the performance of the two machine learning methods, we observed similar patterns. Differencing improved performance, especially in queueing scenarios and situations where additional complexity was introduced into the data generation process. Without additional complexity, differencing showed minimal impact, with the performance of both methods deteriorating slightly when the differentiated data were used, except for very linear data generation processes. Here, only a slight improvement was observed. This suggests that differencing plays a crucial role in improving the resilience of machine learning methods, especially Random Forests when the data is overlaid with additional noise like a random walk. When comparing the performance of the different time series approaches, we found subtle differences between them. ARIMA and SARIMA showed relatively similar performance in all simulation settings under consideration. Their prediction accuracy was quite consistent without big differences in most situations. Comparing their performance with that of TBATS, the differences are also small and not substantial, suggesting that ARIMA, SARIMA, and TBATS had comparable predictive power in our simulation settings. The additional complexity induced, such as a jump process or random noise, significantly impacts the predictive power. Introducing a jump process leads to increased MSE values for all methods and settings, indicating a significant impact on prediction accuracy. In this scenario, all methods show consistent behavior with strong increasing MSE values for increasing time series lengths. When a noise process is introduced, a more nuanced pattern emerges. For the machine learning approaches, differentiating the data proves beneficial and improves the overall performance. The Random Forest approach with differenced data as input outperforms the other approaches in most scenarios, closely followed by all three time series approaches. A comparison between Random Forests and the time series approaches shows different performance patterns in the different simulation environments. In queueing situations, where the underlying processes are often characterized by complicated dynamics, the Random Forest approach shows superior performance. Furthermore, a notable trend emerges in simulation settings where a Poisson process complements the data generating processes. In these cases, ML methods show improved performance, indicating robustness to the inherent complexity introduced by the Poisson process. The adaptability of ML models to capture and learn from nonlinear patterns may contribute to their effectiveness in scenarios with Poisson process or random walk overlays. However, it is essential to recognize that this beneficial performance of ML methods is not universal. In all other simulation settings, the Random Forest approaches perform comparably or slightly worse than all three time series approaches. In addition to the simulation study, our illustrative data analyses were conducted with a focus on one-step demand forecasting for different products of a logistics company. The results indicate that machine learning algorithms can improve the forecasting performance in this context. In particular, the machine learning methods perform better than or as well as the time series methods for most products.

In the context of data-driven logistics, our results underscore the importance of tailoring time series forecasting methods to the specific characteristics of data sets encountered in different logistics areas. The Random Forest approach, especially when using differentiated data as input, is recommended as an initial benchmark prediction tool, particularly for data sets with a lot of noise or complex patterns. The robustness of Random Forests, combined with their ability to achieve good results without extensive tuning of hyperparameters, makes them a pragmatic choice for various prediction scenarios. Conversely, in situations where interpretability is paramount (e.g., to gain the understanding or trust of users in warehouses or decision makers in SCM) and the data exhibit clear patterns, traditional time series approaches remain a valuable and interpretable option. These approaches often come with faster runtimes and greater resource efficiency, which is also essential in the development of data-driven logistics, e.g., in the case of resource constraints [97,98]. As only one-step forecasts were considered, future simulation studies should investigate whether the same observations can be found for more step forecasting. Also, additional or hybrid methods must be investigated [99,100,101]. Another line of future research needs to compare the methods with respect to uncertainty quantification, i.e., point-wise or simultaneous prediction intervals and regions.