Performance Assessment for Short-Term Water Demand Fore- casting Models at an End-Use Level in Korea

It is crucial to forecast the water demand accurately for supplying water efficiently and stably in a water supply system. In particular, accurately forecasting short-term water demand helps in saving energy and reducing operating costs. With the introduction of the Smart Water Grid (SWG) in a water supply system, the amount of water consumption is obtained in real time through an advanced metering infrastructure (AMI) sensor, which can be used for forecasting the short-term water demand. The models widely used for water demand forecasting include the autoregressive integrated moving average, radial basis function-artificial neural network, quantitative multi-model predictor plus, and long short-term memory. However, there is a lack of research on assessing the performance of models and forecasting the short-term water demand by applying the data on the amount of water consumption by purpose and the pipe diameter of an end-use level of the SWG demonstration plant in each demand forecasting model. Therefore, in this study, the short-term water demand was forecasted for each model using the data collected from the AMI, and the performance of each model was assessed. The Smart Water Grid Research Group installed ultrasonicwave-type AMI sensors in the block 112 located in YeongJong Island, Incheon, and the actual data used for operating the SWG demonstration plant were adopted. The performance of the model was assessed by using the residual, root mean square error (RMSE), normalized root mean square error (NRMSE), Nash–Sutcliffe efficiency (NSE), and Pearson correlation coefficient (PCC) as indices. The water demand forecast was slightly underestimated in models that employed the assessment results based on the RMSE and NRMSE. Furthermore, the forecasting accuracy was low for the NSE due to a large number of negative values; the correlation between the observed and forecasted values of the PCC was not high, and it was difficult to forecast the peak amount of water consumption. Therefore, as the short-term water demand forecasting models using only time and the amount of water consumption have limitations in reflecting the characteristics of consumers, a water supply system can be managed more precisely if other factors (weather, customer behavior, etc.) influencing the water demand are applied.


Introduction
The Smart Water Grid Research Group (SWGRG) operated the smart water grid (SWG) demonstration plant in the block 112 located in YeongJong Island, Incheon from 2017 to 2019 [1]. The SWG is based on the Internet of things, which is the core technology of the 4th Industrial Revolution, and aims to provide an efficient and economic management of water resources and stable supply [2]. The integrated system monitors and collects the data on the amount of water consumption at an end-use level through real-time daily water demand was forecasted using the SARIMA model applied with a sliding window [23], whereas the hourly water demand was distributed with the NN model according to a calendar effect for daily patterns and compared with the ANN model.
If the number of end uses is large in the real-time water consumption data transferred from AMI sensors, a statistical model involves difficulties and complexity with identifying periodicity, whereas an ANN model cannot reflect the independent characteristics derived from the training dataset without considering the time-domain problem [29]. Furthermore, the optimal weight value of an activation function is difficult to find during the training process, and the training time is long. A recurrent neural network (RNN) model, which is deep neural network (DNN) model, rectified these drawbacks by applying multiple hidden layers and the backpropagation algorithm to update the weight [30]. However, RNN models have a gradient vanishing problem [31] in which weight and bias are not significantly changed as the gradient is reduced during the backpropagation step, as the length of the training data being input is lengthened. A long short-term memory (LSTM) [32] model resolved this problem by adding a memory cell state to determine to what extent the past information is forgotten and the information input in the current timestamp is remembered. Some recent studies using an LSTM model focused on power demand forecasting [33][34][35], tourism flow forecasting [36], airline demand forecasting [37], and sales demand forecasting [38]. In particular, Li and Cao [36] reported that the forecast accuracy of an LSTM model is higher than that of the ARIMA model for forecasting tourism flows.
Therefore, this study selected ARIMA as a statistical model, RBF-ANN as an ANN model, QMMP+ as a hybrid model, and LSTM as a DNN model for short-term water demand forecasting. As the input data for forecasting water demand, the hourly water consumption data collected at an end-use level of an SWG demonstration plant of the block 112 located in YeongJong Island, Incheon were used. For assessing the water demand forecasting performance of each model, the forecasted value was comparatively analyzed against the 24-h (one day) observed value. The results of this study can be used as index data for applying a short-term water demand forecasting model to an actual water supply system to which an SWG is applied.

Study Process
The process of assessing the performance of a short-term water demand forecasting model can be divided into three parts: (a) the preprocessing of the data collected from the end-use AMI sensors of a district metered area (DMA) of a water supply and then transferred to the integrated server to be accessible, (b) hourly water demand forecasting using the preprocessed water consumption data, and (c) the performance assessment of the models. This process is illustrated in Figure 1.
Similar to most time-series data, the water consumption data transferred from AMI sensors contain missing values due to a communication error or AMI measurement error in addition to error values that substantially differ from the observed values or other samples [39]. Particularly, missing values often occur in big data when a large number of variables have correlations with respect to time [40]. However, despite considerable research on the imputation of missing values, only a few methods have been proposed for accurately processing the missing values in the big data received from AMI sensors. Rahman et al. [40] suggested an imputation method for big data in which the lagged k-NN and fast Fourier transform are combined; this method demonstrated a high accuracy when the missing values were imputed in the diabetes datasets. Particularly, this method was applied in this study, as it can impute the data omitted for at least one day and is less biased.
First, to find outliers, the local minima and the variables exceeding the maximum discharge capacity per unit time with respect to a pipe diameter are determined and designated as missing in the accumulated water consumption data. For imputing the missing values, the lagged k-NN is applied to find the lagged time and the training dataset to find the k-NN value; then, for integrating the water consumption patterns within the variable, the Fourier transform is approximated and the estimates of the two methods are averaged. The water consumption dataset is categorized into the training and validation datasets for forecasting water demand. In general, the training dataset uses the entire observed values of 70%-90%; however, in this study, among the water consumption data of one year measured at one-hour intervals, 70% of the data were used as the training dataset, whereas the remaining 30% were used as the validation dataset [28]. The water consumption data are standardized for reducing the bias in demand forecasting. After configuring each water demand forecasting model, parameters are estimated using the training dataset (70%). Then, if the estimated parameters satisfy the simulation conditions, the water demand for 24 h ahead is forecasted using the validation dataset (30%). For assessing the performance of each model, the residual, root mean square error (RMSE), normalized root mean square error (NRMSE), Nash-Sutcliffe efficiency (NSE), and Pearson correlation coefficient (PCC) are calculated by comparing with the observed data.

ARIMA
ARIMA, a stochastic model, was first presented by Box and Jenkins [41] in 1976, and is used for forecasting a time series data in various fields. In ARIMA(p, d, q) model, of the observed time series y = 〈 〉 is integrated and differentiated with the Autoregressive (AR) and Moving Average (MA) terms, can be expressed as the following equation (1) [41].
where ∅ ,⋯, ∅ are the coefficients of AR(p) to be estimated that accompany each of the observations in the past periods, is the order of AR terms, ,⋯, are the coefficients of MA(q) will be stationary, q is the order of MA terms, is constant, is back shift operator (e.g. = ), is the order of non-seasonal differences, and is the white noise, which is an error term that satisfies (0, ).
The autocorrelation function (ACF) and the partial autocorrelation function (PACF) are used to determine the orders of p, d, and q for model estimation [42]. In order to apply the ARIMA model and find the optimal orders, we used Matlab's 'estimate function' and forecast water demand using 'forecast function' (Matlab R2020b, MathWorks Inc., Natick, MA, USA).

RBF-ANN
ANNs are used in various fields such as monitoring, control, classification, and forecasting. In particular, RBF-ANN model among ANNs models is very powerful for numerical calculations, and has the advantage of fast learning without weight update by repeated computation, so it has superior robust ability and adaptation capability compared to other ANN models [43]. In the RBF-ANN model, the value ℎ of the output layer is as shown in Equation (2) [43,44].
where is hidden layer nodes, y = 〈 〉 is input data, is the synaptic weight connecting the hidden node and the output layer, is the gaussian function of the hidden layer, is a bias, and are the center vector and radius of the nth hidden node, respectively. At this time, the number of Gaussian neurons in the RBF-ANN model was 92 in the hidden layer, = 1, and 'train function' of MATLAB Neural Network Toolbox package (Matlab R2020b, MathWorks Inc., Natick, MA, USA) was used for weight estimation and updating.

QMMP+
Lopez Farias et al. [28,45] proposed the QMMP+ model, which is a nonlinear timeseries model, for water demand forecasting. The time-series data of water consumption typically have a cyclic consumption pattern for every 24 h, excluding holidays [45]. Therefore, the time-series data of water consumption can be disintegrated as in Eqs. (3) and (4) for distinguishing qualitative ( ) and quantitative ( ) components [28].
where is the length of the accumulation period (e.g., a day, 24 h), and T = (t/γ).
, which is the daily water consumption pattern to be estimated, is applied with the calendar effect (week, holiday), the nearest neighbor rule pattern estimation (NNRPE), and probabilistic selection [28] based on a nonparametric learning algorithm, the nearest neighbor (NN), proposed by Kantz and Schreiber [46].
On the other hand, the daily water demand can be forecasted using the SARIMA model. Therefore, the SARIMA( , , )( , , ) model has seasonality added to the ARIMA model and can be expressed as Eq. (5).  + ⋯ +  ). , , refer to the degree of the seasonal autoregressive (SAR) term, seasonal moving average (SMA) term, and seasonal difference term, respectively, whereas refers to the degree of seasonality.
Considering the cyclical characteristics, was set to 7 (seven days), and holidays were reflected according to a calendar effect. Water demand can be forecasted by multiplying the estimated 24-h-ahead hourly pattern ( ) with the daily water consumption . The QMMP+ model used in this study adopted the MATLAB code proposed by Lopez Farias et al. [28].

LSTM
In 1997, Hochreiter and Schmidhuber [32] introduced the LSTM model, which solved the problem of the information in the hidden layer not being transferred to the last layer due to the vanishing gradient in an RNN model, which is called long-term dependency [47]. The model is used in various fields due to its excellent performance. Similar to the RNN models, the LSTM model is a type of a DNN model having a circulating structure by being connected with edges in which a hidden node has directionality. It is designed to be trained by reflecting the information of the past in the present, regardless of the length of sequential data. However, the module having a repeated chain structure consists of a memory cell state that can select information and three gates (forget, input, and output), unlike RNNs. Each gate outputs either 0 or 1 for the information received from the previous cell and the information input in the current layer by multiplying the sigmoid layer and the vector. If the gate value of 0 is received, the input information is not delivered; however, if 1 is received, the information is delivered to the memory cell state.
The weight matrices that can be trained in the LSTM layer are the input weight ( ), recurrent weight ( ), and bias ( ). The mathematical form of each gate can be expressed as Eqs. (6)-(9) as follows [32].
where , , , and denote the input gate, forget gate, cell candidate, and output gate, respectively, at time step .
represents the original time-series data, is the sigmoid function (1 + ) , and ℎ is the hyperbolic tangent function − 1 (scale from -1 to 1).
The cell state ( ) and output (ℎ ) updated from each gate can be expressed as Eqs. (10) and (11) as follows [32]. where, ⨀ is the matrix operator, i.e., Hadamard product or element-wise multiplication. The last layer of the training process is the regression layer in which the loss function is defined as the half-mean-squared error of forecasted responses, and the corresponding equations can be expressed as Eq. (12).
where ℎ * is the forecasted point, is the input point, and is the quantified deviation between the forecasted and observed values.
The loss function is reduced, and the weight of each gate is updated by a gradient matrix using the differential in the backpropagation of training. As an optimizer of the loss function, Adam, which is the combination of AdaGrad and Momentum, was used and can be expressed as in Eq. (13) [48].
where, is the weight of the model at iteration step , is the learning rate, and is a constant for which a value of 10 is commonly used. Furthermore, are the momentums using inertia; thus, the decay rates and are 0.9 and 0.999, respectively [48].  [49]. Therefore, the number of repetitions is set such that the values of the loss function are not reduced further.

Performance Assessment
For assessing the short-term water demand forecasting of each model, the residual, RMSE, NRMSE, NSE, and PCC were selected as the assessment indices in this study (Eqs. (14)- (18)).
where denotes the length of time (24 ℎ), is the forecasted result, is the mean of the forecasted result, is the observed water consumption, and is the mean of the observed hourly water consumption. _ and _ are the maximum and minimum observed values, respectively.
The residual represents the difference in the forecast of daily water demand, which can be used for determining whether the forecast is underestimated or overestimated. Moreover, the RMSE is a typical assessment method for comparing the observed and forecasted values, which facilitates identifying the errors in the actual water consumption. The NRMSE is a percentage error suitable for determining the quantitative characteristics of the forecasted values. However, RMSE has a considerable influence on the observed values, as it is a scale-dependent error [50]. Therefore, NSE and PCC were used as additional assessment indices. NSE is a coefficient often used in the field of hydrology for verifying the accuracy of forecasted values via a comparison with the observed values. A negative value of this coefficient indicates that the forecasted value is poor or is not consistent with the observed values. A value closer to 1 indicates a higher forecasting accuracy [51]. PCC measures the accuracy of the correlation between two variables that are linearly correlated; a value closer to 1 is more linear and thus indicates that the observed and forecasted values are consistent [52].

Study Area
The study area is the block 112, which is located in Unseo-dong and Unbuk-dong of YeongJong Island, Incheon, to which water is supplied through a single submarine pipeline from GongChon Water Purification Plant in Incheon. The total area of the block is 17.41 km 2 , and the population served is approximately 17,000. The total number of customers is 958 within the block, and a service reservoir and pumping station are also present. The SWGRG [1] installed 527 ultrasonic-wave-type smart water meters in the block for the first time in South Korea, and operated the SWG demonstration plant from 2017 to 2019 (Figure 2). The pipe diameter can be categorized into eight different types ranging from 15 mm to 100 mm, which can be further categorized into household use (387), general use (138), and bathroom use (2) according to the water rate criteria. Specifically, the number of 15-mm diameter household-use pipes is 279, which is approximately 52.9% of the total number of water consumers, whereas that of 20-mm diameter pipes is 12.5%, constituting 68.9% of the total number of water consumers (527). The total length of a pipe conduit is approximately 55 km, and the daily water consumption is approximately 8,000 m 3 . The accumulated water consumption data read by the AMI sensors are transferred (at one-hour intervals) to the server of the Water Operation Center of Incheon Waterworks Headquarters in real time.

Dataset Description
As the dataset for forecasting water demand, the accumulated water consumption data received from the 527 ultrasonic-wave-type AMI sensors in the block 112 of YeongJong Island from 00:00 on January 1, 2018 to 00:00 on January 1, 2019 were used. The accumulated water consumption data are updated every hour and transferred to the central server; as shown in Figure 3, the data exhibit a (a) stair-shaped pattern or a (c) nonlinear pattern and contain missing values or erroneous data. The data in (a) and (c) can be represented as the water consumption at one-hour intervals as shown in (b) and (d), which can be used as the input data for forecasting water demand.   The local minima and the maximum discharge capacity with respect to a pipe diameter are designated as missing values for discerning outliers in the annual water consumption data (one-hour intervals) of 527 AMIs. The minimum number of missing values was 77 (0.88%), whereas the maximum number of missing values was 8,119 (92. 68%) of each AMI; the overall average missing rate of 527 AMIs was 5.40% (249,276) ( Figure 4). Water demand forecasting may be affected if the missing rate is high due to errors in the data received during the actual operation of the SWG demonstration plant. Therefore, the problem of missing values must be resolved for the proper management of a water supply system. 10 AMIs were selected for forecasting water demand for which the missing values were set to 10% or less to minimize the influence of errors in the simulation results. Moreover, for a comparison with the forecasted values, the missing data were not included in the 24-h observed values of the validation dataset; the pipe diameters ranged from 15 mm to 32 mm, and the purpose of use included households, restaurants, churches, supermarkets, senior community centers, and laundries. Table 1 lists the details of the AMI dataset. The total water demand (TWD) being forecasted ranges from 0.634 m 3 /day at the minimum to 12.206 m 3 /day at the maximum. Generally, the amount of water consumption increases as the pipe diameter increases, and the water consumption was the highest for restaurants, churches, and laundries compared with households.  Furthermore, the water demand needs to be forecasted per hour within the DMA for the operation of a distribution reservoir of a water supply system. Hence, the missing values are imputed using the lagged k-NN and fast Fourier transform in the hourly dataset of 527 AMIs, and the summed (a) daily water demand (year) and (b) daily average water demand per hour (24 h) of a distribution reservoir are shown in Figure 5. The amount of water consumption being forecasted 24 h ahead is 5,108 m 3 /day, and the average water demand shown in (b) exhibits periodicity with respect to 24 h.

AMIs Water Demand Forecasting
For 10 AMIs selected based on the purpose of use (households, churches, supermarkets, daycare centers, senior community centers, restaurants, etc.) and pipe diameter (15 mm-32 mm), the results of forecasting water demand 24 h ahead by using the ARIMA, RBF-ANN, QMMP+, and LSTM models are illustrated in Figure 6.
The pattern of water consumption reflects the characteristics of use. In particular, restaurants (a) have a higher amount of water consumption at night than during the day, whereas the amount of water consumption at night is extremely low for laundries (e), supermarkets (f), and senior community centers (i). Furthermore, the amount of water consumption in households (b), (c), and (j) is large especially during breakfast and dinner times. In terms of water consumption patterns, the observed values cannot be accurately forecasted due to various factors (weather, day, consumer characteristics, etc.), whereas the forecasted values are generally similar to the observation patterns when visually inspected, excluding the ARIMA model, which cannot reflect the periodic nature. However, it is difficult to forecast the peak value of the observed water consumption. Daily water demand forecasting is also important in terms of stably supplying water in addition to water consumption patterns; the differences in the forecasted daily water demand of each AMI are presented in Table 2 by model. Among the 10 AMIs, two ARIMA models (110013044, 110016799), one RBF-ANN model (110013629), one QMMP+ model (110013012), and five LSTM models (110013004,110013074,110016389,110016860,110018932) produced the results that are most similar to the actual water consumption but the results were underestimated (Table 2). Particularly, the ARIMA model cannot reflect the periodic characteristics but can estimate the daily water demand.
The result was closer to the observed value when the RMSE was small, and the LSTM model had better forecast results compared with the other models ( Table 3). The result indicates that the amount of forecasted water demand per hour is most similar to the observed values. For the NRMSE, which is the RMSE divided by standard deviation, the LSTM model showed excellent forecasting results similar to the case of the RMSE. The accuracy was 27.58% for the ARIMA model, and 23.25% and 17.18% for the RBF-ANN and QMMP+ models, respectively ( Table 4).
The forecast accuracy is higher as NSE is closer to 1. Except for the LSTM model, negative values were observed in the AMIs of eight ARIMA models, three RBF-ANN models, and five QMMP+ models, thus exhibiting a poor forecast accuracy. On average, the NSE of the ARIMA model was 0.11, whereas the corresponding values of the RBF-ANN, QMMP+, and LSTM models were 0.34, 0.34, and 0.61, respectively, thus exhibiting a low forecast accuracy (Table 5).
According to the PCC calculation results, the average value was 0.37 for the ARIMA model, 0.64 for the RBF-ANN model, 0.68 for the QMMP+ model, and 0.79 for the LSTM model, thus failing to exhibition a high level of correlation above 0.9 (Table 6).
In terms of the short-term water demand forecasting performance of the models for 10 AMIs in the block 112 of YeongJong Island, the LSTM model demonstrated the best performance, but there was a limitation in estimating the peak amount of water consumption while the daily water consumption was underestimated. In particular, the NSE had a poor forecasting accuracy due to negative values in most of the models, excluding the LSTM model, and none of the models had a correlation above 0.9 in the PCC analysis. It can be inferred that the life cycle of the consumers could not be reflected accurately as the water demand forecasting is solely based on time and the amount of water consumption. Furthermore, weather conditions and consumer events influenced the results in addition to missing values and erroneous values when collecting the data of water consumption from AMIs.

Total Water Demand Forecasting
The results of water demand forecasting for 24 h ahead for the operation of the DMA distribution reservoir are illustrated in Figure 7. A distribution reservoir supplies water from a purification plant through pumps; thus, the electricity cost can be reduced by controlling the supply time based on the forecasting of water demand. Similar to the results of 10 AMIs, the periodic characteristics are reflected adequately to be similar to the observed values when inspected visually, excluding the ARIMA model. The observed daily water consumption is 5,108 m 3 . The forecasted values of each model were 5,275 m 3 for the LSTM model, 5,607 m 3 for the QMMP+ model, 5,503 m 3 for the RBF-ANN model, and 5,433 m 3 for the ARIMA model, which are all overestimated, unlike individual AMIs. However, the LSTM model had the smallest difference of 167.1 m 3 from the observed value (Table  7). Moreover, in terms of the assessment factors, RMSE, NRMSE, NSE, and PCC, the LSTM model produced better results compared with the other models. However, the NSE and PCC of the LSTM model were 0.61 and 0.79, respectively, thus having a poor forecasting accuracy and correlation. Based on the above results, it can be inferred that there is a limitation in forecasting water demand only based on time and the amount of water consumption. In an SWG environment, information that can differentiate the characteristics of consumers can be obtained in addition to the data on direct information, such as the amount of water consumption and usage time. Therefore, forecasting methods reflecting the propensity and characteristics of consumers can be considered. Specifically, a water supply system can be managed more precisely if various factors, such as water usage time, the amount of water consumption, purpose, the number of household members, occupation, and weather (precipitation, humidity, temperature, etc.), are applied for forecasting the water demand.

Conclusions
The SWG technology, which was introduced for a stable and efficient management of water resources in cities, enables the real-time monitoring and collection of the data on water consumption using bilateral communication and AMI sensors. Short-term water demand forecasting is required for reducing energy and operating costs when managing water supply pipe networks. The water consumption data received from the AMIs can be used as basic materials for forecasting the water demand. Despite a large number of shortterm water demand forecasting models, there is a lack of research on assessing the performance of short-term water demand forecasting models at an end-use level for the operation of an SWG. Therefore, this study used the data on hourly water consumption collected by installing 527 ultrasonic-wave-type AMI sensors at end-use in the block 112 in YeongJong Island, Incheon and operating the SWG demonstration plant for water demand forecasting. The data on water consumption of 10 end-use AMIs categorized according to the purpose and pipe diameter and the total water consumption data of 527 AMIs for operating a distribution reservoir were used for short-term water demand forecasting. As the water consumption data are big data containing missing values and errors, which require correction, the lagged k-NN and fast Fourier transform were applied for the imputation of missing values.  4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22  The ARIMA, RBF-ANN, QMMP+, and LSTM models were selected for the short-term water demand forecasting, and the performance of each model was assessed using residual, RMSE, NRMSE, NSE, and PCC as indices. The LSTM model had the best forecasting performance for 10 AMIs in the block 112 of YeongJong Island. The residual underestimated the water demand, the NSE had negative values, which lowered the forecasting accuracy, and the PCC was below 0.9, which indicated a low correlation between the forecasted and observed values. Weather factors, missing and erroneous values of the water consumption amount, and consumer events may have affected the results. Similarly, the LSTM model produced a higher forecasting accuracy than the other models when forecasting the 24-h water demand of the total water consumption of 527 AMIs; however, the daily water demand was overestimated, whereas the NSE and PCC were below 0.9, which indicate a low forecasting accuracy and correlation.
Consequently, the ARIMA, RBF-ANN, QMMP+, and LSTM models are limited in terms of application for the management of a water supply system. In particular, water demand forecasting based only on usage time and water consumption amount entails an underestimation or overestimation of the forecasted values and fails to reflect the peak water consumption amount. However, in an SWG environment, information that can determine the characteristics of consumers can be obtained in addition to the hourly water consumption amount. Therefore, a water supply system can be managed more precisely if various factors, such as water usage time, the amount of water consumption, purpose, the number of household members, occupation, and weather (precipitation, humidity, temperature, etc.), are considered.