Evaluating the Neural Network Ensemble Method in Predicting Soil Moisture in Agricultural Fields

: Soil is an important element in the agricultural domain because it serves as the media that bridges the water consumption and supply processes. In this study, a neural network ensemble (NNE) method was employed to predict the soil moisture to eliminate the effects of random initial parameters of neural network (NN) on model accuracy. The constructed NNE model predicts daily root zone soil moisture continuously for the whole crop growing season and the water consumption and supply processes were separately modeled. The soil proﬁle was divided into multiple layers and modeled separately. Weather data (including air temperature, humidity, wind speed, net radiation, and precipitation), rooting depth, and the hesternal soil moisture of each layer were used as the input. A calibrated root zone water quality model for maize ( Zea mays L.) was used to generate training and evaluation data. The result showed that with 100 randomly initialized NN models, the NNE model achieved an average R 2 of 0.96 and nRMSE of 5.93%, suggesting that the NNE model learned the soil moisture dynamics well and sufﬁciently improved the robustness of soil moisture prediction with high accuracy.


Introduction
Soil moisture is a critical factor for the production of crops or growth of plants. The plant grows well when there is a sufficient supply of soil water to be absorbed by the root; otherwise, the deficit of soil water will damage crop growth or production. Therefore, irrigations are scheduled to regulate soil moisture during the whole crop growing season so that they can escape any water deficit. Managers or apps [1][2][3] commonly schedule irrigations to regulate the soil moisture within an appropriate range and the soil moisture is used as the fundamental indicator [4].
However, it is difficult to precisely measure the soil moisture [5], especially when the field variability is taken into account [6,7]. For a site-specific field, the mean soil water status is the main concern for irrigation scheduling; therefore, other than based on measurements from field sensors [8,9], the soil moisture can also be estimated by water balance equation where the most important water consuming element, evapotranspiration, is calculated [10,11]. To achieve a more precise estimation, agricultural system models, e.g., the root zone water quality model (RZWQM2) [12], and the AquaCrop [13,14], were used, by taking more factors in the soil-plant-atmosphere continuum (SPAC) system into The objectives of this study are (1) to evaluate how precisely a neural network ensemble (NNE) model could achieve when applied in estimating the soil moisture, and (2) to quantify the number of NN models that should be ensembled in the NNE model to achieve a robust prediction accuracy.

Data Collection
An agricultural system model, the root zone water quality model (RZWQM2, version 3.0), was adopted to generate training and evaluation data, including soil moisture and rooting depth. The RZWQM2 has been well-calibrated under a three-year field experiment in Colorado for maize (Zea mays L.) [45], where the irrigation was scheduled by the evapotranspiration and water balance method [10,11]. This calibrated RZWQM2 was also adopted to generate a water stress-based irrigation regime, which was published in Gu et al. [46], where the soil dynamic across the root zone was different from that under the field irrigation regime due to a lower irrigation frequency and higher irrigation quantity at each irrigation event. With two different irrigation regimes, the calibrated RZWQM2 was adopted to provide both training and evaluation data. RZWQM2 simulated daily soil moisture and rooting depth under the field irrigation scheme, as well as the recorded weather data were collected as the field irrigation (FI) dataset for model training, whereas, for evaluation, the data were simulated under the water stress regime, collected as a water stress (WS) dataset. Weather data, including the air temperature, humidity, wind speed, net radiation, and precipitation, were recorded at a standard Colorado agricultural meteorological network (http://ccc.atmos.colostate.edu/~coagmet/ accessed on 28 July 2021) weather station (GLY04) and they are the same for both datasets as well as the meteorological input of the calibrated RZWQM2 model. Both datasets contained data of three crop growing seasons from 2008 to 2010. The soil moisture dynamics under FI and WS irrigation regimes are shown in Figure 1.
The calibrated RZWQM2 model was also served to provide daily soil moisture in seven defined soil layers (0-15, 15-30, 30-60, 60-90, 90-120, 120-150, and 150-200 cm) during the three growing seasons. The 7 soil layers and their properties, including soil moisture at field capacity and wilting point, remained the same in the NNE model, as shown in Ma et al. (2012) [45].

Model Structure
The purpose of the modeling is to precisely predict the daily root zone soil moisture in a continuous manner with a neural network ensemble (NNE) model, which is constructed with several neural network (NN) models [37]. Each of the NN models predicts the daily root zone soil moisture based on the hesternal soil moisture obtained by either the NN model itself or the average of these NN model outputs. The diagram of NNE model is shown in Figure 2. To account for the effects of initial model parameters, those NN models are with random initial model parameters. For each of the NN models, the three-layer multilayer perceptron (MLP) [47] was employed, that is, with one input layer, one hidden layer, and one output layer. The three-layer MLP is expressed in Equations (1) to (3).
where x i is the input variable; w ij , b j , v j , and c are parameters of the MLP model; g j and u j are the input and output of jth hidden neuron, respectively; y is the output soil moisture of a certain soil layer; n 1 is the number of input neurons; n 2 is the number of hidden neurons; f 1 is the activation function (Sigmoid function); and e is the natural constant.
tion regimes are shown in Figure 1. The calibrated RZWQM2 model was also served to provide daily soil moisture in seven defined soil layers (0-15, 15-30, 30-60, 60-90, 90-120, 120-150, and 150-200 cm) during the three growing seasons. The 7 soil layers and their properties, including soil moisture at field capacity and wilting point, remained the same in the NNE model, as shown in Ma et al. (2012) [45].

Model Structure
The purpose of the modeling is to precisely predict the daily root zone soil moisture in a continuous manner with a neural network ensemble (NNE) model, which is constructed with several neural network (NN) models [37]. Each of the NN models predicts the daily root zone soil moisture based on the hesternal soil moisture obtained by either  The soil water dynamic has two different typical characteristics: (1) during the wat consuming process, when water evaporates from the soil surface and extracts from t crop root before transpiring to the atmosphere; and (2) during the water supplying pr cess when an accountable irrigation or rainfall occurs, even though the water consumi process still exists. The soil water content commonly decreases during the water consum ing process, whereas it increases during the water supplying process. To better ident the two opposite soil water dynamic processes, the NN model was separated into tw sub-models: the water-consuming sub-model and the water supplying sub-model, learn the typical characteristics of soil water dynamics, both with the three-layer ML When predicting the soil moisture, one of the two sub-models is selected depending whether there is an efficient water supply amount through irrigation and/or precipitati on the day. In this study, a total amount of 1 mm daily irrigation and precipitation w deemed as efficient, as the water-consuming process is still dominant when there is a ti amount of precipitation. Daily samples with and without water supply (precipitation a irrigation) in the FI dataset were separated to train the water-consuming sub-model a water-supplying sub-model, respectively.
Besides the dynamic during the water-consuming and -supplying processes, the s water dynamic across the soil profile due to different soil properties and water distrib tions along the depth should be accounted for, as the soil moisture is commonly used schedule irrigations, where the irrigation depth is strongly related to the crop root dep Therefore, the soil profile was divided into several layers and the constructed NN mod employed several three-layer MLPs to model the soil water dynamic of each soil lay using a water-consuming or -supplying sub-model. The predicted soil moisture in ea soil layer was then used to calculate the weighted average daily root zone soil moistu using Equation (4), which is the output of the NN model. In the same way, each of the N models with different initial parameters can obtain a weighted average daily soil moistu across the root zone.
For a given day, the weighted average soil moisture across the root zone, i.e., t output of the NN model, was calculated as: where L is the number of soil layers, is the Lth layer soil moisture (cm 3 ·cm -3 ), the depth of the Lth layer (cm), M is the number of soil layers at the deepest simulat root depth, is the Mth layer soil moisture (cm 3 ·cm -3 ) and is the root depth (cm).

NNE model Output mean SM (d)
Output SM -M (d-1) ) / output mean SM (d-1) The soil water dynamic has two different typical characteristics: (1) during the water consuming process, when water evaporates from the soil surface and extracts from the crop root before transpiring to the atmosphere; and (2) during the water supplying process when an accountable irrigation or rainfall occurs, even though the water consuming process still exists. The soil water content commonly decreases during the water consuming process, whereas it increases during the water supplying process. To better identify the two opposite soil water dynamic processes, the NN model was separated into two sub-models: the water-consuming sub-model and the water supplying sub-model, to learn the typical characteristics of soil water dynamics, both with the three-layer MLP. When predicting the soil moisture, one of the two sub-models is selected depending on whether there is an efficient water supply amount through irrigation and/or precipitation on the day. In this study, a total amount of 1 mm daily irrigation and precipitation was deemed as efficient, as the water-consuming process is still dominant when there is a tiny amount of precipitation. Daily samples with and without water supply (precipitation and irrigation) in the FI dataset were separated to train the water-consuming sub-model and water-supplying sub-model, respectively.

Input
Besides the dynamic during the water-consuming and -supplying processes, the soil water dynamic across the soil profile due to different soil properties and water distributions along the depth should be accounted for, as the soil moisture is commonly used to schedule irrigations, where the irrigation depth is strongly related to the crop root depth. Therefore, the soil profile was divided into several layers and the constructed NN model employed several three-layer MLPs to model the soil water dynamic of each soil layer, using a waterconsuming or -supplying sub-model. The predicted soil moisture in each soil layer was then used to calculate the weighted average daily root zone soil moisture using Equation (4), which is the output of the NN model. In the same way, each of the NN models with different initial parameters can obtain a weighted average daily soil moisture across the root zone.
For a given day, the weighted average soil moisture across the root zone, i.e., the output of the NN model, was calculated as: where L is the number of soil layers, θ L is the Lth layer soil moisture (cm 3 ·cm -3 ), D L is the depth of the Lth layer (cm), M is the number of soil layers at the deepest simulated root depth, θ M is the Mth layer soil moisture (cm 3 ·cm -3 ) and D r is the root depth (cm). According to the model structure described above, the model inputs include the weather data, namely the air temperature, humidity, wind speed, net radiation, and precipitation on day d; the root depth on day d; and the soil moisture at different soil layers on the previous day (d − 1), to account for the soil water dynamic across the root zone. The involved weather data is responsible for the rate of water evaporation from the soil surface and transpiration from the plant. The irrigation amount was added to the precipitation as a total water input of the model. The output of the NN model is the soil moisture at different soil layers on day d and then the root zone soil moisture can be calculated by Equation (4). By using these inputs and outputs, it is assumed that the predicted soil moisture is the soil water status at the end of day d, which is consistent with the pre-study in Gu et al. [37]. The structure of each NN model was drawn in Figure 3.
According to the model structure described above, the model inp weather data, namely the air temperature, humidity, wind speed, net rad cipitation on day d; the root depth on day d; and the soil moisture at diffe on the previous day (d − 1), to account for the soil water dynamic across the involved weather data is responsible for the rate of water evaporation from and transpiration from the plant. The irrigation amount was added to the a total water input of the model. The output of the NN model is the soil m ferent soil layers on day d and then the root zone soil moisture can be calcu tion (4). By using these inputs and outputs, it is assumed that the predicte is the soil water status at the end of day d, which is consistent with the pre al. [37]. The structure of each NN model was drawn in Figure 3.

Model Training and Evaluation
An NN model library was first constructed, with 1000 NN mode trained with randomly initialized parameters, and the NNE model with 1 t els selected randomly from the NN model library was evaluated. To accou ation of selected NN models, for each of the NNE models, the NN mode lected randomly 100 times. For example, the NNE model with 50 NN m structed by randomly selecting 50 trained NN models from the NN model average performance of the 50 models was identified as the performan model; this process was conducted 100 times so that the variation of averag could be achieved.
For training the NN model in the NN model library, the input neuron 12 for the water-consuming sub-model and 13 for the water-supplying sub with daily soil moistures of 7 soil layers and root depth, as well as the mete without and with precipitation, respectively. The training object is to achie error between the predicted and target values through adjusting model pa recursive gradient descent method [48]. The cross-validation method was the training process. All model parameters were adjusted at each epoch, were developed by the Deep Learning Toolbox of Matlab R2018b. To impr mance of the NN model, input parameters were regularized to the range fr number of hidden neurons was set as 10 through trial and error. The NNE model was evaluated through comparing the predicted so namic during the three crop growing seasons with the target values in W NNE model predicted the daily soil moisture by calculating the mean val

Model Training and Evaluation
An NN model library was first constructed, with 1000 NN models which were trained with randomly initialized parameters, and the NNE model with 1 to 100 NN models selected randomly from the NN model library was evaluated. To account for the variation of selected NN models, for each of the NNE models, the NN models were also selected randomly 100 times. For example, the NNE model with 50 NN models was constructed by randomly selecting 50 trained NN models from the NN model library, and the average performance of the 50 models was identified as the performance of the NNE model; this process was conducted 100 times so that the variation of average performance could be achieved.
For training the NN model in the NN model library, the input neuron number n 1 is 12 for the water-consuming sub-model and 13 for the water-supplying sub-model, that is, with daily soil moistures of 7 soil layers and root depth, as well as the meteorological data without and with precipitation, respectively. The training object is to achieve a minimum error between the predicted and target values through adjusting model parameters by the recursive gradient descent method [48]. The cross-validation method was used to control the training process. All model parameters were adjusted at each epoch, and programs were developed by the Deep Learning Toolbox of Matlab R2018b. To improve the performance of the NN model, input parameters were regularized to the range from −1 to 1. The number of hidden neurons n 2 was set as 10 through trial and error.
The NNE model was evaluated through comparing the predicted soil moisture dynamic during the three crop growing seasons with the target values in WS dataset. The NNE model predicted the daily soil moisture by calculating the mean value of predicted daily soil moistures from those component NN models. Those component NN models used either the predicted hesternal soil moisture from themselves or the mean value from engaged NN models in the NNE model to calculate the intraday soil moisture (Figure 2).

Evaluation Criteria
The following five criteria were used to evaluate the performance of the predicted value: determination coefficient (R 2 ), normalized mean bias error (nMBE), normalized mean absolute error (nMAE), normalized root mean square error (nRMSE), and mean absolute percentage error (MAPE); this indicates a better prediction accuracy when the R 2 is closer to 1 and nMBE, nMAE, nRMSE, MAPE values are closer to zero. These are calculated as: where N is the number of samples, y is the predicted value of the constructed NN model, y t is the target value (soil moisture from RZWQM2 simulations), and y and y t are the average of the predicted and target values, respectively.

Variation of NNE Model Performance
The variation of NNE model performance was evaluated for the three crop growing seasons, and the result for R 2 and nRMSE is shown in Figure 4. The results show that the NNE model performance becomes stable with the increase of component NN model number. When only one NN model was selected from the NN model library to constitute the NNE model, its performance could be awful with the lowest R 2 at about 0.1 and nRMSE about 35%; even some of them are with relatively high performance with R 2 above 0.9 and nRMSE lower than 10%; the average R 2 is about 0.74 and nRMSE 14%. When the NNE model consists of more than 31 NN models with randomly initialed parameters, its R 2 could be consistently larger than 0.9; to achieve an nRMSE lower than 10%, more than 38 NN models with randomly initialed parameters are suggested. With 100 randomly initialed NN models, the maximum and minimum R 2 was 0.97 and 0.94, respectively, with an average of 0.96, whereas the nRMSE was 7.32% and 4.66%, respectively, with an average of 5.93%.

NNE Model Performance
The NNE model ensembled from 100 randomly initialed NN models was evaluated in two different ways: by calculating the daily average root zone soil moisture from (1) individually predicted NN models (individual way) and (2) average input NN models (average way). For the "individual way", each of the NN models predicts the daily root zone soil moisture based on the hesternal soil moisture predicted by itself, whereas the soil moisture on the planting day was given., whereas, for the "average way", the daily average (for each soil layer) on day d was used in each of the NN models to predict the soil moistures on the next day (day d + 1), and then the average soil moisture on day (d + 1) was calculated and applied as the basis for predicting soil moisture on day (d + 2). Figure 5 shows the predicted soil moisture dynamic of the NNE model under the "individual way" of ensemble, and its performance criteria are listed in Table 1. The figure shows that the outline of the daily results with 95% confidence becomes wider in the dimension of soil moisture value along the crop growing season, which indicates that each individual NN model is accumulating errors or consistently over-/under-estimating the soil moisture dynamic. However, the average daily soil moisture, calculated from the daily results with 95% confidence of 100 NN models, showed a high consistence with the target.

NNE Model Performance
The NNE model ensembled from 100 randomly initialed NN models was evaluated in two different ways: by calculating the daily average root zone soil moisture from (1) individually predicted NN models (individual way) and (2) average input NN models (average way). For the "individual way", each of the NN models predicts the daily root zone soil moisture based on the hesternal soil moisture predicted by itself, whereas the soil moisture on the planting day was given., whereas, for the "average way", the daily average (for each soil layer) on day d was used in each of the NN models to predict the soil moistures on the next day (day d + 1), and then the average soil moisture on day (d + 1) was calculated and applied as the basis for predicting soil moisture on day (d + 2). Figure 5 shows the predicted soil moisture dynamic of the NNE model under the "individual way" of ensemble, and its performance criteria are listed in Table 1. The figure shows that the outline of the daily results with 95% confidence becomes wider in the dimension of soil moisture value along the crop growing season, which indicates that each individual NN model is accumulating errors or consistently over-/under-estimating the soil moisture dynamic. However, the average daily soil moisture, calculated from the daily results with 95% confidence of 100 NN models, showed a high consistence with the target. Note: R 2 is determination coefficient, nMBE is normalized mean bias error, nMAE is normalized mean absolute error, nRMSE is normalized root mean square error, and MAPE is mean absolute percentage error. The valid line shows the outline of the daily results with 95% confidence, and the daily average soil moisture was calculated using those results within 95% confidence. Each NN model predicted daily soil moisture based on the hesternal value from its own. DAP is day after plant; RZWQM2-WS represents the water stress (WS) method with RZWQM2 simulations.
With the same 100 NN models used under the "individual way" of the ensemble, the result of the NNE model under the "average way" is shown in Figure 6. The corresponding evaluation criteria are listed in Table 2. The result shows that the output of the NNE model generally fits well with the target value during the main crop growing season but underestimate the soil moisture in certain periods, including those after 120 days after planting, when the maize is approaching the "end of grain filling" [phase number 79 on Target value (RZWQM2-WS) NN mean with 95% confidence NN predicted Figure 5. Predicted soil moistures across the root zone of 100 neural network (NN) models with random initial model parameters and given initial soil moisture values, and their daily averages against the target values. The valid line shows the outline of the daily results with 95% confidence, and the daily average soil moisture was calculated using those results within 95% confidence. Each NN model predicted daily soil moisture based on the hesternal value from its own. DAP is day after plant; RZWQM2-WS represents the water stress (WS) method with RZWQM2 simulations. With the same 100 NN models used under the "individual way" of the ensemble, the result of the NNE model under the "average way" is shown in Figure 6. The corresponding evaluation criteria are listed in Table 2. The result shows that the output of the NNE model generally fits well with the target value during the main crop growing season but underestimate the soil moisture in certain periods, including those after 120 days after planting, when the maize is approaching the "end of grain filling" [phase number 79 on the BBCH scale (Biologische Bundesanstalt, Bundessortenamt und CHemische Industrie)] and during the maturity period [46]. The average R 2 for the tested three years was 0.9470, which is slightly lower than the result shown in Table 1, where the average R 2 is 0.9717. For the years 2008 and 2010, the NNE model performances under two ensemble ways are similar; the "average way" of the ensemble achieved a relatively higher decrease of performance in 2009 compared to the "individual way", where R 2 and nRMSE are 0.9184 and 8.13% compared to 0.9694 and 4.20% under the "individual way", respectively. The higher error occurred mainly because of the over-estimation during 50-60 of DAP and its subsequent effect until about DAP = 80 in 2009. A higher predicting error also occurred at the later period of crop growing season, e.g., in 2008 and 2009 under the "average way" and in 2008 and 2010 under the "individual way". This is probably due to the error accumulation. Compared to the error with R 2 ranging from 0.94 to 0.99 for predicting one-day-ahead soil moisture in [33], at about 0.91 for spatial predicting in [49], and from 0.92 to 0.96 for continuous predicting and above 0.99 for one-day-ahead prediction in [37], the error in this study for continuous predicting is fine. However, for either ensemble way, the NNE model presented a relatively high performance for the tested three years, compared to most randomly initialed NN models (shown in Figure 4 when the NN model number is 1, and in Figure 5). This is because the error of an ensemble model is not simply equal to the average error of those NN models that constitute the ensemble model but is related with the difference of each NN model [50]. When each NN model is independent from other constitute NN models, the ensemble model can be more precise. Therefore, the NNE model constitute of randomly initialized NN models achieved a higher performance even under an open-loop strategy, and the predicted root zone soil moisture could be used to schedule irrigations with a soil moisturebased irrigation scheme [4]. A precise soil moisture prediction is required in an open-loop irrigation control system because otherwise the irrigation would be applied before or after the soil moisture reaches the threshold, thereby applying an excessive quantity of water or decreasing the crop yield [37]. The predicted accuracy at a later period of the crop growing season is less important for irrigation scheduling because the crop commonly requires less water during this period and irrigations are not necessary.
that each of them continuously predicted soil moisture based on the hesternal average.

Conclusions
It is concluded that the neural network ensemble method can sufficiently improve the robustness of soil moisture prediction while achieving a high accuracy. In this study, more than 38 NN models with randomly initialed model parameters are suggested in an NNE model to achieve a high model performance. With 100 randomly initialed NN mod- Figure 6. Predicted soil moistures across the root zone of 100 neural network (NN) models with random initial model parameters and given initial soil moisture values, and their daily averages against the target values. The daily average soil moisture was calculated using the 100 NN model results with 95% confidence. Each NN model predicted daily soil moisture based on the hesternal average value. DAP is day after plant, RZWQM2-WS represents the water stress (WS) method with RZWQM2 simulations. Figures 5 and 6, for most periods of the crop growing season, the increasing rate and decreasing slope of soil moisture dynamic fit well with that of the target curve, which indicates that the soil moisture dynamic of the NNE model ensembled in both ways well described the influence of atmosphere, soil situation, and root depth. That is, by taking into account these factors associated with the SPAC system, the constructed NNE model using MLP, including water consuming and water supplying sub-models, was capable of accurately estimating the soil moisture dynamic, even though with limited training samples. In this study, the training samples for water consuming and water supplying sub-models of each NN model are 363 and 164 in total, respectively.

Conclusions
It is concluded that the neural network ensemble method can sufficiently improve the robustness of soil moisture prediction while achieving a high accuracy. In this study, more than 38 NN models with randomly initialed model parameters are suggested in an NNE model to achieve a high model performance. With 100 randomly initialed NN models, the NNE model achieved an average R 2 of 0.96 and nRMSE of 5.93%, suggesting that the NNE model is capable of precisely estimating the soil moisture dynamic over the whole crop growing season, without any feedback from field measurement.
Two ensemble ways were investigated, namely, the "individual way" and "average way" with respect to the calculation of the daily root zone soil moisture. For both ensemble ways, the NNE model showed a high performance (robustness and accuracy) for the three years tested. Though with an open-loop prediction strategy, the trained NNE model predicted the soil moisture dynamic well during the main crop growing season by taking SPAC factors into consideration.
Since the NN model in this study used a three-layer MLP, which is conventional and widely used, the proposed robust NNE model can be easily and widely applied to field controllers, which is low cost compared to using a laptop running agricultural system models. Moreover, with the help of soil moisture measurements as feedback, the constructed NN/NNE model can be improved online and immediately applied without professional calibrations when using agricultural system models. The future work should be focused on the evaluation of a real-time irrigation scheduling system based on the proposed NNE soil moisture model.