Performance Evaluation of Five Machine Learning Algorithms for Estimating Reference Evapotranspiration in an Arid Climate

Abstract

The Food and Agriculture Organization recommends that the Penman–Monteith Method contains Equation 56 (PMF) as a widely accepted standard for reference evapotranspiration (ET_o) calculation. Despite this, the PMF cannot be employed when meteorological variables are constrained; therefore, alternative models for ET_o estimation requiring fewer variables must be chosen, which means that they perform at least as well as, if not better than, the PMF in terms of accuracy and efficiency. This study evaluated five machine learning (ML) algorithms to estimate ET_o and compared their results with the standardized PMF. For this purpose, ML models were trained using monthly time series climatic data. The created ML models underwent testing to determine ET_o under varying meteorological input combinations. The results of ML models were compared to assess their accuracy and validate their performance using several statistical indicators, errors (root-mean-square (RMSE), mean absolute error (MAE)), model efficiency (NSE), and determination coefficient (R²). The process of evaluating ML models involved the utilization of radar charts, Smith graphs, heatmaps, and bullet charts. Based on our findings, satisfactory results have been obtained using RBFFNN based on M12 input combinations (mean temperature (T_mean), mean relative humidity (RH_mean), sunshine hours (Sh)) for ET_o estimation. The RBFFNN model exhibited the most precise estimation as RMSE obtained values of 0.30 and 0.22 during the training and testing phases, respectively. In addition, during training and testing, the MAE values for this model were recorded as 0.15 and 0.17, respectively. The highest R² and NSE values were noted as 0.98 and 0.99 for the RBFNN during performance analysis, respectively. The scatter plots and spatial variations of the RBFNN and PMF in the studied region indicated that the RBFNN had the highest efficacy (R², NSE) and lowest errors (RMSE, MAE) as compared with the other four ML models. Overall, our study highlights the potential of ML models for ET_o estimation in the arid region (Jacobabad), providing vital insights for improving water resource management, helping climate change research, and optimizing irrigation scheduling for optimal agricultural water usage in the region.

1. Introduction

Providing water to dry and semi-arid regions has become a worldwide challenge. As the global population rises, countries seek new and improved water supply management methods [1]. Improving water management systems and quantitatively scheduling irrigation is crucial to addressing water scarcity issues. Irrigation scheduling, which may be calculated based on evapotranspiration (ET), refers to providing crops with the ideal quantity and timing of water. ET comprises water evaporation into the atmosphere from the soil surface, groundwater table’s capillary fringe, and land-based water bodies. Water flow from the ground to the atmosphere through plants is also included in ET [2]. ET can be directly estimated using lysimeter, eddy covariance system, surface renewal, flux variation and Bowen ratio methods. The high initial, continuing, and maintenance costs of these methods hinder ET estimation [3,4,5]. Therefore, indirect methods are used to quantify ET. The indirect methods use empirical models, categorized into temperature-based, radiation-based, and mass-transfer-based.

Including an estimate of the reference evapotranspiration (ET_o) is a basic factor in the empirical models. The difficulty in ascertaining the ET for each individual crop is the underlying factor contributing to this issue. Hence, indirect methods are used to estimate ET_o and crop coefficients, which are further utilized to quantify ET for individual crops of interest. The Penman–Monteith (PMF) approach, first introduced by Allen in 1998 and subsequently validated in several climatic conditions [4,6], is now endorsed by the Food and Agriculture Organisation (FAO) of the United Nations and the American Society of Civil Engineers (ASCE) committee. In order to complete PMF ET_o calculations, it is important to possess a comprehensive meteorological data collection for several places around the globe. Although the relative importance, interdependence, and interrelationships among the components were previously unclear, recent studies provided ML-based projections of the PMF ET_o using a substantial amount of meteorological data [7,8]. Ravindran et al. [9] employed the PMF using the available meteorological data to calculate ET_o and then compared it with a deep neural network (DNN) model that only takes sun radiation (Rn) as an input parameter [9]. Zhou et al. [10] investigated the efficacy of machine learning (ML) models, namely deep factorization machine (DeepFM), gradient boosting with categorical feature support (CatBoost), light gradient boosting (LightGBM), extreme gradient boosting (XGBoost), and gradient boosting decision tree (GBDT), for estimating the ET_o. Basagaoglu et al. [7] determined that Rn is the most important meteorological variable for determining ET_o in a semi-arid location. However, sunshine hours (Sh) might be substituted when Rn is unavailable. The analyses of previous studies revealed that Sh exhibits a closer association with Rn than the other meteorological variables and can be used in the absence of Rn [11,12,13]. The current study used the Sh variable in the analysis for ET_o estimation because of the deficiency in the Rn factor.

A major barrier in ET_o estimation using the PMF is the unavailability of several meteorological variables and uncertainty in collected data [14,15]. Despite the availability of critical climatic variables, a scarcity of automated weather stations installed at specific sites necessitates the presence of climatic data in numerous locations. Additionally, the use of obsolete weather stations raises issues with data quality. Data calibration is required since the sensor’s inconsistent performance exposes several variances. In this scenario, the inconsistency of ET_o estimates opens the door for developing alternative techniques that use fewer inputs to calculate ET_o similarly or at least approach the PMF.

Several studies have been conducted that demonstrate an analogy between proposed models based on meteorological data and the PMF. These studies aim to provide alternate ways for ET_o estimates because of the frequent absence of meteorological variables [16,17,18,19]. Despite this, ML based on various algorithms (cuckoo, water wave optimization, coactive neuro-fuzzy inference system, decision support system) has become popular for ET_o modeling in comparison with empirical equations (the PMF, Hargreaves, Turc, Jensen–Haise, Hargreaves–Samani) using limited climatic data [20,21,22,23,24,25,26,27,28]. The ML models provided good output (ET_o) that approached the corresponding empirical equation (PMF) because the optimal selection of input variables against the output factor by adding or eliminating an input variable is considered the main advantage in increasing the efficacy of input versus output relation [29,30,31,32,33,34,35].

The subject of ET_o modeling using ML has gained significant attention in recent academic studies, raising interest among hydrologists and meteorologists across different countries. Yin et al. [36] calculated daily ET_o in a Hilly interior watershed in northwest China and found support vector machines (SVMs) worked best using daily meteorological data. Wen et al. [37] compared ML with four empirical models in an arid region of China and found temperatures (minimum and maximum) as effective parameters for ET_o estimation. Wang et al. [38] analyzed the efficacy of two ML models, namely gene expression programming (GEP) and artificial neural networks (ANNs), in the Karst area of Guangxi Province located in China, for calculating daily ET_o. The research showed that ML outperformed with fewer meteorological inputs. Sanikhani et al.’s [39] study showed that the monthly ET_o in the Isparta and Antalya region (Turkey) was successfully quantified by employing ML models using limited (temperature) data. In an arid region (Sistan and Baluchestan Province) of Iran, Pour et al. [40] used ML with varying input climatic combinations and found that the ML model (SVM) exhibited superior performance in comparison with the other models for the estimation of ET_o. In Jiangxi Province (China), Wu et al. [41] examined the efficacy of ML models using cross-station and synthetic data to measure monthly mean daily ET_o and concluded that tree-based ML outperformed other models. Daily ET_o using limited climatic data was estimated by Saggi and Jain [42] in Punjab province, India. They found deep learning exhibited superior performance compared with the other models. Similarly, Shiri et al. [43] found good performance of ML for ET_o estimation using 29 weather stations in Iran. Tikhamarine et al. [44] compared five ANN-based meta-heuristic algorithms (gray wolf optimizer, multiverse optimizer, particle swarm optimizer, whale optimization algorithm, ant lion optimizer) to estimate monthly ET_o using limited data in the Uttarakhand State (Himalayan Region) of India and found that gray wolf optimizer performed well in comparison with the targeted empirical (valiantzas) model. Likewise, Ferreira et al. [45] found that the performance of ML (ANN, GEP) models was good and close to the PMF with minimal climate data in several regions of Brazil. Granata [46] estimated ET_o using climate data collected from eddy covariance (EC) flux tower stations set up at a location (Floral City) in central Florida with a humid subtropical environment and found that the chosen ML models were capable of ET_o modeling with limited input data in the studied area. Keshtegar et al. [47] compared tree-based and ANN-based ML models with polynomial chaos expansion and the response surface method in the Mediterranean region (Isparta and Antalya) of Turkey for estimating ET_o. Likewise, Nourani et al. [48] estimated ET_o using climatic data in different climatic regions across five countries (Turkey, NC, Iraq, Iran and Libya) by employing ML ensemble based models and found ANN based ML performed good in comparison to other models. Shiri [49] used limited climatic input in ML models for estimating ET_o in an arid region of Iran, and comparison results with six empirical equations showed the supremacy of the chosen ML model.

Recent advances have made it easier to access weather records, yet a lack of meteorological data persists in many locations. Raza et al. [50] systematically reviewed research articles on ET_o estimation using ML under Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and suggested the comparison of ML algorithms for ET_o using fewer inputs, especially in developing countries (Pakistan) due to the lack of climatic data. When conventional methods (like the PMF) cannot be used because of excessive input demands or the unavailability of climatic parameters like Rn [36,37,38,39,40,41,42,43,44,45,46,47,48,49], the task of improving methods that rely on fewer climatic inputs and developing ML models for ET_o estimation with minimal climatic records becomes of great relevance. Kushwaha et al. [51] concluded that ML might capture time series data without discretization when handled correctly and be able to estimate ET_o accurately. Similar findings were obtained by Wu et al. [52] in the Poyang Lake basin in southern China, Roy et al. [53] in the Gazipur District in Bangladesh, Ahmadi et al. [54] in the arid and semi-arid climate of Iran, Sattari et al. [55] in the Corum province of Turkey, and Malik et al. [56] in the Uttarakhand State of India in comparison with the empirical (PMF) method for ET_o modeling.

Meteorologists have always monitored the conditions of the atmosphere that impact the weather, but over time the equipment they use has changed. As technology advanced, our scientists began to use more efficient equipment to collect and use weather data. These technological advances enabled meteorologists to make better predictions faster than ever before. These instruments include Doppler radar, Satellite data, Radiosondes, Automated surface-observing systems, and AWIPS (NOAA’s Advanced Weather Information Processing System). The current study obtained climatic data of Jacobabad Meteorological Observatory (JMO) from the Regional Meteorological Center (RMC) Karachi. It is the official meteorological department which is run by the Pakistani government. The current study obtained climatic data (1987–2016) of Jacobabad Meteorological Observatory (JMO) from the Regional Meteorological Center (RMC) Karachi, Pakistan.

According to Germanwatch, the environmental NGO, Pakistan is the eighth most vulnerable country to extreme weather due to climate change. Natural catastrophes, including floods, droughts, and cyclones, have been catastrophic in recent years, resulting in hundreds of deaths, millions of displaced people, and extensive property damage. Jacobabad is an arid city in the province of Sindh, Pakistan. The city’s administration is split into eight different Union Councils. The average summer temperature is 37 °C, while the winters are mild, with an average temperature of 7.7 °C. Precipitation in the city ranged from a high of 838.7 mm in 2022 to a low of 3.3 mm in 1922, with an annual average of 122.5 mm between 1991 and 2020. During the South Asian heat wave in 2022, the average monthly temperature recorded was 43 °C (109 °F), including four days of temperatures equal to or above 50 °C. People in Jacobabad moved during the city’s hottest months (June–August) to avoid the oppressive heat. City heat-reduction services need personal care from government sectors. Being higher in elevation, Quetta (the northern side of Pakistan) is more appealing to locals during the warmer summer. Access to clean water in Pakistan is unreliable and costly because of widespread shortages and significant infrastructure problems. Consequently, monitoring and managing agricultural water usage in this arid region is crucial.

With the recent development in ET_o modeling based on ML algorithms using less climatic data, the current study explores five ML algorithms’ performances for estimating ET_o in the arid region (Jacobabad) of Sindh province, Pakistan. The main goal of this study is to determine effective climatic variables for ET_o estimation. In the current study, a total of 17 different input models were initially formulated based on different combinations made by using the meteorological inputs which are mainly used in the PMF for ET_o estimation. We tried to accurately estimate ET_o with limited climatic data and the best input combination to calculate ET_o, which can be used as an alternative to the PMF in Jacobabad. The novelty of this study is that a comparison of five ML algorithms, namely Iterative dichotomizer (ID3), gradient boosting (GB), Random forest (RF), multilayer neural network (MLNN), and radial basis function neural network (RBFNN), for ET_o estimation using less climatic data is not yet explored in an arid region (Jacobabad) of Pakistan. In a nutshell, the following are three objectives of this study:

• ET_o estimation using five ML (ID3, GB, RF, MLNN, and RBFNN) algorithms.
• Identifying an effective combination of meteorological inputs for ET_o estimation.
• Performance evaluation of ML algorithms through visualization (radar, heatmap, bullet, and Smith graphs) based on different statistical indices to determine the best one among them.

The findings of the current study practically contributed in a way that the ET_o retrieved from the ML model may be interpolated into monthly and yearly ET_o maps depicting ET_o variations over the studied area. Agronomists, hydrologists, and agricultural engineers may use the ET_o maps to more accurately determine the water needed to irrigate the crops (via drip and spray irrigation).

2. Study Area and Dataset

Pakistan lies between moderate to scorching climatic zones [52]. Temperatures vary widely from location to location, but the climate is generally arid, with scorching summers and frigid winters. The current study obtained climatic data of Jacobabad Meteorological Observatory (JMO) from the Regional Meteorological Center (RMC) Karachi (weblink: https://rmcsindh.pmd.gov.pk/, accessed on date 15 August 2023). The observatory (index number 41715) installed by the RMC is situated at a latitude of 28°18′, longitude of 68°28′, and altitude of 56 m (Figure 1).

Table 1. Summary of 30 years (1987–2016) of climatic data obtained from Jacobabad weather station installed by the RMC.

Variables	Tmin	Tmax	RHmean	Ws	Sh
	°C	°C	%	m/s	h
Mean	20.33	34.23	38.10	1.29	7.79
Standard Error	0.42	0.38	0.63	0.04	0.03
Median	21.50	35.90	37.00	1.24	7.70
Mode	29.10	25.70	33.00	1.11	7.30
Standard Deviation	7.96	7.23	11.88	0.67	0.59
Sample Variance	63.35	52.25	141.21	0.45	0.35
Kurtosis	−1.39	−1.10	−0.52	−0.40	−1.65
Skewness	−0.30	−0.21	0.22	0.28	0.10
Range	25.80	27.20	62.00	3.15	1.60
Minimum	5.20	20.10	11.00	0.00	7.00
Maximum	31.00	47.30	73.00	3.15	8.60
Sum	7318.30	12,322.20	13,717.00	465.89	2805.00
Count	360.00	360.00	360.00	360.00	360.00

displays a summary of 30 years (1987–2016) of climatic variables obtained from the RMC. Furthermore, Figure 2a,b present monthly and interannual changes in the climatic variables utilized in the present study. Figure 2a shows a notable trend wherein the minimum temperature (T_min) exhibited an initial increase from January to July, followed by a slow decline until December. The observed pattern in Tmax was consistent, as shown in Figure 2a. The fluctuation in seasons can be attributed to the solar radiation emitted by the sun, as depicted in Figure 2a. Sh witnessed the summer’s peak and the beginning of winter’s decline. The variable RH_mean exhibited a relationship with both T_max and T_min, as it was seen that the periods of elevated air temperature coincided with the highest levels of humidity. According to the data presented in Figure 2a, the average relative humidity (RH_mean) exhibited its maximum values throughout July to September, which can be attributed to the impact of wind, and there could be much more important meteorological reasons, such as the atmospheric circulation pattern together with the surrounding ocean and topography of the selected regions. During the summer season, it can be observed from Figure 2a that there is an upward trend between wind speed and humidity levels in the air. The summer months exhibited the most notable rise in average and maximum humidity levels. In addition, Figure 2b indicates cyclical conditions of climatic variables starting from 1 January to 31 December for the years 1987 to 2016.

Physical–Geographical Conditions

Jacobabad is geographically situated in a manner that it shares its boundaries with three provinces concurrently. These provinces include the Rajanpur District of Punjab Province and the Nasirabad district of Balochistan Province. The Indus River influences the district, resulting in direct effects on the Kashmore and Kandhkote Tehsils. The Guddu Barrage in the Kashmore Tehsil serves as the point of origination for historical water channels, such as the Pat feeder canal. The total land area of Jacobabad district is 2686 square kilometers. The district encompasses Kacha communities residing in the Kandhkote and Kashmore Talukas, occupying extensive land parcels on both banks of the Indus River. The agricultural output of the Kacha region is limited to Rabi crops due to the annual flooding of the river during the summer. The investigated site exhibits varied topography and variations in elevation, resulting in significant variations in evapotranspiration values. The gradual decrease in elevation from Kashmore to Jacobabad and its surrounding areas has not obstructed the river’s continuous flow. The Indus River has facilitated social and agricultural development by watering and feeding the surrounding landscape, enabling the cultivation of crops twice a year without the need for fallow periods. During the Kharif season, most of the paddy is harvested and rabbit crop wheat is sown, both of which need a lot of water. Nevertheless, the issues of floods and salinization have emerged as significant challenges, resulting in detrimental effects on agricultural land and a decline in crop productivity. Attempts to mitigate the prevailing circumstances involve implementing drainage systems and excavation activities aimed at reducing the water table. However, despite such efforts, the situation continues to progressively deteriorate. The management of sewage systems is a prevalent concern across the western bank, necessitating the implementation of a siphon mechanism to divert the flow of wastewater towards the left bank.

3. Study Methodology

3.1. PMF

The CROPWAT software version 8.0 from the FAO of the United Nations was used to compute the ET_o using meteorological input values. The PMF approach was used to determine the value of ET_o in this software (Weblink: https://www.fao.org/land-water/databases-and-software/cropwat/en/, access date, 20 September 2023). The PMF developed by Allen et al. [4] in 1998 determines ET_o values based on a combination of meteorological and aerodynamic parameters (es, ea, e_min, e_max, Δ, G, and Ɣ). The mathematical notation of the PMF is described as follows:

$${ET}_o=\frac{0.408∆\left(R_n-G\right)+\gamma \times \frac{900}{T_{mean}+273}\times Ws\times \left(e_s-e_a\right)}{∆+Ɣ\left(1+0.34Ws\right)}$$

(1)

where:

ET_o is measured in mm/day; R_n = net radiation at the surface (MJ/m²/day);

G = soil heat flux density (MJ/m²/day); T_mean = air temperature at 2 m height (°C);

e_s = saturation vapor pressure (kPa); e_a = actual vapor pressure (kPa);

Δ = slope of vapor pressure curve (kPa/°C); γ = psychrometric constant (kPa/°C);

Ɣ = 0.000665 × p (kpa); e_s is saturation vapor pressure (kPa); Ws = wind speed.

3.2. Proposed ML Framework

PMF ET_o was used as the benchmark for output (ET_o) values when chosen ML algorithms were applied. For this purpose, we used Waikato Environment for Knowledge Analysis (WEKA) software version 3.9.4. It is a collection of machine learning algorithms for data mining tasks. It contains tools for data preparation, classification, regression, clustering, association rules mining, and visualization. It is an open-source software issued under the GNU General Public License (weblink: https://www.cs.waikato.ac.nz/ml/weka/, accessed date, 5 August 2023). Weka was used in this study, and the following steps were implemented. The dataset used was pre-processed in the first step (1). The data file was prepared in comma-separated values (CSV) file format (2), and it was uploaded to Weka software by using the import feature (3). The optimal parameters related to ML algorithms were selected (4), data were classified (5), and the nearest neighbor was chosen as the estimation function (6). In the cross-validation process, K-means clustering was utilized (7), association rules were performed (8), and, in the last step, the model was evaluated (9). A schematic framework of the ML models suggested for ET_o estimation is shown in Figure 3.

3.2.1. Iterative Dichotomizer (ID3)

The ID3 algorithm relies heavily on the measurement of entropy and information gain. Both entropy (E) and information gain (IG) may be defined in terms of the predicted decrease in entropy (Purity) and the homogeneity (H) of a learning set (Impurity), respectively. E and IG may be written in general mathematical form using Equations (2) and (3), respectively. If p is the fraction of S in class i, then both terms are provided by Equations (4) and (5) in the discrete case.

$$\mathrm{E}=-\mathrm{A}{\log }_2\mathrm{A}-\mathrm{B}{\log }_2\mathrm{B}$$

(2)

$$\mathrm{I}\mathrm{G}={\mathrm{S}}_{\mathrm{b}}-{\mathrm{S}}_{\mathrm{a}}$$

(3)

$${\left(\mathrm{E}\right)}_{\mathrm{b}}=\sum _{\mathrm{i}=1}^{\mathrm{k}}-{\mathrm{p}}_{\mathrm{i}}{\log }_2{\mathrm{p}}_{\mathrm{i}}$$

(4)

$${\left(\mathrm{I}\mathrm{G}\right)}_{\mathrm{b}}=\sum _{\mathrm{i}=1}^{\mathrm{k}}{{\mathrm{p}}_{\mathrm{i}}}^2$$

(5)

where Sb is the entropy of distribution before the split, Sa is the entropy of distribution after the split, (E)b is the entropy for the k-wise classified function, and (IG)_b is the information gain for the k-wise classified function.

The decision model may benefit significantly from simplification by following the top-down greedy (TDG) search process. The TDG process is straightforward and consists mainly of two parts. Using the provided data, initial calculations for each characteristic are made. Second, we determine a function that optimally divides the node. The TDG procedure and the depth of the resulting tree are also critical factors in the categorization of datasets. The ideal value of each characteristic is evaluated and then used to divide the tree node. The E and IG are then compared for each characteristic. The property with the highest IG and lowest E value is chosen as the root node’s primary identifier. IG aims to create a hierarchy in which attributes are ranked from top to bottom, starting at the root node. There is currently insufficient evidence to conclude that the attribute present in each node with a greater value of IG and a lesser value of E is adequate to establish a root node. This hierarchy aims to facilitate the desired moderation of the decision-making process. As a result, we created a forest of little trees with shallow roots. A smaller tree facilitates faster dataset identification. We chose the attribute with the highest E and IG values from the training data and assigned it to the root node. We created smaller subsets of the training data that were assembled into a larger whole. Each attribute value should be consistent across all datasets used to create the subset. Recursion happens until a child node has been formed for each branch. Ultimately, we compared the recorded attribute values to the original attribute.

3.2.2. Gradient Boosting (GB)

Gradient boosting is a method through which a prediction function is increased by repeated use. After that, we add each function’s output according to its relative importance. By taking this measure, prediction will get closer to accuracy, and errors will be eradicated. Because of this helpful property, GB is often represented as a linear model. The ability of this approach to decouple dependent and independent variables is a significant plus. Equation (6) is a numeric representation of the GB model, and it is written as

predicted target = F_o + B₁ × T₁ (X) + B₂ × T₂ (X)………………+ B_m × T_m (X)

(6)

where F_o is the starting value in the series, X is the vector of “pseudo-residual” values remaining at this point in the series, T_x(X) represents the trees fitted to the pseudo-residuals (here, x = 1, 2, 3…….m), B_x is the coefficient of the tree node predicted values (here, x = 1, 2, 3…….m).

The boosting procedure will use a primary tree ensemble mechanism, as seen in Figure 4. It has been discovered that a succession of three trees is connected. Tree 2 receives the residual values that Tree 1 produces. Likewise, the values of Tree 2’s residuals may be found in Tree 3’s structure. Furthermore, there is no apparent connection between Trees 1 and 3. This procedure was repeated until every relevant data subset was included in the phase. Next, the weighting index of each tree is added to the dataset, and the misclassification error is calculated.

There are several ID3-style trees inside a GB ML algorithm. Each of these supplementary trees is connected to the others. Thus, there is a link between the neighboring trees. So, it is a chain of trees connected. In this investigation, we explored the following applications of the GB algorithm: (i) The climate predictor variables and their corresponding input parameters. (ii) Misclassification errors that occurred during model calibration were eliminated using the Huber loss function and the surrogate splitter technique. The K-type classification was employed for dataset segregation, partitioning it into k and k−1 subsets. Following multiple trials and an error-based approach, tree numbers for the GB model at minimum and maximum level were allocated to create a series that enables ensemble processing. The k-test subsets were thoroughly tested, with the k-1 subset serving as a means for instruction. The overall ET_o value was predicted by averaging the values from the various subsets.

3.2.3. Random Forest (RF)

The bagging technique is a collection of untrained decision trees utilizing bootstrap samples. The underlying structure of the method used to build the RF model may be grasped quickly. Treat the whole dataset as if it were N observations. Select the desired subset of N observations from the data collection. Bagging is the term for this operation. Some observations are utilized many times in the bagging process. Thus, not all of them are included in the analysis. Two-thirds of a data collection is often sampled, with the other third classified as “out of bag” (OOB) information. Each tree is built using a random subset of the available data.

The data from the chosen rows are then used to construct a decision tree. It is intriguing to learn that, unlike the ID3 and GB models, the RF model does not need tree pruning. The bagging procedure is performed recursively until all trees have been built. Only a subset of the available predictor variables is used when building a tree. Each node’s potential splitter is chosen randomly from this collection of predictor factors. For instance, if there are six predictor variables, three will be selected randomly to be used as splitter candidates.

Nonetheless, the uninvolved variables might be utilized for another node split in the same tree. Not all of the predictor variables have to be used in the separation. Thus, some predictor variables may need to be used throughout a tree’s structure. This procedure is repeated until a wooded area has been established. After a subset has been chosen, the scoring process involves cycling over each tree in the forest in turn with each data row. The scoring method generates projected values that are used in the development of terminal nodes. Using several techniques, the RF model yields accurate predictions in regression and classification analyses. When doing a regression analysis, the typical score predicted by all trees is determined. The “votes” for the best variable in a classification study come from the variables predicted for each tree. The most popular predictor variable is used to provide a score for each row in the tree. There are two forms of randomness in an RF algorithm. First, a subset of rows from the data is used as an input for each tree, and then a predictor variable is utilized to determine how to divide up the nodes in the tree. These supplementary characteristics significantly boost the RF model’s overall predictive accuracy when added to forecasts. The following procedures were used to create the TF model for this investigation:

• Climatic variables were chosen as predictors/inputs.
• The sample data constituted 70% of the total dataset, separated using the randomization function.
• Analysis was performed using dataset (OOB), and residual values were stored separately.
• The ET_o obtained from each tree as output was gathered and stored.
• The mean values of the output variable (ET_o) were computed as a whole.

Figure 5 shows the procedure for using ML algorithms that make decisions. As a starting point, we feed in data on temperature ranges, humidity averages, wind speeds, and sunshine totals. Separate sets of data, known as train and testing, were created. Second, we tuned ideal parameters to improve the performance of the ID3, GB, and RF algorithms used to calculate ET_o.

Table 2. Tuned values of parametric variables for decision-based ML algorithm.

Decision-Based ML Algorithm	Parametric Variables
	Tree Numbers	Splitter	Node Size
Iterative Dichotomizer (ID3)	12	22	04
Gradient Boosting (GB)	14	26	06
Random Forest (RF)	18	29	08

overviews the best possible ID3, GB, and RF parameters.

3.2.4. Multilayer Neural Network (MLNN)

An input layer, a hidden layer, and an output layer make up the MLNN architecture. There are a certain number of neurons in each layer, all of which communicate in the same way. Predictor variables from the previous layer are stored in each neuron, as are the number of outputs to the next layer. The structure’s overall effectiveness depends on the contributions of each layer. Weights establish the precise nature of the connections between neurons in different layers. The results are supposed to be carried by these kilos from one layer to the next. Each neuron in the input layer sends its output to a corresponding neuron in the hidden layer. Just as the output of one neuron in the hidden layer becomes the input of another neuron in the output layer, the reverse occurs in the output layer. Figure 6 depicts an MLNN architecture with a single hidden layer. The input layer neurons are all linked to one another in the hidden layer, while the hidden layer neurons are only linked to one another in the output layer. Input and output layers do not have any “direct” connections between their neurons. The name “feed-forward neural network model” comes from the unidirectional nature of the data flow it facilitates. The following procedures were used to determine ET_o using the MLNN:

• Climate-related input variables were chosen as predictors.
• Sigmoid and linear activation functions were used in the input-hidden and hidden-output layers.
• The weight connection between interconnected neurons was adjusted via the adoption of a back-propagation procedure.
• The kernel functions utilized were Traditional Conjugate Gradient (TCG) and Scaled Conjugate Gradient (SCG).
• The ET_o was estimated as an output.

3.2.5. Radial Basis Function Neural Network (RBFNN)

There are three distinct layers in the RBFNN: the input, the hidden, and the output. Simply put, the input layer is made up of a large number of neurons that are linked to the hidden layer through weighted connections. These weights convey the values from one layer to the next. The input layer neurons utterly unconnected to one another form the output layer. Both a hidden layer and an output layer need linear and transfer functions. Activation functions are another term for these mathematical operations. A parametric vector accompanies the neurons in the buried layer. This vector, known as the hidden layer’s center, may be considered a weighted index. The distance between an input vector and the hidden layer’s center is quantified using the Euclidean distance (E). The RBFNN network used in this analysis was developed as follows. We used V-fold cross-validation to ensure the integrity of the dataset, tuned the RBFNN network to optimal performance by selecting a population size of 200 and a maximum generation of 20, and used the radial basis function’s activation function in both the hidden and output layers. Figure 7 depicts a schematic representation of the RBFNN algorithm used in this investigation, illustrating the input–output link between climate variables and ET_o.

3.2.6. Selection of Meteorological Input Combinations

A total of 17 meteorological input combinations were used as input, which are tabulated in

Table 3. Seventeen scenarios for determining the best input combinations to estimate ET_o.

Input Combination	Symbol
Tmin, Tmax, RHmean, Ws, Sh	M1
RHmean, Sh	M2
RHmean, Sh, Ws	M3
RHmean, Ws	M4
Tmax,Tmin, Sh, Ws	M5
Tmax, RHmean, Sh, Ws	M6
Tmax, RHmean, Ws	M7
Tmax, Tmin, RHmean, Sh	M8
Tmean, RHmean, Sh	M9
Tmin, RHmean, Ws	M10
Tmin, RHmean, Sh, Ws	M11
Tmean, RHmean, Ws	M12
Tmax, Tmin, RHmean, Ws	M13
Tmean, RHmean, N, Ws	M14
Tmean, RHmean	M15
Tmean, Sh	M16
Tmean, Ws	M17

Note(s): T_min, minimum temperature; T_max, maximum temperature; RH_mean, mean relative humidity; Ws, wind speed; Sh, sunshine hours.

. It can be perceived that M1 contains inputs T_min, T_max, RH_mean, Ws, and Sh for ET_o estimation. When M2 is used to estimate ET_o, ML algorithms (ID3, GB, RF, MLNN, and RBFNN) with the RH_mean and Sh variables are considered as inputs. Likewise, Table 3 provides information on the other 15 input combinations (M1 through M17) in the same format as M1 and M2. Previous research evaluated the performance of the ML algorithms using various statistical criteria [40,41,42,43,44,45,46,47,48,49,50]. The ML algorithms’ performance using all 17 input combinations was measured using the statistical indices root-mean-square error (RMSE), determination coefficient (R2), mean absolute error (MAE), mean bias error (MBE), and Nash–Sutcliffe efficiency (NSE).

$$R^2={\left[\frac{\sum _{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{E}\mathrm{T}}_{\mathrm{o}\mathrm{b}\mathrm{s}}-\overline{{\mathrm{E}\mathrm{T}}_{\mathrm{o}\mathrm{b}\mathrm{s}}}\right)\left({\mathrm{E}\mathrm{T}}_{\mathrm{e}\mathrm{s}\mathrm{t}}-\overline{{\mathrm{E}\mathrm{T}}_{\mathrm{e}\mathrm{s}\mathrm{t}}}\right)}{\sqrt{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{E}\mathrm{T}}_{\mathrm{o}\mathrm{b}\mathrm{s}}-\overline{{\mathrm{E}\mathrm{T}}_{\mathrm{o}\mathrm{b}\mathrm{s}}}\right)}^2}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{E}\mathrm{T}}_{\mathrm{e}\mathrm{s}\mathrm{t}}-\overline{{\mathrm{E}\mathrm{T}}_{\mathrm{e}\mathrm{s}\mathrm{t}}}\right)}^2}\right]}^2$$

(7)

$$RMSE=\sqrt{\sum _{i=1}^N\frac{{{(ET}_{obs}-{ET}_{est})}^2}{n}}$$

(8)

$$MAE=\sum _{i=1}^N\frac{{|ET}_{obs}-{ET}_{est}|}{n}$$

(9)

$$NSE=1-\frac{\sum _{i=1}^n{(ET}_{obs}-{ET}_{est})}{\sum _{i=1}^n{(ET}_{obs}-\overline{{ET}_{obs}})}$$

(10)

$$MBE=\frac{1}{n}\sum _{i=1}^n({ET}_{obs}-{ET}_{est})$$

(11)

where ET_obs, ET_est, $\overline{{ET}_{obs}}$, and $\overline{{ET}_{est}}$ are the observed, estimated, average observed, and average estimated ET_o, respectively, and n represents total records.

R² is the goodness-of-fit parameter, which may be from 0 to 1. It is a number between −1 and 1 that has no dimensions. A correlation coefficient near 1 indicates a strong association. The MSE calculates the average squared deviation between observed and anticipated values. When comparing errors between experimental and anticipated values, the root-mean-square error (RMSE) is often utilized. Model performance improves when the MSE, RMSE, and NRMSE values decrease. The NSE index is a regularly used goodness of fit statistic to evaluate a model’s effectiveness. The NSE may be measured from −1 to 1. Lower RMSE and MAE values and greater R² and NSE values indicate a superior model.

4. Study Results

Figure 8, Figure 9, Figure 10 and Figure 11 show the findings as several statistical indices (RMSE, R², MAE, and NSE). The 30-year, 360-record dataset was split into training (252 records) and testing sets (108 records). Each ML model’s performance accuracy over all seventeen scenarios was evaluated using the testing set. The training set was utilized for calibration and model development. The ET_o estimated with the PMF was used as the standard for assessment of both decision-based and neural network-based ML algorithms.

The study considered various meteorological variables’ possible combinations (M₁ = T_min, T_max, RH_mean, W_s, S_h to M₁₇ = T_mean, W_s) to determine their impact on ET_o prediction. The performance of ET_o prediction was evaluated by adding and removing different meteorological variables. The interpretation of the performances was based on a combination of statistical indicators and graphical approaches. Figure 8 presents the performance evaluation of ML models using bullet charts. The structure of statistical metrics for each model in the training and testing phases is provided. It is worth mentioning that all the ML models created attained high levels of accuracy. Nevertheless, after scrutinizing the NSE and R2 values, it can be deduced that the RBFNN algorithms and the M12 and M13 combinations showed the highest precision. Likewise, examining the RMSE and MAE values, it was concluded that the M12 combination exhibited the lowest error rate and yielded the most precise prediction results using the RBFNN ML algorithms.

In Figure 9, the selection of the optimal ML model is evaluated through performance metrics and addressed with heatmap representations. Different data densities represent different levels of relationship and measurement. The combinations M1, M5, M6, M7, M10, M11, M12, M13, and M14 exhibit similar and satisfactory levels of prediction accuracy based on R² and NSE values. However, the highest prediction for ET_o was achieved using RBFFNN algorithms. When evaluating ML model performances based on RMSE and MAE error metrics, the suitable input combination was found to be M12. Additionally, the model combinations M1, M11, M12, and M13 stand out for their low RMSE and MAE and high accuracy (NSE and R²) as compared with other input combinations. Furthermore, when the optimal algorithm is considered, the RBFNN algorithm was tested to yield the highest prediction performance. According to this, the optimal model has RMSE values of 0.30 during training and 0.22 during testing. Additionally, the MAE values for this model are 0.15 during training and 0.17 during testing. When examining the R² and NSE values, they were observed to be 0.98 during the training phase and 0.99 during the testing phase.

Figure 10 shows the performance evaluation of the ML models for ET_o prediction using radar charts. When evaluating the R² and NSE values, it is observed that the models M1, M5, M6, M7, M11, M12, M13, and M14 exhibit high prediction accuracy, with the RBFNN algorithm achieving the highest accuracy. The GB algorithm is also successful as a second-degree model. On the other hand, the highest error (RMSE and MAE) was obtained with the ID3 algorithm. When performing a performance analysis based on RMSE values to determine the optimal model for ET_o prediction, the RBFNN model emerges as the model with the lowest error. The second-best ML model is identified as GB, followed by GB and ID3. Additionally, analyzing the MAE values reveals that the algorithm with the lowest error is the RBFNN, while the second-best ML model was RF, followed by GB and ID3. Overall, RMSE and MAE obtained the highest in the case of ID3 ML algorithms.

Input and output values may be visually converted using a Smith chart. A planar grid is constructed for this purpose, allowing the output value to be calculated for a particular input. The Smith Chart is only a plane with circles superimposed on top to determine the results of each independent variable. In Figure 11, the Smith chart further supports the abovementioned results and explains the efficiency (R², NSE) and error (MAE, RMSE) of the applied ML models. The Smith chart is a particular kind of impedance chart that has parallel series of lines. Lines in the first set, called constant resistance lines, display mutual tangency along the right side of the horizontal diameter and form circles. Constant resistance (j) circles are the conventional name for these spherical formations. The values of the resistances represented by the j circles are shown on the horizontal diameter where the circles connect with the line. The circle of positive resistance is shown on the top side of the horizontal line, while the circle of negative resistance is shown on the bottom side of the horizontal line. It is observed in Figure 10 that statistical indices (R², NSE, MAE, and RMSE) of the ML models lie on the top side of the horizontal line, which corresponds to the positive value ranging between 0 and 1. In addition, it was noted that the RBFNN approaches 1 for R² and NSE while MAE and RMSE decline to 0, indicating the superior performance of the RBFNN as compared with other applied ML models.

Figure 12 shows a scatter plot and temporal variation by comparing ET_o estimated with the RBFNN using the M12 input combination (best model) and standardized PMF in the training and testing phases over the studied region. It is clearly observed in Figure 11 that RBNN trailed well with the PMF and could be used in cases of limited climatic input data. In addition, Figure 13 indicates temporal variation in ET_o estimated with the RBFNN and PMF from 1986 to 2016 in the training and testing phases. It can be concluded that the RBFNN algorithms performed well and coincide with the ET_o values of the PMF.

In addition, the MBE captures the average deviations between two datasets. It has the units of the variable. Values near 0 are the best; negative values indicate underestimation and positive values indicate overestimation. Moreover, RMSE combines the spread of individual errors. Figure 8, Figure 9, Figure 10 and Figure 11 indicate that the MBE produced by ML models at all the 17 input climatic combinations found nearly zero, which indicated the superior performance of the chosen ML models. The results also affirmed the use of ML models in cases of limited climatic data because there is no underestimation and overestimation found in the dataset. It can also be confirmed that the dataset used in the current study has no missing values and no abnormality was detected during data analysis.

5. Study Discussion

Within the scope of the presented study, various machine learning models were evaluated to the peak for the estimation of ET_o values. Accordingly, the RBFFNN model had the highest accuracy and ID3 showed the weakest prediction success. It was deduced that the Tmean, RHmean, and Ws input variables had the highest effect on ET_o estimation. Pal and Deswal [57] employed the M5 model tree method to model daily ET_o in the climatic data of Davis station, which California maintains. The inputs for the model were solar radiation, average air temperature, average relative humidity, and average wind speed. The M5 model tree model successfully predicted meteorological data and ET_o values, as evidenced by the results of this research. The outputs of this research support the presented study. The study conducted by Vaz et al. [58] employed machine learning and deep neural networks to construct a model for evapotranspiration, utilizing only a limited number of weather variables, including temperature, humidity, and wind. The presented research coincides with this research, as both establish the Random forest algorithm’s potential to provide satisfactory outputs for ET_o estimation. Also, LSTM-ANN recommends a hybrid approach for ET_o estimation. The estimation of ET_o has been predicted by Wang et al. [59] through the utilization of a combination of time granulation computing techniques and gradient boosting decision tree (GBDT) with Bayesian optimization (BO). Subsequently, GBDT is deployed to anticipate evapotranspiration, while BO determines the optimum hyperparameter values from the pared-down granules. The study’s findings align with current research, indicating that the GB algorithm substantiates the sufficiency of ET prediction. The effectiveness of various types of ML algorithms, including tree-based, neural network-based, multifunction-based algorithms, and a combination of ML and physical models, have been investigated in predicting hydrological variables (ET_o, river discharge, precipitation, monitoring droughts) and their related factors [60,61,62,63,64,65,66,67,68,69,70,71,72,73,74]. Wang et al. [59] analyzed temperature data from several different climatic stations located in Pakistan [60]. The TB model produced outperforming results (R² and NSE = 1.00, MAE and RMSE = 0.26 and 0.37) when an input combination based only on temperatures (T_max and T_min) was used. The study’s outputs support the current study in terms of tree-based algorithms showing high performance in ET_o model estimation. However, ET_o estimation established only with the maximum and minimum temperatures does not overlap in delivering the highest performance.

In the current research, T_mean, RH_mean, and Sh were critical indicators of ET₀ in our study. The study’s findings supported the assertion made by [75] that increased air moisture content causes relative humidity to have greater impacts in wet locations; as a result, when the aridity index increases, air moisture content is constrained, and its effects are less. Temperature and relative humidity were discovered to be the most important predictors of ET_o in a study conducted by Estévez et al. [76]. Eslamian et al. [77] investigated the effect of weather parameters on ET_o estimation in Esfahan province. The study concluded that T_min, Sh, and RH_mean found effective parameters on ET_o estimation in this region. Similarly, it was observed by [45] that climatic variables related to RH_mean significantly influenced the ML modeling of ET_o. Including RH_mean in ML models increased performance by up to 24%. These earlier observations support our finding in the selection of the best input combination.

However, it is recommended to employ ML over empirical and locally calibrated models in cases where climatic data are unavailable, inconsistent, or of poor quality. Calibration of ML models in the training phase is critical to avoid over- or underestimation of ET_o values. ET_o is underrated with more training data, but it is overestimated with less training data. The use of ML models requires a sufficient amount of input data for good calibration. Therefore, in order to test the efficacy of the ML models, the current study evaluated ML models using various set of climatic variables. The data requirements for ET_o estimation using the PMF and ML models are displayed in

Table 4. Data requirements for ET_o estimation using PMF and ML models.

Chosen Method	Climatic Variables								Aerodynamic Factors
	Tmin	Tmax	Tmean	RHmin	RHmax	RHmean	Ws	Sh	Rn, es, ea, emin, emax, Δ, G, and Ɣ
PMF	▀▀	▀▀	▀▀	▀▀	▀▀	▀▀	▀▀	▀▀	▀▀
ML models	X	X	▀▀	X	X	▀▀	▀▀	X	X

. The PMF may be observed in Table 4 to depend on numerous characteristics that are difficult to obtain, especially in the Sindh and Balochistan areas (underdeveloped provinces of Pakistan). As an alternative to the PMF approach, ML models use fewer parameters that yield ET_o-value approaches to the PMF which aid in crop scheduling and irrigation planning. In Table 4, the parameters required for ET_o estimates are denoted by “▀▀”, while “X” denotes that they were not employed in the corresponding method. Sabino and Souza [78] indicated that changes in solar radiation, relative humidity, and wind speed are the main driving forces that impact the ET_o. In addition, RHmean and Ws have higher sensitivity indices during the dry season, which is also affirmed in our study area as it is arid in nature.

6. Conclusions

This study evaluated the performance of various decision tree-based machine learning models for estimating ET_o values at the Jacobabad, Sindh, Pakistan station. It also aims to analyze the effect of various model input combinations on the ET_o prediction. Model performances were evaluated according to Smith chart heatmap, radar chart, and bullet chart results. As a result of the analysis, M2 (RH_mean, Sh), M3 (RH_mean, Sh, Ws), M4 (RH_mean, Ws), M8 (T_max, T_min, RH_mean, Sh), M9 (T_mean, RH_mean, Sh), M15 (T_mean, RH_mean), and M16 (T_mean, Sh) model combinations exhibited lower levels of prediction success compared with the remaining model combinations. When evaluating model performances based on statistical criteria, the most suitable model combination is M12 (T_mean, RH_mean, Ws). In general, satisfactory results have been obtained in the models where WS and T values are used together as inputs for ET_o estimation. The RBFFNN model, which exhibits the most precise estimation results, demonstrates RMSE values of 0.30 during the training phase and 0.22 during the testing phase, according to the findings. In addition, during training and testing, the MAE values for this model are 0.15 and 0.17, respectively. The R² and NSE value analysis revealed that they were 0.98 and 0.99 in the training and testing phases, respectively. The conducted analyses have provided empirical evidence indicating that the ID3 model demonstrates relatively inferior performance in comparison with alternative models.

Limitations, Suggested Improvements, and Future Directions

The suggested ET_o ML models can only be used in the research area and only with the available meteorological data. Therefore, it is required to design comparable or innovative ML models using less or similar meteorological data to examine how well the established ML models function in various places. In addition, the present study’s suggested ET_o model must be calibrated and appropriately trained before being used in other locations. Furthermore, ML does not have physical processes, and the user is just aware of the input and the anticipated output of the model. Therefore, developing an appropriate ML model is complex without understanding functional criteria. Because the dataset was divided at random, over-fitting and under-fitting issues may arise during the ML model’s training and calibration.

The following steps for this research will include developing further ET_o models using EM and ELM that consider a more comprehensive range of climate types (from humid to desert) and, if applicable, the impacts of climate change. Since extensive and trustworthy data on ET_o modeling is necessary for successfully planning, managing, and controlling water resource systems, gap-in-filling strategies must be actively pursued. Regional models are critical because they develop local models in regions with little data. Crop water needs (CWRs) may be used to manage crucial data about ground and surface water resource planning and management, regional water use analysis, water allocation, water consumption, and water rights. Water resource planning for seasonal variation (summer, winter, autumn, and spring) can benefit from estimating ET_o in different months for determining CWRs. While this study focused on arid climates, future research should include other climate types to examine climatic variability in greater depth.