Modelling Efﬁciency of Electric Utilities Using Three Stage Virtual Frontier Data Envelopment Analysis with Variable Selection by Loads Method

: Electric utility regulators and policy makers implement incentive-based regulation to improve electric utilities efﬁciency or to manage the cost of electricity. However, poorly implemented regulation may produce undesired results such as low reliability or poor quality of service. More-over, the competition within the electricity sector is likely to be low because of the high barriers to entry, vertically integrated electric utilities, and high capital requirements. Therefore, benchmarking exercises allow policy makers and regulators to gauge the relative efﬁciency of electric utilities and help them to reward or penalize the electric utilities accordingly. In this study, we examined the variables that signiﬁcantly inﬂuence the efﬁciency of electric utilities and developed an optimum method to measure the efﬁciency of the electric utilities. The results of the efﬁciency measurement were then used to rank the electric utilities. The result of this study indicates that there are 13 variables that signiﬁcantly affect the efﬁciency score of electric utilities and three stage virtual frontier data envelopment analysis (3S-VF-DEA) is the optimum method to measure the efﬁciency of the electric utilities.


Introduction
The electricity generation industry is often regarded as an essential service for a nation since a reliable electricity supply is vital for economic growth. Traditionally, this industry tends to be vertically integrated and has a high barrier of entry for new players. Therefore, policy makers and regulators around the world have implemented various regulations and legislations to regulate the electric utilities and improve the competition in this industry [1]. Vertically integrated electric utilities manage or operate the generation, transmission, and distribution of electricity supply. However, regulators in some countries such as the United States of America [2], United Kingdom [3] and Germany [4] have unbundled vertically integrated utilities into electricity generation, transmission, and distribution businesses. This purpose of this paper is to measure the efficiency of electric utilities and benchmark the electric utilities against their peers. Benchmarking of electric utilities is an under-researched subject, and this study may improve our understanding of this subject. Apart from that, this study also to tries to improve existing benchmarking studies by eliminates bias in variables selection by incorporating variable selection by loads method. Lastly, this study also tries to determine the optimum method to measure the efficiency of electric utilities by exploring methods such as data envelopment analysis (DEA), virtual frontier DEA (VF-DEA), three stage DEA (3S-DEA) and three stage virtual frontier DEA (3S-VF-DEA). Section 1 of this paper briefly introduces regulation in electric industry and the remainder of this paper is organized as follows, Section 2 describes some of the efficiency Similarly, the United Kingdom Electricity Supply Industry (ESI) in England and Wales was under public ownership from 1948 to 1990. It was vertically integrated and was operated by Central Electricity Generating Board (CEGB) during this period. CEGB was restructured and privatized in 1990 by dividing it into four (4) companies such as National Power, PowerGen, National Grid and Nuclear Electric. The successful reform of ESI later became a model for power sector reform for the rest of the world [15].
Likewise, Norway also reformed their electricity market by enacting the Energy Act in 1990. Alongside New Zealand and United Kingdom, Norway was one the first to deregulate and liberalize its electricity sector. The main driver for electricity market deregulation in Norway was an increasing dissatisfaction with the performance of the electricity sector, particularly with regard to an excessive investment in capacity that exceeded the demand [16].

Relationship between Regulation and Efficiency of Electric Utilities
Legislation alone may not be sufficient to improve the efficiency or performance of electric utilities. Therefore, regulators and policy makers have also been implementing incentive-based regulations to encourage the electric utilities to improve their efficiency and deliver expected quality of service to the customer. The regulators are implementing such measures to improve the overall cost competitiveness, increase technical efficiency and strengthen operational performance [17].
Even though regulators and policy makers implement incentive-based regulation to improve the electric utilities efficiency or to manage the cost of electricity, poorly implemented regulation may produce undesired results such as low reliability or poor quality of service. For example, regulations that encourages the electric utilities to lower the cost of electricity may unintentionally encourage the electric utilities to reduce their spending on maintenance that may lead to lower service quality or reliability. Studies have shown that such poorly implemented regulations may have caused major power outage to parts of Canada and United States of America in 2003 [18].
Similarly, the Brazil Electric Authority Agency (ANEEL) enforced a regulation where Brazilian electric distribution companies are compelled to produce zero Economic Value Added (EVA). EVA means that the electric utilities' earnings shall be equal to their capital costs and ANEEL may reduce the electricity tariff if the electric utilities are generating higher than expected profit [19]. Although ANEEL may have good intentions to manage the cost of electricity, however such regulations will reduce the profitability of the electric utilities and will not attract new investors.
Likewise, Pakistan implemented the Water and Power Development Authority (WAPDA) Act in 2008 to meet future energy demands, improve operational performance and increase efficiency. However, this legislation did not improve the efficiency of electric utilities as expected because the legislation did not address underlying issues such as poor collection, technical losses, non-technical losses, undersized grid, low retail price and, inadequate capital investment [17].

Modelling the Efficiency Measurement and Benchmarking
There are various methods or models such as data envelopment analysis (DEA), DEA-Malmquist index, stochastic frontier analysis (SFA) and slack-based measure (SBM) that can be used to measure the efficiency of electric utilities.
DEA is a linear programming model that was introduced by Charnes, Cooper and Rhodes in 1978 and was later improved to account for variable return to scale (VRS) in 1984. DEA is a deterministic method because it measures efficiency as distance from the efficient frontier without considering statistical noise [20].
The Malmquist index was developed by Malmquist (Wu et al., 2017). It is a distance function that uses technical efficiency change (TEC) and technical change (TC) components. The Malmquist index can be calculated from both DEA and SFA methods [21]. Meeusen and Brook developed SFA in 1977 as a parametric method that integrates error terms such as inefficiency and statistical noise. The main advantage of SFA is its ability to measure efficiency while simultaneously considering the presence of statistical noise [22].
Traditional DEA models such as Charnes-Cooper-Rhoder (CCR) and Banker-Charnes-Cooper (BCC) models ignore slack adjustments and cannot distinguish efficient DMUs effectively. Tone (2001) developed a non-radial SBM model based on DEA-CCR model that are more accurate because it can eliminate deviation of efficiency measurement caused by the difference of radial selection. In contrast to CCR or BCC, SBM deals directly input excess and output shortfalls. Tone (2001) also stated that SBM is a product of input and output inefficiencies [23].
However, the selection of efficiency measurement methods and variables are often influenced by quality and availability of data [24]. Therefore, the efficiency measurement methods and variables are being improved continuously and existing efficiency measurement methods may not be perfect for all conditions.

Variables Selection Methodologies
Selection of variables is critical because it can significantly influence electric utilities efficiency measurement and score. The efficiency score may vary significantly if regulators or policy makers do not consider all pertinent variables or if they decide to give higher weightage to certain variables [25].
Moreover, the variables chosen by the policy makers or regulators may not be perfect and may include extraneous variables. Therefore, it is important that the variables are selected correctly because inclusion of irrelevant variables or exclusion of significant variables may lead to inaccurate efficiency score [26]. Additionally, the selection of variables may affect methods such as DEA by altering its frontier and may produce inaccurate efficiency scores [27].
Similarly, studies have shown that accurate selection of variables will eliminate or reduce inclusion irrelevant variables and will reduce the size of efficiency measurement method. The reduction of variables may also produce more accurate efficiency scores [28]. Apart from that, exclusion of significant variables may also produce unexplained gap between estimated and measured efficiency score [29].
Some of the variables that were used in energy sector related efficiency measurement studies are listed in Table 1 below. Although, studies have shown that variables must be selected carefully because it will affect the efficiency score and influence incentivebased regulation efforts [30], but the selection of the variables are often influenced by socio-economic structure of a country and availability or quality of data [12]. There are several variables selection method that can be used to select the significant variables. For example, researchers such as Omrani, Beiragh, and Kaleibari investigated the efficiency of 37 electric distribution companies in Iran using DEA, principal component analysis (PCA) and game theory. Initially, the number of variables was too big compared to the number of decision making units (DMUs) and therefore, the researchers used PCA to reduce the number variables and improve the accuracy of the DEA method [36].
Another method that can be used to select the significant variables is efficiency contribution measure (ECM). ECM method works by comparing two DEA methods; first a DEA method with the test variable and a second DEA method without the test variable. The efficiency scores of both methods will be evaluated using binomial statistical test to determine if the test variable significantly affects the efficiency score. The process is repeated ether via addition or removal of the other variables. Apart from ECM, regression-based test can also be used to select significant variables. In this method, the initial efficiency score will be calculated using known variables. After that, the efficiency is regressed against potential variables to determine if those variables are statistically significant. These steps are repeated for all potential variables until all significant variables are found [27].
Researchers such as Madhanagopal and Chandrasekaran used a genetic algorithm (GA) to reduce the number of variables to measure the efficiency of the Indian banking sector. They used DEA with GA to reduce the number of variables from 13 to 6 [37].
More recently, Fernandez-Palacin, Lopez-Sanchez and Munoz-Marquez used variable selection by loads method to reduce the number of variables in DEA. They used this method to measure and rank research activities carried out by Spanish public universities. Compared to other variables reductions methods, the variable selection by loads method can detect bias and eliminate them. Therefore, this method may prevent inclusion of irrelevant variables in the efficiency calculation and produce a more accurate efficiency score. The authors of this method have created an online application at http://knuth.uca. es/DEA (accessed on 8 June 2021) [26].
Apart from that, researchers have utilized multiplier restrictions such as assurance regional analysis and the cone ratio approach to reduce the number of efficient DMUs and improve the discriminatory factor. However, such multiplier restrictions require prior information such as previous experience to restrict the multiplier in a specific range and may not produce feasible solution without such prior information [38].
The variable selection by loads method (ADEA Selection Method) will be used in this study to measure the efficiency of electric utilities since it is able to reduce the number of variables without bias and improve DEA method's discriminatory factor.

Efficiency Measurement Methods
DEA is one of the earliest efficiency measurements methods, developed by Charnes, Cooper and Rhodes in 1978 and published in the European Journal of Operations Research. This model is known as CCR model and assumes constant return to scale (CRS) assumption. In CRS assumption, production output increases or decreases proportionally to the changes in the inputs.
DEA measures the efficiency of DMUs by creating a non-parametric envelopment frontier over data points and uses it to identify efficiency and inefficient DMUs. Let's assume that we have n firms which are represented as DMUj (j = 1,2, . . . ,n). Each firm produces s outputs yrj (r = 1,2, . . . ,s) and uses m inputs xij (i = 1,2, . . . ,m). Therefore, efficiency of DMU o can be formulated as shown below [39]: subject to This formulation above can be re-expressed in linear programming as shown below: . . ,m, r = 1, . . . ,s Referring to Figure 1 below, assume that a production functions requires two (2) kinds of resources. These resources are labelled as x1 and x2. D1-D5 are efficient DMUs because they are on the production frontier. This formulation above can be re-expressed in linear programming as shown below: Referring to Figure 1 below, assume that a production functions requires two (2) kinds of resources. These resources are labelled as x1 and x2. D1-D5 are efficient DMUs because they are on the production frontier. The BCC model [40] was developed by Banker, Charles and Cooper in 1984. This model revised the CCR model by adopting VRS assumption [41]. In VRS assumption, production output does not increase proportionally to the changes in the inputs. In this study, an input oriented radial DEA-BCC model with VRS was used to calculate the electric utilities' efficiency. The linear programming for the BCC input-oriented model can be formulated as shown below: Ѳ subject to Ѳ ∑ , 1 … The BCC model [40] was developed by Banker, Charles and Cooper in 1984. This model revised the CCR model by adopting VRS assumption [41]. In VRS assumption, production output does not increase proportionally to the changes in the inputs. In this study, an input oriented radial DEA-BCC model with VRS was used to calculate the electric utilities' efficiency. The linear programming for the BCC input-oriented model can be formulated as shown below: Min θ . . ,n Efficiency of the DMUs can also be calculated using input-oriented model or outputoriented model [42]. The output-oriented model shall be used when we have more control over the outputs and input oriented model shall be used when we have more control over the inputs.
Although DEA can measure the efficiency of the DMUs, however DEA will not be able to differentiate or rank the DMUs on the production frontier [43]. In such situations, a virtual frontier can be added to overcome these limitations as shown in Figure 2. The virtual frontier is formed from a set of virtual or dummy DMUs that are created by reducing the input variables by a factor of 0.95 and increasing the output variables by a factor of 1.05 [44]. In this example, V1-V5 will form a new production frontier where D1-D5 will be inefficient, and this will allow DEA to differentiate the efficiency of D1-D5. nology factor significantly influence the carbon efficiency. However, structure factors do not significantly influence the transportation industry's carbon efficiency [44].
The same researchers improved the VF-DEA method and used the virtual frontier benevolent DEA cross efficiency model (VFB-DEA) to measure the energy efficiency of airlines industry from 2008-2012. The collected data from 11 airlines worldwide and concluded that capital efficiency significantly influences the energy efficiency of the airlines [43]. Although VF-DEA method can differentiate efficient DMUs, however this method is not able to eliminate environmental or other random factors. Therefore, researchers have developed Three Stage DEA (3S-DEA) to eliminate these environmental or random factors [30]. The steps for calculating efficiency using 3S-DEA method is as shown below: (1) First stage: Measure efficiency of DMUs using DEA method, (2) Second stage: Using the DEA efficiency score from the previous step, conduct stochastic frontier analysis (SFA) to exclude environmental or random factors and adjust the inputs, (3) Third stage: Measure the efficiency of DMUs using DEA methods and adjusted inputs [45]. Researchers such as Cui and Li have used the so-called virtual frontier DEA (VF-DEA) method to measure the carbon efficiency of transportation industry. They collected data from 15 countries from year 2003-2010 and concluded that management and technology factor significantly influence the carbon efficiency. However, structure factors do not significantly influence the transportation industry's carbon efficiency [44].
The same researchers improved the VF-DEA method and used the virtual frontier benevolent DEA cross efficiency model (VFB-DEA) to measure the energy efficiency of airlines industry from 2008-2012. The collected data from 11 airlines worldwide and concluded that capital efficiency significantly influences the energy efficiency of the airlines [43].
Although VF-DEA method can differentiate efficient DMUs, however this method is not able to eliminate environmental or other random factors. Therefore, researchers have developed Three Stage DEA (3S-DEA) to eliminate these environmental or random factors [30]. The steps for calculating efficiency using 3S-DEA method is as shown below: (1) First stage: Measure efficiency of DMUs using DEA method, (2) Second stage: Using the DEA efficiency score from the previous step, conduct stochastic frontier analysis (SFA) to exclude environmental or random factors and adjust the inputs, (3) Third stage: Measure the efficiency of DMUs using DEA methods and adjusted inputs [45].
In the first stage, the efficiency of the DMUs is measured using DEA-BCC method. However, the efficiency scores are affected by effects such as managerial inefficiencies environmental affects, and statistical noise [46].
In the second stage, regression analysis is conducted with SFA to adjust these effects. The total input slack is identified as S ni and it is the dependent variable in the SFA regression model. The independent variables in the SFA regression model are the element s of the t observable environmental variables, Z i = Z 1i , Z 2i , . . . ,Z ti . The N separate regression model is expressed as follows: S ni = f n (Z i ; β n ) + (v ni + µ ni ), n = 1, . . . ,N; i = 1, . . . ,I Referring to the model above, N is number of firms and I is the number of inputs, f n (Z i ; β n ) is the deterministic feasible slack frontier, β n is the environmental vector for estimation and, (v ni + µ ni ) is the composite error that includes statistical noise and managerial inefficiencies. The input variables of each DMU are then adjusted to SFA regression model results. The adjusted model is expressed as follows: In this model, X A ni is the adjusted input value from the observed input quantity X ni . In the third stage, the efficiency of the DMUs is re-measured using the adjusted inputs and DEA-BCC method.
Researchers such as Shyu and Chiang used the 3S-DEA method to measure efficiency of 123 bank branches in Taiwan and discovered that environmental factors or statistical noise significantly affects the efficiency of the bank branches [47].
Likewise, Iparraguirre and Ma measured the efficiency of senior people's social care in 148 English councils using 3S-DEA method. They found that the efficiency of the social care services is very high but the efficiency is negatively influenced by eligibility criteria and eligibility assessment costs variables [48].
Like the DEA method, 3S-DEA method cannot discriminate or differentiate between efficient DMUs. Therefore, researchers such as Cuia and Li have improved 3S-DEA by adding a virtual frontier. The improved method is known as three stage virtual frontier DEA (3S-VF-DEA). The steps for calculating the efficiency of DMUs are like VF-DEA as shown below:

•
First stage: Measure efficiency of DMUs using VF-DEA method, • Second stage: Using the DEA efficiency score from the previous step, conduct stochastic frontier analysis (SFA) to eliminate environmental or random factors and adjust the inputs, • Third stage: Measure the efficiency of DMUs using VF-DEA methods and adjusted inputs [44].
Cui and Li used 3S-VF-DEA method to examine the efficiency of transportation energy in 30 Provisional Regions in China. They found that the efficiency of transportation energy is significantly affected by management measures and transport structure.
Some of the recent efficiency measurement methods that have been employed by the researchers are shown in Table 2 below. The table shows that researchers have employed improved DEA methods such as value-based DEA, DEA with game theory and PCA, two stage virtual frontier dynamic DEA, and DEA cross efficiency methods. To measure the efficiency of electric distribution in terms of cost reduction and also with respect of overall regulatory framework.

DEA Window Analysis
DEA window analysis calculates the average efficiency of CCR and BCC models and it can be used to measure efficiency of DMUs over time [21]. In window analysis, the same DMU in a different period is treated as a different DMU and a moving average method is used to choose different reference set to determine the relative efficiency of each DMU. This method is beneficial when there are limited number of DMUs because this method can increase the number of DMUs and improve discriminatory power [55].
Consider a set of N (n = 1,2, . . . N) DMUs in T (t = 1,2, . . . T) period of time and each DMU has r kind of inputs and s kind of outputs. The input vector (X t n ) and output vector (Y t n ) are represented as follows: If the window starts at the time point of k (1 ≤ k ≤ T), and the window width is w (1 ≤ w ≤ T-k), then the input X kw and output Y kw are represented as follows: Replacing the above inputs and outputs in the DEA models will generate the results of DEA window analysis.

Benchmarking in Electricity Industry
Benchmarking is an exercise that includes measuring the efficiency of electric utilities and ranks them based on the measured efficiency scores. The competition within the electricity sector tends to be low because of the high barriers to entry, vertical integration, and high capital requirements. Therefore, the benchmarking exercise will allow the policy makers and regulators to gauge the relative efficiency of electric utilities and ensure that the electric utilities are operating in an efficient manner [56].
Other than that, policy makers and regulators can implement incentive-based revenue and use benchmarking exercise to reward efficient utilities and penalize inefficient electric utilities. Such exercises will encourage the electric utilities to continuously improve themselves and provide the best quality of service to the consumer [25].
There are several studies that evaluates the efficiency of electric utilities and benchmarks them against their peers. Researchers such as Giannakis, Jamasbb, and Pollitt measured the efficiency of electric utilities in United Kingdom using hybrid DEA and Malmquist index using information from year 1991-1992 and 1998-1999. Their study revealed that there is no significant relationship between high quality service and electric utilities that are cost efficient. They also concluded that benchmarking exercise should include quality of service variables since cost only approaches may not produce accurate results [1].
Apart from measuring and comparing the efficiency of electric utilities against their peers, regulators and policy makers can also use benchmarking exercise as a tool to determine electricity rates or tariffs. However, as highlighted by the researchers above, the efficiency measurement and benchmarking must be accurate so that it will allow the policy makers and regulators to determine an accurate electricity tariff [57].
For example, the Finnish Energy Market Authority (EMV) is one the pioneers that implemented benchmark regulation. Initially EMV employed DEA as part of its regulatory model in 2005 and adopted SFA in 2008. However, EMV revised its regulatory model and adopted stochastic semi-nonparametric envelopment of data (StoNED) after its DEA and SFA method was criticized by electricity distribution utilities and other energy industry players [58].
Similarly, Norway implemented its Energy Act in 1991 that laid the foundation for its electricity sector deregulation. However, in Norway, the electricity market was deregulated but not privatized and largely remains under public ownership. The electricity sector in Norway is regulated by Norwegian Water Resources and Energy Directorate (NVE) and this regulator has mainly used DEA for its efficiency measurement [59].
Researchers also have used DEA methods to measure the efficiency and benchmark thermal power plants in India. The study shows that stated owned power plants are less efficiency compared to privately owned power plants. Apart from that, the study also concludes that smaller power plants are less efficient compared to medium and large thermal power plants [60].
Some of other benchmarking exercise or studies that were carried out are as listed in Table 3 below.

Methodology
The data analysis that was employed in this study is illustrated in Figure 3 below. The data analysis begins with data collection from secondary sources that are available online and free to access without any restrictions.
After that, the financial information was converted to local currency and adjusted to the value at year 2018. Next, the data was normalized and DMUs with extreme outliers were removed from the dataset.
Afterwards, the number of input and output variables were reduced using variable selection by loads method. The efficiency of all DMUs were then calculated using DEA, VF-DEA, 3S-DEA and 3S-VF-DEA methods and the efficiency scores were used to determine the discriminatory power. Next, a Monte Carlo simulation was performed to determine the mean absolute deviation (MAD) of the methods and the method with the highest discriminatory factor and consistent MAD score was selected as the optimum method.
Subsequently, a window analysis was performed to calculate average efficiency scores of the electric utilities. The electric utilities were then benchmarked or ranked using the average score of the optimum method to determine their efficiency compared to their peers.

Data Collection
The data for this study was collected from various publicly available secondary sources such annual reports, statistical databases, stock exchange publications and regulatory publications as shown in Table 4 below. Secondary data from year 2000 to 2018 for 21 electric utilities from countries such as United States of America, Ireland, Germany, Indonesia, Thailand, and Malaysia were collected for this study. The data were collected from the secondary sources above and each of the electric utility were identified as Utility-1 to Utility-21.

Data Collection
The data for this study was collected from various publicly available secondary sources such annual reports, statistical databases, stock exchange publications and regulatory publications as shown in Table 4 below.  Secondary data from 19 other utilities from the Philippines, Australia, Singapore, Thailand, China, United Kingdom, Austria, Portugal, United States of America, Vietnam, Cambodia, Laos, European Union, France, Spain, and Korea we collected during this study. However, those data were excluded from the analysis due data quality issues such as missing data, incomplete data, and language issues.

Variable Reduction
Initially 10 input variables and 15 output variables as listed in Table 5 were investigated for this study. However, researchers such as Nataraja and Johnson have stated that higher number of variables will diminish DEA method's accuracy and discriminatory power [27]. Other researchers also stated that the total number of input and output variables should be less than half of the number of DMUs [67].
Therefore, the variables in Table 5 will be reduced using variable selection by loads method. This method reduces the number of variables by following these steps: (1) Calculate efficiencies of DMUs by using input oriented DMU and with all potential variables, (2) Calculate the loads of the variables, (3) Eliminate the variable with the smallest load if the load is below the specified threshold, (4) Repeat step (1) until all variables' load are above threshold.
An online application to calculate the above steps are available at http://knuth.uca.es /DEA (accessed on 8 June 2021) and this application was used in this study to reduce the number of variables.

Benchmarking
In this step, the efficiency of the electric utilities was calculated using the optimum method with variables that have been reduced by using variable selection by loads method. The average efficiency of the electric utilities was calculated using window analysis method for the period of 2000-2018. After that, an electric utility was selected and benchmarked against its peers for the same period.

Findings
For this study, a total of 399 observations with 10 input variables and 15 output variables were collected from 21 electric utilities. The electric utilities are from various countries such as United States of America, Malaysia, Thailand, Indonesia, Germany, and Ireland.
Secondary data from other electric utilities from other countries were explored too but were excluded from this study due to inconsistent or incomplete data. The collected secondary data also revealed that emission related data such as SO 2 and CO 2 emission are not available in older datasets.
The secondary data was collected over a period of 19 years from 2000 to 2018 because investments in electric utilities are generally long-term investments and majority of electric utilities' assets have long lifespans. For example, coal power plants can operate up to 50 years [68], whereas distribution transformers have a lifespan of 75 years [69].
A dataset that with a short time frame may not reveal the actual effectiveness of the managerial decisions related to the investments and may produce inaccurate efficiency scores. Therefore, a dataset with a longer time frame may reveal the effectiveness of the managerial decision and may be more accurate.
The extreme outliers were identified and removed from this dataset using boxplot via IBM SPSS Statistics version 27. A total of 46 outliers were removed from the original dataset and the remaining 353 observations were used for in the variable reduction step.

Variable Reduction
The descriptive analysis for the variables from Table 5 are shown in Table 6 below. The variables from Table 5 were reduced using variable selection by load method. Looking at the results in Table 7, there is a significant variable reduction between step number 8 and 9. The load at step is 0.5007 which is better than the recommended value of 0.6. Therefore, the selected variables are as listed in step number 9 and shown in Table 8 below. The input and output variables include various dimensions such as financial, asset, service area, total manpower, electricity interruptions and share earnings.

Method Selection
Using the selected variables, the efficiency of electric utilities was calculated using DEA, VF-DEA, 3S-DEA, and 3S-VF-DEA methods. The discriminatory power of the methods or the ability of the methods to differentiate the efficiency scores of the DMUs is shown in the Table 9 below.
Based on the Table 9, DEA was not able to differentiate the efficiency scores of 65.43% of the DMUs. Likewise, VF-DEA, 3S-DEA and 3S-VF-DEA were not able differentiate the efficiency score of 38.76%, 65.48% and 26.35% of the DMUs, respectively.
Therefore, 3S-VF-DEA method with variable selection by loads method has the highest discriminatory power because it can differentiate the efficiency score of 73.65% of the DMUs.
A Monte Carlo simulation was also carried out to investigate the performance of the methods under different conditions. The Monte Carlo simulations were developed and simulated using R language and R Studio version 1.3.1093. The Monte Carlo simulation is shown in the Figure 4 below and the simulation is repeated for 50,100,150,200,250,350 and 400 DMU using random datasets that has a mean of 0.5 and standard deviation of 1.96.  MAD scores for the Monte Carlo simulation is shown in Table 10 below. The MAD scores indicate that DEA, VF-DEA, 3S-DEA, and 3S-VF-DEA methods were consistent without any significant deviations.    Table 10 below. The MAD scores indicate that DEA, VF-DEA, 3S-DEA, and 3S-VF-DEA methods were consistent without any significant deviations.
The 3S-VF-DEA has the highest discriminatory power followed by VF-DEA. However, 3S-VF-DEA was able to eliminate statistical noise and therefore was able to produce a more accurate efficiency scores compared to VF-DEA. Based on the findings above, the 3S-VF-DEA method was selected as the optimum method for this study.

Benchmarking
The average score for 21 electric utilities from year 2000-2018 was calculated using window analysis method. A total of 10 windows was analyzed for this study as shown in Table 11. This first window is from 2000 to 2012, whereas the last window is from 2009-2018. The difference of average score between DEA and 3S-DEA is not significant because the statistical noise is small. Similarly, the difference of average score between VF-DEA and 3S-VF-DEA is not significant for the same reason.
Referring to Table 12 and 3S-VF-DEA method, Utility-6 was the most efficient utility with a score of 0.949994 and the least efficient electric utility was Utility-16 with an efficiency score of 0.855454.
For example, the efficiency of Utility-4 s efficiency compared to its peers is shown in Figure 5 below. This benchmarking example indicates that the efficiency of Utility-4 remained higher than the average score of its peers from year 2000 to 2018.  For example, the efficiency of Utility-4′s efficiency compared to its peers is shown in Figure 5 below. This benchmarking example indicates that the efficiency of Utility-4 remained higher than the average score of its peers from year 2000 to 2018.

Conclusions
This study was able to determine the variables that significantly influence the efficiency of electric utilities, establish the optimum method to measure the efficiency of electric utilities and benchmark the electric utilities data using window analysis method.
The study indicates that there are 13 variables that significantly affect the efficiency score of electric utilities such as listed below. These variables encompass various dimensions such as financial, asset, service area, total manpower, electricity interruptions and share earnings:

•
Operations cost over total asset • Operations cost over revenue The study also concludes that 3S-VF-DEA is the optimum method to measure the efficiency of the electric utilities because it has the highest discriminatory power, consistent MAD scores and was able to eliminate statistical noise.
Lastly, the study also concludes that Utility-6 was the most efficient electric utility for the period of 2010-2018 and Utility-16 was the least efficient electric utility for the same period.

Methodological and Practical Contributions
The main objective of this study was to measure the efficiency of electric utilities and benchmark the electric utilities against their peers. Since benchmarking of electric utilities is under-researched, this study was able to improve our understanding of this subject by identifying the significant variables that affect the efficiency of electric utilities and determining the optimum method to measure the efficiency of those utilities.
This study also improves the current electric utilities' efficiency measurement and benchmarking methods by incorporating the variable selection by loads method that was able to eliminate bias in the selection of variables.
Apart from that, this study also enables regulators and policy makers to use the method to predict or simulate electric utilities' efficiency by adjusting the expected or predicted input and output variables. Such simulation exercises will allow the regulators and policy makers to focus their efforts on variables that will significantly impact the efficiency of the electric utilities.
Similarly, the electric utilities can use the method to measure and predict their own efficiency by tweaking the input and output variables. The outcome of the simulations can be used by the electric utilities to realign their initiatives and activities to achieve better efficiency scores.

Limitations of the Study and Future Research
The time taken to gather secondary data for this study was longer than expected because the data was cross-checked with other sources to ensure its accuracy. Operational data for variables such as maximum demand (MW), generation capacity (MW), distribution network length (km) and number of distribution substations were excluded from the analysis due data quality issues such as inconsistent data, missing data, incomplete data, and language issues. Other than that, some data related to CO 2 and SO 2 emission was not available in datasets older than year 2005. Therefore, such variables were excluded from this study.