Quantifying Electricity Supply Resilience of Countries with Robust Efficiency Analysis

The interest in studying energy systems’ resilience is increasing due to a rising awareness of the importance of having a secure energy supply. This growing trend is a result of a series of recent disruptions, among others also affecting electricity systems. Therefore, it is of crucial importance for policymakers to determine whether their country has a resilient electricity supply. Starting from a set of 12 indicators, this paper uses data envelopment analysis (DEA) to comprehensively evaluate the electricity supply resilience of 140 countries worldwide. Two DEA models are applied: (1) the original ratio-based Charnes, Cooper, and Rhodes (CCR) model and (2) a novel hybrid framework for robust efficiency analysis incorporating linear programming and Monte Carlo simulations. Results show that the CCR model deems 31 countries as efficient and hence lacks the capability to differentiate them. Furthermore, the CCR model considers only the best weight vectors for each country, which are not necessarily representative of the overall performance of the countries. The robustness analysis explores these limitations and identifies South Korea, Singapore and Canada as the most resilient countries. Finally, country analyses are conducted, where Singapore’s and Japan’s performances and improvement potentials are discussed.


Introduction
Electricity is a crucial commodity to foster the economic development and well-being of a country [1]. Governments are increasingly aware of the need to improve the energy efficiency of electricity production (i.e., decrease the total amount of energy required to produce the desired quantity of electricity), as it leads to better supply security and reduced greenhouse gas emissions. Even though recently the global average efficiency slowly improved [1], major electricity supply disruptions still happen (e.g., the 2012 India blackout [2] or the 2015 Turkey blackout [3]). The financial, social and environmental consequences of such disruptions can cause great damage to the economy of a country, its government and citizens [4,5]. Resilience aims at minimizing the impact of these adverse consequences by defining pre-and post-event strategies, making outages less likely or smaller in extent [6,7].
A resilient electricity supply is fundamental to guarantee a well-functioning modern society [8].
In this regard, one of the key interests of policymakers is to assess how their country performs compared to others. This kind of international comparison supports them in identifying improvement potentials and quantifying achievements or progress towards predefined objectives and targets [9]. Many international country performance assessments are indicator-based [10], because indicators are suitable to model multi-dimensional problems [11]. Countries are commonly ranked based on the indicators only or on aggregated measures, sometimes also called indices or composite indicators, that combine the individual indicators [12]. Within the energy sector, a wide range of operational research methods are used to build such indices [13]. Examples include the analytical hierarchy process (AHP) [14,15], technique for order preference by similarity to ideal solution (TOPSIS) [16], outranking methods such as the preference ranking organization method for enrichment evaluation (PROMETHEE) [17] or elimination et choix traduisant la réalité (ELECTRE) [18], weighted averages [19], multi-attribute value/utility theory (MAVT/MAUT) [20] and data envelopment analysis (DEA) [21].
In the context of electricity supply resilience, comparative country evaluations and rankings are missing [22]. Hence, building upon the framework proposed by   [23] and Gasser et al. (2020) [24] to define a set of 12 indicators that cover resilience holistically, this paper assesses the electricity supply resilience of 140 countries following a security of supply perspective. Due to the fact that the indicators have positive and negative preference orders, and the requirement to not involve preferences from decision-makers, DEA, a family of non-parametric methods to derive efficiencies of decision-making units (DMUs) [25], is particularly suitable to rank the countries. In fact, for DEA, the indicator weights are endogenously determined (directly calculated from the performance matrix itself) [26]. The final scores of the DMUs are commonly called efficiencies, as they represent the ratio of the weighted output indicators' performances to the weighted input indicators performances. Hence, a DMU that is deemed efficient is also resilient, as the indicator set represents electricity supply resilience. DEA efficiency and resilience have thus equal meanings in this context and the terms can be used interchangeably.
The DEA methodology hereby developed allows the following important questions to be answered, which are relevant to (inter-) governmental agencies as well as research institutions:

1.
What are the best performing (i.e. most resilient) countries and what are the reasons for this achievement (see Section 4.1)? 2.
Why are some countries inefficient, how can they improve their scores and which are their benchmarks (see Section 4.2)? 3.
How robust is the performance of the countries (see Section 4.3)? 4.
What is the univocal ranking of the countries (see Section 4.4)? 5.
How well does a country perform in comparison to another one (see Section 4.5)? 6.
How does the performance of countries vary according to changes in selected indicators (see Section 4.6)?
The current paper is organized as follows. In Section 2, a detailed literature review about energy-related country comparisons using DEA is provided, which leads to the formulation of the research gaps. Section 3 describes the case study and the methodology. The latter includes the original ratio-based Charnes, Cooper, and Rhodes (CCR) DEA model [27] and a hybrid framework for robust efficiency analysis incorporating linear programming and Monte Carlo simulations [28]. The CCR model was considered because it is the most commonly used DEA model. The hybrid framework explores the limitations of the CCR model by performing a robustness assessment through selecting random weight vectors obtained via a Hit-And-Run algorithm. To the authors' best knowledge, the present study represents the first application of such an analysis to a country ranking. In Section 4, comparative results answering the research questions are presented and discussed. Furthermore, improvement potentials for Singapore and Japan are analyzed. Section 5 provides the main conclusions of the study.  2 and NO x air-pollutants by looking at the evolution of environmental performance over time [46]. They found that the environmental efficiency of European countries had improved over the period 1993-2010. The same three air-pollutants were also analyzed by Camarero et al. (2013) and they reached similar conclusions, with the exception that the eco-efficiency of NO x emissions did not improve [43]. Regarding energy efficiency improvements, Wang et al. (2019) compared the CO 2 emissions in relation to GDP growth from 25 countries and found that India and China are the two worst countries in terms of energy efficiency [54]. A similar study focusing on energy recovery from waste for European Union (EU) member states was produced by Halkos and Petrou (2019) [47]. Furthermore, Robaina-Alves et al. (2015) derived the efficiencies of European countries based on the maximization of the ratio between the GDP (desired output) and GHG emissions (undesired output) [52]. Their key finding was that since the ratification of the Kyoto Protocol, countries have taken steps to reduce emissions and this is reflected in the evolution of the eco-efficiency level of some countries.
Several other publications address energy-efficiency issues in general. Cui et al. (2014) found that energy efficiency is mostly driven by investments into energy technologies research and tax exemptions for technology companies [45]. Based on the analysis of 28 European countries, Hsieh et al. (2019) recommend that "the EU's strategy for environmental energy improvement should be to pay attention to the benefits of renewable energy utilization, reducing GHG emissions, and enhancing the development of renewable energy utilization to help achieve the goal of lower GHG emissions" [48]. Apergis et al. (2015) show that capital-intensive countries are more energy efficient than labor-intensive ones [34]. Zhang et al. (2011) compare 23 developing countries according to their total-factor energy efficiency, which is defined as the ratio between the targeted energy input and the actual energy input [56]. Similarly, Chang (2014) studies the difference between the targeted and actual energy intensities of 27 EU member countries in order to make conclusions about the potentials for improvement [44]. With a data set of 57 countries, Liou and Wu (2011) found that economic development is interrelated with energy use efficiency and CO 2 emission control [50]. Furthermore, Zhou et al. (2014) used the DEA to rank 30 administrative regions of China according to the energy efficiency of their transport sector [58]. Results show that the Eastern area generally performs better than the Central and Western areas. Song et al. (2013) found that the economies of Brazil, Russia, India, China and South Africa (BRICS) have low energy efficiencies but the trend is increasing quickly [53]. Cai et al. (2019) quantify the carbon emissions of 280 Chinese cities to find that only nine of them are efficient [42]. Results show that coastal regions are performing better than central and western regions. Finally,  develop novel energy-efficiency measures that seem to handle undesirable outputs better and are more effective at identifying inefficient production behaviour [60]. Their data set consists of 32 countries.
Economic efficiency has also been analyzed as the capacity of an economy to produce higher GDP for a given total energy input. In particular, Bampatsou et al. (2013) study the effect of different energy mixes and find that adding nuclear energy into a country's energy mix affects negatively its economic efficiency, due to fewer efforts invested in energy saving and conservation [41].
In summary, as seen in Table 1, most of these studies deal with energy or eco-efficiency. Out of the 25 studies analyzed (including the present one), 15 of them (60%) use the same set of inputs: labor force, capital stock and energy consumption [34,39,40,42,44,[47][48][49][50][52][53][54]56,57,59]. Furthermore, out of these 15 studies, four consider only the GDP as an output [44,49,53,56], while the rest consider the GDP as a desirable output and GHG emissions as an undesirable output [34,39,40,42,47,48,50,52,54,57,59]. Within DEA country comparisons, efficiency is, therefore, usually measured as a minimization of the labor force, capital stock, and energy consumption (inputs) in order to maximize the GDP (desirable output) and minimize the GHG emissions (undesirable output). Further studies use only a subset of these indicators, such as only the energy consumption as an input and the GDP as an output [41], or only the GHG emissions as an input and the GDP as an output [43,46]. Overall, only five studies do not directly use the GDP as an output [37,38,45,55,58]. Three studies differentiate between fossil fuel and non-fossil fuel energy consumption [38,51,55]. The fossil fuel energy consumption would be classified as an input (hence to be minimized), while the non-fossil fuel or renewable sources are classified as outputs (hence to be maximized).
Finally, to the authors' best knowledge, no study using DEA to comprehensively rank countries with regard to their electricity supply resilience performance has been published. This implies that a broader set of indicators needs to be considered, including grid reliability, accident risks, diversity of generation and availability of production technologies. Furthermore, the present study uses DEA models to assess ranking robustness, and to explore scenario-based improvement strategies. Therefore, the research gaps filled by the present study are: 1.
The use of DEA models to develop rankings that represent the electricity supply resilience of countries.

2.
The development of novel DEA algorithms to better understand why some countries are efficient and others are not. These new models are applied for the first time in a real-life case study. 3.
The examination of ranking stability by means of robustness analysis. 4.
The study of country-specific improvement strategies from an optimization point of view.

Case Study Description and Methodology
In this section, first the general scope of the case study and the indicator set selection process are described. Second, the quantification of the data and preparation for DEA are presented. Finally, the DEA concept, its notation and formulas are explained.

Indicator Set Selection, Quantification and Data Set Preparation
The first step was to conduct a literature review in order to identify relevant indicators. As shown by   [23], most of the indicators used in the security of electricity supply studies are related to resilience too. Hence, the abundance of security of supply studies is a promising starting point (e.g., [10,[61][62][63][64][65]). On top of these, further ones related to resilience were considered as well (e.g., [8,66]). This resulted in an extensive list of resilience-related indicators. Subsequently, all of these were assessed according to four quality criteria [67]: (1) their relevance to resilience, (2) the credibility of the data source, (3) the availability of the data and (4) the comparability of the data between countries. The more countries in the list, the more likely data becomes unavailable or incomparable. Thus, as the present case study is about the electricity sector, the limiting data was the electricity production by fuel type. The most comprehensive data for this comes from the International Energy Agency (IEA) and is available for 140 countries worldwide [68]. Therefore, the final set of 140 countries consists of 12 indicators fulfilling the four assessment criteria (see Table 2) [24,69]. The indicators cover resilience holistically, i.e., both the pre-and post-event phases, and represent, among others, the quality, reliability and interconnectivity of the electricity system, the generation diversity, the fuel supply security and self-sufficiency, the available financial resources, the equivalent availability factor (EAF), the average outage times and geopolitical factors such as corruption, government effectiveness and political stability. Overall, this study covers more than 96% of the world's population and 99.6% of the world's electricity consumption.
The data come from a variety of reliable, credible and widely recognized sources, ranging from governmental agencies to international organizations and private companies [68,[70][71][72][73][74][75]. Furthermore, the values of each indicator originate from a unique source, making it homogeneous and comparable. However, there are some missing values, which were inserted as the mean of the other values in order not to distort the data [76]. Inserting the mean of the other values is one of the three most standard techniques employed for dealing with data incompleteness in different scientific disciplines [77]. The other two procedures consist in (1) excluding the incomplete cases and (2) replacing the unknown value with the entire range of all possible values on a given indicator (input or output). These two other procedures were neglected for the following reasons. First, the countries with missing performances were not excluded from the analysis as the data was not available only for a limited subset of inputs or outputs, whereas for the vast majority of countries it has been reliably collected. In the context of DEA, the practical usefulness and credibility of the results increase with the greater number of DMUs. Considering that we have 140 countries for only 12 indicators, the ratio country to indicators is very high [32,78]. This increases the discriminatory power of DEA, ultimately resulting in less bias. In addition, by using the novel hybrid framework for robust efficiency analysis (i.e., a Hit-And-Run Monte Carlo simulation), subjectivity is again minimized. Second, analyzing the entire range of admissible values, it would have markedly deteriorated the robustness of the results. For example, such a hypothetical country would be allowed to attain the best and the worst performances on a given indicator. Due to the application of a ratio-based efficiency model, such a unit would attain the extreme comprehensive performances (e.g., the first and the last ranks depending on the precise performance adopted in a given scenario) only because of treating the missing values in this particular way.
The next step in the preparation of the data for further analysis is the normalization. In fact, the algorithm used in this research is a weighted sum over another weighted sum (called the ratio-based efficiency measure, see formula 1), which makes normalization a necessary step to render the different measurement units comparable [79]. The normalization method adopted in the present case study is the target one, which consists in dividing all the indicators by their maximum value, because it conserves the ratios. Hence, each indicator has a maximum value of 1 and a minimum value in the range [0, 1]. Additionally, the ratios are conserved (see Table S2 in the electronic supplementary information (ESI)).
Finally, DEA requires the indicators to be classified into either being inputs or outputs (see formula 1). As the present case study represents a general benchmarking problem, where DEA is employed for decision-making, the inputs are the indicators with a negative preference order (i.e., to be minimized) and the outputs are the ones with a positive preference order (i.e., to be maximized) [78]. Hence, the classification of the indicators is straightforward and univocal. This results in a total of 4 inputs (i 1 , i 2 , i 6 and i 11 ) and 8 outputs (i 3 , i 4 , i 5 , i 7 , i 8 , i 9 , i 10 and i 12 ) (see Table 2).

Ratio-Based Efficiency Analysis with the Charnes, Cooper, and Rhodes (CCR) Model
DEA is a method for assessing efficiencies of DMUs, which in the present case study are countries. Considering a set of DMUs (D = {DMU 1 , . . . , DMU K }, where K is the number of DMUs), the efficiency of DMU o ∈ D is calculated as the ratio between virtual output and virtual input, which are quantified as weighted sums of individual outputs and inputs, respectively [27]. The equation is: where: In a standard DEA setting, the aim is to divide the DMUs into efficient and inefficient ones. For this purpose, one has to find for each DMU o ∈ D a weight vector that maximizes its efficiency score. Hence, the ratio-based efficiency analysis with the CCR model consists in solving the following primal optimization problem [27]: By definition, the DMUs with efficiency score E * o equal to 1 are considered as efficient. The rest of the DMUs are inefficient (efficiency scores between 0 and 1 exclusive), because other DMUs or their conical combination achieve higher scores under the same conditions. Note that E * o indicates a multiplier that should be applied to all inputs x mo , m ∈ IN so that DMU o ∈ D becomes efficient (e.g., in case E * o = 0.8, DMU o would become efficient by decreasing its inputs by 20%). Moreover, DEA allows to identify benchmarks to be followed and improvement strategies for the inefficient DMUs. These can be determined by solving the following dual optimization problem: On the one hand, for an efficient DMU o ∈ D, θ o = 1 and λ k=o = 1. On the other hand, for an inefficient DMU o ∈ D, all DMUs with λ k > 0 are contained in the reference set of DMU o and can be used for constructing a hypothetical reference unit with greater or equal outputs and lower inputs than DMU o . The differences between the inputs of such a reference unit and DMU o indicate the improvements of inputs that are expected from DMU o for attaining the efficiency. Overall, the CCR model allows tackling research questions 1 and 2.
The aforementioned analysis represents an input-oriented perspective. It derives the required reduction of inputs, if any, that would ensure efficiency (i.e., the best ratio between the virtual outputs and inputs for at least one feasible vector of weights associated with these factors), assuming that the outputs of a given DMU remain unchanged. Note that in DEA, it is also possible to conduct an output-oriented analysis, hence finding the improvements of outputs needed for reaching the efficiency while holding the current amount of inputs (for details, see [27]).

In-Depth Analysis of Status of Efficiency
Explanation of the efficiency status requires construction of arguments, which can be used to justify its validity and logic. In this section, the task of generating explanations of the outcomes of DEA in view of the following procedures is considered: • in case DMU o ∈ D is efficient, identification of the minimal subsets of indicators that make it efficient (such minimal subsets of inputs and outputs are called efficiency reducts); • in case DMU o ∈ D is inefficient, identification of the smallest subsets of other DMUs that underlie its inefficiency (such minimal subsets of DMUs are denoted as efficiency constructs).
It is to be noted that the efficiency reducts and constructs are new methodological developments, as explained in the following sentences. On the one hand, to determine all efficiency reducts for some efficient DMU o ∈ D, an additive method is implemented (see Algorithm 1). It consists of a progressive verification if DMU o is efficient when using different subsets of inputs IN and outputs OUT, while starting with the smallest ones, and eliminating from further consideration the proper supersets (a superset is a set that includes another set. For example, i 1 , i 2 and i 3 is a superset of i 1 and i 3 , and i 2 and i 3 .) of these subsets of indicators that already guaranteed the efficiency [80]. For each efficient DMU o there exists at least one efficiency reduct (in the worst-case scenario, it contains all inputs and outputs). IO = all subsets containing at least one input from IN and at least one output from OUT ordered with respective to the increasing cardinality 2: for each IO k ∈ IO do 3: Solve equation (2) for DMU o ∈ D with inputs and outputs reduced to IO k to derive an optimal solution Remove all supersets of IO k from IO 7: end if 8: end for On the other hand, to identify an efficiency construct for some inefficient DMU o ∈ D, the aim is to find a subset of other DMUs that once removed from the analysis would make DMU o efficient. This can be attained by solving the following mixed-integer linear programming (MILP) problem: where C is a large positive constant. If b k = 1, DMU k needs to be eliminated to make DMU o efficient. Hence, the optimal solution of the above MILP (denoted with *; e.g., f * w ) indicates one of the efficiency constructs IC w = DMU k ∈ D : b * k = 1 . It is possible to identify other constructs by adding the constraints that forbid finding again the solutions found in the previous iterations (w, w − 1, . . . , 1):

Robust Efficiency Analysis
The standard ratio-based efficiency analysis derives the efficiency scores from the best-case scenario for each DMU. As a result, such scores may not be representative, because they may only be achieved for a very limited number of weight vector combinations, which-in addition-are different for each DMU. Therefore, some DMUs may be considered as efficient even though they do not perform particularly well in general. Moreover, the subsets of efficient and inefficient DMUs can be large, and standard DEA methods offer poor arguments to discriminate within these subsets [78]. For this purpose, a variety of measures that reflect how DMUs perform across all feasible vectors of input/output weights are accounted for [28]. These results refer to three different perspectives: cardinal ratings (efficiency scores; answers research question 3), pairwise one-on-one comparisons (preference relations; answers research question 4), and ordinal comparisons of all DMUs (efficiency ranks; answers research question 5). Specifically, linear programming (LP) to derive the following exact outcomes is used: • maximal E * o and minimal E o, * efficiency scores for DMU o ∈ D attained in the set of all feasible input/output weights (note that E * o corresponds to the score derived from the standard analysis); • a necessary efficiency preference relation N E , which holds for a pair (DMU o , DMU k ) ∈ D × D in case DMU o attains efficiency at least as good as DMU k for all feasible input/output weights; • the best R * o and the worst R o, * efficiency ranks for DMU o ∈ D, which are derived from the analysis of, respectively, minimal and maximal subsets of DMUs that attain better efficiency than DMU o for some feasible input/output weights.
The measures convey useful knowledge on the performances of DMUs in the most and least advantageous scenarios as well as for all feasible weight vectors combinations. Nonetheless, the difference between extreme outcomes can, in general, be quite large, whereas the necessary relation can leave many DMUs incomparable. For this reason, a stochastic efficiency analysis based on the Monte Carlo (MC) simulation to estimate the probability of different outcomes is applied [82]. Hence, a large representative set of feasible weight vectors (v, u) S (with W being the number of samples) is derived, using a dedicated algorithm such as Hit-And-Run [83]. Hit-And-Run samplers have been proven to perform well for problems of larger sizes [83]. Note that each vector from the feasible weight space is assigned equal chances to be hit (uniform distribution). For each (v, u) ∈ (v, u) S , the efficiency score E o (v, u) for each DMU o ∈ D is computed, which allows us to approximate the following stochastic acceptability indexes: • an efficiency acceptability interval index EAII(DMU o , b i ), which is the share of feasible weight vectors for which DMU o ∈ D attains an efficiency score in the This represents the distribution of scores, providing the performance robustness assessment that answers research question 3; which is the share of feasible weight vectors for which DMU o is not worse than DMU k in terms of the efficiency score, i.e., . This answers research question 4 as it indicates how well countries perform in comparison with each other; • an efficiency rank acceptability index ERAI(DMU o , r), which is the share of feasible weight vectors for which DMU o ∈ D attains r-th rank. This answers research question 5 as it allows to rank the countries; Most importantly, all of the above results indicate how stable the scores, rankings, and relations observed for different DMUs are across all feasible weight vectors including those that are disadvantageous for each DMU. Hence, it is complementary to the CCR model as it provides a more likely and plausible representation compared to the standard efficiency analysis. Moreover, these outcomes offer arguments (e.g., average efficiencies or ranks), which enable univocal rankings. For the computational details, discussion on the properties and detailed interrelations between the outcomes computed with LP and MC simulation, see [28]. Overall, the robust efficiency analysis allows us to explore research questions 3, 4 and 5.

Results and Discussion
In this section, the results are discussed according to the research questions formulated in Section 1. The main interest of decision-makers might be knowing which countries are the best. Therefore, the results of the CCR model are given in Table 3. According to this model, 31 out of the 140 countries are deemed as efficient (having a value of 1). All of these countries have at least some inputs and outputs performing well enough so that with specific weight vectors no other countries do better.
Among the inefficient countries, Togo, Benin, Namibia and the Democratic Republic of Congo (DRC) score the lowest, with maximal CCR efficiencies of 0.040, 0.245, 0.268 and 0.373, respectively. This means that even in the best case, i.e., with the most advantageous weight vector combination, other countries perform significantly better. While the CCR model provides a clear differentiation of scores for inefficient countries, it does not allow us to differentiate the efficient ones, with 31 receiving an efficiency of 1. Therefore, building a univocal ranking is impossible with the CCR model. The differentiation between efficient countries comes in the context of ranking robustness assessment presented in Sections 4.3 and 4.5.
After the efficient countries were identified, in the next step it was analyzed why they are efficient. This can be done by identifying the individual indicators that make the corresponding country efficient, i.e., efficiency reducts, as shown in Table 4. For example, it is possible to find priorities (weight vectors) that make the United States of America (USA) efficient when considering inputs 1 and 6 and outputs 5, 7, 8 and 9 only (the numbers correspond to the indicator numbers). In fact, on the inputs, the USA's SAIDI (i 1 ) of 0.6 hours per customer per year and its electricity import dependence (i 6 ) are well-performing. On the outputs, the USA's electricity production mix (i 5 ) is diverse, its EAF (i 7 ) is high and both its GDP per capita (i 8 ) and insurance penetration (i 9 ) are comparatively high on an international scale. In other words, these indicators represent the strengths of the USA and no other country performs better under such priorities. However, this efficiency might not be obvious to reach as the USA requires, under specific weight vectors, the combination of at least two inputs and at least four outputs. Considering another example, Singapore can become efficient under certain weight vectors if inputs 1 and 2 and output 4 are considered. This efficiency reduct corresponds to the SAIDI, where Singapore is the best country in the world as it only experiences less than a minute of electricity supply interruption per customer per year [84], the low severe accidents risks, indicating that Singapore's electricity production mix is safe from the point of view of human fatalities, and its outstanding political stability and absence of violence/terrorism. Hence, Singapore requires only two inputs and one output to reach efficiency, indicating that these are stronger compared to those from the USA.
Furthermore, a rather surprising result is given by Libya. Considering its low indicator performances (e.g., SAIDI of 1883.4 hours per customer per year, high corruption and low political stability, GDP per capita, insurance penetration, government effectiveness and ease of doing business), it is unexpected that Libya still is deemed efficient. However, as it has the best ratios of i 5 /i 6 or i 7 /i 6 , these subsets of indicators still make it efficient for specific weighting vectors. Knowing which countries are the best might be the first requirement of a decision-maker, but not all countries perform at the top. Therefore, it is necessary to also look at the inefficient countries, including the reasons why they are inefficient and how they can improve themselves, which leads to Section 4.2 below.

Why
Are Some Countries Inefficient, How Can They Improve Their Scores and What Are Their Benchmarks? Table 5 shows the projections of inefficient countries onto the efficiency frontier. For example, the projection for Uruguay represents 0.691 of Canada and 0.240 of Sweden (the sum of the shares does not necessarily have to be equal to 1, because conical combinations of existing units are tolerated). This means that the closest virtual country to Uruguay, that is situated on the efficiency frontier, is composed of 0.691 times the indicator values of Canada plus 0.240 times the indicator values of Sweden. The distance to this virtual country represents the closest path for Uruguay to become efficient. In other words, Canada and Sweden are its benchmarks. The higher the share, the closer the original country already is to its benchmark. In this example, Uruguay is already closer to Canada compared to Sweden. The same analysis can be made with other countries. For example, Denmark can become efficient with contributions from six countries and Japan from four. Furthermore, the necessary improvements for all inputs or outputs that need to be applied to make a certain country efficient are given in Table 6. The values are negative for the inputs, as they need to be decreased, and positive for the outputs because they have to be increased. These necessary improvements have to be applied to all the inputs together or all the outputs together. Table 6. Necessary improvements to make a country efficient. These improvements need to be achieved on all inputs cumulatively or all outputs cumulatively. The full table for all countries is available in the ESI, Table S5. Finally, as DEA is a measure of efficiency in relation to other DMUs, it is also possible to become efficient when the country list is changed. Table 7 represents the efficiency construct to be removed in order to make a given country efficient. In the case of Uruguay, it is inefficient because Canada is included in the analysis. In other words, even under the most favorable conditions for Uruguay, Canada will perform better. Another example is Denmark, which has a high level of electricity supply resilience at an absolute level, but it can do even better as under similar conditions, Switzerland scores higher. Furthermore, for Denmark, Japan and Uruguay, only one country has to be removed to make them efficient. This indicates that these three countries are close to being efficient. On the contrary, for example, the Democratic Republic of Congo (DRC) requires the removal of 119 countries to become efficient (see Table S6). In other words, the DRC is far away from the efficiency frontier. It is important to note that, for policymaking, the efficiency constructs should not be misused. In fact, it would not make sense to adapt the list of countries in order to artificially increase the score of a country of interest. The efficiency constructs should rather be seen as a tool to identify benchmarks towards which inefficient countries should aim for.
The results presented in Sections 4.1 and 4.2 demonstrated that the CCR model successfully identified (1) the benchmarks, (2) the leading indicators for each country from an efficiency perspective, (3) the distance from the efficiency frontier, (4) the required indicator improvements to make inefficient countries efficient and (5) the list of countries making others inefficient. However, one has to bear in mind that these findings are based on the most advantageous weight vectors for each country, clearly representing best-case scenarios.
Therefore, the analysis is extended using robust efficiency analysis models to address the following questions: • What is the average or most likely performance of a country? What is the expected distribution of its performance (Section 4.3)? This answers research question 3.   Table S6. Identifying benchmarks allows us to analyze weaknesses and develop successful policies. However, the results of Sections 4.1 and 4.2 are based on the CCR model, i.e., a best-case scenario, as explained in Section 3.4. Decision-makers might be interested in a more unbiased view of the results in which ranking robustness is assessed. This leads to Section 4.3 below.

How Robust Is the Performance of the Countries?
Instead of analyzing only the best-case scenario, the Hit-And-Run Monte Carlo-based robust efficiency model accounts for the wide variability of preference models and it calculates a distribution of performance scores of the countries, thus providing a measure of robustness [83,85]. Table 8 shows the results for the maximum and average performance of the scores obtained with 10,000 model runs. Furthermore, the efficiency acceptability interval indices (10 bins of equal size) are displayed. The sum of the efficiency acceptability interval indices per country is equal to 100%. The table shows only a selection of 45 from the 140 countries, ordered by decreasing values of average efficiency. According to this model, South Korea, Singapore and Canada are the most efficient countries as in 64.1%, 47.8% and 34.7% of the simulations, respectively, their efficiency is situated in [0.9, 1]. As these countries perform well on multiple inputs and multiple outputs, their efficiency score is high for many weight vectors. On the other end of the spectrum, Togo, the DRC and Nigeria have, for more than 98% of the Monte Carlo simulations performed, efficiencies situated in [0, 0.1]. In Togo's or the DRC's case, neither of their indicators is well-performing. Regarding Nigeria, only its EAF (i 7 ) performs at an average level, which in comparison with other countries, still makes it one of the worst-performing ones. Table 8. Maximum and average efficiency scores for the robust efficiency analysis model, as well as the efficiency acceptability interval indices. The number after the countries' names corresponds to their ranks computed based on the average efficiency. The full table for all countries is available in the ESI, Table S7.   When it comes to the distribution of performance, South Korea, Singapore and Canada consistently rank at the top, as they attain efficiencies higher than 0.7 for most of the simulations. This indicates that these three countries clearly outperform the others leading to lower efficiencies for all other countries. In the middle, there are countries that show more balanced distributions of scores. For example, Germany, Sweden, New Zealand and France attained efficiencies in all intervals and the sum of their three highest interval values represent between 37% and 40%. Hence, the efficiency of these countries is highly dependent on the weight vector considered. If more weight is placed on their well-performing indicators, they will score higher, and reciprocally. This aspect is not revealed by the CCR model, as it only retains the single most advantageous weight vector. When analyzing France in more detail, it can be seen that all its indicators are performing well except its political stability and absence of violence/terrorism (i 4 ), where it ranks 60th, and its relatively low electricity mix diversity (i 5 , 77% of its electricity production comes from nuclear energy). Hence, France mostly performs well, but there exist some disadvantageous weight vectors, resulting in more balanced distribution of scores. Finally, about half of the countries are clustered at the bottom. In fact, for most weight vectors, these countries attain efficiencies situated in [0, 0.1] and only rarely exceed 0.5.
The maximum efficiency column in Table 8 corresponds to the extreme value of the 10,000 Monte Carlo simulations (the convergence of the results was verified by increasing the number of simulations stepwise and the confidence interval of 95% for Stochastic Multicriteria Acceptability Analysis (SMAA) analyses is not exceeded [85]). While the CCR model deemed 31 countries as efficient, the stochastic analysis only identified 13 as efficient, indicating that the Monte Carlo simulation could not find the best scenario for each country based on the chosen number of runs. Both scores would of course be equal, if an infinite amount of simulations would have been performed [28]. Also, as the CCR model is a linear optimization problem, the simulation-based maximum efficiencies will always be lower or equal to the CCR efficiencies. The difference of both scores, along with the efficiency acceptability interval indices, provides an indication of the likelihood of finding weight vectors that result in a high country score. The largest differences between the CCR (best-case) and the simulation-based maximum efficiency were found for Venezuela, Kenya and Luxembourg with differences of 0.430, 0.332 and 0.310, respectively. Among the countries considered efficient by the CCR, the largest differences were found for Luxembourg, Haiti and Algeria (0.310, 0.264, 0.231). This means that, even though these countries can be efficient based on the CCR model, the corresponding weight vectors are limited. This indicates that relying on the CCR model only might be misleading, as some countries are considered efficient even though efficiency is reached for very few and unlikely weight vectors.
Finally, Table 8 also shows the average efficiency, taken as the arithmetic average of all the simulations. Once again, South Korea, Singapore and Canada are the most efficient countries with average efficiency scores of 0.911, 0.852 and 0.668, respectively. On the lower end is again Togo, the DRC and Nigeria. It is important to note that this average efficiency can be used to rank the countries as it is highly unlikely that two or more countries have equal values. Furthermore, a ranking based on the average makes more sense because it is not just driven by the best-case scenario of the CCR model.
Interestingly, even though Libya, among others, reaches maximal efficiency in the CCR model, its average efficiency is extremely low (0.035). In fact, it is lower than numerous countries that do not reach efficiency. Therefore, Libya reaches unitary efficiency only for a very small number of randomly selected weight vectors and hence should not be seen as a country with an overall high electricity supply resilience. As shown in Table 4, only three efficiency reducts make it efficient. These are indicators {5, 6}, {6, 7} and {2, 6, 12}. In particular, its electricity import dependence is excellent as it produces much more electricity than it consumes and, therefore, can, in the case of shortage, easily cover its own demand. But this particular situation does not mean that Libya's electricity supply is resilient holistically, as 8 out of its 12 indicators perform poorly. On the other end, Denmark is the inefficient country with the highest average efficiency (0.489). In fact, Denmark actually scores high on all of its indicators, except for its EAF (i 7 , 49% of Denmark's electricity production comes from wind energy which has a low EAF) and its insurance penetration (i 9 ). Nevertheless, for each weight vector, there are still other countries that perform better. However, based on the average efficiency, one can safely state that Denmark's electricity supply resilience is higher than Libya's. Hence, compared to the CCR model, the stochastic model allows us to distinguish which countries are more robustly deemed efficient than others, even though they all may be considered as efficient in the CCR model. Furthermore, the simulation-based analysis ranks many inefficient countries higher than several efficient ones in the CCR model. Therefore, for policymaking, it is crucial to consider the CCR model in combination with the robust efficiency model, as it gives a more realistic and broader interpretation.
To confirm the robustness of these results, a sensitivity analysis was performed which investigated how the outputs of a model are affected by varying its inputs and which inputs have the highest effect on the results. If the variations in the inputs result in small variations of the outputs, then the model is considered robust. In the present study, the sensitivity analysis was applied on the Monte Carlo simulation by removing each indicator one by one and keeping the others. The Spearman's rank correlation coefficients (rho) was calculated between the average country efficiencies for the complete indicator set and one indicator removed at a time. It can be seen from Table 9 that all the coefficients are very high and significant at the 0.01 level. The lowest value is for i 2 , where the correlation factor between the vectors of average efficiencies is ca. 0.897. This still represents a very high correlation and the general trend in the ranking is preserved. Thus, it can be concluded that the results are robust. Table 9. Spearman's rank correlation coefficients (rho) between vectors of average efficiencies when removing the 12 indicators one by one. All correlation coefficients are significant at the 0.01 level. The distribution of performance of the countries presented in this section shows how robust the results are, which is a key interest of decision-makers. Once the robustness is analyzed, it is important for decision-makers to verify the rank of the countries. In fact, a certain score could lead to a high, average or low rank, depending on the performance scores of the other countries. In Section 4.4, it will become clear that there is no one-to-one relationship between scores and ranks.

What Is the Univocal Ranking of the Countries?
The previous three sections discussed country scores. However, a certain rank can be achieved by different scores and a certain score can result in different ranks. In order to analyze the rank distribution, this section shows the country ranks computed with the simulation-based Monte Carlo analysis (see Table 10). For the optimal weight vectors, the 31 efficient countries obviously have rank 1 as their best. All the inefficient countries rank at best second. Additionally, the expected rank (ER o ) allows us to differentiate all countries, including the efficient ones. As with the average efficiency, the ER o can be used to build a univocal country ranking. Therefore, the ER o once again confirms that the top place is taken by South Korea (ER o of 3.6). As shown in Section 4.3, the close followers are Canada and Singapore (ER o of 4.1 and 4.6, respectively). In conformity with previous results, the DRC is the country with the lowest ER o (137.9). Table 10. Best and expected ranks for selected countries obtained by the simulation-based Monte Carlo analysis. The three highest efficiency rank acceptability indices (ERAI) are shown with their corresponding ranks. The ERAIs are expressed in %. The number in parenthesis after a country's name is its overall rank according to the expected rank ER0. The full table for all countries is available in the ESI, Table S9. Furthermore, it is interesting to analyze the distribution of the ranks. The countries with the most concentrated ranks are the top-and bottom-performing ones (e.g., South Korea, Singapore, the DRC, Canada and Nigeria, among others). In fact, these are clustered at the top or at the bottom, hence it is more likely for them to have large ERAIs. On the contrary, the United Arab Emirates (UAE), Costa Rica and Luxembourg show only small ERAIs. In fact, the sum of their three largest ERAIs is only 6.8%, 7.7% and 7.8%, respectively. An extreme case is Libya, which according to the weight vectors considered, can attain every possible rank. However, Libya's ER o is 126.8, indicating that it most likely does not rank high. It can be seen that Denmark performs much better than Libya as its ER o is 12.4. Switzerland, the country that makes Denmark inefficient, ranks slightly better at 8.0. Finally, the tie between Niger and Syria is also reflected through their almost identical ER o (99.1 and 98.9, respectively).

Country Best (R o *) Expected (ER o ) Rank ERAI Rank ERAI Rank ERAI
After the analysis of scores and ranks of the countries, it would be valuable for decision-makers to identify close competitors. These would help to identify strengths and weaknesses, therefore supporting the development of realistic targets and appropriate policies. This leads to Section 4.5 below.

How Well Does One Country Perform in Comparison to Another?
Pairwise efficiency outranking indices (PEOI) are used to compare the performance of countries between each other. It represents the share of simulations where a country performs at least as good or better than another (see Table 11). PEOIs become extremely useful for identifying close competitors. In country rankings, relevant comparisons can be made in smaller peer groups, i.e., with similarly performing or geographically neighboring countries. For example, Syria and Niger are in a tie (50.1%). Hence, these two countries can benchmark their performance and closely monitor each other over time.
If a country turns out to outperform its peer, then the policies of the two countries can be analyzed and successful strategies identified. Table 11. Pairwise efficiency outranking indices (PEOI) for selected pairs of countries. The values indicate the shares (in percentage) of weight vector samples for which a country has an efficiency score not worse than another (i.e., at least as good). Due to the large number of countries in the data set, the complete table is available in the ESI, Table S8. Countries hereby shown include the best-and worst-performing ones, ties and some of the ones discussed in this paper. Furthermore, as expected, the top-performing countries (i.e., South Korea, Singapore and Canada) only rarely score lower than other countries. Similarly, on the other end of the spectrum, Togo, the DRC and Nigeria rarely achieve higher efficiencies than the other countries. Furthermore, even though Libya is deemed efficient, it performs better than Denmark in ca. 1% of the simulations only (the CCR model deemed Denmark as inefficient). Once again, this confirms that Denmark has a better electricity supply resilience than Libya.

Country
Another interesting case is how Switzerland performs compared to the top three countries. Switzerland wins against Singapore and South Korea in ca. 22%-24% of the simulations, but wins against Canada for only 2.9% of the cases, even though the average efficiency of Canada is lower than that from South Korea and Singapore. This is due to the fact that the Euclidian distance between Switzerland's and Canada's indicator performances is smaller, meaning that Switzerland and Canada have more similar indicator values compared to Singapore and South Korea. Hence, for the weight vectors that are advantageous for Switzerland, Canada almost always performs slightly better, whereas South Korea and Singapore do not.

How Does the Performance of Countries Vary According to Changes in Selected Indicators?
The previous sections presented results on the performance of countries, their potentials for improvement and their position in a univocal ranking. In this section, DEA is used as a means to make country-specific improvement potential evaluations. In this way, policymakers can detect early warning signals and explore different future pathways, leading to more effective decisions and subsequent implementation of strategies to reach the targets. In the present study, two types of country analyses were applied: 1.
Obtain a new country ranking, based on updated indicator values according to specific scenarios (Singapore, Section 4.6.1).

2.
Determine the minimal required improvements on the indicators in order to become an efficient country (Japan, Section 4.6.2) 4.6.1. Country Analysis: Singapore's Electricity Supply Resilience Located in Southeast Asia, the small sovereign city-state and island country of Singapore is often referred to as the Switzerland of Asia (e.g., [86]). It portrays a high standard of living [87], a strong economy [88] and political stability [89]. Furthermore, Singapore is one of the largest and most competitive financial centers in the world [90]. Additionally, it is one of the world's top five oil trading and refining hubs [91] and is home to the world's second busiest container port [92]. Regarding the energy sector, Singapore imports mainly petroleum products, crude oil and natural gas. Furthermore, natural gas is the source for 95.2% of the electricity produced [93]. Currently, the Singaporean government is trying to diversify its energy supply, in order to be able to better cope with supply disruptions and price increases [94]. Its Economic Strategies Committee (ESC) published a report aiming at ensuring energy resilience and sustainable growth [95]. It contains five key strategies: (1) diversifying the energy sources, (2) enhancing infrastructure and systems, (3) increasing energy efficiency, (4) strengthening the green economy and (5) pricing energy right.
Overall Singapore, deemed as efficient with the CCR model (see Table 3), has the second highest average efficiency (see Table 8) and the third best expected rank (see Table 10). Its outstanding infrastructure is ranked the second best in the world [96], which is reflected by its low SAIDI (i 1 ) of less than a minute per customer per year, and by the fact that there are almost no fatalities related to electricity production (i 2 ). Furthermore, being among the top performers in controlling the levels of corruption (i 3 ) and political stability (i 4 ), Singapore has a stable environment that enables clear policymaking and transparent directives. However, its electricity generation mix diversity (i 5 ), consisting of 95.2% of natural gas, makes it particularly vulnerable to potential disruptions. Additionally, on its electricity import dependence (i 6 ), Singapore's electricity grid is currently not strongly connected in the region, even though this might change in the near future as there are growing efforts to establish electricity interconnections in Southeast Asia, in particular between Singapore, Malaysia and Indonesia [97]. Nowadays, Singapore basically produces what it consumes. If there were a shortage for any reason, only limited amounts of electricity could be imported from its neighbors. Furthermore, its EAF (i 7 ) is high, which is attributed to the fact that the electricity is mainly produced by natural gas (natural gas has an EAF of 0.85 [74]). Also, Singapore excels with a high GDP per capita (i 8 ) and an outstanding government effectiveness (i 10 ). This is the result of a series of successful developments undertaken in the past decades [98]. However, its insurance penetration as a percentage of GDP (i 9 ), a central part for accelerating recovery processes [99], is not expected to increase in the future [75]. Regarding the average outage time (i 11 ), Singapore is the top-performing country having almost no interruptions at all and even if there happens to be one, it usually is so short that it is hardly noticeable by the population. Lastly, Singapore's ease of doing business (i 12 ) is already the second highest in the world [100].
Based on these premises, two scenario analyses were developed and discussed in the following two sections (Sections 4.6.2 and 4.6.3).

Scenario 1: 8% Solar Photovoltaic Electricity Production
One of the weaknesses of Singapore's electricity supply is its generation mix diversity (i 5 ) consisting of 95.2% of natural gas, which makes it particularly vulnerable to potential disruptions. To address this issue, one of the government's current strategies is to diversify its sources [101], by for example increasing the share of renewables [102]. In particular, by 2030, Singapore has a potential of producing 8% of its electricity by solar photovoltaics (PV). This change affects the following indicators: • i 2 ; improvement from 0.124 to 0.115 fatalities/GWeyr: solar PV has lower fatality rates than natural gas [71]. • i 5 ; improvement from 0.11 to 0.22: replacing natural gas generation by solar PV improves the mix diversity. • i 7 ; deterioration from 0.85 to 0.79: solar PV has a lower EAF than natural gas [74].
Therefore, as the performance of i 2 and i 5 increases, but that from i 7 decreases, it is not yet clear if this will have a positive effect on Singapore's electricity supply resilience. However, results show that, in this particular case, it is advisable to pursue this strategy as Singapore's score effectively improves (see Table 12). In fact, even though Singapore keeps its second position for the average efficiency and the third rank according to the ERAIs, its performance improved, hence reducing the gap to South Korea and Canada.
Overall, this shows that it is not possible to predict a priori if an increase of the share of renewables is good or bad for resilience. It has to be studied on a case-by-case basis. In the present example, it turned out that increasing the share of solar PV generated electricity to 8% is positive for Singapore's electricity supply resilience. The second scenario for Singapore assumes that in the year 2030 it not only reaches an 8% share of solar PV, but that the following two indicators change as well: • i 6 ; improvement from 0.98 to 0.92: Singapore's electricity grid is currently not strongly connected in the region, but its import dependence is expected to decrease as a result of planned interconnections with Malaysia and Indonesia [97] and according to the projected production and consumption in 2030 [103]. • i 8 ; improvement from 51,809 to 67,360 USD/capita: according to predictions, Singapore's GDP will increase to 67,360 USD/capita in 2030 [104].
With the new values for i 2 , i 5 , i 6 , i 7 and i 8 , Singapore's expected performance in 2030 will improve even more (see Table 13), and it will overtake South Korea and Canada, i.e., reaching the first position as the most resilient country in the world regarding electricity supply. In the three decades following 1960, Japan's economy has boomed with average GDP growth rates of up to 12.9% [105] and is currently the third-largest in the world [106]. This lead it to become the fifth best country worldwide, according to a global index from the U.S. News and World Report [107]. Its population enjoys the highest life expectancy in the world [108] proving that the quality of life is high [87]. Japan also ranks particularly high for entrepreneurship (2nd [107]) which portrays its numerous innovations and comparatively high number of patent applications [109]. Regarding the energy sector, Japan has gone through tremendous changes since the 2011 Tōhoku earthquake and tsunami [110]. In fact, due to the Fukushima Daiichi nuclear disaster, the nuclear produced electricity was replaced almost instantaneously by mostly imported oil, gas and coal [68], making Japan more vulnerable to supply disruptions. As this option comes with risks and drawbacks (e.g., import dependence, environmental concerns [111], financial burden [112]), the Japanese government published a revised version of its Strategic Energy Plan (SEP) in 2014 [113]. Its goals are to ensure a stable supply, enhance economic efficiency on the premise of safety and pursue environmental suitability.
Based on these premises, Japan's low SAIDI (i 1 ) and low fatality rates related to electricity production (i 2 ) reflect the overall high quality of its infrastructure [96]. Furthermore, having low levels of corruption (i 3 ) [114] and high political stability (i 4 ), Japan has a stable environment that enables clear policymaking and transparent directives. Even though Japan's performance on the electricity mix diversity (i 5 ) is far from alarming, it shows slightly lower scores compared to some years ago, as its mix diversity has since the Fukushima Daiichi nuclear disaster increasingly been dependent on imported fossil fuels. Regarding the electricity import dependence (i 6 ), due to its geographical location, Japan's electricity grid is currently isolated [115], which means that the production simply follows the consumption pattern. Recently, an interconnected Northeast Asia (NEA) grid has received increasing attention. However, modest economic benefits are a major problem for its implementation [115,116].
Additionally, if this plan were to be realized, it is likely that Japan would overall import electricity which would further increase its dependence on its neighbors. Furthermore, its EAF (i 7 ) is high, which is attributed to the fact that the electricity is mainly produced by fossil fuels (Fossil fuels have EAFs of 0.85 [74]). Also, Japan shows high GDP per capita (i 8 ) and government effectiveness (i 10 ). Its insurance coverage (i 9 ) is expected to grow [75], as a result of increased awareness of potential losses due to frequent recent natural catastrophes [117]. In fact, Japan is the fourth most exposed country in the world [118]. This is probably also the reason why its average outage time (i 11 ) is long (4 hours), even though there is on average only one interruption per citizen every tenth year [100]. Lastly, Japan currently ranks 34th in the ease of doing business ranking [100].
Overall, Japan performs well on most of its indicators. Its average efficiency is 0.263 (42nd, see Table S7) and it ranks 55th according to its expected rank. However, it still does not reach efficiency (CCR score of 0.991, see Table 3). Based on these premises, Section 4.6.5. describes Japan's scenario analysis and its results. 4.6.5. Scenario 3: Required Electricity Generation Portfolio to Make Japan Efficient Unlike the two scenarios for Singapore, where the original calculations were made with an updated data set, the scenario for Japan is an optimization scenario, where the aim was to find the minimal improvement that makes Japan efficient. The first step was to determine which indicators should be varied. The second step consisted in calculating the minimum required performance changes for these indicators in order to make Japan efficient. The chosen scenario investigated if by varying only the electricity generation portfolio Japan can become efficient. As a consequence, the minimum improvements for indicators i 2 , i 5 and i 7 were calculated to make Japan efficient (see Table 14). These improvements were obtained by running the CCR model and considering a constant, proportional improvement over the three indicators. The third step was to calculate to what electricity generation portfolio these new values correspond. Considering the 10 fuel sources that are currently producing electricity in Japan (see Table 15), there are numerous portfolios that result in the required performance on indicators i 2 , i 5 , i 7 . From an optimization point of view, this is equivalent to an underdetermined system (3 indicators for 10 technologies; fewer equations than unknowns). Hence, for each of the portfolios fulfilling the constraints on the three indicators of Table 14, its Euclidian distance was calculated. Performing the calculations for 10 million randomly selected portfolios, Table 15 shows the 10 closest portfolios to Japan's current one (smallest Euclidian distances). In fact, these represent the ones that require the least amount of change in order to make Japan efficient.
Although the 10 portfolios do not show large differences, they can provide different policy perspectives. For example, if the target is to reduce the amount of fossil fuels (coal, oil and natural gas), then portfolios 3 and 10 are most suitable. These two portfolios come with an increase of biomass (biofuels and waste combined), nuclear, hydropower and geothermal electricity, whereas solar PV and wind decrease. Portfolio 8 is the only one that increases the share of solar PV, which already today is on a sharp rise [68]. However, no portfolio jointly increases the shares of solar PV and wind. Overall, portfolio 8 is closest to the Japanese government's goals [113, 119,120], as (1) it decreases the amount of coal and oil, (2) only slightly increases the share of natural gas (currently the cleanest fossil fuel, especially if used in combination with carbon capture and storage [121]), (3) increases the shares of biomass, solar PV, geothermal and nuclear electricity, but (4) decreases the share of hydropower and wind electricity.

Conclusions and Policy Implications
Starting from a set of 12 indicators, this study uses two DEA models to assess the electricity supply resilience of 140 countries. First, the classical CCR model deemed 31 countries as efficient (score of 1), and hence resilient. For these countries, it is possible to find at least one weight vector under which no other country performs better. To gain insights into these efficient countries, a novel algorithm that allows us to calculate their efficiency reducts was developed. This demonstrated which minimal combinations of indicators can make a country efficient. Furthermore, another novel algorithm was developed to identify the efficiency constructs of each inefficient country. In other words, the minimal subsets of countries that make it inefficient was computed.
Second, a robust efficiency analysis was applied. To the authors' best knowledge, the present study represents the first application of such an analysis to a country ranking. A distribution of efficiency scores for each country is calculated, which provides information about ranking stability as it depicts the likelihood of a country scoring in a certain performance bin. Additionally, it allows calculating both the average efficiency and the expected rank of a country that can be used to establish a univocal country ranking. The robustness analysis also allows computing the pairwise efficiency outranking indices.
Finally, scenario analyses for Singapore and Japan were carried out. For Singapore, the analysis consisted in verifying if its current energy policies lead to an even higher resilience, even though Singapore is already efficient according to the CCR model. Results showed that increasing electricity production from solar PV is beneficial for Singapore's electricity supply resilience. In contrast, as Japan is an inefficient country, an optimization problem was solved to determine the minimal required improvement on selected indicators in order to make it efficient. From a policymaking perspective, this is equivalent to finding the optimal way to allocate resources in order to increase its rank. By strictly considering technologies that are already producing electricity, results showed that it is possible to reach efficiency by only slightly changing the production shares.
Overall, this study showed that combining the CCR model, including its efficiency reducts and constructs, with the robust efficiency analysis provides a holistic assessment methodology that can be applied to the present electricity supply resilience assessment of 140 countries, but also similar problems in other domains to support robust decision-making by stakeholders. In fact, even though the CCR model is the most widely used, its results are limited and can be misleading. While the CCR model provides a clear differentiation of scores for inefficient countries, it does not differentiate between the efficient ones. Therefore, building a univocal ranking is impossible. Furthermore, the CCR model provides a best-case scenario, as it computes the most advantageous weight vector for each country separately. As a result, such scores may not be representative, because they might only be achieved for a very limited number of weight vector combinations. Therefore, the authors believe that by using the hereby developed methodology, policymakers would have a broader view of how the alternatives under study perform. Many policies are indeed based on the results of indices obtained by aggregating average values without considering uncertainty or robustness of the results. This might lead to ill-informed decisions. Accounting for uncertainty in input data and problem structure brings a dynamic component to the usual indices that are static.
By considering the CCR and robust efficiency analysis simultaneously, decision-makers can identify close competitors. This provides important learning lessons from comparable countries (so-called benchmarks). Furthermore, this methodology stimulates a multi-disciplinary approach when considering improving the overall performance of a country. In fact, as the indicators are interrelated, multiple specialists should share knowledge in order to tackle the complexity of today's world. Through collaboration between multiple parties, including research institutions, industry and governmental agencies, it would be possible to develop improvement plans and policies to reach predefined targets. The methodology proposed in this paper could provide an interactive discussion platform to lead the decision-making process.