Energy Policy Concerns, Objectives and Indicators: A Review Towards a Framework for Effectiveness Assessment

: This work presents a review that aims to characterize the policy evaluation practices regarding the public policies on energy, with a focus on the metrics: concerns, objectives, and indicators. As key novelty, emphasis was put into finding attributes and metrics that can be used to assess effectiveness, not only efficacy or efficiency. The concerns and objectives were organized into four categories: Institutional, Environmental, Economic, and Social. For every category, detailed and condensed concerns were identified. It was attempted to find indicators for every condensed concern, which resulted in 15 core indicators.

As a more nuanced concept than efficacy, effectiveness thus has the potential to encompass the components of taking into consideration contextual variables and of assessing "success" in a quite modulated manner. It can even be advocated that, while privileging the evaluation of the achievement of results, a truly effective program should assess those results concerning the contextual variables (geographical, economic, and cultural) rather than by just comparing it to the initial goals of the program. This is so because these often are not declared and, even when they are, their choice is inherently somewhat arbitrary. Effectiveness can thus incorporate some dimension of efficiency into the assessment.

Materials and Methods
The academic databases Sciencedirect, Google Scholar, and JSTOR, together with the web browser Google were used for gathering documents. All the sources were useful to identify documents that characterize energy policy effectiveness. The expressions under which the search was done were "Energy policy evaluation" (academic and open-source); "Energy policy effectiveness" (academic source); "Energy policy assessment" (academic source); "Energy policy efficiency" (academic source). The selection of the documents to be reviewed was done by their title, abstract content, and keywords. After doing so, the criteria for the document selection for the analysis was having at least one concern that assesses energy policies. A summary of this process is presented in Table 1.
The detail of each search source is presented in Appendix A, for which reference and order numbers have been provided and that have been used for the organization and analysis of the documents presented in the document. Analysis: Geographical Scope When analyzing the scope or level of application of the energy policies evaluations (local, regional, national, international, or a combination among them), it is noted in Table 2 that the highest number of the reviewed documents regards the national level (with 40 documents), with a considerable difference above the other alternatives. The national scope is followed by the international scope (with eleven documents). Some documents regard all the levels of application or a combination of more than one level. Another observation is that only three locally focused documents were found.
Concerning the time frame shown in Table 3, almost all of the retrieved documents state or enabled us to infer the period of the evaluation (summing 53 documents), against 13 documents where that was not possible. The time framing of the process, according to the reviewed documents, can be ex ante, in progress, or ex post. The ex post energy policy evaluation approach was the most often found (34 documents), followed by the ex ante approach (10 documents). It can be noticed that only two of the reviewed documents ("Moving Toward Energy Efficiency: A Results-Driven Analysis of Utility-Based Energy Efficiency Policies", Theel and Westgaard, 2017 [13] and "Evaluating Consistency in Environmental Policy Mixes through Policy, Stakeholder, and Contextual Interactions Lieu et al., 2018 [14] declared a combined approach (with more than one possible timing period). Table 4 shows the participation of stakeholders in the EP evaluation. Stakeholder participation is related to the involvement of actors along the evaluation process. Less than one-third of the documents retrieved report some form of participation from stakeholders (19 out of 66 documents).   The methodological approaches for evaluation were also reviewed and are presented in Table 5. As expected, very diverse approaches were found from document to document. The qualitativebased methodology was found as the most popular approach (19 documents). Descriptive and qualitative analysis methods have been considered as qualitative-based. The second more frequent approach is statistical (17 documents). Scenario and modeling techniques have also been identified (12 documents). The category others include the approaches that were found only once e.g., costbenefit analysis (CBA), return on assets (ROA), etc. In Table 6, it can be seen that when analyzing the motivation stated in the reviewed documents, the most common one is to conduct a policy evaluation itself, observed in 34 documents. Four documents clearly defined their interest in supporting policymakers, while 16 documents target the evaluation of policy success. It can be noted the interrelation between the identified motivations.

Energy Policy Concerns and Objectives
The identification of the concerns most frequently included in the literature and how they are defined led to forming a list of more than 60 concerns. They have been classified into four broader categories, and some subcategories have been formed.

Environmental Concerns
As shown in Table 7, two subcategories were defined: greenhouse gas (GHG) and other environmental impacts. The burden among the categories could be very thin, as some concerns are related to both, as renewable energy (RE) sources related concerns (e.g., RE standards, RE Deployment and RE Tech Efficiency).
Greenhouse gas (GHG) emissions reduction was found as the predominant concern for evaluating energy policies under an environmental perspective and gathered under the subcategory Environmental-GHG. The GHG emissions concern consideration has been incentivized by the continuous and reinforced influence produced by the definition of international agendas, national targets, agreements, and other international efforts to reduce GHG emissions and minimize climate change.
Other impacts include the concerns that assess the conservation and improvement of the environment, excepting GHG. RE related concerns (e.g., RE standards, RE Deployment and RE Tech Efficiency) have been included in this subcategory.  [14].

Economic Concerns
A large set of concerns was found under the economic category, as shown in Table 8. It should be noticed that almost all the reviewed documents included at least one aspect under this category. The economic dimension regards all the concerns that assess policies in monetary terms, from which two subcategories were formed.
The Economic-cost subcategory includes the concerns related to direct monetary influence, particularly related to costs and expenses. Both levels, macroeconomic and microeconomic, have been assessed; e.g., the energy price is included, referring to the economic aspects in a micro economical view. The gross domestic product (GDP) impacts of energy decisions refer to the macro level of an economy.
Regarding Economic-competitiveness, this subcategory brings together market orientation concerns (e.g., attractiveness of investment, market growth and market maturity aspects). The market characteristics provide information that allows the comparison among different markets, informing about their national and international competitiveness. Regarding the Social category, all concerns with a social orientation have been gathered in Table  9, including the concerns in which equity is recognized. Two sub-categories were defined. Firstly, Social-equity include concerns related to social sustainability factors, as the acceptance of certain energy technologies and the impacts of subsidies on energy bills. It can be noticed how equity has been considered in several documents, being a key component of distributive energy justice, and that should contribute to new policy initiatives. Some other concerns include energy democracy, energy equity and consumer surplus. This last is an important reference to measure the welfare of consumers [29].
The second subcategory, Social-employment, as can be inferred from its name, is the subcategory to account for the employment rates affected by the energy policies. The knowledge and skills (expertise) of the workforce have also been included in this subcategory. Finally, under the institutional concerns assessment (Table 10), the variety of concerns found through the reviewed papers was larger than in the previous categories; however, no subcategories were created, due to a lack of a clear structure. The identified concerns describe the policy institutional framework, including concerns as flexibility with policy instruments, actors, and institutional environment, among others.
There is a recent recognition of the importance of the institutional context. Concerns like political commitment, policy choice, and citizen engagement characterize policies from an institutional perspective.
Even if the institutional performance does not measure policy success itself, it contributes to explain the policy environment. Institutional context tends to be assessed qualitatively rather than quantitatively. The lack of quantitative procedures to evaluate some dimensions of the institutional context provides the opportunity to develop scientific-based methodology proposals that enable to evaluate the concerns found under this category. Table 10. Energy Policy Institutional concerns found.
Policy framework (policy environment and policy process)  [63].
Robustness, efficiency Table 11 shows the number of appearances per concern by subcategory. It is noted that Economic-cost is the subcategory that was found more often in the reviewed documents, reflecting the importance given to it by the consulted sources. The second most popular subcategories are Governance and Social equity, which regards the energy impacts on society and energy distributional impacts. Table 12 shows an analysis of the categories. It was noted that most documents mention more than one category (42 documents). From the remaining 24 documents (single category approach), 12 documents report an Economic orientation, being the largest one. Table 11. Number of appearances per concern by subcategory.

Category
Quantity of Documents

Identification of Gaps
After the review of the concerns that have been used to characterize the effectiveness of the energy policies, some areas have been identified as not fully covered regarding an integral review of EP effectiveness characterization.
A list of missing concerns has been elaborated (Table 13) in which additional concerns have been included. The criterion followed to add concerns was if the outcomes of the concern could be modified by a policy and their relevance for the ep effectiveness characterization.
Not all the subcategories needed extra concerns, e.g., the Social subcategories (Social-Equity and Social-Employment) have been considered as complete, as the concerns identified through the review do provide a wide perspective of the social approach.
Under the environmental category, the concern of environmental protection has been often found through the revised documents. However, it has been considered relevant to detail it through more specific concerns (as water and land resources) to be able to better measure them. They are relevant concerns for the environmental evaluation of policy and closely related to the impacts of energy on the environment.
The externalities pricing is a concern that can be reported under the environmental or economic category, due to their interrelationship in the concern itself. For the monetarization process that externalities involve, it has been decided to keep this concern under the economic category.
It was found a lack of consideration of some aspects that may be relevant for the policy appraisal, as available funds (for technology, innovation, and infrastructure investment), the economic situation of the private sector, and the economic situation by regions, economic and fiscal parameters (like inflation, escalation, exchange rates, and taxes). All these concerns can be gathered as economic background. Additionally, the grid state is a relevant concern for its relation to the energy service provision.
Finally, the concern under the institutional category (Amount and quality of Staff) intends to involve a relevant aspect of the institutional framework: the governing human resources. They differ from Knowledge and skill of the workforce (reported in the Social concerns), as this last regards the technical workforce involved in the policy implementation, while the second one, regards the governance-related workforce.

Inferred Energy Policy Objectives and Condensed Concerns
With the intention of better organizing the concerns and facilitating the identification of their evaluation metrics, a condensation process was taken to eliminate redundancies, conglomerating them into condensed concerns. The condensed concerns have been identified after questioning where there could be an impact if the extracted concern does not comply. In addition, for every condensed concern, its end objective has been identified (as shown in Figure 1). The identification of the ends objective aims to put the review more in line with the typical multi-criteria decision making (MCDM) language and to ease the further identification of indicators.
The economic extracted concerns are the largest in number when compared with other categories. It was possible to narrow down the initial list to three condensed concerns: affordability, accessibility, and economic competitiveness.
Affordability includes all the concerns that may affect the state of being cheap enough for people to be able to pay for electricity or other forms of energy [79]. Accessibility refers to the fact of energy or electricity being able to be reached or obtained easily [80].
Competitiveness brings together the concerns that relate to being able to compete successfully with other companies, countries, organizations, etc. [81].
The concerns under the environmental spectrum have been condensed through two concerns. The first one reflects the policy's impact on the environment. Its ends objective relates to environmental sustainability, defined as the quality of causing little or no damage to the environment and therefore able to continue for a long time [82]. The second condensed concern denotes the impacts on climate change.
In the social category, Equity, as the situation in which everyone is treated fairly and equally [83], gathers the majority of the social concerns. Health is however kept as a separated condensed concern.
Governance effectiveness and efficacy as a single condensed concern included all the concerns under the institutional perspective. Governance is defined as how an organization is managed at the highest level and the systems for doing so [84] and assessed through its effectiveness and efficacy.
The organization of the extracted and condensed concerns is documented in Figure 1 The economic, environmental, and social categories include more than one condensed concern.

Indicators' Relevance and Types
The relevance of indicators has been recognized in the energy field [85][86][87][88][89]. Agents have created ways to assess different dimensions of the energy industry through indicators, giving them several functions and purposes [90][91][92][93]. They can be used as assessment metrics, providing a comparison between the defined objectives and their achievement, therefore assessing the effectiveness of the state of systems, which can be applied to assess the success of policies.
Indicators can be differentiated by function, serving as monitoring, guideline, or communication instrument. With a monitoring purpose, indicators evaluate the policy progress towards their objective's achievement. When indicators are used to monitor the status of an energy policy, they can be helpful for the improvement of the policy. An in progress evaluation enables the identification of the need for modifying the track to achieve the settled objectives in case the progress is not fully satisfactory.
At the beginning of the policy cycle, indicators can provide a deeper orientation of where to add more resources or where to focus while designing a policy. They may allow a deeper understanding of some elements and even help to identify relations that may not be shown through basic statistics or data. Indicators, as communication instruments, can be useful tools for policymakers to express elaborated ideas or concepts to other relevant stakeholders of the sector or out of it; e.g., the indicators issued by national governments to inform regional or local authorities.
Indicators can also be differentiated by whether they are individual or composite (also known as aggregated). The first ones are commonly used to show the presence or state of a situation or condition individually for a topic [94]. Despite the usefulness of individual indicators, sometimes the information provided may not be enough for the purposes of the evaluation or an integrative assessment of more than one element. Hence, the condition of the state of a topic may require the evaluation of more than one criterion, requiring the use of composite indicators. For some interrelated information, composite indicators are a feasible alternative. They are the result of individual indicators being compiled into a single index to summarize complex and multi-dimensional situations [95]. Some well-known examples of composite indicators in the energy field are the energy poverty and energy access indicators. For energy deployment decisions, policymakers require to know the state of the energy sources, which involves different concerns. That may require an aggregated indicator. For this example, aggregated and individual indicators may be required to contribute to providing information about the energy sources, including why and how they were successful, the social perceptions around them, together with the economic details and other relevant concerns [24]. When searching the corresponding indicators for every condensed concern through the retrieved documents, some concerns could not be matched through the recovered documents. A detailed search was done to find at least one indicator for the concerns. This led to finding additional documents listed in Appendix B. For this search, the concerns themselves were used as keywords, together with the term "indicator", "assessment" and "evaluation".

Indicator Identification
In Section 4 all the extracted concerns were organized into condensed concerns, for which the indicator identification process was followed, identifying, at least one indicator for every condensed concern. The end objective of each concern has been enlisted, and at least one indicator as an assessment metric of each objective has been identified. Table 14 shows the indicators for the condensed concerns with an economic orientation. For all the concerns, more than one indicator has been identified.  Table 15 shows the proposed indicators for the condensed concern with environmental orientation, environmental sustainability, for which three alternatives were identified; the proposed indicators for the two condensed concerns with social orientation are shown in Table 16; the indicators for the Governance effectiveness and efficacy condensed concern are shown in Table 17.  [89].
Composed by GHG emissions from energy production over GDP  [87].
Indicators that Summarize dataset of the quality of governance 6. Discussion

Contributions of the Review
Wide variety of documents (66 documents) that meet the eligibility criteria have been analyzed. Each of them presents an alternative to characterize energy policies. The data gathered derives from multiple sources that attempt to give a solution to the same question: how to evaluate energy policy effectiveness?
The results generated a wider perspective on the different concerns that assess energy policy effectiveness. Four core categories (Economic, Environmental, Social, and Institutional) were organized and synthesized through more specific categories (eight condensed concerns organized by their ends objective), presented in Figure 2. It was found that five documents of the review [21,27,30,32,52] consider at least one extracted concern under each of the four core categories, being the most complete documents presented in this review. Nevertheless, no single document covered the seven condensed concerns that have been identified. Therefore, our results are an expansion of those approaches, giving the possibility to apply a more integral evaluation.
The identification of the core metrics of the concerns was also done, leading to the exploration of the energy policy indicators spectrum. The main contribution of this review is the organization of attributes focused on assessing energy policy effectiveness in a more detailed way. The Energy Trilemma [99] metrics have been very useful to identify options to assess some of the identified condensed concerns; however, the Trilemma lacks the consideration of one of the core categories resulting from this review, the Institutional one.
The potential of the data from this review gives the possibility to construct more complete and integral assessment procedures and to reframe already existing energy policy evaluation methodologies with the addition of new attributes (and its metrics).
This review can be directly useful for the national and international current governance situation, in which different levels of authority are implementing approaches for climate change mitigation, whereas EP evaluation is required, e.g., for the SDGs achievement assessment, a set of indicators currently exists [106]. However, a policy effectiveness evaluation tool or methodology (in coherence with the SDGs) may contribute to assess in more detail every policy. Therefore, this report contributes to establishing a base for the future development of a policy effectiveness evaluation methodology that supports the alignment of policies to achieve the SDGs or any other international, national, or local frameworks.

Limitations of the Review
The search considered the key search engines Elservier, Google, Google scholar, and JSTOR, so in principle, all major publications obeying the criteria were covered. However, theoretically, it could happen that the consideration of additional databases (e.g.,CORE, Bielefeld Academic Search Engine (BASE), Directory of Open Access Journals (DOAJ), Social Science Research Network (SSRN), World Wide Science, Semantic Scholar, etc.) could provide some additional results. In addition, the nonpublished documents (studies non-accepted for publication) have not been reviewed; however, this situation does not pose a serious threat to the validity of the review [107]. The search protocol applied has a reasonable degree of easiness and reachability, resulting in a sound number of documents that could be gathered.
The data generated is suitable to be analyzed with statistical methods or other quantitative techniques as HOMA [108] or MASEM [109]. Meta-analysis techniques have not been applied in this study, which has a qualitative nature. However, useful information may be retrieved in the future by such kind of analysis.

Conclusions
This review aimed to characterize the policy evaluation practices in the Energy field, with a focus on the metrics: concerns, objectives, and indicators. In addition to efficacy and efficiency, an emphasis was put into trying to find indicators that can assess effectiveness. Effectiveness enables a more holistic, field-tested, and modulated assessment of the achievements.
It was found that from the different categories (Environmental, Economic, Social, and Institutional), the Economic category is the one most often included, being present in nearly all documents (36 out of 66). Ex post is the most frequent evaluation type in terms of the moment (34 out of 66). Regarding the level of disaggregation reported for EP evaluations, local studies were found in three documents, while the national ones had the lead (40 out of 66). Considerations like the stakeholder's participation, evaluation motivation, and methodological approach varied with less focus across the gathered documents.
The extracted concerns were condensed in a more compact list that eliminated redundancies, organizing them by their end objective. This resulted in seven condensed concerns: accessibility; affordability; competitiveness; impact on climate change; other impacts on the environment; equity; and health.
A total of 15 indicators were identified as possible alternatives to fully characterize the concerns: 2 for affordability, 2 for accessibility, 3 for economic competitiveness, 3 for environmental impacts, 2 for equity, 1 for health, and 2 for governance effectiveness and efficacy. This list is a considerable expansion from what could be found in any single previous document. It is proposed as a holistic assessment of the effectiveness of energy policies. The information organized through this review could be the object of further future works, e.g., on the computation of every single indicator-either ex ante or ex post, but with the goal of enabling ex ante estimation as much as possible. This would further support the elaboration of solid policies, adapted to the local institutional and geographical contexts.

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix B
The indicators' documents with general energy policy goals from which the indicator identification analysis was initially done is presented in Table 1. The additional energy policy indicators' documents are registered in Table 2.