How Efficient Is the Cohesion Policy in Supporting Small and Mid-Sized Enterprises in the Transition to a Low-Carbon Economy?

Funds from the European Union that are devoted to fostering a low-carbon economy are aimed at assisting Member States and regions in implementing the required investments in energy efficiency, renewable energy, and smart distribution electricity grids, and for research and innovation in these areas. In this context, we assessed the implementation of these funds in small and medium-sized enterprises across different beneficiary countries and regions of the European Union. Therefore, this study uses a non-radial slack-based data envelopment analysis model coupled with cluster analysis that covers multiple aspects of evaluation, including two inputs and two outputs, to assess 102 programs from 22 countries. Overall, we were able to ascertain that there are 25 efficient operational programs that remain robustly efficient, whereas 51 remain robustly inefficient for data perturbations of 5 and 10%. Under the current output level, there was almost no input surplus. Therefore, to promote a low-carbon economy, operational program managers should concentrate on solving the problems behind the poor results achieved, both in terms of greenhouse gas emissions reduction and the pace of the programs’ implementation.


Introduction
Curbing the impact of climate change represents one of the major concerns of the current European Union (EU) policy agenda. The 2050 climate and energy framework, which comprises EU-wide targets and policy objectives, proposes cuts of at least 55% in greenhouse gas (GHG) emissions (from 1990 levels) and a 40% utilization of renewable sources in the EU's energy mix, set to be achieved by 2030. Hence, the EU strategy for tackling the impacts of climate change requires transitioning to a low-carbon economy (LCE). Simply put, an LCE is an economy where the structure is supported by activities associated with low levels of carbon dioxide emissions into the atmosphere [1]. In this framework, the EU policies, primarily its cohesion policy, provide support in the form of the promotion of low-carbon investments. The number and quality of low-carbon projects, besides the required structures and capacities of regions, will depend upon coordinated national and regional policies and actions. In addition, the significance of cohesion policy programs is extremely reliant on the global funding that has been assigned to this low-carbon thematic identified, thus providing additional information that can be used in the intermediate monitoring and process evaluations; (3) it considers the distinct regional characteristics of the OPs, i.e., it accounts for the differences between the regions that have below 75% of the EU's gross domestic product (GDP), which are eligible for considerably greater financial assistance from the cohesion policy (less developed regions), the regions just above this threshold, i.e., regions of transition (75% ≤ GDP ≤ 90%), and more developed regions (GDP > 90%), which are grouped into three distinct clusters. This paper is structured as follows. Section 2 presents a literature review on the cohesion policy program evaluations that specifically address LCE concerns, also describing works that have applied DEA in the assessment of LCE efficiency in a broad context. Section 3 clarifies the key definitions and concepts regarding the methodology used to evaluate the implementation of the OPs under scrutiny. Section 4 describes the main assumptions considered in the choice of the input and output factors that are used in the efficiency evaluation of OPs, also providing some statistical information on the data used. Section 5 examines the results obtained and explores potential policy implications. Section 6 presents our conclusions and proposes potential recommendations based on the results obtained.

Literature Review
Ever since 2015, more than 1000 assessments have been conducted by EU MS, focusing on distinct funds, themes, and regions, monitoring the progress of implementation, and/or evaluating the impact of interventions, both referring to the 2007-2013 and 2014-2020 programming periods [13].
The number of evaluations conducted by MS differs broadly (see Figure 1). This is the result of considerable disparities in the amount and type of investment funding, the number of programs in each MS, and the methodology suggested in the assessment plans (see Figure 1). Besides, while some countries prefer to conduct a large number of smaller assessments, others prefer to conduct aggregate evaluations.

Number of evaluations
Methodologies employed Most of these assessments are focused on implementation matters and evaluate progress regarding target achievement, being mainly concerned with the alignment of the projects and actions with the programs' objectives and with the effectiveness and  IT  DE  CZ  ES  FR  UK  HU  RO  LT  GR  SK  AT  SE  BG  NL  IE  LV  PT  EE  HR  SI  DK  BE  CY  Most of these assessments are focused on implementation matters and evaluate progress regarding target achievement, being mainly concerned with the alignment of the projects and actions with the programs' objectives and with the effectiveness and efficiency of their implementation. They also assess whether the existing funding is spent wisely or not and if the targets that were established, particularly those of the performance framework, are achieved. The impact assessments are performed later in the program cycle when most actions have already taken place and have also generated impacts. The distribution of assessments across thematic objectives (TO) by country is depicted in Figure 2. As can be seen from Figure 2, most of the evaluations focus on social themes (TO 8, TO 9 and TO 10).
cycle when most actions have already taken place and have also generated impacts.
The distribution of assessments across thematic objectives (TO) by country is depicted in Figure 2. As can be seen from Figure 2, most of the evaluations focus on social themes (TO 8, TO 9 and TO 10).
Since the MS have dedicated about EUR 60 billion from European structural and investment (ESI) funds to the investment in LCE during 2014 to 2020 (more than twice the amount spent in this area during the previous funding period), our study will be devoted to the assessment of OPs within the framework of TO 4. In this context, Table A1 (available in Appendix A) lists the cohesion policy program evaluations completed by the EU MS, which specifically mention LCE concerns in their abstract summary from 2015 to the present. Although an analysis of Table A1 suggests that the MS are using systematic methods in their assessments of TO 4, namely, desk research, monitoring data/data analysis, interviews, focus groups/facilitated workshops, surveys, and case studies, only a few assessments (mainly devoted to impact evaluations) consider more robust methods (such as statistical methods or other techniques), showing that despite the commitment of MS to enhancing the evaluation of cohesion policy, there is still room for improvement, in terms of the methodological approaches that can be employed.
For example, none of the reports reviewed in Table A1 apply the DEA method in their assessments. When performing an efficiency evaluation using the DEA method, MA will be capable of identifying the OP benchmarks, throughout the programming period, in terms of best practices and the necessary adjustments that need to be made regarding the group of indicators of the performance framework that will enable inefficient OPs to become efficient. Besides, the DEA method has also been used in the assessment of LCE efficiency at national, regional and sectoral levels-see Table 1. Most of the studies reviewed in Table 1 use the SBM-DEA model but do not account for the robustness assessment or uncertainty handling of the results obtained. In addition, to the best of our TO 5-Climate change adaptation, risk prevention and management; TO 6-Preserving and protecting the environment and promoting resource efficiency; TO 7-Sustainable transport and key network infrastructures; TO 8-Employment and labor mobility; TO 9-Social inclusion, combating poverty and any discrimination; TO 10-Education, training and vocational training for skills and lifelong learning; TO 11-Enhancing the institutional capacity of public authorities and stakeholders, and efficient public administration.
Since the MS have dedicated about EUR 60 billion from European structural and investment (ESI) funds to the investment in LCE during 2014 to 2020 (more than twice the amount spent in this area during the previous funding period), our study will be devoted to the assessment of OPs within the framework of TO 4. In this context, Table A1 (available in Appendix A) lists the cohesion policy program evaluations completed by the EU MS, which specifically mention LCE concerns in their abstract summary from 2015 to the present.
Although an analysis of Table A1 suggests that the MS are using systematic methods in their assessments of TO 4, namely, desk research, monitoring data/data analysis, interviews, focus groups/facilitated workshops, surveys, and case studies, only a few assessments (mainly devoted to impact evaluations) consider more robust methods (such as statistical methods or other techniques), showing that despite the commitment of MS to enhancing the evaluation of cohesion policy, there is still room for improvement, in terms of the methodological approaches that can be employed.
For example, none of the reports reviewed in Table A1 apply the DEA method in their assessments. When performing an efficiency evaluation using the DEA method, MA will be capable of identifying the OP benchmarks, throughout the programming period, in terms of best practices and the necessary adjustments that need to be made regarding the group of indicators of the performance framework that will enable inefficient OPs to become efficient. Besides, the DEA method has also been used in the assessment of LCE efficiency at national, regional and sectoral levels-see Table 1. Most of the studies reviewed in Table 1 use the SBM-DEA model but do not account for the robustness assessment or uncertainty handling of the results obtained. In addition, to the best of our knowledge, the SBM-DEA method has not been employed for the evaluation of the implementation of OPs under the LCE theme. Hence, we will specifically address the efficiency assessment of these sorts of OPs through the application of the SBM-DEA method, combined with cluster analysis, and we will also perform robustness and sensitivity analyses of the results obtained.
A three-stage approach (1) SBM-DEA model; (2) stochastic frontier approach to eliminate the impacts of external environment variables on these slacks; (3) re-estimation of efficiency with adjusted inputs and outputs.

Methodology
In this work, we use the DEA model proposed by Tone, known as SBM [21]. In comparison to the Charnes-Cooper-Rhodes (CCR) and Banker-Charnes-Cooper (BCC) models (see [22,23]), this model provides a more comprehensive efficiency assessment tool because it is non-radial (assuming that inputs and outputs can vary in a non-proportional manner), and it can be input-, output-and non-oriented. Unlike radial models, which disregard slacks, the SBM model provides information on the enhancements required for the values of each inefficient decision-making unit (DMU)'s input and output, respectively. Differently from the additive model [24], which is also a non-radial efficiency measure model, the SBM model allows computing an efficiency score based on the slacks. Besides this, the SBM model can be coupled with clustering analysis by grouping DMUs according to certain characteristics, to evaluate the DMUs' efficiency based on the cluster frontier, thus reducing the impact of DMUs' heterogeneity on efficiency [25][26][27].

The SBM Output-Oriented Model
Consider the set of n DMUs (DMU 1 , DMU 2 , . . . , DMU n ), where X = [x ij , i = 1, 2, . . . , m, j = 1, 2, . . . , n] is the (m × n) matrix of inputs, Y = [y rj , r = 1, 2, . . . , s, j = 1, 2, . . . , n] is the vector of outputs (s × n) and the rows of these matrices, corresponding to the inputs and outputs of DMU k , are respectively given by x T k and y T k , with T denoting the transposing of a vector. The SBM output-oriented model can be given as [27]: Model (1) employs the constant returns to scale (CRS) assumption. To consider variable returns to scale (VRS), it is only necessary to add the constraint e T λ = 1 to model (1).
In this context, the scale efficiency (σ k ) of DMU k can be given as [27]: where ρ CRS k and ρ VRS k correspond to the efficiency scores of DMU k , based on the SBM model under CRS and VRS, respectively. The value of σ k ranges between 0 and 1, with a larger value indicating a better scale condition.

Let x 1
i = x ik − s − i and y 1 r = y rk + s + r . Tone [28] suggested the following SBM outputoriented model to evaluate the super-efficiency of DMU k : The super-SBM-output-oriented model assesses the efficiency of an efficient DMU regarding the nearest point on the frontier except itself [28,29].
The optimal value of the objective function of model (3) is greater than or equal to 1, i.e., ρ * 1 ≥ 1. Nevertheless, the super-efficiency for DMU k will be 1 even if DMU k is inefficient. Thus, to determine whether DMU k is inefficient or not, both models (1) and (3) must be solved. If DMU k is efficient according to model (1), model (3) should be used to obtain its super-efficiency score.

The SBM Model with Cluster Benchmarking
Traditional DEA models assume that all DMUs have similar characteristics. Therefore, all DMUs are considered to provide the reference set to construct meta-frontiers. In reality, DMUs are not always homogeneous, thus affecting the accuracy of DEA findings [30]. Hence, clustering benchmarking can be useful to deal with heterogeneous DMUs [25,26]. Cluster benchmarking is a technique used for separating a set of DMUs into groups (i.e., clusters) with specific characteristics. The clusters can be defined by using a clustering Sustainability 2022, 14, 5317 8 of 55 method (in the field of statistics) applicable to the problem under evaluation or provided exogenously, using experts' knowledge, or established internally according to the degree of scale efficiency [27]. The DMUs that belong to the same cluster are more alike than those that belong to different clusters [26]. The main goal is to take advantage of the homogeneity of DMUs in the same cluster and the heterogeneity of DMUs in distinct clusters. To perform the efficiency assessment regarding the best practices according to the corresponding clusters, the production frontiers need to be built separately. By comparing the results of the non-grouped DEA model (i.e., by considering all DMUs as a reference set) and the results of self-benchmarking after grouping, the technology gap ratio (TGR) can be obtained. The meta-frontier and the cluster frontiers are obtained considering the output-oriented version of the SBM model (see Model (1)). The TGR k of DMU k is then calculated as [31]: where ρ k meta (VRS) * is the SBM-output-efficiency value of DMU k , computed according to the meta-frontier under VRS, and ρ k cluster (VRS) * is the SBM-output-efficiency value of DMU k obtained according to the cluster frontier under VRS.
The value of the TGR may identify the gap between the cluster frontier and the metafrontier. It is used to evaluate the technical efficiency gap of the same DMU, according to distinct frontiers. Besides this, TGR can further indicate the need for separating different groups [31]. The lower the TGR value is, the bigger the requirement of grouping will be, and vice versa. Since ρ k cluster (VRS) * ≥ ρ k meta (VRS) * , the value of TGR k varies between 0 and 1 [25,26]. A TGR k closer to 1 suggests that there is a small gap between the meta-frontier and the cluster frontier. The meta-frontier shows the underlying technical level of the entire assessed group of individuals, and the cluster frontier depicts the real technical level of each cluster. For example, if we assume that ρ k meta * = 0.6 and ρ k cluster * = 0.8, this implies that the maximum output that could be produced by the DMU k , which belongs to the cluster under analysis, is 75% of the output that is feasible when using the meta-frontier as a benchmark. The higher the value of TGR k , the smaller the gap between the meta-frontier and the cluster frontier and the smaller the gap between the technology used by the DMU and the technology frontier.
Expression (4) allows us to obtain the following decomposition of the efficiency of DMU k for a particular input-output combination: Expression (5) shows that the efficiency of DMU k , measured according to the metafrontier (expressing the current state of knowledge), can be split into the product of the efficiency obtained for the cluster frontier (following the current state of knowledge and the characteristics of the cluster under analysis) and the TGR for the cluster under analysis (which evaluates how close the cluster frontier is to the meta-frontier). Finally, since both clusters and scale efficiencies might significantly influence the outcomes attained, Expression (5) can be further decomposed into: where ρ k cluster (CRS) * is the efficiency value of DMU k obtained according to the cluster frontier under CRS and σ k cluster * is the scale efficiency found for the cluster under evaluation. Hence, if σ k cluster * = 1, then DMU k has no scale demerits, and its slacks are imputed to itself [27]. For example, if σ k cluster * = 0.25, then 75% of the slacks are ascribed to the DMU's scale demerits.

Data and Assumptions
We involved the stakeholders in the choice of a pre-selected set of indicators (the pre-selected set of indicators presented to the stakeholders was based on the scientific literature and on the set of common indicators used by the European Commission in the evaluation of TO 4), by conducting a facilitated workshop on the topic "Evaluating the Co-financed Intervention Policies in Enterprises" with specific policymakers and MA. Besides this, we performed a separate evaluation of regional and national programs, which were grouped into clusters according to the categories of the regions under analysis (i.e., less developed regions, regions of transition and more developed regions). The values considered are cumulative values from different years released on the 19th of November 2021 since these are the most updated data available for the achievement indicators. In our assessment, we have only studied the programs with no missing information (i.e., programs with missing data were eliminated), leading to the consideration of 22 countries and 102 programs (the DMUs).
The input and output factors considered for performing an efficiency assessment of the implementation of the ESI funds devoted to LCE interventions in SMEs were selected from the list of common indicators that are legally required by the EU [32] and are described below (see also Table 2).

Financial Absorption Capacity
According to [33], an efficiency assessment of the deployment of EU structural funds should specifically involve consideration of the capacity of absorption of EU funds by a given region/group of regions/countries. The financial absorption capacity corresponds to the capacity of co-financing by the MS [34]. A higher financial absorption capacity means lower dependency on EU co-financing of the country/region. Besides this, a rise in the EU co-financing rate relieves the burden on the national budget by reducing the MS's initial budgeted contribution to the OP but also diminishes the OP's total value. This indicates that the scale (number or size) of the interventions will be lowered as well, unless the MS keeps investing the funds originally envisioned at the national level. As a result, this may have a negative effect on the aims and outcomes that the OPs may accomplish, which, in turn, impacts the value that the awarded EU funds can deliver [35]. Therefore, the "percentage of EU co-financing", which measures the weight of EU financing on the total financing received (including the national financing), is used as a proxy to measure the financial absorption capacity. To maximize this, the percentage of EU co-financing in total financing should be minimized and, therefore, should be considered as an input [8].

The Pace of the Programs' Implementation
Since its inception, efforts have been made to evaluate and improve the implementation of the cohesion policy [36]. In fact, the goal of enhancing financial performance, or spending capacity, in MS and regions implementing OPs is fundamental. Although the pace of implementation is only one indicator of effectiveness, the extent to which MS and MA are capable of spending their allocated funding effectively and efficiently provides a basic method for assessing the implementation progress.
The assessment of financial implementation is critical because deferred or irregular financial performance in the first half of the programming period generates pressures on MA that may result in a negative effect on efficient and effective OP implementation and closure [36].
Therefore, the financial execution of structural funds, which is a necessary condition for effective policy implementation, should also be incorporated into the analysis, particularly when addressing the pace of the programs' implementation [37]. In this framework, costs that are not eligible or in agreement with the applicable eligibility rules cannot be claimed. These must be validated by an accredited controller; this is the body or person accountable for validating, at the national level, that the co-financed products and services have been delivered, that the related expenditures have been paid and that a particular project conforms to the applicable EU program and national rules. On the one hand, the "total eligible spending" refers to those eligible costs reported by the selected projects that have been validated by this controller. Hence, the higher its value, the higher each project's financial execution, and, thus, it is used herein as an output. On the other hand, the "eligible costs decided" refer to those costs that have financial resources assigned to the projects selected for funding (project pipeline). Therefore, these should be minimized (and perceived as inputs), so that the pace of the programs' implementation is further enhanced.

Energy and Climate Change
TO 4 encourages the transition to an LCE. This goal is driven first and foremost by the climate agenda, which seeks to reduce the harmful impacts of anthropogenic GHG emissions. The scope is largely focused on a shift that necessitates both physical changes (lower net emissions from a technical infrastructure) and behavioral changes (energy savings), both of which are supported by new technologies and solutions [39].
The investment priority devoted to SMEs (investment priority 4b) is focused on promoting energy efficiency and renewable use, with two types of intervention [38,39].
For energy efficiency measures, the values are computed according to the amount of primary energy saved through the adoption of the supported operations in a given year (either one year after project completion or the calendar year after project completion) [39]. The saved energy is expected to replace non-renewable energy generation. The MS's total GHG emission per unit of non-renewable energy output is used to calculate the GHG effect of non-renewable energy [39].
In the case of renewable energy production, the estimated values are computed according to the quantity of primary energy produced by the supported facilities in a particular year (either one year after project completion or the calendar year after project completion) [38]. Renewable energy is expected to be GHG-free [39].
Either way, "GHG reduction" is an indicator that is specifically calculated for interventions directly aiming to increase renewable energy production or to decrease energy consumption through energy-saving measures. This indicator should be maximized and is employed as an output.
Data on these indicators are given in Table A2 in Appendix B.
From an analysis of Table 3, it might be established that the average EU co-financing support was 60.77%, 78.672% for the less developed regions, 61.43% for the regions of transition and 49.10% for the more developed regions, respectively. The overall average financial execution rate (i.e., the ratio between the total eligible spending and the total eligible cost) is low (32.33%) and it is even lower for the regions of transition (31.2%), presenting slightly higher values (but still below 50%) for the more developed regions and less developed regions, with values of 39.6% and 39.2%, respectively. Finally, the highest average decrease of GHG is attained (in decreasing order of magnitude) for the regions of transition, followed by the less developed and more developed regions. A requirement of DEA refers to input and output factors that should hold an isotonic relationship, which can be validated by correlation analysis [40]. There is a need to guarantee that the link between inputs and outputs is not inconsistent. Increasing the value of any input whilst holding other factors constant should not diminish any output but should rather lead to an expansion in the value of at least one output. If the correlation between input and output factors is positive (and significant), this indicates that the factors verify an isotonic relationship. In this case, since the normality assumption for the application of the tests for the significance of Pearson's correlation is not verified, we opted for obtaining the Spearman correlation coefficients and the corresponding significance tests-see Table 4. The results obtained corroborate the isotonic relationship of the inputs and outputs used in the analysis.

Discussion of Results
The results were computed with the MaxDEA 8 Ultra software. The values of the TGR, together with the technical efficiencies (CRS and VRS) obtained for the distinct region categories and the meta-frontier, were computed for all OPs. Basic descriptive statistics for these measures are shown in Table 5.  Figure 3 depicts the efficiency scores based on the meta-and cluster frontiers, respectively.  Figure 3 depicts the efficiency scores based on the meta-and cluster frontiers, respectively.
It is worth noting that the cluster frontiers of more developed regions and transition regions were almost tangential to the meta-frontier. The average values of the TGR vary from 0.392 (for less developed regions) to 0.896 (for more developed regions), suggesting the existence of a huge gap between the two frontiers, particularly in the case of the less developed regions (based on the cluster frontier, the number of efficient DMUs-score equals to 1-increased from 2 to 13). Overall, the number of OPs considered to be efficient increased from 19 (meta-frontier) to 35 (cluster frontier)-see Figure 4. This result is mostly influenced by the TGR reached by LCE OPs in less developed regions, which produce, on average, only about 39.2% of the potential output, given the technology available for this type of OP as a whole (22 countries are represented)-see Table 5. Nevertheless, more developed regions produce, on average, 89.6% of the potential  It is worth noting that the cluster frontiers of more developed regions and transition regions were almost tangential to the meta-frontier. The average values of the TGR vary from 0.392 (for less developed regions) to 0.896 (for more developed regions), suggesting the existence of a huge gap between the two frontiers, particularly in the case of the less developed regions (based on the cluster frontier, the number of efficient DMUs-score equals to 1-increased from 2 to 13). Overall, the number of OPs considered to be efficient increased from 19 (meta-frontier) to 35 (cluster frontier)-see Figure 4.  From an analysis of Figure 5, it can be ascertained that the difference in the average EU co-financing between efficient and inefficient OPs was not significant. Although the mean funds devoted to the eligible cost of efficient OPs (EUR 1,126,456,826.26) were slightly higher than that of inefficient ones (EUR 881,248,217.61), the mean eligible spending of efficient OPs (EUR 477,458,910.63) was substantially higher than that of inefficient ones (EUR 283,346,601.70). Furthermore, the mean reduction of GHG emissions of efficient OPs was 248% higher than that attained by inefficient OPs. This result is mostly influenced by the TGR reached by LCE OPs in less developed regions, which produce, on average, only about 39.2% of the potential output, given the technology available for this type of OP as a whole (22 countries are represented)see Table 5. Nevertheless, more developed regions produce, on average, 89.6% of the potential outputs (Table 5), whereas the regions of transition produce, on average, 77% of the potential outputs (Table 5). Figure 4 shows the number of OPs at different subintervals of super-efficiency scores, based on cluster-and meta-frontiers, respectively. According to the cluster frontiers, 35 OPs were relatively efficient, suggesting that 34.31% of these were maximizing their outputs. Overall, more developed regions and regions of transition show higher room-for-improvement potentials of 57% and 51%, respectively, while less developed regions had a smaller room for improvement of 48%. Besides, on average, about 10%, of the adjustments required to make the non-efficient OPs of less developed regions efficient are due to scale demerits, whereas 16% and 14% are attained for the regions of transition and more developed regions, respectively.
From an analysis of Figure 5, it can be ascertained that the difference in the average EU co-financing between efficient and inefficient OPs was not significant. Although the mean funds devoted to the eligible cost of efficient OPs (EUR 1,126,456,826.26) were slightly higher than that of inefficient ones (EUR 881,248,217.61), the mean eligible spending of efficient OPs (EUR 477,458,910.63) was substantially higher than that of inefficient ones (EUR 283,346,601.70). Furthermore, the mean reduction of GHG emissions of efficient OPs was 248% higher than that attained by inefficient OPs.

Potential Improvements
The SBM model allows us to identify in inefficient DMUs the potential improvements that inputs and outputs should undergo to become efficient. These outcomes are illustrated in Figures 6, 7 and 8, sorted both by operational program and by region category. The decrease in GHG emissions has the greatest improvement potential, which could improve by about 514% (i.e., the reduction of GHG emissions can increase on average from 16,719 to 102,577 tonnes of CO2 eq.) according to the current input levels. Either way, the regions of transition and the more developed regions show the highest room-for-improvement potentials of 641% and 553%, respectively, whereas the lowest improvement potential belongs to the less developed regions, i.e., 278%. The pace of the programs' implementation also shows an overall room for improvement (27%), particularly for the regions of transition (49%), followed by more developed and less developed regions, with potentials for improvement of 20% and 18%, respectively. Under the current output level, there was almost no input surplus, although more developed regions show better potential for improving their dependence on EU co-financing (−7%) and eligible costs (−8%). Hence, to foster an LCE, OPs managers should focus on solving problems that further enhance both GHG reduction and the pace of the programs' implementation.

Potential Improvements
The SBM model allows us to identify in inefficient DMUs the potential improvements that inputs and outputs should undergo to become efficient. These outcomes are illustrated in Figures 6-8, sorted both by operational program and by region category. The decrease in GHG emissions has the greatest improvement potential, which could improve by about 514% (i.e., the reduction of GHG emissions can increase on average from 16,719 to 102,577 tonnes of CO 2 eq.) according to the current input levels. Either way, the regions of transition and the more developed regions show the highest room-for-improvement potentials of 641% and 553%, respectively, whereas the lowest improvement potential belongs to the less developed regions, i.e., 278%. The pace of the programs' implementation also shows an overall room for improvement (27%), particularly for the regions of transition (49%), followed by more developed and less developed regions, with potentials for improvement of 20% and 18%, respectively. Under the current output level, there was almost no input surplus, although more developed regions show better potential for improving their dependence on EU co-financing (−7%) and eligible costs (−8%). Hence, to foster an LCE, OPs managers should focus on solving problems that further enhance both GHG reduction and the pace of the programs' implementation.     Tonnes of CO2 eq.) Decrease of GHG (Tonnes of CO2 eq.) Projection(Decrease of GHG (Tonnes of CO2 eq.))

Robustness Analysis
Traditional DEA methods use crisp values for both the inputs and outputs. Nevertheless, in real-world problems, the values used to instantiate the input and output data are occasionally subject to uncertainty. In this context, imprecise or vague data can be conveyed within interval ranges, as ordinal (rank-order) data or as fuzzy numbers [41]. Out of the approaches available in the literature that include the applications of fuzzy set theory in DEA, the α-level approach is possibly the most popular one. This approach consists of converting the fuzzy DEA model into a pair of parametric programs to find the lower and upper bounds of the α-level of the membership functions of the efficiency scores. The MaxDEA software converts the DEA model that has fuzzy inputs/outputs with bounded intervals into a pair of standard DEA models, so that the lower and upper boundaries of the efficiency scores are computed.
This sort of analysis is particularly important to evaluate the robustness of the results obtained, particularly if the programs are still underway. Hence, it is possible to anticipate the impact that potential changes on the output levels might have on the efficiency scores of OPs, given their current levels of input. In particular, we consider that the perturbations in the value of each factor are within an interval range. This interval is found by applying a common tolerance to the output factors, such that = (1 − ) ≤ ≤ (1 + ) = , where L and U designate the lower and upper boundaries, respectively. Besides this, we simultaneously consider data perturbations according to a worst-case scenario and a best-case scenario. While the former presumes increased outputs for all other DMUs and decreased outputs for the DMU under assessment (i.e., the efficiency of DMUk declines and the efficiency of all the other DMUs improve), the latter supposes the reverse situation. In this context, a DMU is robust to changes in its output factors if it remains efficient (or inefficient), and it thus can be stated as robustly efficient (or robustly inefficient) for the tolerance considered.
If we consider a potential change of the outputs within the tolerance of δ = 5% with the current inputs, 27 programs remain robustly efficient (9, 10 and 8 belong to more developed, transition, and less developed regions, respectively), 59 remain robustly

Robustness Analysis
Traditional DEA methods use crisp values for both the inputs and outputs. Nevertheless, in real-world problems, the values used to instantiate the input and output data are occasionally subject to uncertainty. In this context, imprecise or vague data can be conveyed within interval ranges, as ordinal (rank-order) data or as fuzzy numbers [41]. Out of the approaches available in the literature that include the applications of fuzzy set theory in DEA, the α-level approach is possibly the most popular one. This approach consists of converting the fuzzy DEA model into a pair of parametric programs to find the lower and upper bounds of the α-level of the membership functions of the efficiency scores. The MaxDEA software converts the DEA model that has fuzzy inputs/outputs with bounded intervals into a pair of standard DEA models, so that the lower and upper boundaries of the efficiency scores are computed.
This sort of analysis is particularly important to evaluate the robustness of the results obtained, particularly if the programs are still underway. Hence, it is possible to anticipate the impact that potential changes on the output levels might have on the efficiency scores of OPs, given their current levels of input. In particular, we consider that the perturbations in the value of each factor are within an interval range. This interval is found by applying a common tolerance δ to the output factors, such that where L and U designate the lower and upper boundaries, respectively. Besides this, we simultaneously consider data perturbations according to a worst-case scenario and a bestcase scenario. While the former presumes increased outputs for all other DMUs and decreased outputs for the DMU under assessment (i.e., the efficiency of DMU k declines and the efficiency of all the other DMUs improve), the latter supposes the reverse situation. In this context, a DMU is robust to changes in its output factors if it remains efficient (or inefficient), and it thus can be stated as robustly efficient (or robustly inefficient) for the tolerance considered.
If we consider a potential change of the outputs within the tolerance of δ = 5% with the current inputs, 27 programs remain robustly efficient (9, 10 and 8 belong to more developed, transition, and less developed regions, respectively), 59 remain robustly inefficient and the remaining 16 are potentially efficient-see Figure 9. For a tolerance of δ = 10%, 25 programs remain robustly efficient (8, 10 and 7 belong to more developed, transition, and less developed regions, respectively), whereas 51 become robustly inefficient and 26 become potentially efficient. Overall, if the outputs suffer an increase from 5% to 10%, the potential for improvement increases, particularly for less developed regions, from 3% to 17%, whereas for the regions of transition, it just increases from 9% to 10%. Finally, with this change in outputs, more developed regions also have a high room-for-improvement value from 12% to 33%-see Figure 10. Table 6 provides some characteristics of the robustly efficient OPs for both tolerances. From an analysis of this table, it can be seen that the three OPs that are more often viewed as benchmarks in their corresponding clusters are also those more often viewed as benchmarks in the meta-frontier (these are OPs from Ireland, Italy, and Spain). Besides this, it can also be concluded that less developed regions are the ones that usually prefer to perform more intermediate assessments (see, e.g., the cases of the OPs from Poland and Lithuania). There is only one operational program belonging to a less developed region (see the case of Lithuania) that also serves as a benchmark for the other programs in the meta-frontier (but this is just one time). Finally, it can also be established that despite the variety of the number of evaluations conducted by each MS for each robustly efficient operational program, most of them perform some kind of evaluation. Specifically, the majority of MS perform monitoring (43) and process (34) evaluations, with only a few (6) presenting impact evaluations.

Sensitivity Analysis
Since the SBM-DEA approach is a non-parametric method, a distinct approach is usually used to perform sensitivity analysis. This consists of eliminating one factor of evaluation (input or output) at a time and assessing the level of change attained thereby in terms of efficiency [42].
In addition, because the SBM-DEA model used in the assessment of the OPs is outputoriented, we have considered two simple regression models in which the dependent variables are the scores obtained when omitting one output at a time, and the independent variable is the original score. Through these models, it is possible to obtain the slope and the corresponding coefficient of determination (or r-square). The sensitivity of efficiency to the changes in the variables (i.e., outputs) can thus be identified by the gap between the value 1 and the slope of the simple regression function [25,43], meaning that the sensitivity of efficiency to changes in the outputs increases with this gap.
The results of the sensitivity analysis are presented in Figure 11 and Table 7. The factor that shows higher impacts on efficiency is GHG reduction, since the omission of this variable leads to the highest value of |1-slope|, 0.590, whereas the total eligible spending has a smaller impact on efficiency. These results suggest that the type of interventions targeted for funding are critical for producing desirable results regarding an LCE.

Sensitivity Analysis
Since the SBM-DEA approach is a non-parametric method, a distinct approach is usually used to perform sensitivity analysis. This consists of eliminating one factor of evaluation (input or output) at a time and assessing the level of change attained thereby in terms of efficiency [42].
In addition, because the SBM-DEA model used in the assessment of the OPs is output-oriented, we have considered two simple regression models in which the dependent variables are the scores obtained when omitting one output at a time, and the independent variable is the original score. Through these models, it is possible to obtain the slope and the corresponding coefficient of determination (or r-square). The sensitivity of efficiency to the changes in the variables (i.e., outputs) can thus be identified by the gap between the value 1 and the slope of the simple regression function [25,43], meaning that the sensitivity of efficiency to changes in the outputs increases with this gap.
The results of the sensitivity analysis are presented in Figure 11 and Table 7. The factor that shows higher impacts on efficiency is GHG reduction, since the omission of this variable leads to the highest value of |1-slope|, 0.590, whereas the total eligible spending has a smaller impact on efficiency. These results suggest that the type of interventions targeted for funding are critical for producing desirable results regarding an LCE.

Policy Implications
Although 50% of the OPs remain robustly inefficient for both tolerances, from the analysis of Table 8 (further information is obtainable from Tables A3 and A4 in Appendices C and D, respectively), it can be concluded that 73% of the inefficient OPs do not need adjustments in EU co-financing. Nevertheless, there are at least four OPs (from Figure 11. Sensitivity analysis for GHG reduction (a) and total eligible spending (b). The y-axis of each graph represents the recalculated efficiency by omitting one variable at a time, and the x-axis represents the original efficiency score. The solid blue lines represent the lines of the best fit.

Policy Implications
Although 50% of the OPs remain robustly inefficient for both tolerances, from the analysis of Table 8 (further information is obtainable from Tables A3 and A4 in Appendix A, respectively), it can be concluded that 73% of the inefficient OPs do not need adjustments in EU co-financing. Nevertheless, there are at least four OPs (from more developed regions) that require a meaningful reduction of EU co-financing (of between 34% and 54%) to become efficient. Particularly in such cases, the management structures should consider reprogramming the OPs to reallocate EU co-funding from more developed regions to less developed regions. In terms of the funding assigned to eligible costs, 88% of inefficient OPs do not require adjustments; however, there are four funds (3 from more developed regions and 1 from a region of transition) that require a reduction of funding of between 20% and 32% (see Tables A3 and A4). Then again, MA should have enough flexibility to decide how to change the coverage of the funding, thus contemplating other less developed regions and/or other investment priorities, if needed. One of the critical issues that preclude OPs' efficiency is their implementation pace. There are 11 OPs that need to more than double their performance and 7 that need to increase it by more than 75%. In these cases, it is important to understand the true reasons behind the numbers. Some OPs evaluations suggest that some firms have withdrawn their subsidies, probably because of their difficulties in obtaining bank credit for co-financing. In such situations, the MA should be able to support firms in finding other sources of funding, e.g., venture capital funds, business angels and crowdfunding, also facilitating the conditions for bringing other institutional investors. If none of these possibilities is viable, funding should be transferred to regions or other investment priorities with more capacity to spend. Besides, to accelerate the implementation of the OPs, best practices from other countries need to be considered and bureaucratic requirements for accessing funding should be reduced. Management structures should find ways of making project implementation more efficient by promoting the simplification of procedures for preparing and submitting the application for payment requests. The reduction of the implementation failure rate should also involve additional opportunities for direct consultation and better support and guidance of applicants. Finally, it can be ascertained that GHG emission reduction is actually the indicator that needs greater attention since 59 OPs need to improve their performance substantially in terms of this factor. Further analysis is required, namely, regarding the possibility of replacing or flanking the MA with a competent alternative body or even offering specific training and the exchange of best practices to improve the choice of the projects selected for funding. OPs should ensure better links between policy objectives, actions undertaken and indicators, and, if needed, promote greater involvement of key stakeholders in the planning phase.
Overall, the MA should monitor closely the execution of OPs, especially those indicators that have implementation delays, and revise the allocation of funding if needed.

27%
Need to finance projects that make some contribution to meeting climate-related challenges Focus on reducing current energy consumption of businesses; expand the types of renewable sources eligible for funding and define the rules for the joint use of the ERDF with other subsidies.

78%
Need to finance projects that make a large contribution to meeting climate-related challenges Promote greater involvement of key stakeholders in the planning phase.

30,847%
Assess and change investment strategy Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.

Conclusions
This work aims to evaluate the efficiency of the implementation of structural funds in different OPs from distinct EU beneficiary countries and regions by using data provided by the European Commission. These data involve financial implementation and expected results (targets) in the field of LCEs for the 2014-2020 ESI funds devoted to SMEs under the ERDF. In addition to representing the latest reported information available, these data are well suited for use with non-parametric methods, such as DEA. The main advantage of using this type of approach is the richness of information that they can offer to MA regarding the inefficiency of the Ops, which are compared against their peers. Through DEA, the benchmarks of inefficient OPs are also identified and pertinent information can be obtained thereby regarding the best practices to follow to achieve greater efficiency.
Although DEA has indisputable advantages over other traditional approaches (e.g., microeconomic studies, which use control groups, and case study evaluation), there is still a lack of scholarly attention regarding its use in the framework of structural funds efficiency assessment. Therefore, one of the novelties of our research consists of using the SBM-DEA model, combined with cluster analysis, to evaluate the efficiency of the implementation of structural funds dedicated to the promotion of an LCE in SMEs.
The use of the SBM measure employed herein, besides identifying the benchmarks and the adjustments required (given by the slacks) to improve the implementation of these funds, also enables exploiting the differences between the distinct regions' categories in a single model. This approach also involves the use of two approaches in the robustness assessment of the results obtained, something which is not usually performed even in other broader studies within the framework of LCE. This type of analysis is particularly relevant if the programs are still in progress, enabling MA to foresee the impact that potential adjustments on the output levels might have on the efficiency scores of OPs, given their levels of inputs.
Differently from other tools and methods that are specifically used for the ex-post or ex-ante evaluation of cohesion policies, the DEA method also enables us to evaluate the efficiency of OPs' implementation during the programming period, so that the required policies can be adopted within the time needed for making the necessary adjustments during the time horizon in progress.
DEA models can easily be adjusted to evaluate other thematic objectives, as long as the rule of thumb established by [17] is retained, i.e., the number of DMUs under assessment should be at least the double of the number of input and output factors considered.
Overall, regarding the first research question ("Which factors require special attention for reaching an efficient implementation of the funds devoted to fostering an LCE in the EU?"), we were able to conclude that: (1) MA should pay special attention to GHG emission reduction, since 59 OPs need to further improve their performance substantially. In such cases, consideration should be given to the prospect of replacing or supplementing these MAs with a capable alternative organization/group of experts, or perhaps providing particular training and exchanging best practices to improve the selection of projects for funding. (2) The other main important factor that precludes OPs' efficiency is their implementation rate. Specifically, there are 11 OPs that have to more than double their performance and 7 who must enhance it by more than 75%. Either way, it is vital to understand the reasons behind these numbers. There are evaluation reports that suggest that some enterprises have withdrawn their subsidies, most likely due to issues related to bank credit. In this context, MA should be able to assist enterprises in obtaining other funding sources, while also easing the criteria for bringing in other institutional investors. Furthermore, in order to expedite the implementation of the OPs, best practices from other countries should be explored, and bureaucratic barriers to getting financing should be minimized. Management structures should seek ways to improve project implementation by supporting the simplification of procedures for submitting payment requests, also providing enhanced assistance and guidance. Concerning our second research question ("Which OPs were more often viewed as benchmarks during the programming period under evaluation?"), we were able to establish that: (1) The OPs more often viewed as benchmarks (either in the meta-or cluster frontiers) were, by decreasing order of importance, the "Southern and Eastern Regional Programme-IE-ERDF", "Liguria-ERDF", "Multi-regional Spain-ERDF", "Pays de la Loire-ERDF/ESF", "Investments in Growth and Employment-AT-ERDF and "Cantabria-ERDF". (2) There is only one OP from a less developed region (see the case of Lithuania) that also serves as a benchmark (i.e., as a reference of best practices) for the other OPs in the meta-frontier (but just one time).
In what regards our third research question ("Were the OPs robustly efficient in the face of potential changes of the performance framework indicators used?"), we were able to find that: (1) 50% of LCE OPs were robustly inefficient for the tolerances considered in the analysis, highlighting the difficulty of further improving the efficiency of these OPs.
In terms of the fourth research question ("Which type of regions managed to attain higher LCE performance"?), we were able to ascertain that: (1) Our findings appear to support the apparent paradox of the EU Cohesion Policy, since the OPs which are more often viewed as benchmarks according to the meta-frontier belong to more developed regions. In fact, if we had not grouped our OPs according to the region type, we would only have 2 efficient OPs from the less developed regions (instead of the 13 obtained with the cluster frontier analysis). (2) The TGR reached by LCE OPs in less developed regions suggests that these produce, on average, only about 39.2%, of the potential output, given the technology available for this type of OPs in EU countries. Therefore, it appears that there is a positive association between the efficiency of the implementation of the OPs and the more advantageous socioeconomic circumstances of the regions where the OPs are implemented.
Finally, we are also aware that although the performance framework provides a group of common indicators, a match between the data gathered for the achievement indicators and the data from financial implementation is not fully possible. This is particularly true regarding the investment priority devoted to SMEs (investment priority 4b), which is focused on promoting energy efficiency and renewable energies, that has data for the achievement indicators, but it does not have data with that level of detail in the financial implementation. Moreover, the data reported is often lacking, thus leading to the consideration of a smaller number of indicators and OPs in our assessment.
Further research should be conducted to evaluate the efficiency of the implementation of other OPs in different TO, eventually considering other performance framework indicators.

Conflicts of Interest:
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.       Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.

Slightly improve implementation
Promote the simplification of procedures for preparing and submitting applications and for payment requests.
Assess and change investment strategy.
Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding. Assess and change investment strategy.
Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.

Mecklenburg-Vorpommern-ERDF
T EU co-financing should be slightly reduced Allow some flexibility in reallocation of EU co-funding between regions.
No adjustments required for eligible funding

No suggestions
Slightly improve implementation Promote the simplification of procedures for preparing and submitting applications and for payment requests.
Assess and change investment strategy.
Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.

Integrated
Regional Programme-RO-ERDF T EU co-financing should be slightly reduced Allow some flexibility in reallocation of EU co-funding between regions.

Eligible funding
should be reduced Allow more flexibility in reallocation of funding between regions and Investment Priorities.

Strongly improve implementation
Envisage specific training and exchange of best practice for managing bodies to improve implementation.
Assess and change investment strategy.
Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.  Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding. Assess and change investment strategy.
Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.