Next Article in Journal
On the Use of Copula for Quality Control Based on an AR(1) Model
Next Article in Special Issue
Conditional Time-Varying General Dynamic Factor Models and Its Application to the Measurement of Volatility Spillovers across Russian Assets
Previous Article in Journal
A Characterization of GRW Spacetimes
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Efficiency Evaluation of Regional Environmental Management Systems in Russia Using Data Envelopment Analysis

V.A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences, 65 Profsoyuznaya St., 117997 Moscow, Russia
College of Information Technologies and Computer Sciences, National University of Science and Technology “MISiS”, 4 Leninsky Ave., Bldg. 1, 119049 Moscow, Russia
Author to whom correspondence should be addressed.
Mathematics 2021, 9(18), 2210;
Submission received: 19 August 2021 / Revised: 4 September 2021 / Accepted: 7 September 2021 / Published: 9 September 2021
(This article belongs to the Special Issue Quantitative Analysis and DEA Modeling in Applied Economics)


The concept of eco-efficiency has recently become an issue of great importance due to the growing trend of environmental degradation, and many approaches based on Data Envelopment Analysis (DEA) are used in the literature to evaluate the environmental performance of economic systems. However, research to date has paid little attention to the possibility of extending the DEA approach to the problem of measuring the economic feasibility of eco-efficiency improvement. The main aim of this study is to evaluate the efficiency of investments focused on improving the eco-efficiency of the regional economy in Russia using the DEA approach. The various types of costs for environmental protection measures are considered as inputs and the annual decrease in specific environmental impacts of the regional economy are considered as outputs of DEA models. This is different from previous research, which generally focused on environmental efficiency only, omitting the integration of economic aspects in eco-efficiency measures. This study compares three different modifications of basic DEA models in the context of technical complexity and practical feasibility. The results show that the efficiency of regional environmental management in many Russian regions has a great potential for improvement. From a practical point of view, the Slack-Based Measure (SBM) model provides the most accurate results for policy applications. Unlike other ratings, the DEA-SBM model may stimulate an optimization of environmental protection spending and the introduction of technological and organizational eco-innovations.
91B76; 90B30; 90C05
JEL Classification:
C61; C67; Q52; Q56; R11

1. Introduction

The concept of eco-efficiency, first introduced in 1990 by Schaltegger and Sturm [1], has become an issue of great importance due to the growing need for reducing the global trend of environmental degradation. Because of its multidimensional nature, measuring the eco-efficiency of production systems with the help of Data Envelopment Analysis (DEA) in recent years has become the mainstream in the scientific literature [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]. In most studies, environmental efficiency (or eco-efficiency) is considered as the ability of a production system to minimize undesirable outputs (negative externalities of production activity) with fixed inputs (material and human resources) and maximize desired outputs (indicators of the economic performance of the production system) [2,5,6,9,10,11,13]. Therefore, DEA models summarize various desirable and undesirable effects of the production system in a single efficiency index. As a rule, eco-efficiency is evaluated regionally [2,5,6,7,10,14] or sectorally [8,9,11,12,13]. Such inputs of Decision-Making Units (DMUs) as energy (volume of energy consumption), labor (number of employed), capital (volume of investment in fixed assets), land (amount of fertile land used), or water (volume of freshwater consumption) are often considered as inputs. Gross regional product or the gross output of the industry are considered as desirable outputs, while the volumes of various types of pollution as undesirable outputs. Efficiency scores of DMUs, calculated by DEA, provide valuable information for formulating policies for more sustainable development of economical systems under the study.
However, research to date has paid little attention to the possibility of extending the DEA approach to the problem of measuring the economic feasibility of eco-efficiency improvement. Only a few studies include the investments in minimization of negative environmental impacts into considerations while designing DEA model [7,16,17]. These previous papers have provided important information on some technical problems that can arise in the process of calculation of efficiency scores in such extended models. For instance, if we consider improvements in eco-efficiency (which appear due to investments) as outputs of the DEA model, in reality for some DMUs they can be negative. If we apply a two-stage model for production systems eco-efficiency estimation on stage 1 and efficiency of investments in pollution treatment measurement on stage 2, we face the problem of aggregation of two efficiency scores in one objective function.
In addition to the above-mentioned DEA-characteristic issues, we must also note that the basic models for eco-efficiency calculations found in the literature also have several flaws. First, basic DEA models do not put any restrictions on the weights of inputs and outputs. Therefore, in many cases equally effective DMUs can have an opposite burden on the environment: one can produce the minimum amount of emissions into the atmosphere, then the other produces, for instance, the minimum amount of solid waste. However, these DMUs can be severely sub-optimal in other environment categories. Second, basic DEA models can evaluate too many DMUs as efficient ones. In other words, they have poor discrimination of efficiency. Due to these features, such models are insufficient to investigate realistic problems and not helpful for the formulation of policy by local governments or industry leaders.
Therefore, despite the fact that extensive research has been carried out on measuring eco-efficiency with DEA-framework, there is still a lot of room for improvement.
The purpose of this study is twofold. The first aim is to evaluate the efficiency of investments focused on improving the eco-efficiency of the regional economy using DEA approach. In this study, as inputs we consider various types of costs for environmental protection measures: (1) investments in fixed assets aimed at reducing pollution and (2) current costs for environmental protection measures. As outputs, we consider the annual decrease in specific environmental impacts of the regional economy. This is different from previous research, which generally focused on environmental efficiency only, omitting the integration of economic aspects in eco-efficiency measures. From a practical perspective, the purpose of the study is to evaluate the efficiency of environmental management at the regional level. The second aim of the study is to compare three different modifications of basic DEA models in the context of technical complexity and practical feasibility. As modifications, we propose to eliminate the problem of unequal consideration of various environmental aspects in the overall assessment of the efficiency of the regional environmental management by introducing weights. Moreover, we also consider two possible ways to work with negative outputs.
The rest of this paper is structured as follows. The literature review is presented in Section 2. Section 3 introduces the models and methods used for efficiency evaluation. Section 4 describes the data and variables, and then reports the results of DEA efficiency evaluation. Finally, Section 5 provides discussions on the results and their main implications.

2. Literature Review

In environmental management, the decision process plays a vital role in implementing policies to promote development based on eco-efficient economic growth [18,19,20,21,22,23,24]. In this regard, for benchmarking and environmental performance assessment different approaches are used, which include parametric (stochastic frontier analysis) [25,26,27,28], nonparametric (data envelopment analysis), and multicriteria decision-making (MCDM) [29,30,31,32,33] approaches. In some studies on environmental efficiency assessment, MCDM methods are combined with DEA, e.g., Zhou et al. [34] used the TOPSIS method for assessment of inputs and outputs indicators and then studied the spatiotemporal variation of the comprehensive eco-efficiency of 21 cities. Martín-Gamboa et al. [35] presented a review of life cycle approaches coupled with data envelopment analysis within multicriteria decision analysis for sustainability assessment of energy systems. Keshavarz and Toloo [36] proposed a hybrid DEA and multi-attribute decision-making (MADM) approach to sustainability assessment.
Nevertheless, for environmental efficiency evaluation, as confirmed by the literature mentioned in this review, the DEA approach has become more common in recent years. There exist a number of excellent literature reviews devoted to the analysis of environmental efficiency studies based on DEA. Song et al. [4] investigate the achievements of the theoretical and practical basis of environmental policy analysis using DEA. Sueyoshi et al. [37] systematically summarize previous research efforts on DEA applied to energy and environment from the 1980s to 2010s, including concepts and methodologies on DEA environmental assessment. Mardani et al. [38] reviewed and classified 145 papers devoted to the application of DEA models in the assessment of environmental and energy economics. Xu et al. [39] provide a literature review on energy efficiency evaluation based on DEA. This paper deals indirectly with environmental efficiency evaluation, including energy efficiency evaluation of regions, and therefore is also of some interest.
However, since these reviews were published, a number of researches have appeared in the scientific literature. Therefore, this review is mainly focused on recent papers. Moreover, in this section, we have reviewed articles devoted principally to the study of regions. The papers mentioned in our literature review are summarized in Table 1.
Russian regions have large variability in their area and population. The differences become more considerable after the territorial reform 2003–2008 which led to the merging of regions in East Russia. It must be recognized that the area of some regions in Russia exceeds many European countries. Therefore, the literature review is limited not only by studies of the regions, but also includes papers where the environmental efficiency of countries is evaluated. Jebali et al. [44] examine the energy efficiency determinants in the Mediterranean countries. A number of papers [41,52,59,62] evaluate the evolution of environmental performance in member countries of the European Union (EU). Grigoroudis and Petridis [51] present a study of environmental efficiency measurement that concerns 108 countries from all over the world.
Several studies are devoted to the analysis of the environmental efficiency of cities. Such papers have also attracted some interest as large cities are often considered autonomous territorial units under the administrative division. Zhou et al. [34] estimated the eco-efficiency of 21 cities in Guangdong province, China. Li et al. [61] carried out mining eco-efficiency measurement for 14 prefecture-level cities in Guangxi Province, China. Zhou et al. [57] discuss eco-efficiency of 48 cities in Bohai Rim mainly focused on the role of industrial structure upgrades. Perhaps the most extensive study (by the number of regions) considered here was carried out by Bianchi et al. [58] and covers 282 European regions at NUTS-2 level.
As for the territorial aspect, note that most of the recent studies investigate the regions of China; the EU countries are in second place. The environmental efficiency of Russian regions has not been sufficiently studied using the DEA approach; only a few works there exist in the DEA literature in this area. In addition, these studies are limited, as they did not consider all the regions, but only those included in the Central Federal District [54] and Southern Federal District [66].
Usually, the inputs and outputs of the DEA model are chosen a priori by researchers. They are mainly dictated by the purpose of the study and are determined by the availability of data sources. In [42], the authors use Hellwig’s method and coefficient of determination to ensure an appropriate combination of environmental and economic indicators of the eco-efficiency evaluation.
A second-stage analysis is often performed with the help of various regression methods. Shi [55] and Zhu et al. [64] use regression analysis to identify the relationship between the efficiency scores and its determinant factors. Deng et al. [40] explore the factors that influence water use efficiency with a panel data model. Zhou et al. [34], among other methods, also used a panel regression model to determine factors influencing eco-efficiency. In [13,45,61], Tobit regression and truncated regression models are applied. Moutinho and Madaleno [62] analyze the relationship of the obtained DEA scores from the possible influencing factors, namely eight different types of pollutants, with the help of a fractional regression model. Jebali et al. [44] in the second-stage analysis applied a parametric bootstrap procedure to the truncated regression of DEA bias-corrected efficiency scores on environmental variables. Furthermore, in the second stage, Zhao et al. [47] and Zhou et al. [57] calculated a Moran’s I that represents a measure of spatial autocorrelation.
Often, researchers break down the eco-efficiency evaluation in several stages. Zhang et al. [46] adopted a three-stage approach to eliminate the interference of the external environment and statistical noise. Gómez-Calvet et al. [41] conducted a two-stage analysis which develops environmental performance indicators in the first stage for each pair country–year and evaluates its evolution in the second. Zhao et al. [56] employed a three-stage DEA approach combining BCC model [67] and the stochastic frontier analysis (SFA) model.
In the network DEA model, the efficiency measurement process is also split into several stages. Zhao et al. [47] use two-stage network DEA to consider undesirable output: the first stage describes the water utilization process and the second stage characterizes the pollutant treatment process. Shi [55] considers the internal structure of the production system to evaluate the environmental efficiency in the same way, i.e., production subsystem and pollutant treatment subsystem. Zhu et al. [65] divided energy and environment efficiency analysis into two sub-stages: energy production sub-stage and energy utilization sub-stage. Wang et al. [63] proposed a dynamic network DEA approach that considers the dynamic features of network DEA. This approach takes into account that capital investments in the current period are used to establish fixed assets for two stages, which can also be used in the later period, thus affecting its two-stage eco-efficiency.
Most studies employ undesirable outputs as the assessment of environmental efficiency involves the use of indicators that have a negative impact on the environment and, therefore, should be reduced. The traditional radial (BCC and CCR models) and non-radial slack-based measure (SBM) models are commonly applied for environmental efficiency assessment (see, i.e., in [43,48,49]), but there exist some studies that use the directional distance function (DDF) methods [41]. Bianchi et al. [58] used a hyperbolic distance function measure that considers improvements on the input and output direction simultaneously. A newly proposed and rather interesting approach is proposed in [50]; Zhang et al. applied an intermediate model, which is a combination of radial and non-radial approaches.
In order to evaluate the efficiency change over time, several approaches were applied. In [54], dynamic problems of eco-efficiency are explored with the help of Malmquist index estimation. Piao et al. [53] accomplished dynamic environmental performance analysis using Malmquist–Luenberger (ML) index. Wang et al. [13] used the global Malmquist–Luenberger (GML) productivity index to overcome some difficulties with ML index. Lu et al. [52] applied the dynamic DEA model to evaluate the intertemporal eco-efficiency. Wang et al. [63] combined dynamic and network DEA approaches into one model to evaluate industrial eco-efficiency and assess internal ineffective sources. Window analysis is used in [50,60,66].
The vast majority of previous studies use absolute values of indicators in models to analyze environmental efficiency. The main difference between this work and others is that we do not use absolute values of indicators, but their increments as variables. This leads to the values of the indicators are no longer strictly positive and can now take negative values. As a result, calculations based on DEA models become more complicated, as negative values must be taken into account.

3. Methodology

3.1. DEA Background

DEA is a powerful tool to evaluate the relative efficiency of a set of homogeneous decision making units (DMUs). It was originated by Charnes, Cooper, and Rhodes [68] based on seminal work of Farrell [69], where the linear programming technique to estimate a production frontier was first developed. In the DEA model, the relative efficiency score of any DMU is determined as a measure of the relative improvements in inputs and outputs between the DMU and the efficient frontier. In order to construct the efficient frontier, a collection of DMUs is used. The efficiency score can be evaluated by minimization of input keeping its output levels constant (input-oriented model), or by maximization of output keeping the inputs at the same rate (output-oriented model).
Consider a set of n observed DMUs: DMU j | j = 1 , , n . Suppose each DMU j consumes a number of inputs X j = ( x 1 j , x 2 j , , x m j ) T and produces multiple outputs Y j = ( y 1 j , y 2 j , , y r j ) T . It is assumed that vectors X j and Y j are non-negative and has at least one positive component. The production possibility set T is determined as a set { ( X , Y ) | the outputs Y 0 can be produced from the inputs X 0 } . The boundary of PPS represents the frontier which is used to evaluate DMUs against each other.
We start with the Charnes, Cooper, and Rhodes (CCR) model [68] which is one of the most popular DEA models [70]. From the literature review, we recognize that a CRS assumption is mostly used in studies at the region and country level. Besides its popularity, this model fits the features of our task, because output variables that we take for regional environmental management systems assessment express not absolute values but their growth rates. Therefore, the model with constant returns to scale is more suitable in this case. The production possibility set of the CCR model can be written in the form
T = { ( X , Y ) | X j = 1 n X j λ j , Y j = 1 n Y j λ j , λ j 0 , j = 1 , , n } ,
The output-oriented CCR model for efficiency evaluation of DMU k is written as follows [71]:
max η s . t . j = 1 n X j λ j X k , j = 1 n Y j λ j η Y k , λ j 0 , j = 1 , , n .
Efficiency score for CCR model is evaluated as 1 / η , where η is optimal objective value of model (2). From the model (2), it follows that η 1 and, consequently, efficiency score of a DMU is from 0 to 1. The unit is considered efficient if the efficiency score is equal to one, and inefficient if less than one. The efficiency score may be seen as a distance from the DMU k to the efficient frontier.
Another popular DEA model proposed by Banker, Charnes, and Cooper (BCC) [67] exhibits variable returns to scale and their output-oriented case can be written in the form
max η s . t . j = 1 n X j λ j X k , j = 1 n Y j λ j η Y k , j = 1 n λ j = 1 , λ j 0 , j = 1 , , n .

3.2. Weights Restrictions

Weights restriction are used in DEA in order to incorporate implicit judgments in the dual form for modeling production trade-offs. For model (2), the dual output-oriented CCR model can be written in the form
min v T X k s . t . v T X j u T Y j 0 , j = 1 , , n , u T Y k = 1 , v 0 , u 0 .
The dual model (4) provides another way of looking at the problem (2). Dual variables u and v are often called weights in DEA, because the efficiency score of DMU is defined as the ratio of a virtual output (weighted sum of outputs i = 1 r u i y i k ) to a virtual input (weighted sum of inputs i = 1 m v i x i k ). In the model (4), dual variables are supposed to be non-negative, and therefore they can be equal to zero in the optimal solution. This means that some variables are ignored in the efficiency evaluation. In order to overcome this situation, it is proposed to insert weight restrictions into the model. A broad overview of DEA models with weight restrictions is presented in [72].
Many of the models proposed in the DEA literature assume that constraints are imposed on actual weights. However, the application of such models in practice is often difficult, as the cost prices and/or production costs are not always known. A more promising approach is when the constraints are defined as the ratio of the input/output weights to the virtual input/output [73].
L s u s y s j i = 1 r u i y i j U s .
A similar weights restriction could be applied to inputs. Note that constraints are DMU-specific. Therefore, such restrictions are added with respect to all the DMUs.
The resulting model, call it CCR-AR, is a generalization of the assurance region method [74,75]. In this case, the upper and lower bounds of the specified constraints show the relative importance of each indicator relative to the others. Another way to improve the frontier is described in [76]. This method is based on including artificial production units and rays in the primal space of inputs and outputs. An algorithm for improving the envelopment in the DEA models using this approach were proposed in [77].

3.3. Non-Radial Model

One of the generally recognized models for efficiency evaluation in the presence of negative indicators is the non-radial model. Currently, there exist a large number of different non-radial DEA models to cope with negative data, see, i.e., in [78,79]. In particular, Portela et al. [80] showed that the CCR model cannot be applied for datasets where negative data can exist and the other type of technologies with variable returns to scale assumption should be considered.
In this paper, we used the approach described in [81], hereinafter referred to as the SBM model. This model is quite general; it takes into account the presence of negative data, allows us to set weights for each variable, and also makes it possible to rank efficient DMUs using the super-efficiency method [82,83]. The model of Lin et al. [81] is adopted from the well-known RAM model [84] which is used the following input and output ranges:
R i = max 1 j n x i j min 1 j n x i j , i = 1 , , m , R s + = max 1 j n y s j min 1 j n y s j , s = 1 , , r .
As a result, the following SBM model was proposed in [81]:
min 1 + i I μ i w i / R i 1 s O ν s w s + / R s + s . t . x i k j = 1 , j k n x i j λ j w i , i I , y s k j = 1 , j k n y s j λ j + w s + , s O , j = 1 , j k n λ j = 1 , λ j 0 , j = 1 , , n , j k , w s + R s + , r O , w s + , w i 0 , s O , i I ,
where I = i | R i > 0 , i = 1 , , m , O = r | R s + > 0 , s = 1 , , r , μ i , and ν s are positive weights determined by decision-maker’s (DM) opinion and satisfying i I μ i = 1 and r O v r = 1 . Model (6) is more general than RAM model, as μ i and ν s can take any values while in RAM model they are equal to 1 / m and 1 / s , respectively.
In model (6), the projection onto the frontier may not be Pareto efficient. Thus, a second stage should be accomplished in order to identify strongly Pareto efficient projection:
min 1 i I μ i s i / R i 1 + i O ν i s i + / R i + s . t . x i k = j = 1 , j k n x i j λ j w i + s i , i I , y i k = j = 1 , j k n y i j λ j + w i + s i + , i O , j = 1 , j k n λ j = 1 , λ j 0 , j = 1 , , n , j k , s i 0 , i I , s i + 0 , i O .
Problems (6) and (7) are fractional and can be solved using Charnes and Cooper’s transformation [85] to the equivalent linear programming problems. Problem (6) is transformed to the following problem:
min t + i I μ i w ^ i / R i s . t . t x i k j = 1 , j k n x i j λ ^ j w ^ i , i I , t y s k j = 1 , j k n y s j λ ^ j + w ^ s + , s O , t s O ν s w ^ s + / R s + = 1 , j = 1 , j k n λ ^ j = t , λ j 0 , j = 1 , , n , j k , w ^ s + t R s + , s O , w ^ s + , w ^ i 0 , s O , i I .
The optimal solution of (6) is obtained by
w i = w ^ i / t , i I , w s + = w ^ s + / t , s O , λ j = λ ^ j / t , j = 1 , , n , j k ,
where λ ^ j , w ^ i , w ^ r + , t is optimal solution of (8).
In a similar way, problem (7) is reduced to the following linear programming problem:
min t i I μ i s ^ i / R i s . t . t x i k = j = 1 , j k n x i j λ ^ j t w i + s ^ i , i I , t y i k = j = 1 , j k n y i j λ ^ j + t w i + s ^ i + , i O . t + i O ν i s ^ i + / R i + = 1 , j = 1 , j k n λ ^ j = t , λ ^ j 0 , j = 1 , , n , j k , s ^ i 0 , i I , s ^ i + 0 , i O .
From the solution of (10), possible input excesses s i and output shortfalls s i + can be determined as follows:
s i = s ^ i / t , i I , s i + = s ^ i + / t , i O .

3.4. Weight Restrictions Adjustment

The weight restrictions in the CCR-AR model are specified exogenously. Moreover, there is no universal rule on how to choose them in a particular case. For each model, the weight restrictions are determined individually depending on the available data and the DM’s opinion. Therefore, to adjust the model, we recommend following the procedure described below (Listing 1).
Listing 1. Evaluation of regional environmental management systems.
Listing 1. Evaluation of regional environmental management systems.
Input: Inputs and outputs for all DMUs
Output: Efficiency scores
1: procedureAdjustWeightRestrictions
2:  Set initial weight restrictions and solve CCR-AR model (4) and (5)
3:  if DM does not agree with efficiency scores of some DMUs then
4:    do
5:     Construct sections of the frontier for these DMUs
6:     Identify the reasons for inconsistencies
7:     Determine new weights restrictions and solve CCR-AR model
8:    while The efficiency scores are not following the DM’s opinion
9:  end if
10: end procedure
Therefore, the process of decision-making for efficiency evaluation in regional environmental management systems can be described as follows.
First, we choose the CCR-AR model and set initial weights restrictions. If DM has no initial preferences, he/she can make them large enough so that they do not have a significant impact. At the second step, efficiency scores are calculated.
If DM disagrees with some of the obtained efficiency scores, then we construct two- and/or three-dimensional sections of the production possibility set in order to reveal the reasons for such inconsistencies. Algorithms for constructing such sections are given in [86,87]. The graphical representation of frontier sections provides a visual way to study a multidimensional production possibility set, which increases the intuition of the decision-maker, helps to produce more effective decisions, and reduces the decision-making time.
After the weights restrictions are corrected, the model is calculated again and DM checks new efficiency scores. If obtained efficiency scores are not in question, then the procedure stops; otherwise, the steps of correcting the weights are repeated.
Note that MCDM methods may be used for weights adjustment. This question requires a deep study and is not the purpose of this paper. Therefore, we consider it as a future research direction. The procedure describes the process of weights adjustment using a single DM’s opinion; for multiple DMs, it is necessary to use special methods of group decision-making [88,89].
In SBM model (8), DM can adjust weights μ i and ν s . Therefore, in order to fit the model to the DM’s opinion, the procedure for SBM model can be written in a similar way and thus omitted.

4. Empirical Analysis

4.1. Data Source and Variables Selection

The data for the efficiency evaluation of regional environmental management systems were collected from the Russian Federal State Statistics Service’s regional statistical data [90,91]. The dataset includes all regions of Russia, with the exception of those in which the current costs of environmental protection measures are below the minimum of maintaining the operability of the regional environmental management system (10 million rubles were taken for the minimum threshold level). The efficiency scores of regional environmental management systems were calculated over the period from 2011 to 2017 in two-year steps, i.e., in 2011, 2013, 2015, and 2017 for three models: the regular CCR model with no weighting and scale shift restrictions (CCR), the CCR model with interval weighting and scale shift restrictions (CCR-AR), and the SBM model.
According to the Russian Federal State Statistics Service, the investments in fixed assets aimed at environmental protection and rational use of natural resources include the cost of construction, reconstruction (including the expansion and modernization) of facilities, the purchase of machinery, equipment, and vehicles. The current expenses on environmental protection include the costs of following activities: the maintenance and operation of fixed assets for environmental protection; measures for the preservation and recovery of the natural environment; the treatment of production and consumption wastes; the organization of control over emissions (discharges), production, and consumption wastes; environmental research; and environmental education.
As inputs, we considered the share of investments in fixed assets aimed at reducing environmental pollution ( x 1 ) and the share of current costs in the region’s gross regional product (GRP) ( x 2 ). As outputs, we consider indicators of a decrease in the specific level of pollution to the atmosphere, soil, and water, i.e., the difference in specific pollution before ( t 1 ) the investments of both types (in fixed capital and current expenses) and after ( t 2 ).
y 1 , k = Δ k E P = E P t 1 , k G R P t 1 , k E P t 2 , k G D P t 2 , k ; y 2 , k = Δ k E T = E T t 1 , k G R P t 1 , k E T t 2 , k G D P t 2 , k ; y 3 , k = Δ k S W = S W t 1 , k G R P t 1 , k S W t 2 , k G D P t 2 , k ; y 4 , k = Δ k W = W t 1 , k G R P t 1 , k W t 2 , k G D P t 2 , k ; y 5 , k = Δ k W T = W T t 1 , i G R P t 1 , i W T t 2 , k G R P t 2 , k ; y 6 , k = Δ k R = R t 2 , k R t 1 , k ,
where E P is an emissions to the atmosphere from stationary sources (thousands tones), G R P is a gross regional product in 2011 prices (millions rubles), E T is an emissions to the atmosphere from road transport (thousands tones), S W is a discharge or untreated sewage (millions m 3 ), W is a fresh water consumption (millions m 3 ), W T is a waste generation (tones), R is a share of recycling and reuse of waste (%), and k is an index of the region. Max, min, and mean statistics of variables are shown in Table 2.

4.2. Weights Selection

We set the weight restrictions in the form of the ratio of the input/output weight coefficient to the virtual input/output. Such a model is a generalization of the assurance regions method (denoted in the work as CCR-AR). In this case, the upper and lower boundaries of the specified constraints show the relative significance of each indicator in relation to the others. In the CCR-AR model, for working with negative outputs, we perform a shift of the scale of indicators. For each indicator having negative values, the shift value was selected in such a way that all values were strictly positive.
As the second way of working with negative outputs in this paper, we used the non-radial SBM model. This model is quite general; it takes into account the presence of negative data, allows one to set weight coefficients for each indicator, and makes it possible to rank effective objects using the super-efficiency method.
As discussed earlier, many outputs have negative values, so in order to apply a model with restrictions on virtual weight coefficients (the CCR-AR model), it is necessary to perform a shift transformation of the corresponding variables. For each indicator that has negative values, the shift value was selected in such a way that all values were strictly positive.
After performing several steps of adjusting weights, the following estimates were obtained. The weight restrictions for the outputs in the CCR-AR model are shown in Table 3 (second column); they are applied for all the DMUs. The values of the weights for the inputs and outputs of the SBM model we used in our computations are shown in Table 3 (third column).

4.3. DEA Evaluation Results

According to determined weights, CCR-AR and SBM models were applied to the dataset, and efficiency scores were evaluated. Comparing the results obtained, note that the traditional CCR model without weight restrictions and the CCR-AR model provides very close efficiency scores (Table 4). Although the results of the SBM model correlate with the results of calculations based on the CCR model and the CCR-AR model, the value of the correlation coefficient is significantly lower than in the previous case.
A detailed analysis of the results obtained allows us to conclude that the CCR and CCR-AR models are more sensitive to the model inputs; according to these models, the regions with the lowest environmental costs are efficient. Low or zero input makes the region efficient even if some outputs have negative values. For example, in the Bryansk Oblast in 2011, the values of investments in fixed assets are zero. The result of environmental protection activities in 2011 was a slight improvement in all environmental indicators, except for the intensity of waste generation. Nevertheless, the environmental management system of the region is recognized as efficient according to the CCR model and the CCR-AR model, see Table 5. Similarly, the environmental management system of Moscow in 2011 was efficient, although the indicators of the intensity of waste generation in the region also deteriorated over the observed period. However, the region is efficient due to the low share of costs (investment in fixed assets is 0.06% of GRP, current costs also 0.06% of GRP). At the same time, the Lipetsk Oblast, which also has the only negative indicator of the intensity of waste generation, has the lowest efficiency, as the indicators of the region’s costs for environmental protection measures are among the highest: investments in fixed assets are 1.37% of GRP, and current costs are 3.25% of GRP.
The SBM model demonstrates a higher sensitivity to outputs: for example, the Stavropol Krai and the Arkhangelsk Oblast in 2011, as regions with the most effective environmental management systems, actually have all positive outputs. However, the regions following them in the performance rating already have one, two, or even three negative outputs, so the interpretation of the results of this model is controversial.
The spatial distribution of efficiency scores for CCR-AR and SBM models are showed in Figure 1 and Figure 2, respectively. Table 4 shows that the conventional CCR efficiency scores are very close to the CCR-AR model, and thus spatial distribution diagrams for the CCR model are not included in the paper. Although the results of the SBM model correlate with the results of calculations using the CCR model and the CCR-AR model, the value of the correlation coefficient is significantly lower than in the previous case.
Looking at the results as a whole, we can conclude that the efficiency of regional environmental management has deteriorated during the period under the study. Most likely, this is due to the deterioration of the economic situation in Russia in the period after 2014 due to the imposition of sanctions: incomplete utilization of production facilities, lack of funds for the introduction of more resource-efficient technologies, etc.
The highest potential for improvement in order to achieve efficiency in environmental management have two largest Russian regions in Eastern Siberia: Krasnoyarsk Krai and Yakutia. Therefore, special attention should be paid to the study of problems in the environmental management system in these regions. The regions that demonstrate the highest indicators of the effectiveness of environmental management systems (Tuva Republic, Ivanovo Oblast, and Moscow) are also worthy of closer study in order to identify and analyze the best practices and technologies of environmental management.
A more detailed analysis of the results allows us to conclude that the CCR and CCR-AR models are more sensitive to model inputs: according to these models, regions with the lowest environmental protection costs are effective. Low or zero cost values of at least one type immediately make the region efficient, even if some of the model outputs have negative values. The SBM model demonstrates a higher sensitivity to outputs, for example, regions, which are recognized as efficient according to the results of calculations, rarely have all outputs positive. However, the regions following them in efficiency ratings already have one, two, and even three negative outputs, so an interpretation of the results of this model is controversial. According to the findings above, some policy implications may be proposed for the evaluation of efficiency in regional environmental management systems in Russia.

5. Discussion and Conclusions

5.1. Discussion and Future Research Directions

In many previous studies, various DEA models are used in order to evaluate environmental efficiency. However, the efficiency evaluation of regional environmental management systems considered in this paper differs from other DEA applications because it does not use absolute variables, but changes over a certain period. As these changes may be not positive due to environmental degradation, negative values of variables are a common occurrence. At the same time, traditional DEA models have certain difficulties with negative data. To apply them, it is necessary to make some data transformations, which would inevitably be subjective. As a result, efficiency scores would be biased and decisions based on the use of such estimates would be unsubstantiated and possibly flawed. Therefore, from a mathematical point of view, the SBM models are more attractive for environmental efficiency problems with negative data. To perform the calculations, we used a fairly new model of Lin et al., which allows us to calculate a non-radial measure and at the same time correctly processes negative variables. In addition, the model used allows us to rank efficient units by calculating the super-efficiency score. It also has some attractive properties—it is monotonous, unit-invariant, and translation-invariant for both inputs and outputs.
The limitations of this study are as follows: (1) the choice of inputs and outputs of the model was limited due to the peculiarities of the statistical accounting of various aspects of environmental management in Russia. For example, Russian statistics do not separate greenhouse gas emissions, but consider them in one indicator along with all other emissions of gaseous pollutants. Therefore, it is not possible to analyze the effectiveness of regional environmental management in such an important area as combating climate change. (2) For the same reason (features of statistical accounting), we cannot compare the effectiveness of environmental management in Russian regions with other countries, for example, with the EU countries, although such comparisons are of significant practical interest. (3) As the study was limited to CRS models, it did not provide the information to analyze the difference in the technical efficiency and scale efficiency of regional systems of environmental management, which also is an important topic for decision-making. Despite its limitations, the study certainly adds to our understanding of the problems with inefficient investments in environmental protection measures in Russian regions.
A natural progression of this work is to analyze the efficiency of regional environmental management taking into account the peculiarities of ecological problems of each region. It can be done by assigning different weight restrictions to the outputs depending on whether the improvement of the ecological situation in this area is a priority for the region or not. Another direction of future research is might be an introduction of an artificial object into the sample, which demonstrates the highest possible investment efficiency at the current level of technological development. The introduction of such an artificial object changes the efficiency frontier and makes it possible to determine not only the efficiency of regions in relation to each other, but also their efficiency in comparison with those regions that use the best available technologies in all areas of environmental management.

5.2. Conclusions

Although the primary applicable result of this research is a rating of Russian regions, this result differs significantly from other Russian ecologic ratings by its methodology. The existing ratings normally evaluate either the eco-efficiency of the regional economy (e.g., the ecological rating of the Analytical Credit Rating Agency [92]), or use the expert estimates of regional events that have any connection to ecology (for example, the national economic rating of the national “Green Patrol” organization [93]). Apart from regional ratings, there are also some ratings of Russian cities, which evaluate the work of municipal governments on improving waste management and planting of greenery (e.g., the combined rating of the political “All-Russia People’s Front” movement and the Ministry of Natural Resource and Ecology [94]). However, none of the methodologies used in these ratings account for the efficiency of spending on environmental protection. Thus, a direct comparison of our three DEA models with results of other ratings is impossible, and doing so will not allow determining which of the models better reflects reality. This could be considered a limitation of our research.
Nevertheless, accounting for the differences in calculations of efficiency measures in each of our models, we consider the SBM model to provide the most accurate results. First, this model does not allow for a situation where some ecologic effects have an overly large weight coefficient and some others have an overly small one. In other words, we cannot consider regions that had huge successes in one area of environmental protection and complete failures in other areas as efficient. Second, this model has a higher discriminating power, which allows using strict ranking.
As for practical application of our methodology for creating ratings of regional ecologic management efficiency, we hold a positive viewpoint, as all necessary indicators for rating calculation are provided by official statistical organizations and are openly available. Our rating is based on quantitative indicators, can be calculated independently of regional authorities, and excludes any subjective influences. As any other official rating, it could be used for stimulating regional authorities to take genuine action in the sphere of ecology. However, unlike other ratings, the DEA-SBM model may also stimulate an optimization of environmental protection spending and, therefore, an introduction of technological and organizational eco-innovations.
The results of DEA models provide a Big Picture in the field of environmental management at the regional level. Taking into account the extreme non-transparency of this area in Russia (for example, the absence of publicly available information on what the budgetary funds allocated for environmental protection were spent on), they allow any independent researcher to identify the most problematic regions. A further detailed analysis of the environmental management processes will be required in order to determine the true reasons of inefficiency; however, DEA modeling helps to point out the potential corruption or technological violations in the region.

Author Contributions

All of the authors contributed to publishing this article. Conceptualization, S.R.; methodology, S.R. and A.L.; software, A.L. and I.L.; validation, S.R. and A.R.; formal analysis, S.R.; visualization, I.L. and A.L. All authors have read and agreed to the published version of the manuscript.


This work was supported by the Russian Science Foundation, project No. 17-11-01353 (implementation of DEA approach).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.


The authors also thank the academic editors and anonymous reviewers for their guidance and constructive suggestions.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Schaltegger, S.; Sturm, A. Ökologische Rationalität: Ansatzpunkte zur Ausgestaltung von ökologieorientierten Managementinstrumenten. Die Unternehm. 1990, 44, 273–290. [Google Scholar]
  2. Bian, Y.; Yang, F. Resource and environment efficiency analysis of provinces in China: A DEA approach based on Shannon’s entropy. Energy Policy 2010, 38, 1909–1917. [Google Scholar] [CrossRef]
  3. Bian, Y.; He, P.; Xu, H. Estimation of potential energy saving and carbon dioxide emission reduction in China based on an extended non-radial DEA approach. Energy Policy 2013, 63, 962–971. [Google Scholar] [CrossRef]
  4. Song, M.; An, Q.; Zhang, W.; Wang, Z.; Wu, J. Environmental efficiency evaluation based on data envelopment analysis: A review. Renew. Sustain. Energy Rev. 2012, 16, 4465–4469. [Google Scholar] [CrossRef]
  5. Lin, B.; Du, K. Energy and CO2 emissions performance in China’s regional economies: Do market-oriented reforms matter? Energy Policy 2015, 78, 113–124. [Google Scholar] [CrossRef] [Green Version]
  6. Wang, K.; Wei, Y.M.; Zhang, X. A comparative analysis of China’s regional energy and emission performance: Which is the better way to deal with undesirable outputs? Energy Policy 2012, 46, 574–584. [Google Scholar] [CrossRef]
  7. Wu, J.; Lv, L.; Sun, J.; Ji, X. A comprehensive analysis of China’s regional energy saving and emission reduction efficiency: From production and treatment perspectives. Energy Policy 2015, 84, 166–176. [Google Scholar] [CrossRef]
  8. Welch, E.; Barnum, D. Joint environmental and cost efficiency analysis of electricity generation. Ecol. Econ. 2009, 68, 2336–2343. [Google Scholar] [CrossRef]
  9. Hoang, V.N.; Alauddin, M. Input-Orientated Data Envelopment Analysis Framework for Measuring and Decomposing Economic, Environmental and Ecological Efficiency: An Application to OECD Agriculture. Environ. Resour. Econ. 2012, 51, 431–452. [Google Scholar] [CrossRef]
  10. Wang, Z.; Feng, C. A performance evaluation of the energy, environmental, and economic efficiency and productivity in China: An application of global data envelopment analysis. Appl. Energy 2015, 147, 617–626. [Google Scholar] [CrossRef]
  11. Nabavi-Pelesaraei, A.; Abdi, R.; Rafiee, S.; Mobtaker, H.G. Optimization of energy required and greenhouse gas emissions analysis for orange producers using data envelopment analysis approach. J. Clean. Prod. 2014, 65, 311–317. [Google Scholar] [CrossRef]
  12. Liu, J.; Zhang, J.; Fu, Z. Tourism eco-efficiency of Chinese coastal cities—Analysis based on the DEA-Tobit model. Ocean Coast. Manag. 2017, 148, 164–170. [Google Scholar] [CrossRef]
  13. Wang, X.; Ding, H.; Liu, L. Eco-efficiency measurement of industrial sectors in China: A hybrid super-efficiency DEA analysis. J. Clean. Prod. 2019, 229, 53–64. [Google Scholar] [CrossRef]
  14. Moutinho, V.; Fuinhas, J.A.; Marques, A.C.; Santiago, R. Assessing eco-efficiency through the DEA analysis and decoupling index in the Latin America countries. J. Clean. Prod. 2018, 205, 512–524. [Google Scholar] [CrossRef]
  15. Matsumoto, K.; Chen, Y. Industrial eco-efficiency and its determinants in China: A two-stage approach. Ecol. Indic. 2021, 130, 108072. [Google Scholar] [CrossRef]
  16. Wu, Y.; Chen, Z.; Xia, P. An extended DEA-based measurement for eco-efficiency from the viewpoint of limited preparation. J. Clean. Prod. 2018, 195, 721–733. [Google Scholar] [CrossRef]
  17. Ratner, S.; Zaretskaya, M. Evaluating Efficiency of Russian Regional Environmental Management Systems. Qual.-Access Success 2020, 21, 120–125. [Google Scholar]
  18. Yang, L.; Yang, Y. Evaluation of eco-efficiency in China from 1978 to 2016: Based on a modified ecological footprint model. Sci. Total Environ. 2019, 662, 581–590. [Google Scholar] [CrossRef]
  19. Yamasaki, J.; Ikaga, T.; Itsubo, N. Eco-Efficiency Assessment of Japanese Municipalities Based on Environmental Impacts and Gross Regional Product. Sustainability 2019, 11, 4045. [Google Scholar] [CrossRef] [Green Version]
  20. Voinea, C.L.; Hoogenberg, B.J.; Fratostiteanu, C.; Bin Azam Hashmi, H. The Relation between Environmental Management Systems and Environmental and Financial Performance in Emerging Economies. Sustainability 2020, 12, 5309. [Google Scholar] [CrossRef]
  21. Liu, H.; Yang, R.; Zhou, Z.; Huang, D. Regional Green Eco-Efficiency in China: Considering Energy Saving, Pollution Treatment, and External Environmental Heterogeneity. Sustainability 2020, 12, 7059. [Google Scholar] [CrossRef]
  22. Li, X.N.; Feng, Y.; Wu, P.Y.; Chiu, Y.H. An Analysis of Environmental Efficiency and Environmental Pollution Treatment Efficiency in China’s Industrial Sector. Sustainability 2021, 13, 2579. [Google Scholar] [CrossRef]
  23. Ma, J.; Yin, Z.; Guo, Z. Meta-Evaluation for the Evaluation of Environmental Management: Standards and Practices. Sustainability 2021, 13, 2567. [Google Scholar] [CrossRef]
  24. Ge, X.; Xu, J.; Xie, Y.; Guo, X.; Yang, D. Evaluation and Dynamic Evolution of Eco-Efficiency of Resource-Based Cities—A Case Study of Typical Resource-Based Cities in China. Sustainability 2021, 13, 6802. [Google Scholar] [CrossRef]
  25. Cuesta, R.A.; Lovell, C.K.; Zofío, J.L. Environmental efficiency measurement with translog distance functions: A parametric approach. Ecol. Econ. 2009, 68, 2232–2242. [Google Scholar] [CrossRef]
  26. Orea, L.; Wall, A. Measuring Eco-efficiency Using the Stochastic Frontier Analysis Approach. In Advances in Efficiency and Productivity; Aparicio, J., Lovell, C.A.K., Pastor, J.T., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 275–297. [Google Scholar] [CrossRef]
  27. Orea, L.; Wall, A. A Parametric Approach to Estimating Eco-Efficiency. J. Agric. Econ. 2017, 68, 901–907. [Google Scholar] [CrossRef]
  28. Moutinho, V.; Madaleno, M. Assessing Eco-Efficiency in Asian and African Countries Using Stochastic Frontier Analysis. Energies 2021, 14, 1168. [Google Scholar] [CrossRef]
  29. Gumus, S.; Egilmez, G.; Kucukvar, M.; Park, Y.S. Integrating expert weighting and multi-criteria decision making into eco-efficiency analysis: The case of US manufacturing. J. Oper. Res. Soc. 2016, 67, 616–628. [Google Scholar] [CrossRef]
  30. Kumar, A.; Sah, B.; Singh, A.R.; Deng, Y.; He, X.; Kumar, P.; Bansal, R. A review of multi criteria decision making (MCDM) towards sustainable renewable energy development. Renew. Sustain. Energy Rev. 2017, 69, 596–609. [Google Scholar] [CrossRef]
  31. Siksnelyte, I.; Zavadskas, E.K.; Streimikiene, D.; Sharma, D. An Overview of Multi-Criteria Decision-Making Methods in Dealing with Sustainable Energy Development Issues. Energies 2018, 11, 2754. [Google Scholar] [CrossRef] [Green Version]
  32. Yao, S. Fuzzy-based multi-criteria decision analysis of environmental regulation and green economic efficiency in a post-COVID-19 scenario: The case of China. Environ. Sci. Pollut. Res. 2021, 28, 30675–30701. [Google Scholar] [CrossRef]
  33. Vlachokostas, C.; Michailidou, A.; Achillas, C. Multi-Criteria Decision Analysis towards promoting Waste-to-Energy Management Strategies: A critical review. Renew. Sustain. Energy Rev. 2021, 138, 110563. [Google Scholar] [CrossRef]
  34. Zhou, C.; Shi, C.; Wang, S.; Zhang, G. Estimation of eco-efficiency and its influencing factors in Guangdong province based on Super-SBM and panel regression models. Ecol. Indic. 2018, 86, 67–80. [Google Scholar] [CrossRef]
  35. Martín-Gamboa, M.; Iribarren, D.; García-Gusano, D.; Dufour, J. A review of life-cycle approaches coupled with data envelopment analysis within multi-criteria decision analysis for sustainability assessment of energy systems. J. Clean. Prod. 2017, 150, 164–174. [Google Scholar] [CrossRef]
  36. Keshavarz, E.; Toloo, M. A hybrid data envelopment analysis and multi-attribute decision making approach to sustainability assessment. Expert Syst. 2020, 37, 1–17. [Google Scholar] [CrossRef]
  37. Sueyoshi, T.; Yuan, Y.; Goto, M. A literature study for DEA applied to energy and environment. Energy Econ. 2017, 62, 104–124. [Google Scholar] [CrossRef]
  38. Mardani, A.; Streimikiene, D.; Balezentis, T.; Saman, M.Z.M.; Nor, K.M.; Khoshnava, S.M. Data Envelopment Analysis in Energy and Environmental Economics: An Overview of the State-of-the-Art and Recent Development Trends. Energies 2018, 11, 2002. [Google Scholar] [CrossRef] [Green Version]
  39. Xu, T.; You, J.; Li, H.; Shao, L. Energy Efficiency Evaluation Based on Data Envelopment Analysis: A Literature Review. Energies 2020, 13, 3548. [Google Scholar] [CrossRef]
  40. Deng, G.; Li, L.; Song, Y. Provincial water use efficiency measurement and factor analysis in China: Based on SBM-DEA model. Ecol. Indic. 2016, 69, 12–18. [Google Scholar] [CrossRef]
  41. Gómez-Calvet, R.; Conesa, D.; Gómez-Calvet, A.R.; Tortosa-Ausina, E. On the dynamics of eco-efficiency performance in the European Union. Comput. Oper. Res. 2016, 66, 336–350. [Google Scholar] [CrossRef]
  42. Masternak-Janus, A.; Rybaczewska-Błażejowska, M. Comprehensive Regional Eco-Efficiency Analysis Based on Data Envelopment Analysis: The Case of Polish Regions. J. Ind. Ecol. 2017, 21, 180–190. [Google Scholar] [CrossRef]
  43. Chen, L.; Jia, G. Environmental efficiency analysis of China’s regional industry: A data envelopment analysis (DEA) based approach. J. Clean. Prod. 2017, 142, 846–853. [Google Scholar] [CrossRef]
  44. Jebali, E.; Essid, H.; Khraief, N. The analysis of energy efficiency of the Mediterranean countries: A two-stage double bootstrap DEA approach. Energy 2017, 134, 991–1000. [Google Scholar] [CrossRef]
  45. Wang, J.; Zhao, T. Regional energy-environmental performance and investment strategy for China’s non-ferrous metals industry: A non-radial DEA based analysis. J. Clean. Prod. 2017, 163, 187–201. [Google Scholar] [CrossRef]
  46. Zhang, J.; Liu, Y.; Chang, Y.; Zhang, L. Industrial eco-efficiency in China: A provincial quantification using three-stage data envelopment analysis. J. Clean. Prod. 2017, 143, 238–249. [Google Scholar] [CrossRef]
  47. Zhao, L.; Sun, C.; Liu, F. Interprovincial two-stage water resource utilization efficiency under environmental constraint and spatial spillover effects in China. J. Clean. Prod. 2017, 164, 715–725. [Google Scholar] [CrossRef]
  48. Huang, J.; Xia, J.; Yu, Y.; Zhang, N. Composite eco-efficiency indicators for China based on data envelopment analysis. Ecol. Indic. 2018, 85, 674–697. [Google Scholar] [CrossRef]
  49. Rybaczewska-Błażejowska, M.; Masternak-Janus, A. Eco-efficiency assessment of Polish regions: Joint application of life cycle assessment and data envelopment analysis. J. Clean. Prod. 2018, 172, 1180–1192. [Google Scholar] [CrossRef]
  50. Zhang, A.; Li, A.; Gao, Y. Social Sustainability Assessment across Provinces in China: An Analysis of Combining Intermediate Approach with Data Envelopment Analysis (DEA) Window Analysis. Sustainability 2018, 10, 732. [Google Scholar] [CrossRef] [Green Version]
  51. Grigoroudis, E.; Petridis, K. Evaluation of National Environmental Efficiency Under Uncertainty Using Data Envelopment Analysis. In Understanding Risks and Uncertainties in Energy and Climate Policy: Multidisciplinary Methods and Tools for a Low Carbon Society; Doukas, H., Flamos, A., Lieu, J., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 161–181. [Google Scholar] [CrossRef] [Green Version]
  52. Lu, C.C.; Lu, L.C. Evaluating the energy efficiency of European Union countries: The dynamic data envelopment analysis. Energy Environ. 2019, 30, 27–43. [Google Scholar] [CrossRef]
  53. Piao, S.R.; Li, J.; Ting, C.J. Assessing regional environmental efficiency in China with distinguishing weak and strong disposability of undesirable outputs. J. Clean. Prod. 2019, 227, 748–759. [Google Scholar] [CrossRef]
  54. Ratner, S.V. Dynamic Problems of Estimation of Ecological-Economic Efficiency of Regions Based on Basic Models of Data Envelopment Analysis. Autom. Remote Control 2019, 80, 2245–2258. [Google Scholar] [CrossRef]
  55. Shi, X. Environmental Efficiency Evaluation of Chinese Industry Systems by Using Non-Cooperative Two-Stage DEA Model. Math. Probl. Eng. 2019, 2019, 9187541. [Google Scholar] [CrossRef] [Green Version]
  56. Zhao, H.; Guo, S.; Zhao, H. Provincial energy efficiency of China quantified by three-stage data envelopment analysis. Energy 2019, 166, 96–107. [Google Scholar] [CrossRef]
  57. Zhou, Y.; Kong, Y.; Sha, J.; Wang, H. The role of industrial structure upgrades in eco-efficiency evolution: Spatial correlation and spillover effects. Sci. Total Environ. 2019, 687, 1327–1336. [Google Scholar] [CrossRef] [PubMed]
  58. Bianchi, M.; del Valle, I.; Tapia, C. Measuring eco-efficiency in European regions: Evidence from a territorial perspective. J. Clean. Prod. 2020, 276, 123246. [Google Scholar] [CrossRef]
  59. Hermoso-Orzáez, M.J.; García-Alguacil, M.; Terrados-Cepeda, J.; Brito, P. Measurement of environmental efficiency in the countries of the European Union with the enhanced data envelopment analysis method (DEA) during the period 2005–2012. Environ. Sci. Pollut. Res. 2020, 27, 15691–15715. [Google Scholar] [CrossRef]
  60. Wu, D.; Wang, Y.; Qian, W. Efficiency evaluation and dynamic evolution of China’s regional green economy: A method based on the Super-PEBM model and DEA window analysis. J. Clean. Prod. 2020, 264, 121630. [Google Scholar] [CrossRef]
  61. Li, Y.; Zuo, Z.; Xu, D.; Wei, Y. Mining Eco-Efficiency Measurement and Driving Factors Identification Based on Meta-US-SBM in Guangxi Province, China. Int. J. Environ. Res. Public Health 2021, 18, 5397. [Google Scholar] [CrossRef]
  62. Moutinho, V.; Madaleno, M. A Two-Stage DEA Model to Evaluate the Technical Eco-Efficiency Indicator in the EU Countries. Int. J. Environ. Res. Public Health 2021, 18, 3038. [Google Scholar] [CrossRef]
  63. Wang, Q.; Tang, J.; Choi, G. A two-stage eco-efficiency evaluation of China’s industrial sectors: A dynamic network data envelopment analysis (DNDEA) approach. Process Saf. Environ. Prot. 2021, 148, 879–892. [Google Scholar] [CrossRef]
  64. Zhu, C.; Zhu, N.; Shan, W.U.H. Eco-Efficiency of Industrial Investment and Its Influencing Factors in China Based on a New SeUo-SBM-DEA Model and Tobit Regression. Math. Probl. Eng. 2021, 2021, 5329714. [Google Scholar] [CrossRef]
  65. Zhu, Q.; Chen, J.; Li, F. A comprehensive analysis of China’s regional energy and environment efficiency from supply chain perspective. Int. J. Log. Res. Appl. 2021, 1–16. [Google Scholar] [CrossRef]
  66. Ratner, S.; Ratner, P. DEA-based dynamic assessment of regional environmental efficiency. Appl. Comput. Sci. 2017, 13, 48–60. [Google Scholar] [CrossRef]
  67. Banker, R.D.; Charnes, A.; Cooper, W.W. Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag. Sci. 1984, 30, 1078–1092. [Google Scholar] [CrossRef] [Green Version]
  68. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
  69. Farrell, M.J. The measurement of productive efficiency. J. R. Stat. Soc. 1957, 120, 253–281. [Google Scholar] [CrossRef]
  70. Emrouznejad, A.; Yang, G. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016. Socio-Econ. Plan. Sci. 2018, 61, 4–8. [Google Scholar] [CrossRef]
  71. Cooper, W.W.; Seiford, L.M.; Tone, K. Data Envelopment Analysis. A Comprehensive Text with Models, Applications, References and DEA-Solver Software, 2nd ed.; Springer Science and Business Media: New York, NY, USA, 2007. [Google Scholar] [CrossRef] [Green Version]
  72. Allen, R.; Athanassopoulos, A.; Dyson, R.; Thanassoulis, E. Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions. Ann. Oper. Res. 1997, 73, 13–34. [Google Scholar] [CrossRef]
  73. Wong, Y.H.B.; Beasley, J.E. Restricting Weight Flexibility in Data Envelopment Analysis. J. Oper. Res. Soc. 1990, 41, 829–835. [Google Scholar] [CrossRef]
  74. Thompson, R.G.; Singleton, F.D.; Thrall, R.M.; Smith, B.A. Comparative Site Evaluations for Locating a High-Energy Physics Lab in Texas. Interfaces 1986, 16, 35–49. [Google Scholar] [CrossRef]
  75. Thompson, R.G.; Dharmapala, P.; Rothenberg, L.J.; Thrall, R.M. DEA/AR efficiency and profitability of 14 major oil companies in U.S. exploration and production. Comput. Oper. Res. 1996, 23, 357–373. [Google Scholar] [CrossRef]
  76. Krivonozhko, V.E.; Utkin, O.B.; Safin, M.M.; Lychev, A.V. On some generalization of the DEA models. J. Oper. Res. Soc. 2009, 60, 1518–1527. [Google Scholar] [CrossRef]
  77. Krivonozhko, V.E.; Førsund, F.R.; Lychev, A.V. Improving the frontier in DEA models. Dokl. Math. 2016, 94, 715–719. [Google Scholar] [CrossRef]
  78. Emrouznejad, A.; Anouze, A.L.; Thanassoulis, E. A semi-oriented radial measure for measuring the efficiency of decision making units with negative data, using DEA. Eur. J. Oper. Res. 2010, 200, 297–304. [Google Scholar] [CrossRef]
  79. Sharp, J.A.; Meng, W.; Liu, W. A modified slacks-based measure model for data envelopment analysis with ‘natural’ negative outputs and inputs. J. Oper. Res. Soc. 2007, 58, 1672–1677. [Google Scholar] [CrossRef]
  80. Portela, M.C.A.S.; Thanassoulis, E.; Simpson, G. Negative data in DEA: A directional distance approach applied to bank branches. J. Oper. Res. Soc. 2004, 55, 1111–1121. [Google Scholar] [CrossRef] [Green Version]
  81. Lin, R.; Yang, W.; Huang, H. A modified slacks-based super-efficiency measure in the presence of negative data. Comput. Ind. Eng. 2019, 135, 39–52. [Google Scholar] [CrossRef]
  82. Andersen, P.; Petersen, N.C. A Procedure for Ranking Efficient Units in Data Envelopment Analysis. Manag. Sci. 1993, 39, 1261–1264. [Google Scholar] [CrossRef]
  83. Adler, N.; Friedman, L.; Sinuany-Stern, Z. Review of ranking methods in the data envelopment analysis context. Eur. J. Oper. Res. 2002, 140, 249–265. [Google Scholar] [CrossRef]
  84. Cooper, W.W.; Park, K.S.; Pastor, J.T. RAM: A Range Adjusted Measure of Inefficiency for Use with Additive Models, and Relations to Other Models and Measures in DEA. J. Product. Anal. 1999, 11, 5–42. [Google Scholar] [CrossRef]
  85. Charnes, A.; Cooper, W.W. Programming with Linear Fractional Functionals. Nav. Res. Logist. Q. 1962, 9, 181–186. [Google Scholar] [CrossRef]
  86. Volodin, A.V.; Krivonozhko, V.E.; Ryzhikh, D.A.; Utkin, O.B. Construction of three-dimensional sections in data envelopment analysis by using parametric optimization algorithms. Comput. Math. Math. Phys. 2004, 44, 589–603. [Google Scholar]
  87. Afanasiev, A.P.; Krivonozhko, V.E.; Lychev, A.V.; Sukhoroslov, O.V. Multidimensional frontier visualization based on optimization methods using parallel computations. J. Glob. Optim. 2020, 76, 563–574. [Google Scholar] [CrossRef]
  88. Jin, F.; Garg, H.; Pei, L.; Liu, J.; Chen, H. Multiplicative Consistency Adjustment Model and Data Envelopment Analysis-Driven Decision-Making Process with Probabilistic Hesitant Fuzzy Preference Relations. Int. J. Fuzzy Syst. 2020, 22, 2319–2332. [Google Scholar] [CrossRef]
  89. Jin, F.; Cao, M.; Liu, J.; Martínez, L.; Chen, H. Consistency and trust relationship-driven social network group decision-making method with probabilistic linguistic information. Appl. Soft Comput. 2021, 103, 107170. [Google Scholar] [CrossRef]
  90. Rosstat. Regiony Rossii. Sotsial’no-Ekonomicheskie Pokazateli (Russian Regions. Socio-Economic Indicators). Available online: (accessed on 10 June 2020).
  91. Rosstat. Ohrana Okruzhayushchej Sredy v Rossii (Environmental Protection in Russia). Available online: (accessed on 10 June 2020).
  92. Analytical Credit Rating Agency. Komu na Rusi Dyshat’ Horosho (Who Can Breathe Well in Russia). Available online: (accessed on 11 August 2020).
  93. All-Russian Public Organization “Green Patrol”. Nacional’nyj Ekologicheskij Rejting (National Environmental Rating). Available online: (accessed on 11 August 2020).
  94. All-Russia People’s Front. Rejting Ekologicheskogo Razvitiya Gorodov Rossii 2017 (Environmental Development Rating of Russian Cities 2017). Available online: (accessed on 11 August 2020).
Figure 1. Spatial distribution of CCR-AR model results.
Figure 1. Spatial distribution of CCR-AR model results.
Mathematics 09 02210 g001
Figure 2. Spatial distribution of SBM model results.
Figure 2. Spatial distribution of SBM model results.
Mathematics 09 02210 g002
Table 1. Summary of the recent literature on environmental efficiency evaluation of regions using DEA.
Table 1. Summary of the recent literature on environmental efficiency evaluation of regions using DEA.
Ref.DMUsRTSDEA Model and MethodologyApplication Area
[40]31 provinces in ChinaCRSSBM model and factor analysisWater use
[41]27 EU countriesCRSTwo-stage efficiency measurement based on DDF modelEnvironmental performance
[42]16 Polish regionsCRSCCR model and Hellwig’s method and coefficient of determinationEnvironmental efficiency
[43]30 regions in ChinaCRSSBM model based on undesirable outputsEnvironmental efficiency of industry
[44]24 Mediterranean countriesVRSSBM model and two-stage double bootstrapEnergy efficiency
[45]30 regions in ChinaVRSSBM model and Tobit regression and truncated regression modelsEnergy-environmental performance
[46]30 provinces in ChinaVRSThree-stage analysis based on BCC model & SFAIndustrial eco-efficiency
[47]31 provincial administrative regions in ChinaCRSTwo-stage US-SBM model & spatial autocorrelation analysis using Moran’s IWater resource utilization efficiency
[48]30 Chinese provincesCRSMeta-frontier Super-US-SBM modelComposite eco-efficiency indicators
[49]16 Polish regionsVRSBCC modelLife cycle assessment
[50]30 provinces in ChinaCRSIntermediate approach and performance indicesSocial sustainability
[34]21 cities in Guangdong province, ChinaCRSSuper-SBM and panel regression modelsEstimation of eco-efficiency
[51]108 countriesCRSSlack-based models with undesirable outputsNational environmental efficiency
[52]28 EU countriesVRSSBM model and dynamic DEAEnergy efficiency
[53]30 provinces in ChinaVRSModels with strong and weak disposability of undesirable outputs and Malmquist–Luenberger indexEnvironmental efficiency
[54]17 regions of Central Federal District, RussiaCRSCCR model and Malmquist indexEcological-economic efficiency
[55]30 regional industry systems in ChinaCRSTwo-stage CCR model and regression analysisEnvironmental efficiency
[13]30 provinces in ChinaCRSSBM model and GML productivity index and panel Tobit modelAir pollution emission efficiency
[56]30 Chinese provincesVRSThree-stage DEA approach based on BCC model and SFAEnergy efficiency
[57]48 cities in Bohai Rim, ChinaCRSSuper-US-SBM model and Moran’s IEco-efficiency
[58]282 European regionsVRSHyperbolic distance function measure and metafrontier approachEco-efficiency
[59]28 EU countriesCRSEnvironmental Efficiency IndexEnvironmental efficiency
[60]30 provinces of ChinaVRSSuper-PEBM model and window analysisGreen economic efficiency
[61]14 prefecture-level cities in Guangxi Province, ChinaCRSMeta-US-SBM model and Tobit ModelMining industry eco-efficiency
[62]27 EU CountriesCRSCCR model and fractional regression modelEco-efficiency
[63]30 regions in mainland ChinaVRSDynamic network DEA approach based on the SBM modelEco-eficiency
[64]30 provinces in ChinaCRSSuper-US-SBM model and regression analysisEco-efficiency of industrial investment
[65]30 province-level regions of Chinese mainlandVRSTwo-stage DEA approach based on BCC modelEnergy and environment efficiency
Notes. RTS: returns to scale; CRS: constant returns to scale; VRS: variable returns to scale; SBM: slack-based measure; DDF: directional distance function; US-SBM: SBM with undesirable outputs; Super-SBM: SBM with super efficiency; Super-US-SBM: SBM with undesirable outputs and super efficiency; Super-PEBM: epsilon based measure based on Pearson correlation coefficient; Meta-US-SBM: meta-frontier SBM with undesirable outputs and super efficiency.
Table 2. Dataset statistics.
Table 2. Dataset statistics.
x 1 −0.02.04−0.250.30
x 2 1.44 × 10 3 3.25−0.500.37
y 1 3.69 × 10 4 6.64 × 10 4 2.51 × 10 5 1.00 × 10 4
y 2 6.00 × 10 4 1.12 × 10 3 6.50 × 10 6 1.04 × 10 4
y 3 5.04 × 10 4 5.79 × 10 4 3.90 × 10 5 9.34 × 10 5
y 4 2.34 × 10 3 3.22 × 10 3 1.59 × 10 4 4.88 × 10 4
y 5 1690.76 643.24 15.00 138.11
y 6 0.904 0.946−0.03250.243
Table 3. Weights in CCR-AR and SBM models.
Table 3. Weights in CCR-AR and SBM models.
VariableUpper/Lower Bound of the Weight Restrictions in CCR-AR ModelWeight in SBM Model
x 1 —share of investments in fixed assets aimed at reducing environmental pollution0.5
x 2 —share of current costs in the region’s GRP0.5
y 1 —difference in the intensity of emissions from stationary sources0.1/0.30.182
y 2 —difference in the intensity of emissions from road transport0.1/0.30.182
y 3 —difference in the intensity of discharge of untreated sewage0.1/0.30.182
y 4 —difference in the intensity of fresh water consumption0.05/0.10.09
y 5 —difference in the intensity of waste generation0.1/0.30.182
y 6 —difference in the share of recycling and reuse of waste0.1/0.30.182
Table 4. Correlation of the efficiency scores in CCR, CCR-AR, and SBM models.
Table 4. Correlation of the efficiency scores in CCR, CCR-AR, and SBM models.
CCR ModelCCR-AR ModelSBM Model
CCR model1
CCR-AR model0.98528151
SBM model0.66221650.62951271
Table 5. Regions with the highest and lowest values of the efficiency score (comparison of 3 models).
Table 5. Regions with the highest and lowest values of the efficiency score (comparison of 3 models).
CCR ModelCCR-AR ModelSBM Model
Regions with the highest values of the efficiency score
Bryansk Oblast-11Moscow-11Stavropol Krai-11
Moscow-11Dagestan-11Astrakhan Oblast-13
Dagestan-11Dagestan-13Arkhangelsk Oblast-11
Dagestan-13Tuva Republic-15Orenburg Oblast-13
Ivanovo Oblast-15Bryansk Oblast-11North Ossetia-Alania-13
Tuva Republic-15Ivanovo Oblast-15Dagestan-11
Ivanovo Oblast-17Ivanovo Oblast-17Tuva Republic-17
Novosibirsk Oblast-17Moscow-13Moscow Oblast-11
Moscow-13Adygea-15Kabardino-Balkar Republic-11
North Ossetia-Alania-13Tuva Republic-13Chuvash Republic-11
Adygea-15Moscow-15Kemerovo Oblast-15
Kostroma Oblast-15Novosibirsk Oblast-17Kostroma Oblast-15
Tuva Republic-13Moscow-17Kemerovo Oblast-17
Moscow-15Mari El Republic-15Karelia-11
Kabardino-Balkar Republic-11Tuva Republic-17Ivanovo Oblast-15
Regions with the lowest values of the efficiency score
Murmansk Oblast-15Astrakhan Oblast-13Murmansk Oblast-13
Murmansk Oblast-13Murmansk Oblast-15Kemerovo Oblast-13
Bashkortostan-11Vologda Oblast-13Krasnoyarsk Krai-13
Vologda Oblast-13Sakha (Yakutia) Republic-17Tver Oblast-11
Sakha (Yakutia) Republic-17Bashkortostan-13Sakha (Yakutia) Republic-17
Kemerovo Oblast-13Murmansk Oblast-13Sakha (Yakutia) Republic-13
Bashkortostan-13Sakha (Yakutia) Republic-13Krasnoyarsk Krai-15
Krasnoyarsk Krai-17Krasnoyarsk Krai-17Vologda Oblast-13
Sakha (Yakutia) Republic-13Murmansk Oblast-17Perm Krai-15
Murmansk Oblast-17Kemerovo Oblast-13Volgograd Oblast-15
Krasnoyarsk Krai-11Krasnoyarsk Krai-11Komi Republic-15
Krasnoyarsk Krai-15Krasnoyarsk Krai-15Khabarovsk Krai-11
Krasnoyarsk Krai-13Krasnoyarsk Krai-13Komi Republic-13
Volgograd Oblast-15Volgograd Oblast-15Amur Oblast-13
Lipetsk Oblast-11Lipetsk Oblast-11Lipetsk Oblast-11
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Ratner, S.; Lychev, A.; Rozhnov, A.; Lobanov, I. Efficiency Evaluation of Regional Environmental Management Systems in Russia Using Data Envelopment Analysis. Mathematics 2021, 9, 2210.

AMA Style

Ratner S, Lychev A, Rozhnov A, Lobanov I. Efficiency Evaluation of Regional Environmental Management Systems in Russia Using Data Envelopment Analysis. Mathematics. 2021; 9(18):2210.

Chicago/Turabian Style

Ratner, Svetlana, Andrey Lychev, Aleksei Rozhnov, and Igor Lobanov. 2021. "Efficiency Evaluation of Regional Environmental Management Systems in Russia Using Data Envelopment Analysis" Mathematics 9, no. 18: 2210.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop