Next Article in Journal
Acknowledgement to Reviewers of Systems in 2015

Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

# A Hierarchical Aggregation Approach for Indicators Based on Data Envelopment Analysis and Analytic Hierarchy Process

by
Faculty of Management, Laurentian University, Sudbury, ON P3E 2C6, Canada
Submission received: 24 November 2015 / Revised: 15 December 2015 / Accepted: 12 January 2016 / Published: 20 January 2016

## Abstract

:
This research proposes a hierarchical aggregation approach using Data Envelopment Analysis (DEA) and Analytic Hierarchy Process (AHP) for indicators. The core logic of the proposed approach is to reflect the hierarchical structures of indicators and their relative priorities in constructing composite indicators (CIs), simultaneously. Under hierarchical structures, the indicators of similar characteristics can be grouped into sub-categories and further into categories. According to this approach, we define a domain of composite losses, i.e., a reduction in CI values, based on two sets of weights. The first set represents the weights of indicators for each Decision Making Unit (DMU) with the minimal composite loss, and the second set represents the weights of indicators bounded by AHP with the maximal composite loss. Using a parametric distance model, we explore various ranking positions for DMUs while the indicator weights obtained from a three-level DEA-based CI model shift towards the corresponding weights bounded by AHP. An illustrative example of road safety performance indicators (SPIs) for a set of European countries highlights the usefulness of the proposed approach.

## 1. Introduction

Individual indicators are multidimensional measures that can assess the relative positions of entities (e.g., countries) in a given area [1]. A Composite indicator (CI) is a mathematical aggregation of individual indicators into a single score. Two simple but popular aggregation methods in the context of multi-criteria decision-making (MCDM) are the weighted sum (WS) method and the weighted product (WP) method [2]. Some researchers have recently pointed out that the WP method may have some advantages over the WS method in CI construction [3,4,5]. However, the assignment of weights to indicators is still a main source of difficulty in the application of these methods. Fortunately, the recent methodological advances in operations research and management science (OR/MS) have provided us with two powerful tools, namely data envelopment analysis (DEA) and analytic hierarchy process (AHP), which can be used as weighting and aggregation tools in CI construction.
Data Envelopment Analysis is a nonparametric method to assess the relative efficiency of a group of DMUs based on their distance from the best-practice frontier. In this method each DMU can freely choose its own weights to maximize its performance [6].
The standard DEA models are formulated using multiple inputs and multiple outputs of DMUs. The application of this group of models in CI construction can be found in [7,8,9]. However, in recent years much more attention has been focused on the application of a new group of DEA models in the field of composite indicators which is known as the “benefit of the doubt” (BOD) approach. In the BOD approach, all indicators are treated as outputs without explicit inputs, i.e., the property of “the larger is the better” [10,11,12].
In light of the possibility of neglecting the priority of various indicators, some critics have questioned the validity and stability of CIs obtained via DEA. Decision makers (DMs), in some contexts, have value judgments concerning the relative priority of indicators that should be taken into account in CI construction.
Alternatively, AHP is a systematic MCDM method to generate the true or approximate weights based on the well-defined mathematical structures of pairwise comparison metrics.
The application of AHP in CI construction provides a priori information about the relative priority of indicators [13,14,15]. AHP usually involves three basic functions: structuring complexities, measuring on a ratio-scale and synthesizing [16]. One of the advantages of AHP is its high flexibility to be combined with the other OR/MS techniques [17]. AHP can be combined with DEA in different ways. The most common approach is the estimation of parameters of weight restrictions on the DEA models. AHP estimates the appropriate values for the parameters in the absolute weight restrictions [18], relative weight restrictions [19,20,21,22,23], virtual weight restrictions [24,25] and restrictions on changes of input (output) units [26].
There are a number of other methods that do not necessarily apply additional restrictions to a DEA model. Such as converting the qualitative data in DEA to the quantitative data using AHP [27,28,29,30,31,32,33,34], ranking the efficient/inefficient units in DEA models using AHP in a two stage process [35,36,37], weighting the efficiency scores obtained from DEA using AHP [38], weighting the inputs and outputs in the DEA structure [39,40,41,42], constructing a convex combination of weights using AHP and DEA [43] and estimating missing data in DEA using AHP [44].
The recent studies by Pakkar [45,46,47,48,49,50] demonstrate the effects of imposing weight bounds on the different variants of DEA models using AHP. To this end, AHP has been applied in single-level DEA models [47,48,49,50] and two-level DEA models [45,46]. Due to the complexity of the hierarchical structures of indicators, this paper applies AHP into an additive three-level DEA model in the context of CI construction. Theoretically, the approach proposed in this paper may also be considered as the additive form of the multiplicative three-level DEA-based CI approach to constructing CIs proposed by [51]. In a three-level hierarchy, the indicators of similar characteristics can be grouped into sub-categories and further into categories. A three-level DEA model entirely reflects the characteristics of a generalized multiple level DEA model developed in [52,53]. Since the proposed approach uses AHP in an additive three-level DEA-based model, it contributes to the set of methods currently available for CI construction.

## 2. Methodology

This research has been organized to proceed along the following stages (Figure 1):
• Computing the composite value of each DMU using one-level DEA-based CI model (4). The computed composite values are applied in three-level DEA-based CI model (6).
• Computing the priority weights of indicators for all DMUs using AHP, which impose weight bounds into model (6).
• Obtaining an optimal set of weights for each DMU using three-level DEA-based CI model (6) (minimum composite loss η).
• Obtaining an optimal set of weights for each DMU using model (6) bounded by AHP (maximum composite loss κ). Note that if the AHP weights are added to model (6), we obtain model (10).
• Measuring the performance of each DMU in terms of the relative closeness to the priority weights of indicators. For this purpose, we develop parameter-distance model (11). Increasing a parameter in a defined range of composite loss we explore how much a DM can achieve its goals. This may result in various ranking positions for a DMU in comparison to the other DMUs.
Figure 1. A hierarchical aggregation approach for indicators using a three-level Data Envelopment Analysis (DEA) and Analytic Hierarchy Process (AHP).
Figure 1. A hierarchical aggregation approach for indicators using a three-level Data Envelopment Analysis (DEA) and Analytic Hierarchy Process (AHP).

#### 2.1. DEA-Based CI Model

A DEA-based CI model can be formulated similar to a classical DEA model in which all data are treated as outputs without explicit inputs [54]. In the following, and in line with the more common CI terminology, we will often refer to outputs as “indicators”. In order to eliminate the scale differences between all (output) indicators, and moreover, to ensure that all of them are in the same direction of change the normalized counterparts of indicators, using the distance to reference method, are computed as follows [1]:
$y r j = y ^ r j y ^ r ( max ) , y ^ r ( max ) = max { y ^ r 1 , y ^ r 2 , ... , y ^ r n } for desirable indicators$
$y r j = y ^ r ( min ) y ^ r j , y ^ r ( min ) = min { y ^ r 1 , y ^ r 2 , ... , y ^ r n } for undesirable indicators$
where $y r j$ is the normalized value of (output) indicator $r$ ($r = 1 , 2.. , s$) for DMU $j$ (j = 1, 2,…, n). Now assume that all DMUs have unit input $i$ ($i = 1 , 2 , ... , m$). Then the fractional CCR-DEA model can be developed as follows [55]:
$Max C I k = ∑ r = 1 s u r y r k ∑ i = 1 m v i$
$∑ r = 1 s u r y r j ∑ i = 1 m v i ≤ 1 ∀ j$
$u r , v i > 0 ∀ r , i$
where $C I k$ is the composite indicator of DMU under assessment. $k$ is the index for the DMU under assessment where $k$ ranges over 1, 2,…, $n$. $v i$ and $u r$ are the weights of input i ($i = 1 , 2 , ... , m$) and (output) indicator r ($r = 1 , 2.. , s$). The first set of constraints assures that if the computed weights are applied to a group of $n$ DMUs, ($j = 1 , 2 , ... , n$), they do not attain a composite score of larger than 1. The second set of constraints indicates the non-negative conditions for the model variables. Introducing the constraint $∑ i = 1 m v i = 1$ and performing the operation of substitution, an equivalent linear model can be formulated as follows:
$Max C I k = ∑ r = 1 s u r y r k$
$∑ r = 1 s u r y r j ≤ 1 ∀ j$
$u r > 0 ∀ r .$
Model (4) looks like a DEA model without inputs that extends the standard DEA methodology to the field of CI construction.

#### 2.2. Three-Level DEA-Based CI Model

We develop a three-level DEA model to aggregate the performance of indicators under the (sub) category they belong to by a weighted-average method (Figure 2). Let $y l l ′ r j$ be the value of indicator $r$ ($r = 1 , 2 , ... , s$) of sub-category $l ′$ $( l ′ = 1 , 2 , ... , S ′ )$ of category $l$ $( l = 1 , 2 , ... , S )$ for DMU $j$ ($j = 1 , 2 , ... , n$) after normalizing the original data. Let $u l l ′ r$ be the internal weight of indicator $r$ of sub-category $l ′$ of category $l$ while $∑ r = 1 s u l l ′ r = 1$. Then the value of sub-category $l ′$ of category $l$ for the DMU $j$ is defined as $y l l ′ j = ∑ r = 1 s u l l ′ r y l l ′ r j$. Let $p l l ′$ be the internal weight of sub-category $l ′$ of category $l$ while $∑ l ′ = 1 S ′ p l l ′ = 1$. Then the value of category $l$ is defined as $y l j = ∑ l ′ = 1 S ′ p l l ′ y l l ′ j$. Let $p l$ be the weight of category $l$. To develop a linear model, the new multiplier of indicator $r$ of sub-category $l ′$ of category $l$ is defined as: $u l l ′ r ′ = p l p l l ′ u l l ′ r$. Similarly, the new multiplier of sub-category $l ′$ of category $l$ is defined as: $p ′ l l ′ = p l p l l ′$. Consequently, a linear three-level DEA model for indicators can be developed as follows:
$Max C I k = ∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r k$
$∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r j ≤ 1 ∀ j$
$∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l ∀ l$
$∑ r = 1 s u l l ′ r ′ = p l l ′ ′ ∀ l , l ′$
$u l l ′ r ′ , p l l ′ ′ , p l > 0 ∀ l , l ′ , r .$
Figure 2. A three-level DEA framework for hierarchical indicators.
Figure 2. A three-level DEA framework for hierarchical indicators.
We develop our formulation based on the generalized distance model [56,57] in such a way that the hierarchical structures of indicators, using a weighted-average approach, are taken into consideration [52,53]. Let $C I k *$ ($k = 1 , 2 , ... , n$) be the best attainable composite value for the DMU under assessment, calculated from model (4). We want the composite value $C I k ( u l l ′ r ′ )$, calculated from the set of weights $u l l ′ r ′$, to be closest to $C I k *$. The degree of closeness between $C I k ( u l l ′ r ′ )$ and $C I k *$ is measured as $D t = [ C I k * − C I k ( u l l ′ r ′ ) t ] 1 / t$ with $t ≥ 1$, where $D t$ is a distance measure and $t$ represents the distance parameter. Our definition of “closest” is that the largest distance is at its minimum. On the other hand, the largest distance completely dominates when $t = ∞$. For $t = ∞$, the distance measure is reduced to $D ∞ = max u l l ′ r ′ { C I k * − C I k ( u l l ′ r ′ ) }$. Hence we choose the form of the minimax model: $min u l l ′ r ′ max k = 1 , ... , n { C I k * − C I k ( u l l ′ r ′ ) }$ to minimize a single deviation which is equivalent to the following linear model:
$Min η$
$C I k * − ∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r k ≤ η$
$∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r j ≤ C I j * ∀ j$
$∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l ∀ l$
$∑ r = 1 s u l l ′ r ′ = p l l ′ ′ ∀ l , l ′$
$u l l ′ r ′ , p l l ′ ′ , p l > 0 ∀ l , l ′ , r$
$η ≥ 0.$
Model (6) identifies the minimum composite loss $η$ (eta) needed to arrive at an optimal set of weights. The first constraint ensures that each DMU loses no more than $η$ of its best attainable composite value, $C I k *$. The second set of constraints satisfies that the composite values of all DMUs are less than or equal to their upper bound of $C I j *$.
Two sets of constraints are added to model (6): $∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l$ and $∑ r = 1 s u l l ′ r ′ = p l l ′ ′$, where $u l l ′ r ′$ are indicator multipliers. This implies that the sum of weights under each (sub-) sub-category equals to the weight of that (sub-) sub-category. It should be noted that the original (or internal) weights used for calculating the weighted averages are obtained as $u l l ′ r = u l l ′ r ′ / p l l ′ ′$ and $p l l ′ = p l l ′ ′ / p l$.

#### 2.3. Prioritizing Indicator Weights Using AHP

Model (6) identifies the minimum composite loss $η$ (eta) needed to arrive at a set of weights of indicators by the internal mechanism of DEA. On the other hand, the priority weights of indicators, and the corresponding (sub) categories are defined out of the internal mechanism of DEA by AHP (Figure 3).
Figure 3. The AHP model for prioritizing indicators.
Figure 3. The AHP model for prioritizing indicators.
In order to more clearly demonstrate how AHP is integrated into the three-level DEA-based CI model, this research presents an analytical process in which indicator weights are bounded by the AHP method. The AHP procedure for imposing weight bounds may be broken down into the following steps:
Step 1: A decision maker makes a pairwise comparison matrix of different criteria, denoted by $A$, with the entries of $a l q$ $( l = q = 1 , 2 , ... , S )$. The comparative importance of criteria is provided by the decision maker using a rating scale. Saaty [16] recommends using a 1–9 scale.
Step 2: The AHP method obtains the priority weights of criteria by computing the eigenvector of matrix $A$ (Equation (7)), $w = ( w 1 , w 2 , ... , w S ) T$, which is related to the largest eigenvalue, $λ max$.
$A w = λ max w$
To determine whether or not the inconsistency in a comparison matrix is reasonable the random consistency ratio, $C . R .$, can be computed by the following equation:
$C . R . = λ max − N ( N − 1 ) R . I .$
where $R . I .$ is the average random consistency index and $N$ is the size of a comparison matrix. In a similar way, the priority weights of (sub-) sub-criteria under each (sub-) criterion can be computed. To obtain the weight bounds for indicator weights in the three-level DEA-based CI model, this study aggregates the priority weights of three different levels in AHP as follows:
$u ¯ l l ′ r = w l e l l ′ f l l ′ r , ∑ l = 1 S w l = 1 , ∑ l ′ = 1 s ′ e l l ′ = 1 and ∑ r = 1 s f l l ′ r = 1$
where $w l$ is the priority weight of criterion $l$ ($l = 1 , ... , S$) in AHP, $e l l ′$ is the priority weight of sub-criterion $l ′$ $( l ′ = 1 , 2 , ... , S ′ )$ under criterion $l$ and $f l l ′ r$ is sub-sub-criterion $r$ ($r = 1 , ... , s$) under sub-criterion $l ′$.
In order to estimate the maximum composite loss $κ$ (kappa) necessary to achieve the priority weights of indicators for each DMU the following linear program is proposed:
$Min κ$
$u l l ′ r ′ = α u ¯ l l ′ r ∀ l , l ′ , r$
$C I k * − ∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r k ≤ κ$
$∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r j ≤ C I j * ∀ j$
$∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l ∀ l$
$∑ r = 1 s u l l ′ r ′ = p l l ′ ′ ∀ l , l ′$
$u l l ′ r ′ , p l l ′ ′ , p l > 0 ∀ l , l ′ , r$
$κ ≥ 0 , α > 0.$
The first sets of constraints change the AHP computed weights to weights for the new system by means of a scaling factor $α$. The scaling factor $α$ is added to avoid the possibility of contradicting constraints leading to infeasibility or underestimating the relative composite scores of DMUs [58]. The optimal solution to model (10) produces a set of weights for indicators that are used to compute the performance of DMUs.
It should be noted that incorporating absolute weight bounds, using AHP, for indicator weights in a DEA-based CI model is consistent with the common practice of constructing composite indicators. According to this practice, the priority weights of indicators can be used directly in an aggregation function to synthetize indicators’ values into composite values [1]. In addition, this form of placing restrictions on indicator weights simply allows us to identify a specific range of variation between two systems of weights obtained from models (6) and (10).

#### 2.4. A Parametric Distance Model

We can now develop a parametric distance model for various discrete values of parameter $θ$ such that $η ≤ θ ≤ κ$. Let $u l l ′ r ′ ( θ )$ be the weights of indicators for a given value of parameter $θ$, where indicators are under sub-category $l ′$ $( l ′ = 1 , 2 , ... , S ′ )$ of category $l$ ($l = 1 , 2 , ... , S$). Let $u ′ l l ′ r * ( κ )$ be the priority weights of indicators under sub-category $l ′$ of category $l$, obtained from model (10). Our objective is to minimize the total deviations between $u l l ′ r ′ ( θ )$ and $u ′ l l ′ r * ( κ )$ with the shortest Euclidian distance measure subject to the following constraints:
$Min Z k ( θ ) = ( ∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s ( u l l ′ r ′ − u ′ l l ′ r * ( κ ) ) 2 ) 1 / 2$
$C I k * − ∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r k ≤ θ$
$∑ l = 1 S ∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ y l l ′ r k ≤ θ ≤ C I j * ∀ j$
$∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l ∀ l$
$∑ r = 1 s u l l ′ r ′ = p l l ′ ′ ∀ l , l ′$
$u l l ′ r ′ , p l l ′ ′ , p l > 0 ∀ l , l ′ , r .$
Because the range of deviations computed by the objective function is different for each DMU, it is necessary to normalize it by using relative deviations rather than absolute ones [59]. Hence, the normalized deviations can be computed by:
$Δ k ( θ ) = Z k * ( η ) − Z k * ( θ ) Z k * ( η )$
where $Z k * ( θ )$ is the optimal value of the objective function for $η ≤ θ ≤ κ$. We define $Δ k ( θ )$ as a measure of closeness which represents the relative closeness of each DMU to the weights obtained from model (10) in the range [0, 1]. Increasing the parameter $( θ )$, we improve the deviations between the two systems of weights obtained from models (6) and (10) which may lead to different ranking positions for each DMU in comparison to the other DMUs. It should be noted that in a special case where the parameter $θ = κ = 0$, we assume $Δ k ( θ )$ = 1.

## 3. A Numerical Example: Road Safety Performance Indicators

In this section we present the application of the proposed approach to assess the road safety performance of a set of 13 European countries (or DMUs): Austria (AUT), Belgium (BEL), Finland (FIN), France (FRA), Hungary (HUN), Ireland (IRL), Lithuania (LTU), Netherlands (NLD), Poland (POL), Portugal (PRT), Slovenia (SVN), Sweden (SWE) and Switzerland (CHE). The data for eleven hierarchical indicators that compose SPIs for these countries have been adopted from [52]. The eight SPIs related to alcohol and speed are undesirable indicators while the three SPIs related to protective systems are desirable ones. The resulting normalized data based on Equations (1) and (2) are presented in Table 1.
Taking the percentage of speed limit violation on rural roads as an example, Slovenia performs the best (1.000) while Poland the worst (0.014) and all other countries’ values lie within this interval.
The results of the AHP model for prioritizing hierarchical SPIs as constructed by the author in Expert Choice software are presented in Table 2. One can argue that the priority weights of SPIs must be judged by road safety experts. However, since the aim of this section is just to show the application of the proposed approach on numerical data, we see no problem to use our judgment alone.
Table 1. Normalized data on the eleven hierarchical safety performance indicators (SPIs).
Table 1. Normalized data on the eleven hierarchical safety performance indicators (SPIs).
CountriesAlcoholSpeedProtective Systems
% of Drivers above Legal Alcohol Limit In Roadside Police Tests% of Alcohol Related FatalitiesMean SpeedSpeed Limit Violation (%)Seat BeltChild Restraint
MotorwaysRural RoadsUrban RoadsMotorwaysRural RoadsUrban RoadsDaytime Seatbelt Wearing Rate in Front Seats of Light Vehicles (%)Daytime Seatbelt Wearing Rate in Rear Seats of Light Vehicles (%)Daytime Usage Rate of Child Restraints (%)
AUT0.1160.4630.9380.7810.8020.7660.0510.2540.9040.6990.863
BEL0.0680.6540.8460.7430.7680.3480.0290.2220.7990.4880.729
FIN0.5930.1360.9630.7290.9070.4090.0230.3230.9110.9220.716
FRA0.2630.1230.9330.7870.8380.5050.0370.3181.0001.0000.937
HUN0.2790.2830.9550.7930.8170.3620.0330.2300.7270.5010.433
IRL0.2370.1190.9450.7620.7241.0000.0320.2230.9010.9140.857
LTU0.5550.3211.0000.7130.71407890.0250.3180.6090.3660.404
NLD0.0811.0000.8990.7400.8810.4540.0200.2340.9590.8900.758
POL0.0910.4380.8060.6970.6470.2900.0150.1650.7990.5890.905
PRT0.1370.6100.8470.6180.9190.3020.0140.3600.8810.5740.591
SVN0.1220.0780.9641.0000.7130.4801.0000.1630.8740.5510.672
SWE1.0000.3570.8830.7170.8700.2410.0190.2590.9730.9271.000
CHE0.2770.2300.9430.7571.0000.7100.0431.0000.8870.8050.895
Table 2. The AHP hierarchical model for SPIs.
Table 2. The AHP hierarchical model for SPIs.
Objective LevelCriteria LevelSub-Criteria LevelSub-Sub-Criteria Level
$w 1 =$ 0.2727
% of drivers above legal alcohol limit
$e 11 =$ 0.333
% of drivers above legal alcohol limit
$f 111 =$ 1.000
% of alcohol-related fatalities
$e 12 =$ 0.667
% of alcohol-related fatalities
$f 121 =$ 1.000
Speed
$w 2 =$ 0.5454
Mean speed
$e 21 =$ 0.60
Mean speed of vehicles on motorways
$f 211 =$ 0.081
Mean speed of vehicles on rural roads,
$f 212 =$ 0.342
Mean speed of vehicles on urban roads,
$f 213 =$ 0.577
Speed limit
violations
$e 22 =$ 0.40
% of vehicles exceeding the speed limit
on motorways
$f 221 =$ 0.081
% of vehicles exceeding the speed limit on
$f 222 =$ 0.342
% of vehicles exceeding the speed limit on urban roads
$f 223 =$ 0.577
Protective systems $w 3 =$ 0.1818Seat belt
$e 31 =$ 0.40
Daytime seatbelt wearing rate in front seats of light vehicles (%)
$f 311 =$ 0.60
Daytime seatbelt wearing rate in rear seats of light vehicles (%)
$f 312 =$ 0.40
Child
Restraint
$e 32 =$ 0.60
Daytime usage rate of child restraints (%)
$f 321 =$ 1.000
Solving model (6) for the country under assessment, we obtain an optimal set of weights with minimum composite loss $( η )$. Since the raw data are normalized, the weights obtained from this model are meaningful and have an intuitive explanation. As a result, we can later set meaningful bounds on the weights in terms of the relative priority of indicators. Taking Austria as an example in Table 3, with the composite value of one obtained from model (4), we can observe that the alcohol-related fatality rate is 26 times less important than the mean speed of vehicles on motorways and about 3.2 times less important than the daytime usage rate of child restraints. Clearly, the other indicators are ignored in this assessment by assigning zero weights which are equivalent to excluding those indicators from the analysis. This kind of situation can be remedied by including the opinion of experts in defining the relative priority of indicators.
Table 3. Optimal weights of hierarchical SPIs obtained from model (6) for Austria.
Table 3. Optimal weights of hierarchical SPIs obtained from model (6) for Austria.
Weights of CategoriesWeights of Sub-CategoriesWeights of Sub-Sub-Categories
$p 1 =$ 0.0361$p 11 ′ =$ 0.0000$u 111 ′ =$ 0.0000
$p 12 ′ =$ 0.0361$u 121 ′ =$ 0.0361
$p 2 =$ 0.9415$p 21 ′ =$ 0.9415$u 211 ′ =$ 0.9415
$u 212 ′ =$ 0.0000
$u 213 ′ =$ 0.0000
$p 22 ′ =$ 0.0000$u 221 ′ =$ 0.0000
$u 222 ′ =$ 0.0000
$u 223 ′ =$ 0.0000
$p 3 =$ 0.1160$p 31 ′ =$ 0.0000$u 311 ′ =$ 0.0000
$u 312 ′ =$ 0.0000
$p 32 ′ =$ 0.1160$u 321 ′ =$ 0.1160
$η = 0.0000$
It should be noted that the composite value of all countries calculated from model (6) is identical to that calculated from model (4). Therefore, the minimum composite loss for the country under assessment is $η = 0$ (Table 4). This implies that the measure of relative closeness to the AHP weights for the country under assessment is $Δ k ( η ) = 0$. On the other hand, solving model (10) for the country under assessment, we adjust the priority weights of hierarchical SPIs obtained from AHP in such a way that they become compatible with the weights’ structure in the three level DEA-based CI models. Table 5 presents the optimal weights of hierarchical SPIs as well as its scaling factor for all countries.
Table 4. Minimum and maximum losses in composite values for each country.
Table 4. Minimum and maximum losses in composite values for each country.
Countries$C I k *$ηκ
AUT1.0000.0000.158
BEL0.9380.0000.127
FIN1.0000.0000.173
FRA1.0000.0000.184
HUN1.0000.0000.284
IRL1.0000.0000.254
LTU1.0000.0000.270
NLD1.0000.0000.023
POL0.9550.0000.224
PRT0.9780.0000.139
SVN1.0000.0000.208
SWE1.0000.0000.040
CHE1.0000.0000.000
Table 5. Optimal weights of hierarchical SPIs obtained from model (10) for all countries.
Table 5. Optimal weights of hierarchical SPIs obtained from model (10) for all countries.
Weights of CategoriesWeights of Sub-CategoriesWeights of Sub-Sub-Categories
$p 1 =$ 0.4089$p 11 ′ =$ 0.1362$u 111 ′ =$ 0.1362
$p 12 ′ =$ 0.2727$u 121 ′ =$ 0.2727
$p 2 =$ 0.8178$p 21 ′ =$ 0.4907$u 211 ′ =$ 0.0397
$u 212 ′ =$ 0.1678
$u 213 ′ =$ 0.2831
$p 22 ′ =$ 0.3271$u 221 ′ =$ 0.0265
$u 222 ′ =$ 0.1119
$u 223 ′ =$ 0.1888
$p 3 =$ 0.2726$p 31 ′ =$ 0.1090$u 311 ′ =$ 0.0654
$u 312 ′ =$ 0.0436
$p 32 ′ =$ 0.1636$u 321 ′ =$ 0.1636
$α =$ 1.4993
Note that the priority weights of AHP used for incorporating weight bounds on indicator weights in model (10) are obtained as $u ¯ l l ′ r = u l l ′ r ′ α$. Similarly, the priority weights of AHP at criteria level can be obtained as $w l = p l α$ while $∑ l ′ = 1 S ′ ∑ r = 1 s u l l ′ r ′ = p l$ and $∑ r = 1 s u l l ′ r ′ = p l l ′ ′$. In addition, The priority weights of AHP at sub-criteria and sub-sub-criteria levels can be obtained as $e l l ′ = p l l ′ ′ / p l$ and $f l l ′ r = u l l ′ r ′ / p l l ′ ′$, respectively.
The maximum composite loss for each country to achieve the corresponding weights in model (10) is equal to $κ$ (Table 4). As a result, the measure of relative closeness to the priority weights of SPIs for the country under assessment is $Δ k ( κ )$ = 1. Going one step further to the solution process of the parametric distance model (11), we proceed to the estimation of total deviations from the AHP weights for each country while the parameter $θ$ is $0 ≤ θ ≤ κ$. Table 6 represents the ranking position of each country based on the minimum deviation from the priority weights of indicators for $θ = 0$. It should be noted that in a special case where the parameter $θ = κ = 0$ we assume $Δ k ( θ ) = 1.$
Table 6. The ranking position of each country based on the minimum distance to priority weights of SPIs.
Table 6. The ranking position of each country based on the minimum distance to priority weights of SPIs.
Countries$Z * ( η )$Rank
AUT0.3326
BEL0.70511
FIN0.4389
FRA0.3387
HUN0.59110
IRL0.3315
LTU0.3818
NLD0.0212
POL0.71812
PRT0.78213
SVN0.1664
SWE0.0353
CHE0.0001
Table 6 shows that Switzerland (CHE) is the best performer in terms of the $C I$ value and the relative closeness to the priority weights of indicators in comparison to the other countries. Nevertheless, increasing the value of $θ$ from 0 to $κ$ has two main effects on the performance of the other countries: improving the degree of deviations and reducing the value of composite indicator. This, of course, is a phenomenon, one expects to observe frequently. The graph of $Δ ( θ )$ versus $θ$, as shown in Figure 4, is used to describe the relation between the relative closeness to the priority weights of indicators and composite loss for each country. This may result in different ranking positions for each country in comparison to the other countries (Appendix A).
Figure 4. The relative closeness to the priority weights of indicators [∆(θ)], versus composite loss (θ) for each country.
Figure 4. The relative closeness to the priority weights of indicators [∆(θ)], versus composite loss (θ) for each country.
In order to clearly discover the effect of composite loss on the countries’ ranking as shown in Appendix A, we performed a Kruskal-Wallis test. The Kruskal-Wallis test compares the medians of rankings to determine whether there is a significant difference between them. The result of the test reveals that its p-value is quite smaller than 0.01. Therefore, we conclude that increasing composite loss in the whole range [0.0, 0.29] changes the countries’ ranking significantly. Note that at $θ = 0$ the countries can be ranked based on $Z k * ( 0 )$ from the closest to the furthest from the priority weights of SPIs. For instance, at $θ = 0$, Sweden, Ireland and Austria with composite values of one, are ranked in 3rd, 5th and 6th places, respectively, while Belgium, Poland and Portugal with composite values of less than one are ranked in 11th, 12th and 13th places, respectively (Table 4 and Table 6). However, with a small composite loss at $θ = 0.01$, Belgium, Poland and Portugal take 3rd, 6th and 5th places in the rankings, respectively. Using this example, as a guideline, it is relatively easy to rank the countries in terms of distance to the priority weights of SPIs. At $θ = 0.02$, Sweden moves up into 3rd place again while Belgium drops into 4th place. It is clear that both measures, $Z k * ( 0 )$ and $Δ k ( θ )$, are necessary to explain the ranking position of a country.

## 4. Conclusions

We develop a hierarchical aggregation approach based on DEA and AHP methodologies to construct composite indicators. We define two sets of weights of indicators in a three-level DEA framework. All indicators are treated as benefit type. The first set represents the weights of indicators with minimum composite loss. The second set represents the corresponding priority weights of hierarchical indicators, using AHP, with maximum composite loss. We assess the performance of each DMU in comparison to the other DMUs based on the relative closeness of the first set of weights to the second set of weights. Improving the measure of relative closeness in a defined range of composite loss, we explore the various ranking positions for the DMU under assessment in comparison to the other DMUs. To demonstrate the effectiveness of the proposed approach, we apply it to construct a composite road safety performance index for eleven hierarchical indicators that compose SPIs for 13 European countries.

## Conflicts of Interest

The authors declare no conflict of interest.

## Appendix A

Table A1. The measure of relative closeness to the priority weights of hierarchical SPIs [$Δ k ( θ )$ ] vs. composite loss [θ] for each country.
Table A1. The measure of relative closeness to the priority weights of hierarchical SPIs [$Δ k ( θ )$ ] vs. composite loss [θ] for each country.
θAUTBELFINFRAHUNIRLLTUNLDPOLPRTSVNSWECHE
00.00000.00000.00000.00000.00000.00000.00000.00000.00000.00000.00000.00001.0000
RankN/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A1
0.010.07880.30020.08080.05950.07840.04500.04190.43670.19510.24540.04920.25211.0000
Rank83710912132651141
0.020.14790.40180.14970.11900.12950.08630.08230.87330.26400.35720.09830.50411.0000
Rank84710912132651131
0.030.21600.48520.21480.17840.17930.12690.12231.00000.32680.42860.14730.75621.0000
Rank74810912131651131
0.040.28380.56050.27970.23780.22720.16750.16241.00000.38450.51330.19631.00001.0000
Rank74891012131651111
0.050.35120.63120.34440.29700.26790.20810.20231.00000.44180.58460.24521.00001.0000
Rank74891012131651111
0.060.41850.71130.40910.35620.30730.24860.24221.00000.48320.64910.29411.00001.0000
Rank74891012131651111
0.070.48570.78130.47360.41520.34540.28910.28191.00000.51950.71210.34281.00001.0000
Rank74891012131651111
0.080.55270.84640.53790.47400.38180.32960.32151.00000.55560.77390.39131.00001.0000
Rank74891112131651011
0.090.61960.89910.60190.53250.41630.36990.36091.00000.59150.83310.43971.00001.0000
Rank64791112131851011
0.10.68610.93960.66550.59060.44970.41020.40011.00000.62730.88580.48791.00001.0000
Rank64791112131851011
0.110.75200.96920.72830.64810.48290.45040.43911.00000.66280.92900.53581.00001.0000
Rank64791112131851011
0.120.81660.98690.78960.70460.51600.49050.47781.00000.69800.96450.58331.00001.0000
Rank64781112131951011
0.130.87771.00000.84760.75960.54900.53040.51611.00000.73280.98380.63091.00001.0000
Rank1411516192021117111811
0.140.92711.00000.89670.81150.58180.57010.55401.00000.76691.00000.67841.00001.0000
Rank141151619202111711811
0.150.96811.00000.92890.85770.61450.60960.59131.00000.80071.00000.72601.00001.0000
Rank141151619202111711811
0.161.00001.00000.95920.89980.64680.64890.62791.00000.83401.00000.77351.00001.0000
Rank141151620192111711811
0.171.00001.00000.98960.94090.67890.68800.66351.00000.86651.00000.82111.00001.0000
Rank11781211131911011
0.181.00001.00001.00000.98200.71040.72690.69871.00000.89731.00000.86871.00001.0000
Rank11181211131911011
0.191.00001.00001.00001.00000.74150.76550.73351.00000.92531.00000.91621.00001.0000
Rank11111211131911011
0.21.00001.00001.00001.00000.77240.80360.76771.00000.95381.00000.96381.00001.0000
Rank11111211131101911
0.211.00001.00001.00001.00000.80290.84080.80151.00000.97361.00001.00001.00001.0000
Rank11111211131101111
0.221.00001.00001.00001.00000.83290.87720.83491.00000.99191.00001.00001.00001.0000
Rank11111311121101111
0.231.00001.00001.00001.00000.86210.91360.86811.00001.00001.00001.00001.00001.0000
Rank1111131112111111
0.241.00001.00001.00001.00000.89050.95010.90131.00001.00001.00001.00001.00001.0000
Rank1111131112111111
0.251.00001.00001.00001.00000.91800.98650.93451.00001.00001.00001.00001.00001.0000
Rank1111131112111111
0.261.00001.00001.00001.00000.94321.00000.96771.00001.00001.00001.00001.00001.0000
Rank111113112111111
0.271.00001.00001.00001.00000.96691.00000.99991.00001.00001.00001.00001.00001.0000
Rank111113112111111
0.281.00001.00001.00001.00000.99061.00001.00001.00001.00001.00001.00001.00001.0000
Rank11111311111111
0.291.00001.00001.00001.00001.00001.00001.00001.00001.00001.00001.00001.00001.0000
Rank1111111111111

## References

1. OECD. Organisation for Economic Co-operation and Development (OECD). In Handbook on Constructing Composite Indicators: Methodology and User Guide; OECD Publishing: Paris, France, 2008. [Google Scholar]
2. San Cristobal Mateo, J.R. Multi-Criteria Analysis in the Renewable Energy Industry; Springer: London, UK, 2012. [Google Scholar]
3. Ebert, U.; Welsch, H. Meaningful environmental indices: A social choice approach. J. Environ. Econ. Manag. 2004, 47, 270–283. [Google Scholar] [CrossRef]
4. Munda, G.; Nardo, M. Noncompensatory/nonlinear composite indicators for ranking countries: A defensible setting. Appl. Econ. 2009, 41, 1513–1523. [Google Scholar] [CrossRef]
5. Zhou, P.; Ang, B.W. Comparing MCDA aggregation methods in constructing composite indicators using the Shannon-Spearman measure. Soc. Indic. Res. 2009, 94, 83–96. [Google Scholar] [CrossRef]
6. Cooper, W.W.; Seiford, L.M.; Zhu, J. Handbook on Data Envelopment Analysis; Kluwer Academic Publishers: Norwel, MA, USA, 2004. [Google Scholar]
7. Chaaban, J.M. Measuring youth development: A nonparametric cross-country “youth welfare index”. Soc. Indic. Res. 2009, 93, 351–358. [Google Scholar] [CrossRef]
8. Murias, P.; de Miguel, J.C.; Rodriguez, D. A composite indicator for university quality assessment: The case of Spanish higher education system. Soc. Indic. Res. 2008, 89, 129–146. [Google Scholar] [CrossRef]
9. Murias, P.; Martinez, F.; de Miguel, C. An economic wellbeing index for the Spanish provinces: A data envelopment analysis approach. Soc. Indic. Res. 2006, 77, 395–417. [Google Scholar] [CrossRef]
10. Cherchye, L.; Moesen, W.; Rogge, N.; van Puyenbroeck, T.; Saisana, M.; Saltelli, A.; Liska, R.; Tarantola, S. Creating composite indicators with DEA and robustness analysis: The case of the technology achievement index. J. Oper. Res. Soc. 2008, 59, 239–251. [Google Scholar] [CrossRef]
11. Cherchye, L.; Moesen, W.; Rogge, N.; van Puyenbroeck, T. An introduction to “benefit of the doubt” composite indicators. Soc. Indic. Res. 2007, 82, 111–145. [Google Scholar] [CrossRef]
12. Zhou, P.; Ang, B.W.; Poh, K.L. A mathematical programming approach to constructing composite Indicators. Ecol. Econ. 2007, 62, 291–297. [Google Scholar] [CrossRef]
13. Arora, A.; Arora, A.S.; Palvia, S. Social media index valuation: Impact of technological, social, economic, and ethical dimensions. J. Promot. Manag. 2014, 20, 328–344. [Google Scholar] [CrossRef]
14. Dedeke, N. Estimating the weights of a composite index using AHP: Case of the environmental performance index. Br. J. Arts Soc. Sci. 2013, 11, 199–221. [Google Scholar]
15. Singh, R.K.; Murty, H.R.; Gupta, S.K.; Dikshit, A.K. Development of composite sustainability performance index for steel industry. Ecol. Indic. 2007, 7, 565–588. [Google Scholar] [CrossRef]
16. Saaty, T.S. The Analytic Hierarchy Process; McGraw-Hill: New York, NY, USA, 1980. [Google Scholar]
17. Vaidya, O.S.; Kumar, S. Analytic hierarchy process: An overview of applications. Eur. J. Oper. Res. 2006, 169, 1–29. [Google Scholar] [CrossRef]
18. Entani, T.; Ichihashi, H.; Tanaka, H. Evaluation method based on interval AHP and DEA. Cent. Eur. J. Oper. Res. 2004, 12, 25–34. [Google Scholar]
19. Kong, W.; Fu, T. Assessing the performance of business colleges in Taiwan using data envelopment analysis and student based value-added performance indicators. Omega 2012, 40, 541–549. [Google Scholar] [CrossRef]
20. Lee, A.H.I.; Lin, C.Y.; Kang, H.Y.; Lee, W.H. An integrated performance evaluation model for the photovoltaics industry. Energies 2012, 5, 1271–1291. [Google Scholar] [CrossRef]
21. Liu, C.M.; Hsu, H.S.; Wang, S.T.; Lee, H.K. A performance evaluation model based on AHP and DEA. J. Chin. Inst. Ind. Eng. 2005, 22, 243–251. [Google Scholar] [CrossRef]
22. Takamura, Y.; Tone, K. A comparative site evaluation study for relocating Japanese government agencies out of Tokyo. Socio-Econ. Plan. Sci. 2003, 37, 85–102. [Google Scholar] [CrossRef]
23. Tseng, W.; Yang, C.; Wang, D. Using the DEA and AHP methods on the optimal selection of IT strategic alliance partner. In Proceedings of the 2009 International Conference on Business and Information (BAI 2009), Kuala Lumpur, Malaysia, 23–30 June 2009; Academy of Taiwan Information Systems Research (ATISR): Taipei, Taiwan, 2009; Volume 6, pp. 1–15. [Google Scholar]
24. Premachandra, I.M. Controlling factor weights in data envelopment analysis by Incorporating decision maker’s value judgement: An approach based on AHP. J. Inf. Manag. Sci. 2001, 12, 1–12. [Google Scholar]
25. Shang, J.; Sueyoshi, T. Theory and Methodology—A unified framework for the selection of a Flexible Manufacturing System. Eur. J. Oper. Res. 1995, 85, 297–315. [Google Scholar]
26. Lozano, S.; Villa, G. Multiobjective target setting in data envelopment analysis using AHP. Comput. Oper. Res. 2009, 36, 549–564. [Google Scholar] [CrossRef]
27. Azadeh, A.; Ghaderi, S.F.; Izadbakhsh, H. Integration of DEA and AHP with computer simulation for railway system improvement and optimization. Appl. Math. Comput. 2008, 195, 775–785. [Google Scholar] [CrossRef]
28. Ertay, T.; Ruan, D.; Tuzkaya, U.R. Integrating data envelopment analysis and analytic hierarchy for the facility layout design in manufacturing systems. Inf. Sci. 2006, 176, 237–262. [Google Scholar] [CrossRef]
29. Jyoti, T.; Banwet, D.K.; Deshmukh, S.G. Evaluating performance of national R & D organizations using integrated DEA-AHP technique. Int. J. Product. Perform. Manag. 2008, 57, 370–388. [Google Scholar]
30. Korpela, J.; Lehmusvaara, A.; Nisonen, J. Warehouse operator selection by combining AHP and DEA methodologies. Int. J. Product. Econ. 2007, 108, 135–142. [Google Scholar] [CrossRef]
31. Lin, M.; Lee, Y.; Ho, T. Applying integrated DEA/AHP to evaluate the economic performance of local governments in china. Eur. J. Oper. Res. 2011, 209, 129–140. [Google Scholar] [CrossRef]
32. Ramanathan, R. Supplier selection problem: Integrating DEA with the approaches of total cost of ownership and AHP. Supply Chain Manag. 2007, 12, 258–261. [Google Scholar] [CrossRef]
33. Raut, R.D. Environmental performance: A hybrid method for supplier selection using AHP-DEA. Int. J. Bus. Insights Transform. 2011, 5, 16–29. [Google Scholar]
34. Yang, T.; Kuo, C. A hierarchical AHP/DEA methodology for the facilities layout design problem. Eur. J. Oper. Res. 2003, 147, 128–136. [Google Scholar] [CrossRef]
35. Ho, C.B.; Oh, K.B. Selecting internet company stocks using a combined DEA and AHP approach. Int. J. Syst. Sci. 2010, 41, 325–336. [Google Scholar] [CrossRef]
36. Jablonsky, J. Measuring the efficiency of production units by AHP models. Math. Comput. Model. 2007, 46, 1091–1098. [Google Scholar] [CrossRef]
37. Sinuany-Stern, Z.; Mehrez, A.; Hadada, Y. An AHP/DEA methodology for ranking decision making units. Int. Trans. Oper. Res. 2000, 7, 109–124. [Google Scholar] [CrossRef]
38. Chen, T.Y. Measuring firm performance with DEA and prior information in Taiwan’s banks. Appl. Econ. Lett. 2002, 9, 201–204. [Google Scholar] [CrossRef]
39. Cai, Y.; Wu, W. Synthetic financial evaluation by a method of combining DEA with AHP. Int. Trans. Oper. Res. 2001, 8, 603–609. [Google Scholar] [CrossRef]
40. Feng, Y.; Lu, H.; Bi, K. An AHP/DEA method for measurement of the efficiency of R & D management activities in universities. Int. Trans. Oper. Res. 2004, 11, 181–191. [Google Scholar]
41. Kim, T. Extended Topics in the Integration of Data Envelopment Analysis and the Analytic Hierarchy Process in Decision Making. Ph.D. Thesis, Louisiana State University, Baton Rouge, LA, USA, 2000. [Google Scholar]
42. Pakkar, M.S. Using the AHP and DEA methodologies for stock selection. In Business Performance Measurement and Management; Charles, V., Kumar, M., Eds.; Cambridge Scholars Publishing: Newcastle upon Tyne, UK, 2014; pp. 566–580. [Google Scholar]
43. Liu, C.; Chen, C. Incorporating value judgments into data envelopment analysis to improve decision quality for organization. J. Am. Acad. Bus. 2004, 5, 423–427. [Google Scholar]
44. Saen, R.F.; Memariani, A.; Lotfi, F.H. Determining relative efficiency of slightly non-homogeneous decision making units by data envelopment analysis: A case study in IROST. Appl. Math. Comput. 2005, 165, 313–328. [Google Scholar] [CrossRef]
45. Pakkar, M.S. An integrated approach based on DEA and AHP. Comput. Manag. Sci. 2015, 12, 153–169. [Google Scholar] [CrossRef]
46. Pakkar, M.S. Using data envelopment analysis and analytic hierarchy process for multiplicative aggregation of financial ratios. J. Appl. Oper. Res. 2015, 7, 23–35. [Google Scholar]
47. Pakkar, M.S. Measuring the efficiency and effectiveness of decision making units by integrating the DEA and AHP methodologies. In Business Performance Measurement and Management; Charles, V., Kumar, M., Eds.; Cambridge Scholars Publishing: Newcastle upon Tyne, UK, 2014; pp. 552–565. [Google Scholar]
48. Pakkar, M.S. Using DEA and AHP for ratio analysis. Am. J. Oper. Res. 2014, 4, 268–279. [Google Scholar] [CrossRef]
49. Pakkar, M.S. Using data envelopment analysis and analytic hierarchy process to construct composite indicators. J. Appl. Oper. Res. 2014, 6, 174–187. [Google Scholar]
50. Pakkar, M.S. An integrated approach to the DEA and AHP methodologies in decision making. In Data Envelopment Analysis and Its Applications to Management; Charles, V., Kumar, M., Eds.; Cambridge Scholars Publishing: Newcastle upon Tyne, UK, 2012; pp. 136–149. [Google Scholar]
51. Pakkar, M.S. Using DEA and AHP for multiplicative aggregation of indicators. Am. J. Oper. Res. 2015, 5, 327–336. [Google Scholar] [CrossRef]
52. Shen, Y.; Hermans, E.; Brijs, T.; Wets, G. Data envelopment analysis for composite indicators: A multiple layer model. Soc. Indic. Res. 2013, 114, 739–756. [Google Scholar] [CrossRef]
53. Shen, Y.; Hermans, E.; Ruan, D.; Wets, G.; Brijs, T.; Vanhoof, K. A generalized multiple layer data envelopment analysis model for hierarchical structure assessment: A case study in road safety performance evaluation. Expert Syst. Appl. 2011, 38, 15262–15272. [Google Scholar] [CrossRef]
54. Liu, W.B.; Zhang, D.Q.; Meng, W.; Li, X.X.; Xu, F. A study of DEA models without explicit inputs. Omega 2011, 39, 472–480. [Google Scholar] [CrossRef]
55. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the Efficiency of Decision Making Units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
56. Hashimoto, A.; Wu, D.A. A DEA-compromise programming model for comprehensive ranking. J. Oper. Res. Soc. Jpn. 2004, 47, 73–81. [Google Scholar]
57. Mavi, R.K.; Mavi, N.K.; Mavi, L.K. Compromise programming for common weight analysis in data envelopment analysis. Am. J. Sci. Res. 2012, 45, 90–109. [Google Scholar]
58. Podinovski, V.V. Suitability and redundancy of non-homogeneous weight restrictions for measuring the relative efficiency in DEA. Eur. J. Oper. Res. 2004, 154, 380–395. [Google Scholar] [CrossRef]
59. Romero, C.; Rehman, T. Multiple Criteria Analysis for Agricultural Decisions, 2nd ed.; Elsevier: Amsterdam, The Netherlands, 2003. [Google Scholar]

## Share and Cite

MDPI and ACS Style

Pakkar, M.S. A Hierarchical Aggregation Approach for Indicators Based on Data Envelopment Analysis and Analytic Hierarchy Process. Systems 2016, 4, 6. https://doi.org/10.3390/systems4010006

AMA Style

Pakkar MS. A Hierarchical Aggregation Approach for Indicators Based on Data Envelopment Analysis and Analytic Hierarchy Process. Systems. 2016; 4(1):6. https://doi.org/10.3390/systems4010006

Chicago/Turabian Style

Pakkar, Mohammad Sadegh. 2016. "A Hierarchical Aggregation Approach for Indicators Based on Data Envelopment Analysis and Analytic Hierarchy Process" Systems 4, no. 1: 6. https://doi.org/10.3390/systems4010006

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.