Next Article in Journal
Teaching Pre-Service Teachers How to Utilize Web 2.0 Platforms to Support the Educational Needs of Students with Disabilities in General Education Classrooms
Previous Article in Journal
SavingLife: An Educational Technology for Basic and Advanced Cardiovascular Life Support
Article Menu

Export Article

Educ. Sci. 2018, 8(2), 79; doi:10.3390/educsci8020079

The Index Number Problem with DEA: Insights from European University Efficiency Data
University of Duisburg-Essen (PIM) and FOM University of Applied Sciences Essen (ild), 45141 Essen, Germany
Received: 23 April 2018 / Accepted: 28 May 2018 / Published: 1 June 2018


An increasing effort has been put into dealing with the question of time-series analysis regarding institutional efficiency, including in the area of higher education. Universities are important institutions for economies and societies and are expected to provide excellence as well as efficiency in their processes and outputs. This is reflected in the context of an increased global competitive environment by more refined international university rankings. Combining the two areas, this paper points towards a methodological challenge in comparing different ranking datasets for their use in a data envelopment analysis (DEA) Malmquist index time-series efficiency analysis, namely, index-based data compared to additive data. The problem is discussed in a theoretical framework and complemented with an empirical application: calculations for 70 European universities with budget and staff input data and different ranking output data for the timeframe of 2011–2016 show that there is no evidence for a specific index data problem. Important implications regarding university management and higher education policies are outlined. Efficiency improvements among the analyzed universities are significant but also unevenly distributed and not easy to obtain for individual institutions.
DEA Malmquist index; index numbers; ranking data; longitudinal efficiency analysis; efficiency improvement; academic performance

1. Introduction

University institutions play an important role in economic development, innovation, and internationalization, e.g., through their objectives of research, teaching, and third mission, and therefore for societies at large. Steering resources within university systems, as done by higher education politicians regarding public budgets, by university managers within the institutions themselves as well as by stakeholders, such as corporations as research partners, and students as study program participants, is an important task within the economic and management domain. To fulfil this task effectively, deciders have to rely on information regarding the performance of universities, recently, for example, provided by a growing number of national and international university rankings [1,2,3,4]. Rankings have evolved regarding their principal setup, incorporating criticism addressing indicators, institutional inclusion, and data quality, including the discourse on journal publication and the individual researcher level [5,6,7,8]. This also went hand in hand with an increased influence on policies and resource decisions in higher education [9,10,11,12]. Regarding the analysis and use of ranking data as well as for higher education efficiency analysis in general, increasing emphasis is put on the question of dynamic time-series developments. Specific calculation methods, such as data envelopment analysis (DEA) window analysis as well as DEA Malmquist index, are employed for such questions. As Parteka and Wolszczak-Delacz [13] (p. 68) outline, this may overcome shortcomings of former analysis perspectives which had focused mainly on static efficiency analysis results [14,15,16,17,18,19,20,21,22,23]. The underlying technique for efficiency measurement is the DEA introduced by Charnes, Cooper, and Rhodes [24] in the basic form with constant returns to scale (CRS) and extended by Banker, Charnes, and Cooper [25] with variable returns to scale (VRS). DEA applications within the higher education sector as a typical multi-input and multioutput production environment are numerous [26,27,28,29,30,31,32,33]. In addition, the Malmquist index for analysing longitudinal developments over time [34,35,36] has also been applied for universities, e.g., for the Philippines [37] and Australia [38].
This paper addresses the research question of if university ranking data is applicable for longitudinal efficiency analysis endeavors. The specific methodological question therein is, if ranking index data can be used. This is motivated by the potential problem that an increased university input volume could not be met by an increased university output volume if index numbers are used.
This is of importance, as many rankings, as for example the Times Higher Education (THE) ranking as well as the ARWU Shanghai ranking, use index numbers for comparing the performance of universities for any yearly publication [39,40]. In these cases, all performance and evaluation measures are indexed for a maximum value of 100. If such an index number problem would exist for dynamic efficiency analysis, it would restrict the analysis potential of using university ranking data as one of the largest and most comprehensive international datasets. To test for this specific problem, a DEA Malmquist index calculation is applied with three different datasets from (a) the THE ranking with index numbers; (b) the CWTS Leiden ranking without index numbers; as well as (c) the combined case with data from both rankings systems as output indicators. This methodological management science question may also be applied to other industries. For higher education, it is connected to the research discussion of university rankings being a “zero sum game”, as rankings depict only relative positions of institutions among themselves, not the overall (e.g., quality, productivity, excellence) development of the higher education sector (see for example [41] (pp. 195–196) or [42] (p. 45)). Additionally, many researchers also connect this question to the presumably necessary increasing input volumes (budget, staff, further tight resources) in order to stay in the same positions within university rankings, such as, for example, Hazelkorn argues [43] (p. 71). For efficiency analysis matters and methodology, this is connected to the question of industry or structural efficiency as introduced by Farrell [44] (p. 262), [45] (p. 165).
The remainder of this paper is structured as follows: Section 2 describes the characteristics of university rankings and their data, especially for the applied systems of the THE and CWTS university rankings. Section 3 provides the methodological background regarding DEA and the Malmquist index as well as a short insight into index number theory. Section 4 presents the calculation results for the ranking datasets. Section 5 lays down some discussion points before Section 6 closes with conclusions and possible further research questions.

2. Ranking Systems and Research Data

University rankings have been established as a part of the higher education information environment for stakeholders, such as students, university managers, corporations, as well as politicians. However, they are also informative for researchers interested in the performance and international or national comparison of university institutions [46,47,48,49,50]. The following two ranking systems have been used in order to gather longitudinal output data for several universities [51,52,53]:
(A) The Times Higher Education (THE) ranking is one of the most long-standing and acknowledged international university rankings, established in 2004. This ranking has incorporated several changes due to feedback and criticism during the last decade. The THE ranking establishes five evaluation areas, all individually indexed for a maximum of 100. For the 2015/16 ranking, for example, the leading California Institute of Technology received the evaluation results of 95.6 for teaching, 64.0 for international outlook, 97.6 for research, 99.8 for citations, and 97.8 for industry income. This altogether provided the total ranking evaluation of 95.2. Though there have been changes and adaptions in the underlying 13 evaluation indicators, the basic setup of this structure has been continued since the 2011 ranking. Therefore, data can be used in this timeline (2011–2016) in a sensible way. In a detailed breakdown, the 13 indicators are explained as follows in Table 1.
The following thresholds and inclusion criteria are employed by THE, which play an important role in the question of which institutions are listed and which not: Universities are excluded from the ranking if they do not teach undergraduates or if their research output averaged fewer than 200 journal articles per year over the five-year period 2010–2014. In exceptional cases, institutions below the 200-paper threshold are included if they have a particular focus on disciplines with generally low-publication volumes. There are significant and elaborate processes in place regarding data gathering, also including a defined error management approach. This is connected to the “Berlin Principles on Rankings of Higher Education Institutions” (see [55] (pp. 51–53) and [56] (pp. 80–86)). Institutions provide and sign off their institutional data for use in the THE ranking. On the rare occasions when a particular data point is not provided, a low estimate between the average value and the lowest value reported by all institutions is entered (25th percentile of all data values). In addition, a standardization approach for each indicator is used based on the distribution of data within a particular indicator—a cumulative probability function using a version of Z-scoring (see [57] (pp. 91–93) and [58]). Within the applied dataset from THE, not all out of the five indicator values reached a maximum of 100 among the selected 70 European universities, as in some cases the 100 maximum value was attained by a non-European university (Australia, Canada, China, the United States, etc.).
(B) The CWTS Leiden ranking is seen as one of the international rankings featuring the highest quality standards, especially because of the high impact of research, publication, and citation data included [53]. In this analysis, size-independent data is used in order to maintain comparability. In addition, due to input data such as budget size, institutional size is already incorporated. Data has been available for this ranking since 2011 [59]. CWTS data is based on bibliometric statistics from the Web of Science (Thomson Reuters), where the universities in the 2016 edition of the Leiden Ranking are ranked according to their percentage of highly cited publications. A publication therein is considered highly cited if it belongs to the top 1%, 10%, or 50% most cited publications in its field as explained by [60]. This focused basis on publications and citations from one large database is a strength (in terms of comparability and data quality) and also a weakness, e.g., regarding disciplinary bias or quality evaluation (see [11] (pp. 13–14)). Compared to THE ranking data, for CWTS data, there is no index value used but additive data with no upper bound (e.g., citations numbers). From the CWTS dataset, P (publications, with partial share points for coauthors), TCS and TCNS (total citations and total citations normalized), as well as P_top1 and P_top50 (number of publications among the top 1% or 50% most frequently cited) are selected.
For input data gathering, the European ETER project was used, which provides large datasets (among others: budgets, staff, students, graduates, etc.) for the years 2011, 2012, and 2013 in the current version, accessible via the Internet [61]. Concurrent with the time series in the THE ranking datasets, input data total budget and total academic staff was used, connecting the input year 2011 with the output (ranking) data of 2011 and 2012, the 2012 input year with the output data of 2013 and 2014, as well as the 2013 input year with the output data of 2015 and 2016.
University selection for this analysis was established regarding the principle of selecting the European institutions from the 2011 THE ranking, which featured the top 200 universities worldwide; that amounted to 81 universities. Furthermore, 11 institutions had to be excluded due to missing data, some due to missing or inconsistent ranking data (THE or CWTS), the majority due to missing input budget or staff data (ETER), leaving 70 institutions. Data as described in Table 2 for 2 example universities has been used for all 70 universities. In this case, the research question can be highlighted: as the University of Oxford (UK) has increased budget and staff input for 2011–2016 by 20.54% and 13.87%, respectively, output numbers have risen in the indexed THE ranking by 5.80% on average, whereas in the nonindexed CWTS ranking, output has risen by 34.33%. On the other hand, for the University of Würzburg (Germany), numbers are different. With a budget increase by 2.06% and a staff increase by 4.24%, the output rise amounted to 17.51% with THE, but only to 14.08% with CWTS.
Figure 1 highlights the input–output scheme for the analysis, in this case regarding the THE indicators as outputs. Some example correlations are included, especially the input indicator correlation between budget and staff of r = 0.86.
Further correlations between input and output indicators assessed from the whole dataset of 420 units (70 universities and 6 years of data) are outlined in Table 3. Interestingly, high correlation levels are not only obvious between the two input indicators budget and academic staff, but also among output indicators, mainly within the two used ranking systems, for example, between the THE indicators “Teaching” and “Research” (0.890), calling in mind the “Humboldt principle” regarding the unity of teaching and research [62] (p. 274), [63,64], and also among the CWTS indicators publications (P) and citations (TCS, 0.963, and TNCS, 0.980), which is obvious given the fact that the same database is used and publications are a requirement for receiving citations. The same holds true for the correlation of publications with being among the top 1% or top 50% of cited publications (0.921 and 0.995, respectively). Therefore, it can be concluded that especially the CWTS indicators are highly correlated as they stem from the same database and are all connected to initial publications.
Furthermore, high correlations between the two rankings datasets can also be observed: THE indicators “Teaching” and “Research” feature high correlation levels with all CWTS (publication-based) indicators. For “Research”, this is not so much surprising, for “Teaching”, it surely is.
Finally, it can also be of interest to look into low levels of correlation. The THE indicators “International Outlook” as well as “Industry Income” show only weak to nonexistent correlations with the other indicators within THE as well as in CWTS. This can support the hypothesis that these fields are fairly independent and should be covered by separate indicators (as THE argues, for example). It also can be seen as proof for a supposition that these areas do not really belong to the core of academic and university objectives. At least, the negative correlation between citations and industry income can be seen in such a light of “estrangement” between academe and the corporate world. At least, it can be understood from an individual researcher’s perspective, who often enough faces the nontrivial trade-off between time invested in topics interesting from an academic perspective and derived publications on the one hand, and industry-affine questions with connected projects, income, and publications for those topics.

3. Research Methodology and Index Numbers

The data envelopment analysis (DEA) method is based on works of Koopmans regarding the activity analysis concept [65], Debreu [66] and Farrell in terms of the radial efficiency measure [44], as well as the works of Diewert [67]. This led to the specific DEA method suggestion by Charnes, Cooper, and Rhodes in 1978 [24]. Reasons for the increasing use of this efficiency analysis technique in higher education research are the fact that no a priori knowledge about a production function is required, only real-life data is used, and a multitude of inputs can be combined with a multitude of outputs, which is very typical for universities as “multi-product-organisations” [68,69,70,71,72,73].
DEA studies decision making units (DMU), which can be seen as the entities responsible for input, throughput, and output decision making [74] (p. 22). DMU such as, e.g., university institutions, departments, schools, or institutes and research groups, can be evaluated and compared, showing a specific level of decision-making success in terms of overall efficiency. DEA uses a nonparametric mathematical programming approach for the evaluation of DMU efficiency relative to each other. Further, it is assumed that there are several DMU and it is supposed that inputs and outputs comply with these requirements:
For each input and output, there are numerical, positive data for all DMU.
Selected values (inputs, outputs, and the chosen DMU) should depict the interest of decision-makers towards the relative efficiency evaluations.
DMU are homogenous in terms of identical inputs and outputs.
Input and output indicator units and scales are congruent.
Furthermore, two different models can be distinguished. The CCR model, named after the authors Charnes, Cooper, and Rhodes [24], with constant returns to scale, and the BCC model with variable returns to scale [25]. For both models and their efficiency measurement, the following specifications are made [75] (p. 239):
n   the number of DMU to be evaluated
DMUj the jth DMU
m   the number of inputs to each DMU
s   the number of outputs to each DMU
xij   amount of the ith input consumed by DMU j
ykj   amount of the kth output produced by DMU j
eff   abbreviation for efficiency
vi   the weight assigned to the ith input
uk   the weight assigned to the kth output.
eff   DMU jo   =   k = 1 s   μ k   y kjo i = 1 m   v i   x ijo
A basic characteristic of the CCR model is the reduction of a multioutput and multi-input setting to a single (weighted) input and output combination for each DMU. For a certain DMU, measuring its efficiency and comparison with other DMU in the system is enabled. Usually executed by a series of linear programming formulations, DMU performance comparison facilitates a ranking of the different analyzed DMU and scales their relative efficiency from low to high, whereby the latter is defined as efficient. The CCR model contains both mathematical maximization and minimization problems. Detecting DMU relative technical efficiency requires on the one hand detection of each DMU technical efficiency, and on the other hand, the comparison of all DMU efficiencies. These steps are executed in the DEA simultaneous arithmetic operation. The calculation of the DMU efficiency value results from the consideration of the weighted inputs and weighted outputs. With the help of quantified inputs and quantified outputs, DEA generates via a quotient one single efficiency ratio for each DMU. The weighting factors are endogenously determined and allow the pooling of heterogeneous inputs and outputs with different units of measurement in one efficiency ratio. Hereby, each DMU’s weights are considered optimally in order to maximize the efficiency value and determine only the definitely provable inefficiency. With the help of the following figure, the different scales of CCR and BCC are depicted in the case of a single input and single output situation. In Figure 2, H illustrates a scale-efficient DMU (on the border production function). Inspection of DMU K reveals that the distance XJ/XK stands for possible input savings regarding a decline of technical inefficiency, whereas XK/YL represents possible output enlargements regarding the decline of technical inefficiency. The distance XI/XK stands for the gross-scale efficiency and XI/XJ shows the pure-scale efficiency with a corrected input. YL/YM stands for the pure-scale efficiency with corrected output (in case of variable scales).
Calculating DMU efficiency, it can be observed that DMU H has the highest efficiency value. Building the border production function (“envelopment function”) under the assumption of constant returns to scale (CCR) therefore complies with a line through the origin. This function with the assumption of variable returns to scale (BCC) is built by combining the points J, H, and L. As this is the case, the area of production opportunities is limited by the set of convex combinations of realized productions belonging to the border production function. Therefore, in the case of variable returns to scale, DMU J, H, and L are efficient, whereas DMU K is inefficient. In a CCR model, usually only one DMU is found to be efficient, whereas in a BCC model several, DMU are expected to be efficient.
Based on early works of Malmquist [34], Caves, Christensen, and Diewert [12] proposed a calculation of a productivity index in order to shed light on efficiency changes over time. This longitudinal perspective is promising for the DEA method, as most efficiency measurement approaches are directed towards the question of efficiency improvement. To provide usable information for this objective, the index is therefore distinguishing between a technological progress for the whole set of DMU, as, for example, universities—individually adapted for each institution—on the one side, and the technological efficiency on the other side [35]. This second technological efficiency is caused by the organizational and process setup of an institution. The following formula depicts the mathematical distance function algorithm used for the Malmquist index [36,38]:
M O , CRS ( x t , y t ; x t + 1 , y t + 1 ) = D O , CRS t + 1 ( x t + 1 , y t + 1 ) D O , CRS t ( x t , y t ) * [ D O , CRS t ( x t , y t ) D O , CRS t + 1 ( x t , y t ) * D O , CRS t ( x t + 1 , y t + 1 ) D O , CRS t + 1 ( x t + 1 , y t + 1 ) ] 1 2
Regarding the use of index numbers, the basic discussion can be divided into three areas of economic application and discussion: (i) Based on mathematical theory, the first and still foremost application area for index numbers was the question of price and monetary value development over time, as, for example, outlined by Fisher (1911, 1922) [76,77], as well as Divisia [78]. This “basket index” field is still discussed and amended today as an intertemporal application, e.g., with questions regarding testing and quality evaluation of price indices [79,80]; (ii) The second area of index number application in economics is further intertemporal comparisons of organizational or industry quantities such as indexed economic cycle growth and development indices or forecasts [81]; (iii) The third area applies the use of index numbers for an interorganizational or geographical comparison of economic quantities such as production outputs between different corporations or commodity prices between different trading locations and stock markets [82,83]. This third application area has to be connected to the index number application in question here, which is the interorganizational comparison of university performance measures within ranking systems. Many problems have been identified in connection with the use of index numbers starting with, e.g., [84]. Mainly quality and stability problems allocated to the basket or definition problems are on record [85,86], but also problems of numerical comparison due to index number calculation are expressed [87,88].

4. Results

The following results have been obtained in a Malmquist index calculation for the timeframe 2011–2016 regarding the 70 selected European universities. The calculation was conducted with the software package BANXIA Frontier Analyst with an output maximization mode in a BCC model with variable returns to scale.
Three calculation runs are reported, all with the same input data (budget and staff data). (I) First, inputs were combined with five output indicators from the THE ranking (all indexed values); (II) Second, two input indicators were combined with five selected output indicators from the CWTS Leiden ranking (all nonindexed values); (III) Third, the inputs were combined in a DEA Malmquist calculation with 10 output indicators (five from THE as indexed values, five from CWTS ranking results as nonindexed values). No superefficiencies were calculated for these DEA runs, therefore all efficient units are showing a maximum efficiency score of 100.00. Table 4 presents the base efficiencies calculated for the initial year 2011 for all 70 universities.
The reason for applying three different calculation runs is to look into possible differences between efficiency level changes of indexed data and nonindexed ranking data in the two different cases. The third run (III) is a control case where both datasets are combined. This should enable a result regarding the question of if an index data problem is existing in using university ranking data for a DEA longitudinal efficiency analysis.
From Table 4, the following obvious recognitions from the data results have to be stated in order to support a further discussion in the next section. First, the 2011 efficiency values for the three calculation runs feature different values for the 70 universities. This can be expected, as different output indicators lead to different efficiency values in a relative efficiency measurement scheme. For some institutions, such as, e.g., Cambridge or the Karolinska Institute, identical efficiency scores are calculated, as such an institution is leading across a multitude of output indicators.
Second, several universities accomplish decreasing returns to scale or even increasing returns to scale, proving the assumption that the DEA BCC model for variable returns to scale is appropriate for university and higher education efficiency analysis settings.
Third, 17 (respectively 2 and 24) out of 70 universities are calculated as being efficient in regard to the selected two inputs and five (respectively 10 in run III) outputs. This is a comparatively high number, underlined by the average efficiency of all 70 universities of 87.21% (run I). For comparison, do Jiménez-Sáez et al. [89] (p. 235) report a mean efficiency of between 58.2% and 82.2% for Spanish research programs between 1988 and 1999. Therefore, a proposed problem with the index numbers (used in runs I and III) cannot be shown. This is supported by detailed data from Table 5 and Figure 3, indicating that there are differences in the efficiency development (measured by the DEA Malmquist index), as can be expected due to different output indicators. However, these differences do not point towards a significant structural disadvantage of index numbers (with the THE data in run I and run III). Especially, the 2012–2016 mean efficiency improvement is nearly identical (1.0266; 1.0251; 1.0240—bold numbers) for the three calculation runs.
Table 6 reports some cut-outs for Malmquist index values per university in the calculated dataset for run III. From this, the following observations can be noted in order to support the discussion (Section 5). Leading institutions for annual efficiency improvements are, for example, the University of Konstanz (33.20% improvement 2016 on 2015, run III), Bielefeld University (30.06% improvement 2016 on 2015), and the Technical University of Munich (22.87% improvement 2016 on 2015). The fact that the top improvement numbers are reduced significantly in the first ranking positions as well as the fact that, for example, Bielefeld University is also listed among the bottom 30 institutions regarding annual efficiency changes (−19.39% for 2012 on 2011) hints to the assumption that data irregularities or changes in data gathering, e.g., with the ranking systems or within the universities themselves, have led to these high numbers in efficiency changes. Therefore, in the next table, the long-term average changes are also reported.
Further on, Table 7 depicts the average of all year-on-year changes in efficiency between 2011 and 2016 for all universities, ranked according to the average Malmquist index values (total average efficiency change). The following statements may be derived in a first analysis concerning run III (last three columns). The highest level of average improvement over the six-year period was attained by the German RWTH University Aachen (average 1.0748 or 7.48% annually), followed by the University of Copenhagen (Denmark), as well as the University of Tübingen (Germany). Obviously, the individual leaders of efficiency improvement in the categories of “Catch-up” (individual DMU improvement due to closing the gap to the border production frontier) and “Frontier shift” (collective improvement of the border production frontier, compare, e.g., [13] (p. 74) are different from the total efficiency improvement champions, which are the Universities of Exeter and Karlsruhe (Catch-up, besides RWTH), as well as the University of Konstanz, LSE London, and Humboldt-University of Berlin (Frontier shift).
In addition, the leading institutions, regarding average improvement from this perspective (Table 7), are different from the institutions leading on yearly improvements (Table 6). This connects to the fact known widely in management science, and especially with accounting and controlling, that achieving short-term improvements is a totally different game than sustaining long-term enhancements, in this case a six-year period [90,91,92]. The success of German institutions in the sample may be connected to the experience that these institutions had to adapt to the international ranking system, regarding the specific data gathering in this arena, within the analyzed timeframe. It also may fall into the same period when, e.g., due to the German excellence initiative, the first results in increasing international visibility and competitiveness were achieved [4,93].
From Table 8, depicting the statistical characteristics of all Malmquist index values by year (2012–2016) on the previous year for run III, some further descriptions can be outlined (highest values in bold numbers). First, high average levels of improvement are shown for the years 2012 and 2016, but not so much for the years in between. As an absolute value, the overall efficiency improvement of 6.59% for the last reported year of 2016 is high, also compared to other industries. For example, Örkcü et al. [94] (p. 101) report an efficiency improvement of 3.28% for Turkish airports on average from 2009 to 2014, Sueyoshi and Goto [95] (p. 342) describe a 1.10% efficiency improvement for 12 national petroleum companies from 2005 to 2009, and Emrouznejad and Yang [96] (p. 853) calculate a 2.89% efficiency improvement for the Chinese metal industry on average from 2004 to 2012. Again, it has to be emphasized that while these are exceptional values, there are also years, such as 2013 and 2014, with only minimal or zero improvement on efficiency regarding the inputs and outputs reported here. Second, variance and standard deviation with an annual sample among the 70 universities show remarkably high values, especially again in the first (2012) and last (2016) analyzed year.
Specific Malmquist index values for each university per year and for all three calculation runs I–III, as well as detailed numbers on the Malmquist catch-up and frontier shift breakdown, are reported in Appendix A (Table A1).

5. Discussion

The described results can be connected to existing research regarding university efficiency and is discussed as follows. The overall range of efficiency scores for 2011, as outlined in Table 4, is quite small between (17 institutions with) 100.00% and a minimum value of 61.60% (University of Copenhagen, run I, mean 87.21%). Such high levels of efficiency are not necessarily usual, but also not unexpected with 70 decision making units and 7, respectively 12, indicators. For example, the results of Fandel [97] (2007, p. 527), detailing 10 out of 15 German universities within North Rhine-Westphalia as efficient, with a mean efficiency of 92.77%, by using two inputs and three outputs for the 15 DMU. Also, Johnes [98] (p. 281) reports a mean efficiency of 92.51% regarding 130 UK universities with six inputs and three outputs, also with the lowest individual university efficiency levels around 60%. See also the comprehensive review of DEA studies in education by Fuentes, Fuster, and Lillo-Banuls [99] (pp. 91–93) regarding the applied inputs and outputs which are highly influential for the calculated efficiency results (in comparison with the number of observed DMU). This baseline is further analyzed regarding time-series changes in the following years until 2016.
Out of 350 dynamic data points (5 years of change from 2011 for 70 universities, run III), altogether 118 institutions with annual negative changes of efficiency are reported, whereas for 232 instances, a positive improvement of efficiency is recognized. This shows that universities are working hard to improve their efficiency as demanded from stakeholders and taxpayers regarding the inflow of public money into the higher education system. However, it also highlights that efficiency improvement is not an easy task but has to be earned with hard work. There is no “automatism” in efficiency improvement as is, for example, sometimes made to be believed due to technological change, as, e.g., for e-learning [100,101]. The results regarding productivity increases (individual and overall) are within range of existing results, for example, by Parteka and Wolszczak-Derlacz [13] (p. 73), who report an average 4.1% annual increase of productivity for 266 public universities in 7 European countries between 2001 and 2005. In their dataset, universities from Germany, Italy, and Switzerland provided the highest efficiency improvements. However, it has to be recognized that in this case, three inputs (staff, students, budget) were compared only to two outputs (publications and graduates). As also in this research, German universities are reported to have above-average efficiency improvements, which seems to be a stable result. Obviously, the reported analysis results and differences regarding the distinction between general technological progress (“frontier shift”) and its use on the one hand and the individual organizational reasons for efficiency changes (“catch-up”) on the other hand for universities are interesting and should be studied further. In addition, in-depth analysis is required in terms of resource and organizational consequences of such efficiency development results for universities as done, e.g., in the health care or service sector [102,103]. For example, it can be questioned if an institution or a department should receive unequivocal research or teaching funding when long-term negative developments of efficiency are recognized. This is superior to the question applied mainly today of, if due to an existing (low) efficiency status compared to others, restrictions in terms of funding shall be implemented. Whereas an efficiency development can largely be attributed to the institutional responsibility (given stable and comparable circumstances), the static efficiency position compared to other institutions may have a multitude of (external) influences and institutional responsibility is not a given.
As a policy implication, this would hint at an adjusted resource distribution scheme where allocation is connected to the longitudinal efficiency development of institutions. For example, institutions with decreasing efficiencies over time would also receive a reduced amount of resources, whereas institutions increasing their efficiency (change, not absolute level, potentially above a threshold level) would receive in increase in funding, e.g., by state budgets or also competitively distributed research funding and other sources.
Requirements for efficiency analysis with the DEA technique and for ranking endeavors have to be recalled into the academic and public discourse. Mainly, it has to be ensured that the analyzed DMU are actually comparable. This notion can be discussed in different perspectives: (i) From a production theory perspective, the border production function has to be identical or at least the same technical production possibilities have to be available to all compared units and institutions [65]; (ii) From an economic price and market perspective, the used factor prices, as, e.g., for academic personnel, have to be identical or at least comparable; whereas wages for academics are not identical, not even within one country, the argument may be stated that even in a global perspective wages for research and teaching assistants as well as professors as core personnel for universities do not feature too much deviation; (iii) From a higher education research perspective, the main notion is the question of comparable objectives, missions, and profiles. This may be tested mainly against the following research hypothesis. The subject mix and homogeneity within and in between institutions has to be taken into account. Either only broad university institutions calling upon the “universitas” principle are included into comparative analyzes, or the analysis is broken down into subject fields, as is done by most rankings today (i.e., THE and CWTS, but also others such as the Shanghai ARWU ranking).
Altogether, it can be argued that the contribution of this paper in finding no proof for an index data problem for the application of ranking data in DEA efficiency analysis endevors for universities has the following implication for the knowledge and future research directions. Ranking data can be used unequivocally for efficiency analysis projects, independent of the fact if the rankings contain indexed data or aggregate data. This alleviates the application of ranking data for higher education efficiency research and therefore provides an important potential for further analysis, as the data realm of international university rankings is growing every year.
It has to be stressed that efficiency questions and the interest in analyzing and improving the performance, excellence, and output (given more or less fixed inputs) of universities in higher education is not new (see, for example, [104]). However, today the available techniques, such as DEA windows analysis and DEA Malmquist index for a dynamic time-series analysis, the available data due to information technology, as well as chances to compare these data and analyses internationally have improved the level of analysis significantly. To put these available instruments to a good use, this article wants to contribute to the methodological discussion regarding efficiency analyses for higher education.

6. Conclusions and Outlook

This paper has made it obvious that no specific index number problem can be found for longitudinal efficiency analysis calculations for universities based on ranking data. This holds true for the indexed ranking publications of Times Higher Education, as this data (for the timeframe 2011–2016) was analyzed herein in comparison with the CWTS (nonindexed) data. Research and policy implications for the presented and analyzed data include, among others, the following points for further discourse. Ranking data, also within an indexed form, are assumed to be feasible and can acts as a quality output indicator basis for efficiency analysis endeavors, also in a longitudinal time-series analysis with such instruments as, for example, the DEA window analysis or the DEA Malmquist index analysis. As described above, the time-series analysis of efficiency development per institution may be an interesting field for academic analysis as well as a decision basis for university managers, politicians, as well as stakeholders and partners of universities. For example, if a company is thinking about a long-term research cooperation with a university, a look into past long-term efficiency developments at this institution may be well advised and informative in order to protect such a strategic investment. Further, it is has been shown that long-term efficiency improvement is a different playing field from yearly improvements. In the long-term perspective, German universities in particular were faring well within the analyzed timeframe of 2011–2016 and the applied dataset of 70 European universities. From the input and output correlation analysis, interesting results are that larger budgets and staff numbers correlate with higher ranking evaluations in the fields of teaching and research, but less so for citations. Institutions with larger input volumes therefore have at least a larger chance to reach higher output levels in these fields. However, this is definitively not true for the evaluation fields of international outlook and industry income as measured by THE rankings. This may be connected to the fact that achieving productive international as well as industry cooperation may not so much depend on the size of the budget and staff numbers but more on a mind-set within the university regarding industry income and especially a form of flexibility. This is hypothetical, as there is no causal analysis at hand, but at least the correlation numbers for the observed 420 cases may provide some interesting basis for creating and further on testing hypotheses regarding such productivity connections in universities.
Efficiency analysis may provide an elaborate form of quality check towards ranking systems, especially in the proposed time-series form, as with constant inputs for the same institutions and timeframe, rankings are supposed to present similar results. If not, the quality of the ranking performance measurement can be doubted. This has to be enlarged with datasets from other rankings in a timeline perspective, and with that, the problem of establishing a “reference set” or “baseline” performance dataset may arise. Which ranking or dataset would other rankings have to be measures “against”? Such avenues of inquiry are connected to the quality debate regarding evaluations and rankings for universities, as, for example, stated by Bornmann [105], Osterloh and Frey [106], and Harmann [107] for an international perspective. For the national German context regarding the VHB JOURQUAL3 journal rankings in economics and business administration, Eisend [108] and Schrader and Hennig-Thurau [109] discuss this as well as Lorenz and Löffler [110] for the Handelsblatt ranking of business economists. Policy implications point towards the discourse regarding conflicts of interest for university management in pursuing efficiency goals compared to other objectives such as excellence, reputation, or cooperation, as Blackmore points out [111].
Further research is warranted to address, for example, the following five areas regarding university efficiency, rankings, and overall sector performance development: (a) Further ranking data for indexed and nonindex values should be tested in order to enlarge the database for the falsification of an index number problem for university ranking data in longitudinal efficiency analysis projects; (b) The further eligibility of such a dynamic efficiency analysis, e.g., with the DEA Malmquist index for a metaevaluation of rankings, could be tested. This can be an important contribution to ranking system quality; (c) Institutional management implications do earn a further look into possible steering and efficiency improvement measures based on longitudinal efficiency results for individual universities. Sideways comparisons with other knowledge-intensive service industries such as health care, finance and insurance, accounting and consulting, or logistics could be promising; (d) In addition, the possible implications and measures on a policy-systems level of higher education are of high interest given the fact that, e.g., for Germany, the public resources spent within university institutions totals about 48.2 billion Euros or 1.3% of total GDP for 2014 [112] (p. 71). Any fact-based research implying possible changes and improvements in the setup for public funding distribution in the sector might be of high value in an economic perspective; (e) Finally, also a look into the organizational level of research groups and individual researchers regarding the long-term dynamic efficiency development may be very interesting. Much research work already does exist for this in static as well as dynamic output perspectives, e.g., [113,114,115], but little yet regarding a longitudinal efficiency perspective taking inputs into account. This could, for example, connect to the long-standing question of if outside and additional resources are able to improve the efficiency of an individual researcher or if there is no or a marginal effect, as, for example, found by Fedderke and Goldschmidt [116] (p. 479).
Altogether, the question of ranking data as an output database for time-series efficiency analysis in higher education has been proven to be a worthwhile and interesting field of inquiry for higher education research and management.


This research received no external funding.

Conflicts of Interest

The author declares no conflict of interest.

Appendix A

Table A1. Malmquist Index Values (Catch-up and Frontier Shift) 2012–2016, based on 2011.
Table A1. Malmquist Index Values (Catch-up and Frontier Shift) 2012–2016, based on 2011.
UniversityYear *Malmquist IndexCatch-UpFrontier ShiftMalmquist IndexCatch-UpFrontier ShiftMalmquist IndexCatch-UpFrontier Shift
Aarhus University2011
Aarhus University20121.15001.13351.01461.03670.99891.03781.15001.13351.0146
Aarhus University20131.02591.05770.96991.04421.08440.96291.02591.05770.9699
Aarhus University20140.93720.94620.99051.08141.04801.03180.93740.94670.9902
Aarhus University20150.97860.99350.98511.07641.05821.01720.98040.99430.9860
Aarhus University20161.18831.14231.04031.06961.04251.02601.18701.14081.0405
Bielefeld University2011
Bielefeld University20120.80610.82520.97681.00550.99161.01400.80610.82520.9768
Bielefeld University20131.08551.11120.97681.04821.13970.91981.08551.11120.9768
Bielefeld University20140.95290.99260.96001.05471.02501.02900.95290.99260.9600
Bielefeld University20150.92470.92021.00501.02501.04080.98490.92470.92021.0050
Bielefeld University20161.30231.09841.18561.05161.03121.01981.30061.09841.1841
Delft University of Technology2011
Delft University of Technology20121.00241.00001.00240.99360.92821.07051.00271.00001.0027
Delft University of Technology20131.03611.00001.03611.02661.04710.98041.03521.00001.0352
Delft University of Technology20140.96311.00000.96311.04841.02381.02400.96471.00000.9647
Delft University of Technology20151.01101.00001.01101.09451.07441.01871.01331.00001.0133
Delft University of Technology20161.03281.00001.03281.13851.10061.03451.03281.00001.0328
Durham University2011
Durham University20121.02841.00821.02000.96020.92681.03611.02781.00711.0206
Durham University20130.98881.00000.98880.94870.97640.97160.98461.00000.9846
Durham University20140.94531.00000.94531.00450.92661.08410.94651.00000.9465
Durham University20151.01330.97811.03601.11591.13840.98021.01460.98011.0352
Durham University20161.01610.95141.06800.95900.96320.99571.02290.98171.0419
Eindhoven University of Tec.2011
Eindhoven University of Tec.20121.03761.00001.03761.05991.02061.03851.03761.00001.0376
Eindhoven University of Tec.20131.02131.00001.02131.05011.09680.95751.02131.00001.0213
Eindhoven University of Tec.20140.97041.00000.97041.05451.02571.02810.97511.00000.9751
Eindhoven University of Tec.20150.97761.00000.97761.01361.02370.99010.97761.00000.9776
Eindhoven University of Tec.20160.85490.84951.00631.12591.11591.00900.86700.86331.0042
Erasmus Univ. Rotterdam2011
Erasmus Univ. Rotterdam20121.05391.01591.03731.08381.00001.08381.01711.00001.0171
Erasmus Univ. Rotterdam20131.18991.10131.08050.98141.00000.98141.11641.00001.1164
Erasmus Univ. Rotterdam20140.95771.00000.95771.05661.00001.05660.97521.00000.9752
Erasmus Univ. Rotterdam20150.99981.00000.99980.97261.00000.97260.97441.00000.9744
Erasmus Univ. Rotterdam20161.11471.00001.11470.98641.00000.98641.04271.00001.0427
ETH Lausanne2011
ETH Lausanne20121.03311.00001.03311.07861.01441.06331.03271.00001.0327
ETH Lausanne20131.00061.00001.00061.00411.05910.94810.99951.00000.9995
ETH Lausanne20140.98591.00000.98591.06791.03641.03040.99071.00000.9907
ETH Lausanne20151.01081.00001.01081.24241.22681.01271.01541.00001.0154
ETH Lausanne20161.01691.00001.01691.12381.06281.05741.01861.00001.0186
Ghent University2011
Ghent University20121.00831.00021.00811.04551.01311.03201.01461.00691.0077
Ghent University20130.99270.98771.00501.01531.03940.97680.98601.00170.9843
Ghent University20140.99230.99300.99931.04881.02451.02371.01281.00001.0128
Ghent University20150.95040.94321.00761.00760.99681.01090.95450.94421.0110
Ghent University20161.02790.97571.05360.99980.98091.01931.02400.98941.0349
Goethe University Frankfurt2011
Goethe University Frankfurt20121.07811.04471.03201.03681.01861.01791.07811.04471.0320
Goethe University Frankfurt20130.98311.00100.98211.02791.00861.01910.98311.00100.9821
Goethe University Frankfurt20141.08631.08191.00401.03741.00771.02951.08691.08421.0025
Goethe University Frankfurt20150.97590.98500.99071.01171.00951.00220.97600.98320.9927
Goethe University Frankfurt20161.00630.98411.02261.01290.98891.02431.00640.98381.0229
Heidelberg University2011
Heidelberg University20121.10111.05301.04561.02821.01381.01421.10111.05301.0456
Heidelberg University20130.96210.98160.98011.04521.01091.03400.96210.98160.9801
Heidelberg University20141.00651.00641.00001.04551.00671.03861.00651.00641.0000
Heidelberg University20151.07041.07610.99481.03141.01211.01911.07071.07610.9950
Heidelberg University20161.09591.06211.03181.03381.00481.02891.09591.06211.0318
Humboldt University of Berlin2011
Humboldt University of Berlin20121.05991.05431.00531.03571.00001.03571.09171.00001.0917
Humboldt University of Berlin20130.96921.00000.96920.89131.00000.89130.92151.00000.9215
Humboldt University of Berlin20140.89341.00000.89341.01251.00001.01250.95271.00000.9527
Humboldt University of Berlin20151.05251.00001.05250.99871.00000.99871.01301.00001.0130
Humboldt University of Berlin20161.26641.00001.26641.00831.00001.00831.21221.00001.2122
Imperial College London2011
Imperial College London20120.99551.00000.99551.06441.00001.06441.02201.00001.0220
Imperial College London20130.97431.00000.97430.99911.00000.99910.99141.00000.9914
Imperial College London20140.94921.00000.94920.97941.00000.97940.96361.00000.9636
Imperial College London20150.99271.00000.99271.00261.00001.00260.99611.00000.9961
Imperial College London20160.98921.00000.98921.02441.00001.02440.99181.00000.9918
Karlsruhe Institute of Tec.2011
Karlsruhe Institute of Tec.20120.99630.98471.01181.05891.01871.03941.00910.99241.0168
Karlsruhe Institute of Tec.20131.15541.15970.99630.98481.03290.95341.13251.14550.9886
Karlsruhe Institute of Tec.20140.97901.00120.97781.10201.08111.01920.97901.00120.9778
Karlsruhe Institute of Tec.20151.00060.99921.00141.15411.14341.00931.00771.00241.0053
Karlsruhe Institute of Tec.20161.19481.18651.00701.07341.05721.01531.18751.18271.0041
Karolinska Institute2011
Karolinska Institute20121.07191.00001.07191.02391.00001.02391.10801.00001.1080
Karolinska Institute20131.00150.99421.00730.97190.95991.01250.96001.00000.9600
Karolinska Institute20140.93690.99040.94591.02330.98111.04300.96171.00000.9617
Karolinska Institute20150.99720.99651.00071.03061.03031.00040.99561.00000.9956
Karolinska Institute20161.15101.01911.12951.00881.02190.98721.09261.00001.0926
King’s College London2011
King’s College London20121.05621.04411.01151.03550.96951.06811.05901.04031.0179
King’s College London20130.96300.96261.00050.98860.99700.99160.93660.96210.9736
King’s College London20141.03361.05670.97821.05881.07590.98411.03141.03630.9952
King’s College London20151.02201.01331.00861.06931.05261.01591.01711.00521.0119
King’s College London20161.08161.02411.05621.14551.08561.05511.08091.02321.0565
KTH Royal Institute of Tec.2011
KTH Royal Institute of Tec.20121.02121.00001.02120.99000.94771.04461.02121.00001.0212
KTH Royal Institute of Tec.20131.00911.00001.00910.97530.96301.01281.00911.00001.0091
KTH Royal Institute of Tec.20140.97931.00000.97931.06221.02621.03500.97931.00000.9793
KTH Royal Institute of Tec.20150.99191.00000.99191.04341.03271.01030.99241.00000.9924
KTH Royal Institute of Tec.20160.93961.00000.93961.06891.06671.00200.95251.00000.9525
KU Leuven2011
KU Leuven20121.02141.01311.00821.00240.96791.03571.01941.00001.0194
KU Leuven20131.00641.00001.00640.98231.01540.96740.99861.00000.9986
KU Leuven20140.99421.00000.99421.01890.98961.02960.99991.00000.9999
KU Leuven20151.00381.00001.00380.99590.98051.01570.99911.00000.9991
KU Leuven20161.04961.00001.04960.99440.97041.02471.04851.00001.0485
Lancaster University2011
Lancaster University20120.94670.98360.96241.01330.98021.03390.93580.96120.9736
Lancaster University20130.96071.01330.94810.85931.03280.83200.95971.01130.9490
Lancaster University20141.00221.04020.96351.04451.02601.01800.99581.04010.9574
Lancaster University20150.98670.99020.99640.75320.84370.89270.98670.99020.9964
Lancaster University20161.06441.05061.01311.04421.03051.01331.06481.05131.0128
Leiden University2011
Leiden University20120.86900.87790.98981.18601.11011.06840.95460.96350.9907
Leiden University20131.12811.07551.04901.02931.03170.99771.07621.03691.0379
Leiden University20140.99391.04050.95511.03821.01871.01920.99911.00090.9981
Leiden University20150.98580.97361.01250.94840.95620.99180.98450.99540.9890
Leiden University20161.04310.94361.10540.97030.97960.99041.02520.94891.0805
LMU Munich2011
LMU Munich20121.10731.07571.02941.03411.02281.01111.10731.07571.0294
LMU Munich20131.01021.03060.98021.01840.98411.03481.01021.03060.9802
LMU Munich20140.97400.97590.99811.02100.97461.04750.97420.97630.9979
LMU Munich20151.16861.15051.01571.00250.95671.04791.16831.15011.0158
LMU Munich20161.02301.00001.02301.01340.98341.03051.02301.00001.0230
LSE London2011
LSE London20121.13771.00001.13771.08731.03711.04851.13771.00001.1377
LSE London20130.99031.00000.99031.01851.21240.84010.99031.00000.9903
LSE London20140.96421.00000.96420.91320.88841.02780.95551.00000.9555
LSE London20151.02831.00001.02830.96191.89630.50721.02831.00001.0283
LSE London20161.15031.00001.15031.13641.00001.13641.15031.00001.1503
Lund University2011
Lund University20121.03331.03350.99990.98760.96571.02281.01940.97221.0486
Lund University20131.04781.04631.00140.96840.95031.01911.02161.02300.9986
Lund University20140.92170.95560.96451.01080.97901.03250.94670.96430.9817
Lund University20151.02971.01541.01411.01901.00691.01201.03181.03121.0006
Lund University20161.13671.09011.04281.03571.01901.01651.13201.06621.0617
Newcastle University2011
Newcastle University20120.99240.94941.04530.98310.99460.98840.99540.99461.0008
Newcastle University20131.00371.03050.97400.95071.00540.94560.95501.00540.9498
Newcastle University20140.97320.97610.99701.09831.00001.09831.05771.00001.0577
Newcastle University20150.99530.96601.03040.98121.00000.98120.96481.00000.9648
Newcastle University20161.13841.13511.00291.00451.00001.00451.03931.00001.0393
Queen Mary Univ. of London2011
Queen Mary Univ. of London20121.00810.98101.02761.16271.11851.03951.00810.98101.0276
Queen Mary Univ. of London20130.99611.03290.96441.02411.06230.96400.99611.03290.9644
Queen Mary Univ. of London20141.00881.00221.00651.15931.11781.03711.00881.00221.0065
Queen Mary Univ. of London20151.00990.98691.02331.03941.04790.99191.00990.98691.0233
Queen Mary Univ. of London20161.05161.00161.04991.05171.03821.01301.05291.00741.0452
RWTH Aachen University2011
RWTH Aachen University20121.07271.03821.03311.02961.00061.02891.07271.03821.0331
RWTH Aachen University20131.07001.07990.99081.06851.03181.03561.07001.07990.9908
RWTH Aachen University20141.06111.04461.01581.07851.03481.04211.06111.04461.0158
RWTH Aachen University20150.97181.02760.94571.05611.02211.03330.97181.02760.9457
RWTH Aachen University20161.19861.17601.01921.04411.01291.03081.19861.17601.0192
Stockholm University2011
Stockholm University20121.15571.13241.02061.04401.01911.02441.15571.13241.0206
Stockholm University20131.03151.06040.97271.10251.10151.00091.03151.06040.9727
Stockholm University20140.95350.98320.96971.05291.02501.02720.95350.98320.9697
Stockholm University20151.02871.00071.02811.02821.01301.01501.02871.00071.0281
Stockholm University20160.92880.88811.04581.05001.03991.00970.92880.88811.0458
Swedish U. of Agri. Sciences2011
Swedish U. of Agri. Sciences20120.99371.00000.99371.02390.98981.03440.99371.00000.9937
Swedish U. of Agri. Sciences20130.99571.00000.99570.99561.00630.98940.99571.00000.9957
Swedish U. of Agri. Sciences20141.00931.00001.00931.07681.03981.03561.00431.00001.0043
Swedish U. of Agri. Sciences20151.03181.00001.03181.12211.12710.99561.04131.00001.0413
Swedish U. of Agri. Sciences20161.02481.00001.02481.05991.04781.01161.02481.00001.0248
Tec. University of Denmark2011
Tec. University of Denmark20121.00560.98051.02561.05461.01821.03571.00560.98051.0256
Tec. University of Denmark20131.04701.03711.00951.08161.09850.98461.04681.03711.0093
Tec. University of Denmark20141.00610.99691.00921.01751.05120.96801.00630.99811.0083
Tec. University of Denmark20151.00170.99861.00310.96320.94241.02211.00240.99841.0039
Tec. University of Denmark20160.87860.87361.00571.06291.04111.02090.89180.88201.0112
Tec. University of Munich2011
Tec. University of Munich20120.91050.88751.02591.05391.02351.02970.90770.88711.0233
Tec. University of Munich20131.00311.02670.97701.06261.02541.03621.00311.02670.9770
Tec. University of Munich20141.05471.05301.00161.06041.01321.04661.05471.05311.0015
Tec. University of Munich20150.97630.97980.99641.05801.01471.04270.97630.97970.9966
Tec. University of Munich20161.22871.20161.02251.03311.00241.03061.22871.20161.0225
Trinity College Dublin2011
Trinity College Dublin20121.01441.00001.01441.08570.99851.08731.01741.00001.0174
Trinity College Dublin20130.98021.00000.98021.09421.11770.97900.98661.00000.9866
Trinity College Dublin20140.97211.00000.97211.05641.03391.02170.97471.00000.9747
Trinity College Dublin20150.96750.94271.02621.13991.11601.02140.98690.98411.0028
Trinity College Dublin20161.00000.96031.04131.02741.00761.01971.00800.97981.0289
University College Dublin2011
University College Dublin20120.94400.91551.03111.10261.06591.03450.95640.92361.0355
University College Dublin20130.92580.96430.96011.06061.08030.98180.94110.98090.9594
University College Dublin20141.03221.02681.00531.04121.00131.03991.03391.01961.0140
University College Dublin20150.97620.95781.01921.09531.10200.99390.99840.99141.0070
University College Dublin20161.08711.05591.02960.98530.97341.01221.06791.03671.0301
University College London2011
University College London20121.02441.00411.02021.04831.00001.04831.04681.00001.0468
University College London20130.98230.99280.98940.98471.00000.98470.98471.00000.9847
University College London20140.99090.99360.99731.03731.00001.03731.02241.00001.0224
University College London20151.00880.99851.01031.02071.00001.02071.02071.00001.0207
University College London20161.05381.02281.03041.05191.00001.05191.04841.00001.0484
University of Aberdeen2011
University of Aberdeen20120.99300.96931.02441.02970.98131.04930.99580.97351.0229
University of Aberdeen20130.99731.02760.97050.91950.98150.93690.97291.00890.9643
University of Aberdeen20140.98570.96351.02310.97150.91691.05960.99500.97381.0218
University of Aberdeen20151.00110.98531.01610.95080.93491.01711.00210.98401.0184
University of Aberdeen20161.08271.08261.00000.96910.95171.01831.06701.06251.0042
University of Amsterdam2011
University of Amsterdam20121.12701.14320.98581.04351.00671.03651.04591.00751.0381
University of Amsterdam20131.10021.10021.00001.04041.02621.01381.05081.04301.0075
University of Amsterdam20140.96731.01500.95291.05951.03021.02841.03531.01291.0222
University of Amsterdam20151.04041.02571.01431.06161.04771.01331.06201.05061.0108
University of Amsterdam20161.14301.06161.07661.02991.01561.01411.05011.01471.0350
University of Basel2011
University of Basel20121.03411.00601.02800.95240.92151.03351.03121.00381.0273
University of Basel20130.99241.01010.98251.03731.02161.01540.99221.01010.9822
University of Basel20141.02811.00001.02811.11481.07391.03811.02811.00001.0281
University of Basel20150.99641.00000.99641.09351.08931.00380.99901.00000.9990
University of Basel20161.04161.00001.04161.08731.09140.99621.04141.00001.0414
University of Bergen2011
University of Bergen20121.00590.97151.03541.04561.02681.01831.00590.97151.0354
University of Bergen20130.97070.99170.97881.03961.03741.00210.97030.99450.9756
University of Bergen20140.94280.94640.99611.04531.01681.02800.94130.94710.9939
University of Bergen20151.02260.98941.03361.03591.02071.01491.02600.99451.0318
University of Bergen20161.19521.17391.01821.03301.02481.00801.19011.16371.0226
University of Birmingham2011
University of Birmingham20120.94560.95130.99400.98040.95301.02880.96160.94921.0131
University of Birmingham20130.99471.02290.97240.97590.97391.00210.96281.00810.9551
University of Birmingham20141.02661.04860.97890.99280.96751.02611.02741.01231.0149
University of Birmingham20151.04401.02791.01571.02761.00711.02031.04951.03521.0139
University of Birmingham20161.10721.08711.01851.03221.01831.01361.09561.06391.0298
University of Bonn2011
University of Bonn20121.11561.07751.03531.00510.98551.01981.11561.07751.0353
University of Bonn20131.10551.12740.98061.07721.06001.01621.10551.12740.9806
University of Bonn20141.00370.99701.00671.04111.01181.02901.00380.99721.0066
University of Bonn20151.03811.04740.99120.97470.97510.99961.03811.04720.9913
University of Bonn20161.01790.99361.02441.02540.99961.02591.01790.99361.0244
University of Bristol2011
University of Bristol20121.08561.05971.02441.04360.98781.05641.07051.01151.0583
University of Bristol20130.95370.97520.97800.84620.87990.96180.91030.95310.9551
University of Bristol20140.97931.01380.96591.03751.06480.97431.00111.02100.9805
University of Bristol20151.00570.98471.02130.94230.93291.01010.99550.98011.0158
University of Bristol20161.07711.05061.02520.95110.92631.02671.05621.01111.0446
University of Cambridge2011
University of Cambridge20121.02431.00001.02431.00551.00001.00551.01321.00001.0132
University of Cambridge20130.99641.00000.99641.01621.00001.01621.01621.00001.0162
University of Cambridge20140.99521.00000.99521.05131.00001.05131.03031.00001.0303
University of Cambridge20150.98581.00000.98581.01541.00001.01541.01311.00001.0131
University of Cambridge20161.00441.00001.00441.03831.00001.03831.02511.00001.0251
University of Copenhagen2011
University of Copenhagen20121.30031.23251.05501.07751.03911.03701.16591.12271.0385
University of Copenhagen20131.02061.02710.99361.06871.07750.99181.00181.02380.9785
University of Copenhagen20141.00271.00191.00081.08291.04521.03601.05371.03481.0183
University of Copenhagen20150.99240.98771.00481.03901.01791.02081.02731.01021.0170
University of Copenhagen20161.17181.14351.02481.07631.04491.03001.09171.05541.0344
University of Dundee2011
University of Dundee20121.07181.03261.03801.09151.00351.08781.06961.00001.0696
University of Dundee20130.91910.94000.97780.84340.86120.97940.90900.95210.9547
University of Dundee20141.03101.06390.96910.97750.96211.01601.02921.05030.9800
University of Dundee20150.99540.94811.04980.91410.90311.01210.99160.95051.0433
University of Dundee20161.06521.04311.02121.03791.03391.00391.06431.04061.0228
University of East Anglia2011
University of East Anglia20121.09771.11410.98530.97970.99340.98621.12031.10981.0095
University of East Anglia20130.99161.01490.97710.86960.92100.94420.95621.00020.9560
University of East Anglia20140.96231.00480.95770.96270.88801.08420.95430.99310.9609
University of East Anglia20151.02650.96621.06241.04231.02211.01981.02830.97181.0581
University of East Anglia20161.10031.10740.99361.01180.99061.02141.09531.10040.9953
University of Edinburgh2011
University of Edinburgh20121.07181.03071.03981.04761.01691.03021.07181.03071.0398
University of Edinburgh20130.98741.00660.98091.00350.99251.01110.98641.00660.9799
University of Edinburgh20140.95140.96930.98151.03981.01101.02850.95740.97510.9819
University of Edinburgh20151.00681.00541.00140.99890.98381.01541.00370.99951.0042
University of Edinburgh20161.09961.05811.03921.02330.99831.02511.09961.05811.0392
University of Exeter2011
University of Exeter20121.09161.05641.03331.06981.03411.03451.09161.05641.0333
University of Exeter20131.05881.08990.97151.08551.12710.96311.05881.08990.9715
University of Exeter20141.01121.01770.99361.00160.93511.07121.01141.01820.9933
University of Exeter20151.00600.97661.03010.92500.93890.98521.00650.97741.0298
University of Exeter20161.16101.12331.03351.23871.23571.00251.15431.12551.0256
University of Freiburg2011
University of Freiburg20121.02991.01531.01441.03571.00461.03091.02991.01531.0144
University of Freiburg20131.01191.01970.99241.02371.01551.00811.01191.01970.9924
University of Freiburg20141.04131.02321.01771.05851.02401.03361.04131.02321.0177
University of Freiburg20150.98080.98740.99331.02221.02151.00060.98080.98740.9933
University of Freiburg20161.17391.14581.02451.02500.99711.02791.17361.14581.0242
University of Geneva2011
University of Geneva20121.01841.03520.98381.03940.98761.05241.01831.02020.9981
University of Geneva20130.97410.99970.97440.99600.96591.03120.97630.99930.9770
University of Geneva20141.01171.00871.00301.06621.02421.04101.01221.00071.0115
University of Geneva20151.00761.00001.00761.01621.00961.00661.00841.00001.0084
University of Geneva20161.02101.00001.02101.00371.00181.00191.02111.00001.0211
University of Glasgow2011
University of Glasgow20121.11431.07411.03741.03451.00451.02991.11431.07411.0374
University of Glasgow20130.94110.95740.98300.96900.96171.00760.94190.95760.9836
University of Glasgow20141.01171.04360.96951.02270.99621.02661.01831.04980.9700
University of Glasgow20151.04391.01931.02411.05651.05271.00361.04261.02141.0208
University of Glasgow20161.01500.96511.05181.01030.98201.02891.01090.95711.0561
University of Göttingen2011
University of Göttingen20120.90910.87531.03871.24551.18751.04880.90950.87531.0391
University of Göttingen20131.09851.12340.97781.11141.08451.02481.09861.12340.9780
University of Göttingen20140.96760.97190.99560.67410.64771.04070.96600.97190.9938
University of Göttingen20151.02701.02750.99951.02381.01861.00511.02701.02750.9995
University of Göttingen20160.79600.77131.03211.03041.00851.02170.79600.77131.0321
University of Groningen2011
University of Groningen20121.05211.01101.04071.07011.03721.03181.08121.03951.0401
University of Groningen20131.22161.18761.02861.04971.03311.01611.17581.19300.9856
University of Groningen20140.95290.99290.95981.04231.00791.03421.01221.00931.0029
University of Groningen20151.00810.99801.01011.04481.02091.02351.01880.99801.0209
University of Groningen20161.09351.03271.05881.03231.02301.00901.04291.00201.0409
University of Helsinki2011
University of Helsinki20120.99880.97341.02611.00140.96751.03500.99830.96241.0374
University of Helsinki20131.02321.05000.97451.00031.03160.96961.00711.05120.9580
University of Helsinki20140.98451.01130.97351.01930.99161.02791.00641.01850.9881
University of Helsinki20151.02121.00661.01451.04671.03251.01381.02561.00651.0190
University of Helsinki20161.05991.01681.04231.03201.00991.02191.03990.97301.0687
University of Konstanz2011
University of Konstanz20120.91181.00000.91180.92761.00000.92760.91181.00000.9118
University of Konstanz20131.09221.00001.09221.01051.00001.01051.09221.00001.0922
University of Konstanz20140.97441.00000.97441.07911.00001.07910.97441.00000.9744
University of Konstanz20151.00521.00001.00520.92701.00000.92701.00561.00001.0056
University of Konstanz20161.34171.00001.34171.01631.00001.01631.33201.00001.3320
University of Lausanne2011
University of Lausanne20121.07921.06421.01400.99380.93001.06851.07381.05981.0132
University of Lausanne20130.96470.98890.97551.01641.00441.01200.95920.98280.9760
University of Lausanne20141.01761.00951.00811.05541.01961.03511.01891.00911.0097
University of Lausanne20150.99740.98281.01491.02411.02830.99590.99790.98581.0123
University of Lausanne20161.05181.02201.02910.95370.95950.99391.04911.02521.0233
University of Leeds2011
University of Leeds20121.05971.05301.00641.03321.00731.02571.09201.07541.0155
University of Leeds20131.05581.06760.98890.96590.96361.00240.98571.02410.9625
University of Leeds20141.04581.07030.97711.02370.99771.02611.05051.06540.9860
University of Leeds20150.96830.95061.01861.02941.01721.01200.97970.96041.0201
University of Leeds20161.13771.11611.01941.05761.02331.03351.12741.09091.0334
University of Liverpool2011
University of Liverpool20121.07021.06511.00481.02981.00201.02781.09841.08541.0120
University of Liverpool20131.03041.06380.96860.96650.95861.00821.00351.04140.9635
University of Liverpool20141.01251.00441.00801.02240.99271.03001.02121.00941.0116
University of Liverpool20151.06161.04331.01751.04061.02331.01691.06001.04781.0117
University of Liverpool20161.09261.06201.02881.01851.00901.00941.08071.04951.0297
University of Manchester2011
University of Manchester20121.03981.02981.00971.00790.97131.03771.01700.99731.0198
University of Manchester20130.98630.98760.99870.97400.98980.98400.96120.98350.9773
University of Manchester20141.00141.01320.98831.00940.97661.03361.03441.02481.0094
University of Manchester20151.02101.01111.00971.01001.00341.00661.01471.00381.0108
University of Manchester20161.06221.04251.01891.01770.99031.02771.05751.02781.0289
University of Nottingham2011
University of Nottingham20120.96970.96990.99981.05771.02651.03040.99790.98151.0167
University of Nottingham20130.98040.99310.98720.98720.99190.99530.93590.98090.9541
University of Nottingham20140.97820.99640.98171.04411.01881.02491.01781.00131.0165
University of Nottingham20151.00390.99091.01311.03711.02701.00981.02941.01001.0192
University of Nottingham20161.14281.13221.00931.01230.99221.02031.10061.06761.0309
University of Oxford2011
University of Oxford20121.03691.00001.03691.05991.00001.05991.04031.00001.0403
University of Oxford20131.00901.00001.00901.04631.00001.04631.05681.00001.0568
University of Oxford20141.02511.00001.02511.07171.00001.07171.05401.00001.0540
University of Oxford20150.94961.00000.94961.02931.00001.02930.96951.00000.9695
University of Oxford20161.00561.00001.00561.05881.00001.05881.03061.00001.0306
University of Sheffield2011
University of Sheffield20121.06381.05091.01230.98180.95851.02431.06491.01271.0516
University of Sheffield20131.01371.03920.97550.95300.95211.00100.96030.99980.9605
University of Sheffield20140.99401.02320.97141.00550.98111.02481.01391.03050.9839
University of Sheffield20151.01990.99951.02030.99140.97511.01671.02191.00741.0144
University of Sheffield20161.08571.05571.02841.00530.99031.01511.07191.04371.0270
University of Southampton2011
University of Southampton20121.02551.00781.01750.99990.96821.03271.04761.01721.0299
University of Southampton20131.01991.04060.98010.99040.98841.00200.97931.01940.9606
University of Southampton20140.98110.98340.99771.04191.01641.02511.00391.00460.9994
University of Southampton20151.03791.02731.01041.07051.05501.01471.04621.03381.0120
University of Southampton20161.06681.04931.01671.01930.99821.02121.05591.02801.0271
University of St Andrews2011
University of St Andrews20120.94441.00000.94441.07251.00001.07250.96401.00000.9640
University of St Andrews20130.99871.00000.99870.98441.00000.98441.02741.00001.0274
University of St Andrews20140.97561.00000.97560.90391.00000.90390.88991.00000.8899
University of St Andrews20151.03121.00001.03120.99121.00000.99121.03271.00001.0327
University of St Andrews20161.09591.00001.09590.97961.00000.97961.09541.00001.0954
University of Sussex2011
University of Sussex20120.88211.00000.88211.03051.00001.03050.88211.00000.8821
University of Sussex20131.01751.00001.01750.90441.00000.90441.01781.00001.0178
University of Sussex20140.92271.00000.92271.07601.00001.07600.95711.00000.9571
University of Sussex20151.03541.00001.03540.83380.95510.87301.03541.00001.0354
University of Sussex20160.99651.00000.99650.98970.97371.01640.99651.00000.9965
University of Tübingen2011
University of Tübingen20121.13671.07641.05601.01261.00141.01121.13671.07641.0560
University of Tübingen20131.09131.11370.97990.99470.98261.01231.09131.11370.9799
University of Tübingen20140.97290.96591.00721.01780.98891.02920.97370.97121.0025
University of Tübingen20151.05861.06220.99651.02481.02401.00081.05481.05650.9985
University of Tübingen20161.07681.03951.03591.01590.99021.02591.07681.03951.0359
University of Twente2011
University of Twente20121.05090.95011.10600.96310.93961.02501.03410.92871.1136
University of Twente20131.09141.04601.04341.01231.08390.93401.09141.04601.0434
University of Twente20140.99681.05430.94551.19941.13691.05501.00921.08440.9306
University of Twente20151.06351.05941.00390.89120.88721.00441.03981.03001.0095
University of Twente20161.07981.07751.00211.06871.06321.00511.07981.07751.0021
University of Würzburg2011
University of Würzburg20121.02500.98891.03650.99770.96961.02901.02500.98891.0365
University of Würzburg20131.12441.14980.97790.97040.96641.00411.12441.14980.9779
University of Würzburg20141.08601.07881.00671.01850.99041.02841.08601.07881.0067
University of Würzburg20151.05741.06530.99261.02191.00791.01401.05741.06530.9926
University of Würzburg20160.97170.95301.01961.00990.98701.02320.97170.95301.0196
University of York2011
University of York20120.88350.89380.98851.00800.96531.04420.88900.89120.9974
University of York20131.10141.10190.99960.86850.89190.97381.09101.10100.9909
University of York20141.02871.08020.95220.97850.91911.06461.03101.08020.9544
University of York20151.01050.97151.04010.94940.95430.99481.00980.97261.0382
University of York20160.93590.91271.02551.08381.08550.99850.93840.91841.0218
Uppsala University2011
Uppsala University20120.98470.95191.03451.00190.97871.02360.96660.94101.0271
Uppsala University20130.97080.94351.02890.97450.97111.00350.95330.95550.9976
Uppsala University20140.97641.02350.95401.02720.99971.02750.97330.99070.9824
Uppsala University20151.05051.03781.01230.97830.95891.02031.03481.01691.0176
Uppsala University20161.14251.07341.06441.04451.02641.01771.13571.06671.0647
Utrecht University2011
Utrecht University20121.37401.32671.03561.04581.01551.02981.05181.01551.0358
Utrecht University20131.03701.05100.98671.05961.00931.04991.04271.00931.0331
Utrecht University20140.99791.02460.97401.04291.00001.04291.02141.00001.0214
Utrecht University20150.98010.98440.99560.99001.00000.99000.99291.00000.9929
Utrecht University20161.09331.01691.07511.01081.00001.01081.03131.00001.0313
VU University Amsterdam2011
VU University Amsterdam20120.98430.93971.04741.03990.99401.04621.01620.97681.0403
VU University Amsterdam20131.01511.03030.98521.00510.98981.01551.00180.99341.0084
VU University Amsterdam20140.99811.02320.97541.07761.07311.00411.01571.00121.0145
VU University Amsterdam20151.01390.98861.02561.01461.01221.00241.02071.00521.0154
VU University Amsterdam20161.18101.12201.05261.09101.07191.01781.11981.06431.0522
Wageningen University & R.2011
Wageningen University & R.20121.05281.00001.05281.07831.06251.01491.06891.00001.0689
Wageningen University & R.20131.01351.00001.01350.92790.96860.95800.99801.00000.9980
Wageningen University & R.20140.93821.00000.93821.02780.96671.06320.96641.00000.9664
Wageningen University & R.20151.00241.00001.00241.05611.05581.00031.00331.00001.0033
Wageningen University & R.20161.07441.00001.07441.10481.10321.00141.08151.00001.0815


  1. Altbach, P.G.; Knight, J. The internationalization of higher education: Motivations and realities. J. Stud. Int. Educ. 2007, 11, 290–305. [Google Scholar] [CrossRef]
  2. Albers, S. Esteem indicators: Membership in editorial boards or honorary doctorates, discussion of quantitative and qualitative rankings of scholars by Rost and Frey. Schmalenbach Bus. Rev. 2011, 63, 92–98. [Google Scholar] [CrossRef]
  3. Teichler, U. Social Contexts and Systematic Consequences of University Rankings: A Meta-Analysis of the Ranking Literature. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 55–69. [Google Scholar]
  4. Klumpp, M.; de Boer, H.; Vossensteyn, H. Comparing national policies on institutional profiling in Germany and the Netherlands. Comp. Educ. 2014, 50, 156–176. [Google Scholar] [CrossRef]
  5. Groot, T.; García-Valderrama, T. Research quality and efficiency—An analysis of assessments and management issues in Dutch economics and business research programs. Res. Policy 2006, 35, 1362–1376. [Google Scholar] [CrossRef]
  6. Costas, R.; van Leeuwen, T.N.; Bordos, M. A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact. J. Am. Soc. Inf. Sci. Technol. 2010, 61, 1564–1581. [Google Scholar] [CrossRef]
  7. Krapf, M. Research evaluation and journal quality weights: Much ado about nothing. J. Bus. Econ. 2011, 81, 5–27. [Google Scholar] [CrossRef]
  8. Rost, K.; Frey, B.S. Quantitative and qualitative rankings of scholars. Schmalenbach Bus. Rev. 2011, 63, 63–91. [Google Scholar] [CrossRef]
  9. Sarrico, C.S. On performance in higher education—Towards performance government. Tert. Educ. Manag. 2010, 16, 145–158. [Google Scholar] [CrossRef]
  10. Kreiman, G.; Maunsell, J.H.R. Nine criteria for a measure of scientific output. Front. Comput. Neurosci. 2011, 5, 48. [Google Scholar] [CrossRef] [PubMed]
  11. Wilkins, S.; Huisman, J. Stakeholder perspectives on citation and peer-based rankings of higher education journals. Tert. Educ. Manag. 2015, 21, 1–15. [Google Scholar] [CrossRef]
  12. Hicks, D.; Wouters, P.; Waltman, L.; de Rijcke, S.; Rafols, I. The Leiden Manifesto for research. Nature 2016, 520, 429–431. [Google Scholar] [CrossRef] [PubMed]
  13. Parteka, A.; Wolszczak-Derlacz, J. Dynamics of productivity in higher education: Cross-European evidence based on bootstrapped Malmquist indices. J. Prod. Anal. 2013, 40, 67–82. [Google Scholar] [CrossRef]
  14. Dundar, H.; Lewis, D.R. Departmental productivity in American universities: Economies of scale and scope. Econ. Educ. Rev. 1995, 14, 199–244. [Google Scholar] [CrossRef]
  15. Hashimoto, K.; Cohn, E. Economies of scale and scope in Japanese private universities. Educ. Econ. 1997, 5, 107–115. [Google Scholar] [CrossRef]
  16. Casu, B.; Thanassoulis, E. Evaluating cost efficiency in central administrative services in UK universities. Omega 2006, 34, 417–426. [Google Scholar] [CrossRef]
  17. Sarrico, C.S.; Teixeira, P.N.F.L.; Rosa, M.J.; Cardoso, M.F. Subject mix and productivity in Portuguese universities. Eur. J. Oper. Res. 2009, 197, 287–295. [Google Scholar] [CrossRef]
  18. Mainardes, E.; Alves, H.; Raposo, M. Using expectations and satisfaction to measure the frontiers of efficiency in public universities. Tert. Educ. Manag. 2014, 20, 339–353. [Google Scholar] [CrossRef]
  19. Daraio, C.; Bonaccorsi, A.; Simar, L. Efficiency and economies of scale and specialization in European universities—A directional distance approach. J. Infometr. 2015, 9, 430–448. [Google Scholar] [CrossRef]
  20. Millot, B. International rankings: Universities vs. higher education systems. Int. J. Educ. Dev. 2015, 20, 156–165. [Google Scholar] [CrossRef]
  21. Dixon, R.; Hood, C. Ranking Academic Research Performance: A Recipe for Success? Sociol. Trav. 2016, 58, 403–411. [Google Scholar] [CrossRef]
  22. Lee, B.L.; Worthington, A.C. A network DEA quantity and quality-orientated production model: An application to Australian university research services. Omega 2016, 60, 26–33. [Google Scholar] [CrossRef]
  23. Olcay, G.A.; Bulu, M. Is measuring the knowledge creation of universities possible? A review of university rankings. Technol. Forecast. Soc. Change 2016. [Google Scholar] [CrossRef]
  24. Charnes, A.; Cooper, W.W.; Rhodes, E.L. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
  25. Banker, R.D.; Charnes, A.; Cooper, W.W. Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag. Sci. 1984, 30, 1078–1092. [Google Scholar] [CrossRef]
  26. Bessent, A.M.; Bessent, E.W.; Charnes, A.; Cooper, W.W.; Thorogood, N. Evaluation of educational program proposals by means of DEA. Educ. Admin. Q. 1983, 19, 82–107. [Google Scholar] [CrossRef]
  27. Johnes, G.; Johnes, J. Measuring the research performance of UK economics departments: An application of data envelopment analysis. Oxf. Econ. Pap. 1993, 45, 332–347. [Google Scholar] [CrossRef]
  28. Athanassopoulos, A.D.; Shale, E. Assessing the comparative efficiency of higher education institutions in the UK by means of data envelopment analysis. Educ. Econ. 1997, 5, 117–134. [Google Scholar] [CrossRef]
  29. McMillan, M.L.; Datta, D. The relative efficiencies of Canadian universities: A DEA perspective. Can. Public Policy 1998, 24, 485–511. [Google Scholar] [CrossRef]
  30. Ng, Y.C.; Li, S.K. Measuring the research performance of Chinese higher education institutions: An application of data envelopment analysis. Educ. Econ. 2000, 8, 139–156. [Google Scholar] [CrossRef]
  31. Feng, Y.J.; Lu, H.; Bi, K. An AHP/DEA method for measurement of the efficiency of R&D management activities in universities. Int. Trans. Oper. Res. 2004, 11, 181–191. [Google Scholar] [CrossRef]
  32. Johnes, J. Measuring efficiency: A comparison of multilevel modelling and data envelopment analysis in the context of higher education. Bull. Econ. Res. 2006, 58, 75–104. [Google Scholar] [CrossRef]
  33. Kocher, M.G.; Luptacik, M.; Sutter, M. Measuring productivity of research in economics: A cross-country study using DEA. Socio-Econ. Plan. Sci. 2006, 40, 314–332. [Google Scholar] [CrossRef]
  34. Malmquist, S. Index numbers and indifference surfaces. Trabajos De Estadistica 1953, 42, 209–242. [Google Scholar] [CrossRef]
  35. Grifell-Tatje, E.; Lovell, K.C.A.; Pastor, J.T. A quasi-Malmquist productivity index. J. Prod. Anal. 1998, 10, 7–20. [Google Scholar] [CrossRef]
  36. Wang, Y.M.; Lan, Y.X. Measuring Malmquist productivity index: A new approach based on double frontiers data envelopment analysis. Math. Comp. Model. 2011, 54, 2760–2771. [Google Scholar] [CrossRef]
  37. Castano, M.C.N.; Cabana, E. Sources of efficiency and productivity growth in the Philippine state universities and colleges: A non-parametric approach. Int. Bus. Econ. Res. J. 2007, 6, 79–90. [Google Scholar] [CrossRef]
  38. Worthington, A.C.; Lee, B.L. Efficiency, technology and productivity change in Australian universities, 1998–2003. Econ. Educ. Rev. 2008, 27, 285–298. [Google Scholar] [CrossRef]
  39. Times Higher Education. World University Rankings 2015–2016 Methodology. Available online: (accessed on 12 April 2018).
  40. Academic Ranking of World Universities: Methodology. Available online: (accessed on 12 April 2018).
  41. Morphew, C.C.; Swanson, C. On the Efficacy of Raising Your University’s Rankings. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 185–199. [Google Scholar]
  42. Federkeil, G.; van Vught, F.A.; Westerheijden, D.F. Classifications and Rankings. In Multidimensional Ranking—The Design and Development of U-Multirank; van Vught, F.A., Ziegele, F., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 25–37. [Google Scholar]
  43. Hazelkorn, E. Rankings and the Reshaping of Higher Education: The Battle for World-Class Excellence; Palgrave Macmillan: London, UK, 2011. [Google Scholar]
  44. Farrell, M.J. The Measurement of Productive Efficiency. J. R. Stat. Soc. Ser. A 1957, 120, 253–290. [Google Scholar] [CrossRef]
  45. Ylvinger, S. Industry performance and structural efficiency measures: Solutions to problems in firm models. Eur. J. Oper. Res. 2000, 121, 164–174. [Google Scholar] [CrossRef]
  46. Teichler, U. The Future of University Rankings. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 259–265. [Google Scholar]
  47. Shin, J.C. Organizational Effectiveness and University Rankings. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 19–34. [Google Scholar]
  48. Yeravdekar, V.R.; Tiwari, G. Global Rankings of Higher Education Institutions and India’s Effective Non-Presence. Procedia Soc. Behav. Sci. 2014, 157, 63–83. [Google Scholar] [CrossRef]
  49. O’Connel, C. An examination of global university rankings as a new mechanism influencing mission differentiation: The UK context. Tert. Educ. Manag. 2015, 21, 111–126. [Google Scholar] [CrossRef]
  50. Bonaccorsi, A.; Cicero, T. Nondeterministic ranking of university departments. J. Informetr. 2016, 10, 224–237. [Google Scholar] [CrossRef]
  51. Kehm, B.M.; Stensaker, B. University Rankings, Diversity and the New Landscape of Higher Education; Sense Publishers: Rotterdam, The Netherlands, 2009. [Google Scholar]
  52. Hazelkorn, E. Higher Education’s Future: A New Global World Order. In Resilient Universities—Confronting Changes in a Challenging World; Karlsen, J.E., Pritchard, R.M.O., Eds.; Peter Lang: Oxford, UK, 2013; pp. 53–89. [Google Scholar]
  53. Federkeil, G.; van Vught, F.A.; Westerheijden, D.F. An Evaluation and Critique of Current Rankings. In Multidimensional Ranking—The Design and Development of U-Multirank; van Vught, F.A., Ziegele, F., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 39–70. [Google Scholar]
  54. Times Higher Education. World University Rankings 2016–2017 Methodology. Available online:!/page/0/length/25/sort_by/rank/sort_order/asc/cols/stats (accessed on 12 April 2018).
  55. Institute for Higher Education Policy. College and University Ranking Systems—Global Perspectives and American Challenges; Institute for Higher Education Policy: Washington, DC, USA, 2007. [Google Scholar]
  56. Rauhvargers, A. Global University Rankings and Their Impact—EUA Report on Rankings 2013; European University Association (EUA): Brussels, Belgium, 2013. [Google Scholar]
  57. Longden, B. Ranking Indicators and Weights. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 73–104. [Google Scholar]
  58. Selkälä, A.; Ronkainen, S.; Alasaarela, E. Features of the Z-scoring method in graphical two-dimensional web surveys: The case of ZEF. Qual. Quant. 2011, 45, 609–621. [Google Scholar] [CrossRef]
  59. University of Leiden. CWTS Leiden Ranking 2011/12—Methodology; Centre for Science and Technology Studies: Leiden, The Netherlands, 2012. [Google Scholar]
  60. Waltman, L.; van Eck, N.J.; Tijssen, R.; Wouters, P. Moving Beyond Just Ranking—The CWTS Leiden Ranking 2016. Available online: (accessed on 12 April 2018).
  61. ETER Project: European Tertiary Education Register. Available online:>A (accessed on 12 April 2018).
  62. Humboldt, W. Von Über die innere und äussere Organisation der höheren wissenschaftlichen Anstalten zu Berlin. In Gelegentliche Gedanken Über Universitäten; Müller, E., Ed.; Reclam: Leipzig, Germany, 1809; pp. 273–283. [Google Scholar]
  63. Anderson, R. Before and after Humboldt: European universities between the eighteenth and the nineteenth century. Hist. High. Educ. Annu. 2000, 20, 5–14. [Google Scholar]
  64. Ash, M.G. Bachelor of What, Master of Whom? The Humboldt Myth and Historical Transformations of Higher Education in German-Speaking Europe and the US. Eur. J. Educ. 2006, 41, 245–267. [Google Scholar] [CrossRef]
  65. Koopmans, T.C. Analysis of Production as an Efficient Combination of Activities. In Analysis of Production and Allocation, Proceedings of A Conference; Koopmans, T.C., Ed.; Wiley: New York, NY, USA, 1953; pp. 33–97. [Google Scholar]
  66. Debreu, G. The Coefficient of Resource Utilization. Econometrica 1951, 19, 273–292. [Google Scholar] [CrossRef]
  67. Diewert, W.E. Functional Forms for Profit and Transformation Functions. J. Econ. Theory 1973, 6, 284–316. [Google Scholar] [CrossRef]
  68. Cohn, E.; Rhine, S.L.W.; Santos, M.C. Institutions of higher education as multiproduct firms: Economies of scale and scope. Rev. Econ. Stat. 1989, 71, 284–290. [Google Scholar] [CrossRef]
  69. Flegg, T.A.; Allen, D.O.; Field, K.; Thurlow, T.W. Measuring the Efficiency of British Universities: A Multi-period Data Envelopment Analysis. Educ. Econ. 2004, 12, 231–249. [Google Scholar] [CrossRef]
  70. Agasisti, T.; Johnes, G. Beyond frontiers: Comparing the efficiency of higher education decision-making units across more than one country. Educ. Econ. 2009, 17, 59–79. [Google Scholar] [CrossRef]
  71. Ramón, N.; Ruiz, J.L.; Sirvent, I. Using Data Envelopment Analysis to Assess Effectiveness of the Processes at the University with Performance Indicators of Quality. Int. J. Oper. Quant. Manag. 2010, 16, 87–103. [Google Scholar]
  72. Bolli, T.; Olivares, M.; Bonaccorsi, A.; Daraio, C.; Aracil, A.G.; Lepori, B. The differential effects of competitive funding on the production frontier and the efficiency of universities. Econ. Educ. Rev. 2016, 52, 91–104. [Google Scholar] [CrossRef]
  73. Dyckhoff, H.; Clermont, M.; Dirksen, A.; Mbock, E. Measuring balanced effectiveness and efficiency of German business schools’ research performance. J. Bus. Econ. 2013, 39–60. [Google Scholar] [CrossRef]
  74. Cooper, W.W.; Seiford, L.M.; Tone, K. Data Envelopment Analysis—A Comprehensive Text with Models, Applications, References and DEA-Solver Software; Springer: New York, NY, USA, 2007. [Google Scholar]
  75. Ding, L.; Yang, Q.; Sun, L.; Tong, J.; Wang, Y. Evaluation of the Capability of Personal Software Process based on Data Envelopment Analysis. In Unifying the Software Process Spectrum: International Software Process Spectrum; Li, M., Boehm, B., Osterweil, L.J., Eds.; Springer: Berlin, Germany, 2006; pp. 235–248. [Google Scholar]
  76. Fisher, I. The Purchasing Power of Money, Its Determination and Relation to Credit, Interest and Crises; MacMillan: New York, NY, USA, 1911. [Google Scholar]
  77. Fisher, I. The Making of Index Numbers; Houghton Mifflin: Boston, MA, USA, 1922. [Google Scholar]
  78. Divisia, F. L’ Indice Monétaire et la Theorie de la Monnaie. Rev. Econ. Politique 1926, 40, 49–81. [Google Scholar]
  79. Nordhaus, W.D. Quality Change in Price Indices. J. Econ. Perspect. 1998, 12, 59–68. [Google Scholar] [CrossRef]
  80. Pollak, R.A. The Consumer Price Index: A Research Agenda and Three Proposals. J. Econ. Perspect. 1998, 12, 69–78. [Google Scholar] [CrossRef]
  81. Giannetti, B.F.; Agostinho, F.; Almeida, C.M.V.B.; Huisingh, D. A review of limitations of GDP and alternative indices to monitor human wellbeing and to manage eco-system functionality. J. Clean. Prod. 2015, 87, 11–25. [Google Scholar] [CrossRef]
  82. Pal, D.; Mita, S.K. Asymmetric oil product pricing in India: Evidence from a multiple threshold nonlinear ARDL model. Econ. Model. 2016, 59, 314–328. [Google Scholar] [CrossRef]
  83. Petrou, P.; Vandoros, S. Pharmaceutical price comparisons across the European Union and relative affordability in Cyprus. Health Policy Technol. 2016, 5, 350–356. [Google Scholar] [CrossRef]
  84. Frisch, R. Annual Survey of General Economic Theory: The Problem of Index Numbers. Econometrica 1936, 4, 1–38. [Google Scholar] [CrossRef]
  85. Theil, H. Best Linear Index Numbers of Prices and Quantities. Econometrica 1960, 28, 464–480. [Google Scholar] [CrossRef]
  86. Kloek, T.; Theil, H. International Comparisons of Prices and Quantities Consumed. Econometrica 1965, 33, 535–556. [Google Scholar] [CrossRef]
  87. Gilbert, M. The Problem of Quality Changes and Index Numbers. Mon. Lab. Rev. 1961, 84, 992–997. [Google Scholar]
  88. Kloek, T.; de Wit, G.M. Best Linear and Best Linear Unbiased Index Numbers. Econometrica 1961, 29, 602–616. [Google Scholar] [CrossRef]
  89. Jiménez-Sáeza, F.; Zabala-Iturriagagoitia, J.M.; Zofío, J.L.; Castro-Martínez, E. Evaluating research efficiency within National R&D Programmes. Res. Policy 2011, 40, 230–241. [Google Scholar] [CrossRef]
  90. Juo, J.C.; Fu, T.T.; Yu, M.M.; Lin, Y.H. Non-radial profit performance: An application to Taiwanese banks. Omega 2016, 65, 111–121. [Google Scholar] [CrossRef]
  91. Gulati, R.; Kumar, S. Assessing the impact of the global financial crisis on the profit efficiency of Indian banks. Econ. Model. 2016, 58, 167–181. [Google Scholar] [CrossRef]
  92. Lazov, I. Profit management of car rental companies. Eur. J. Oper. Res. 2016, 258, 307–314. [Google Scholar] [CrossRef]
  93. Pasternack, P. Die Exzellenzinitiative als politisches Programm—Fortsetzung der normalen Forschungsförderung oder Paradigmentwechsel? In Making Excellence—Grundlagen, Praxis und Konsequenzen der Exzelleninitiative; Bloch, R., Keller, A., Lottmann, A., Würmann, C., Eds.; Bertelsmann: Bielefeld, Germany, 2008; pp. 13–36. [Google Scholar]
  94. Örkcü, H.H.; Balıkçı, C.; Dogan, M.I.; Genç, A. An evaluation of the operational efficiency of Turkish airports using data envelopment analysis and the Malmquist productivity index: 2009–2014 case. Transp. Policy 2016, 48, 92–104. [Google Scholar] [CrossRef]
  95. Sueyoshi, T.; Goto, M. DEA environmental assessment in time horizon: Radial approach for Malmquist index measurement on petroleum companies. Energy Econ. 2015, 51, 329–345. [Google Scholar] [CrossRef]
  96. Emrouznejad, A.; Yang, G. A framework for measuring global Malmquist-Luenberger productivity index with CO2 emissions on Chinese manufacturing industries. Energy 2016, 115, 840–856. [Google Scholar] [CrossRef]
  97. Fandel, G. On the performance of universities in North Rhine-Westphalia, Germany: Government’s redistribution of funds judged using DEA efficiency measures. Eur. J. Oper. Res. 2007, 176, 521–533. [Google Scholar] [CrossRef]
  98. Johnes, J. Data envelopment analysis and its application to the measurement of efficiency in higher education. Econ. Educ. Rev. 2006, 25, 273–288. [Google Scholar] [CrossRef]
  99. Fuentes, R.; Fuster, B.; Lillo-Banuls, A. A three-stage DEA model to evaluate learning-teaching technical efficiency: Key performance indicators and contextual variables. Expert Syst. Appl. 2016, 48, 89–99. [Google Scholar] [CrossRef]
  100. Alsabawy, A.Y.; Cater-Steel, A.; Soar, J. Determinants of perceived usefulness of e-learning systems. Comput. Hum. Behav. 2016, 64, 843–858. [Google Scholar] [CrossRef]
  101. Chang, V. E-learning for academia and industry. Int. J. Inf. Manag. 2016, 36, 476–485. [Google Scholar] [CrossRef]
  102. Tiemann, O.; Schreyögg, J. Effects of Ownership on Hospital Efficiency in Germany. Bus. Res. 2009, 2, 115–145. [Google Scholar] [CrossRef]
  103. Harlacher, D.; Reihlen, M. Governance of professional service firms: A configurational approach. Bus. Res. 2014, 7, 125–160. [Google Scholar] [CrossRef]
  104. Bottomley, A.; Dunworth, J. Rate of return analysis and economies of scale in higher education. Socio-Econ. Plan. Sci. 1974, 8, 273–280. [Google Scholar] [CrossRef]
  105. Bornmann, L. Peer Review and Bibliometric: Potentials and Problems. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 145–164. [Google Scholar]
  106. Osterloh, M.; Frey, B.S. Academic Rankings between the Republic of Science and New Public Management. In The Economics of Economists; Lanteri, A., Fromen, J., Eds.; Cambridge University Press: Cambridge, UK, 2014; pp. 77–78. [Google Scholar]
  107. Harmann, G. Competitors of Rankings: New Directions in Quality Assurance and Accountability. In University Rankings—Theoretical Basis, Methodology and Impacts on Global Higher Education; Shin, J.C., Toutkoushian, R.K., Teichler, U., Eds.; Springer: Dordrecht, The Netherlands, 2011; pp. 35–53. [Google Scholar]
  108. Eisend, M. Is VHB-JOURQUAL2 a Good Measure of Scientific Quality? Assessing the Validity of the Major Business Journal Ranking in German-Speaking Countries. Bus. Res. 2011, 4, 241–274. [Google Scholar] [CrossRef]
  109. Lorenz, D.; Löffler, A. Robustness of personal rankings: The Handelsblatt example. Bus. Res. 2015, 8, 189–212. [Google Scholar] [CrossRef]
  110. Schrader, U.; Hennig-Thurau, T. VHB-JOURQUAL2: Method, Results, and Implications of the German Academic Association for Business Research’s Journal Ranking. Bus. Res. 2009, 2, 180–204. [Google Scholar] [CrossRef]
  111. Blackmore, P. Universities Face a Choice between Prestige and Efficiency, 5 April 2016. Available online: (accessed on 12 April 2018).
  112. Destatis. Education and Culture—University Finances, 11(4.5) Bildung und Kultur, Finanzen der Hochschulen, Fachserie 11, Reihe 4.5; Statistisches Bundesamt: Wiesbaden, Germany, 2014. [Google Scholar]
  113. Abramo, G.; Cicero, T.; D’Angelo, C.A. A sensitivity analysis of researchers’ productivity rankings to the time of citation observation. J. Informetr. 2012, 6, 192–201. [Google Scholar] [CrossRef]
  114. Baruffaldi, S.H.; Landoni, P. Return mobility and scientific productivity of researchers working abroad: The role of home country linkages. Res. Policy 2012, 41, 1655–1665. [Google Scholar] [CrossRef]
  115. Beaudry, C.; Larivière, V. Which gender gap? Factors affecting researchers’ scientific impact in science and medicine. Res. Policy 2016, 45, 1790–1817. [Google Scholar] [CrossRef]
  116. Fedderke, J.W.; Goldschmidt, M. Does massive funding support of researchers work: Evaluating the impact of the South African research chair funding initiative. Res. Policy 2015, 44, 467–482. [Google Scholar] [CrossRef]
Figure 1. Input and Output Indicators and Correlations.
Figure 1. Input and Output Indicators and Correlations.
Education 08 00079 g001
Figure 2. DEA Frontier Graph for CCR and BBC Models.
Figure 2. DEA Frontier Graph for CCR and BBC Models.
Education 08 00079 g002
Figure 3. Mean Malmquist Index 2012–2016 for Three Datasets (Run I, II, III).
Figure 3. Mean Malmquist Index 2012–2016 for Three Datasets (Run I, II, III).
Education 08 00079 g003
Table 1. Indicator Weights and Definitions for the THE Ranking [54].
Table 1. Indicator Weights and Definitions for the THE Ranking [54].
Output FieldWeightIndicatorWeight
THE Teaching *30.00%Academic reputation survey (THE)15.00%
Doctorates awarded-to-academic staff ratio **6.00%
Staff-to-student ratio4.50%
Doctorate-to-bachelor’s ratio2.25%
Institutional income ***2.25%
THE International Outlook *7.50%International-to-domestic-student ratio2.50%
International-to-domestic-staff ratio2.50%
International collaboration (proportion of research journal publications with at least one international co-author) **2.50%
THE Research *30.00%Academic reputation survey (THE)18.00%
Research income6.00%
Research productivity (publications in Scopus indexed academic journals per scholar) **6.00%
THE Citations *30.00%Number of times a university’s published work is cited by scholars globally, compared with the number of citations a publication of similar type and subject is expected to have. (Bibliometric data supplier Elsevier examined more than 51 million citations to 11.3 million journal articles, published over five years. The data are drawn from the 23,000 academic journals indexed by Scopus and include all indexed journals published between 2010 and 2014. Only three types of publications are analysed: journal articles, conference proceedings and reviews—citations to these papers from 2010 to 2015 are collected.)30.00%
THE Industry Income *2.50%Research income an institution earns from industry ***2.50%
* Indexed value, maximum data value of 100.00. ** Discipline normalised. *** Scaled against staff numbers and normalised for purchasing-power parity.
Table 2. Calculation Data Cut-out for Two Example Universities, Sources: [39,40,61].
Table 2. Calculation Data Cut-out for Two Example Universities, Sources: [39,40,61].
U. Oxford (UK)20112016Change 2011–2016Arith. Mean **U. Würzburg (DE)20112016Change 2011–2016Arith. Mean **
THE Teaching *88.2086.50−1.93% 48.7034.60−28.95%
THE Int. Outlook *77.2094.4022.28% 40.3050.9026.30%
THE Research *93.9098.905.32% 40.9035.80−12.47%
THE Citations *95.1098.803.89% 60.4079.1030.96%
THE Industry Income *73.5073.10−0.54%5.80% 27.9047.9071.68%17.51%
CWTS_P10,701.0013,300.0024.29% 3219.003349.004.04%
CWTS_TCS89,149.00127,888.0043.45% 22,932.0026,866.0017.16%
CWTS_TNCS15,464.0020,373.0031.74% 3748.003998.006.67%
CWTS_P_top1215.00311.0044.65% 36.0051.0041.67%
CWTS_P_top506593.008408.0027.53%34.33% 1883.001899.000.85%14.08%
Budget (Mil. €, Input)1119.771349.7620.54% Budget (Mil. €, Input)806.97823.582.06%
Academic Staff (Input)5375.006120.0013.86%17.20%Academic Staff (Input)3281.003420.004.24%3.15%
* Indexed numbers; ** Arithmetic Mean of all changes for THE (indexed), CWTS (non-indexed) and input group data per university in italics.
Table 3. Total Input and Output Indicator Correlations (n = 420).
Table 3. Total Input and Output Indicator Correlations (n = 420).
n = 420BudgetAcad. StaffTHE TeachingTHE Int. OutlookTHE ResearchTHE CitationsTHE Ind.Inc.CWTS PCWTS TCSCWTS TNCSCWTS P_top1CWTS P_top50
Acad. Staff 1.0000.490−0.1300.3780.1720.1860.6810.6310.6390.5790.656
THE Teaching 1.0000.2280.8900.2310.2130.6900.7240.7330.7460.711
THE Internat. Outlook 1.0000.1730.375−0.1420.0530.1490.1380.2400.092
THE Research 1.0000.1980.2730.7000.7090.7330.7340.718
THE Citations 1.000−0.2520.2960.4230.3680.4080.334
THE Industry Income 1.0000.1670.1280.1780.1730.174
CWTS_P 1.0000.9630.9800.9210.995
CWTS_TCS 1.0000.9850.9590.978
CWTS_TNCS 1.0000.9740.994
CWTS_P_top1 1.0000.949
CWTS_P_top50 1.000
“Loading” (ari. mean r)0.4830.4680.5630.0990.5360.2560.1070.6480.6660.6710.6620.661
Table 4. University Efficiency Scores and Returns to Scale 2011 (Base Year).
Table 4. University Efficiency Scores and Returns to Scale 2011 (Base Year).
UniversityRun I—Eff. ScoreReturns to ScaleRun II—Eff. ScoreReturns to ScaleRun III—Eff. ScoreReturns to Scale
Aarhus University66.00%Decrease45.00%Decrease66.00%Decrease
Bielefeld University83.10%Decrease32.30%Increase83.10%Decrease
Delft University of Technology100.00%Constant54.00%Decrease100.00%Constant
Durham University99.20%Decrease66.70%Increase99.30%Decrease
ETH Lausanne100.00%Constant54.10%Decrease100.00%Constant
Eindhoven University of Technology100.00%Constant45.50%Decrease100.00%Constant
Erasmus University Rotterdam89.40%Decrease100.00%Constant100.00%Constant
Ghent University97.80%Decrease71.40%Decrease99.10%Decrease
Goethe University Frankfurt74.40%Decrease40.10%Decrease74.40%Decrease
Heidelberg University76.30%Decrease53.90%Decrease76.30%Decrease
Humboldt University of Berlin94.90%Decrease100.00%Constant100.00%Constant
Imperial College London100.00%Constant100.00%Constant100.00%Constant
KTH Royal Institute of Technology100.00%Constant58.70%Increase100.00%Constant
KU Leuven98.70%Decrease76.70%Decrease100.00%Constant
Karlsruhe Institute of Technology73.40%Decrease43.90%Decrease73.70%Decrease
Karolinska Institute100.00%Constant100.00%Constant100.00%Constant
King’s College London89.00%Decrease68.00%Decrease91.90%Decrease
LMU Munich80.30%Decrease59.30%Decrease80.30%Decrease
LSE London100.00%Constant47.20%Increase100.00%Constant
Lancaster University92.30%Decrease85.80%Increase94.70%Decrease
Leiden University100.00%Constant82.70%Decrease100.00%Constant
Lund University76.50%Decrease83.20%Decrease84.70%Decrease
Newcastle University90.00%Decrease100.00%Constant100.00%Constant
Queen Mary University of London98.50%Decrease33.60%Increase98.50%Decrease
RWTH Aachen University69.30%Decrease35.00%Decrease69.30%Decrease
Stockholm University82.50%Decrease36.70%Decrease82.50%Decrease
Swedish U. of Agricultural Sciences100.00%Constant38.50%Increase100.00%Constant
Technical University of Denmark98.30%Decrease45.20%Decrease98.30%Decrease
Technical University of Munich87.80%Decrease44.70%Decrease87.90%Decrease
Trinity College Dublin100.00%Constant65.50%Increase100.00%Constant
University College Dublin100.00%Constant50.60%Increase100.00%Constant
University College London97.80%Decrease100.00%Constant100.00%Constant
University of Aberdeen97.70%Decrease74.60%Increase100.00%Constant
University of Amsterdam67.00%Decrease88.10%Decrease88.10%Decrease
University of Basel98.40%Decrease49.50%Decrease98.60%Decrease
University of Bergen81.60%Decrease41.00%Decrease81.60%Decrease
University of Birmingham76.00%Decrease63.10%Decrease81.30%Decrease
University of Bonn69.50%Decrease39.90%Decrease69.50%Decrease
University of Bristol88.10%Decrease92.30%Increase98.30%Decrease
University of Cambridge100.00%Constant100.00%Constant100.00%Constant
University of Copenhagen61.60%Decrease71.90%Decrease71.90%Decrease
University of Dundee96.80%Decrease69.90%Increase100.00%Constant
University of East Anglia82.30%Decrease60.00%Increase84.80%Decrease
University of Edinburgh92.90%Decrease66.10%Decrease92.90%Decrease
University of Exeter77.50%Decrease44.70%Increase77.50%Decrease
University of Freiburg83.40%Decrease37.80%Decrease83.40%Decrease
University of Geneva95.80%Decrease54.20%Decrease98.00%Decrease
University of Glasgow81.30%Decrease59.40%Decrease81.30%Decrease
University of Groningen77.00%Decrease79.60%Decrease79.90%Decrease
University of Göttingen99.20%Decrease53.80%Decrease99.20%Decrease
University of Helsinki80.80%Decrease70.00%Decrease85.40%Decrease
University of Konstanz100.00%Constant100.00%Constant100.00%Constant
University of Lausanne87.20%Decrease54.80%Decrease88.10%Decrease
University of Leeds64.80%Decrease60.00%Decrease67.30%Decrease
University of Liverpool71.80%Decrease55.10%Decrease72.80%Decrease
University of Manchester81.90%Decrease83.80%Decrease87.00%Decrease
University of Nottingham76.60%Decrease63.50%Decrease82.30%Decrease
University of Oxford100.00%Constant100.00%Constant100.00%Constant
University of Sheffield71.80%Decrease69.70%Decrease79.20%Decrease
University of Southampton82.20%Decrease65.50%Decrease84.70%Decrease
University of St Andrews100.00%Constant100.00%Constant100.00%Constant
University of Sussex100.00%Constant100.00%Constant100.00%Constant
University of Twente83.60%Decrease58.50%Decrease85.50%Decrease
University of Tübingen64.70%Decrease44.70%Decrease64.70%Decrease
University of Würzburg65.70%Decrease36.00%Decrease65.70%Decrease
University of York94.00%Decrease67.00%Increase94.30%Decrease
Uppsala University82.20%Decrease73.30%Decrease87.20%Decrease
Utrecht University67.80%Decrease97.60%Decrease97.60%Decrease
VU University Amsterdam87.70%Decrease85.10%Decrease96.20%Decrease
Wageningen University and Research100.00%Constant85.10%Increase100.00%Constant
Eff. DMU17 2 24
Arithm. Mean87.21% 66.20% 89.78%
Min61.60% 32.30% 64.70%
Table 5. Mean Efficiency Improvement Scores in Three Calculation Runs I–III.
Table 5. Mean Efficiency Improvement Scores in Three Calculation Runs I–III.
Total Arithm. Mean1.02661.01701.00951.02511.01251.01391.02401.01241.0114
MI: Malmquist index; CU: Catch-up; FS: Frontier shift.
Table 6. Top 30 and Bottom 30 Annual Malmquist Index 2012–2016 (Cut-out, Sorted from Run III).
Table 6. Top 30 and Bottom 30 Annual Malmquist Index 2012–2016 (Cut-out, Sorted from Run III).
UniversityYearRUN I
Malmquist IndexCatch-UpFrontier ShiftMalmquist IndexCatch-UpFrontier ShiftMalmquist IndexCatch-UpFrontier Shift
University of Konstanz20161.34171.00001.34171.01631.00001.01631.33201.00001.3320
Bielefeld University20161.30231.09841.18561.05161.03121.01981.30061.09841.1841
Tec. University of Munich20161.22871.20161.02251.03311.00241.03061.22871.20161.0225
Humboldt University of Berlin20161.26641.00001.26641.00831.00001.00831.21221.00001.2122
RWTH Aachen University20161.19861.17601.01921.04411.01291.03081.19861.17601.0192
University of Bergen20161.19521.17391.01821.03301.02481.00801.19011.16371.0226
Karlsruhe Institute of Tec.20161.19481.18651.00701.07341.05721.01531.18751.18271.0041
Aarhus University20161.18831.14231.04031.06961.04251.02601.18701.14081.0405
University of Groningen20131.22161.18761.02861.04971.03311.01611.17581.19300.9856
University of Freiburg20161.17391.14581.02451.02500.99711.02791.17361.14581.0242
LMU Munich20151.16861.15051.01571.00250.95671.04791.16831.15011.0158
University of Copenhagen20121.30031.23251.05501.07751.03911.03701.16591.12271.0385
Stockholm University20121.15571.13241.02061.04401.01911.02441.15571.13241.0206
University of Exeter20161.16101.12331.03351.23871.23571.00251.15431.12551.0256
LSE London20161.15031.00001.15031.13641.00001.13641.15031.00001.1503
Aarhus University20121.15001.13351.01461.03670.99891.03781.15001.13351.0146
LSE London20121.13771.00001.13771.08731.03711.04851.13771.00001.1377
University of Tübingen20121.13671.07641.05601.01261.00141.01121.13671.07641.0560
Uppsala University20161.14251.07341.06441.04451.02641.01771.13571.06671.0647
Karlsruhe Institute of Tec.20131.15541.15970.99630.98481.03290.95341.13251.14550.9886
Lund University20161.13671.09011.04281.03571.01901.01651.13201.06621.0617
University of Leeds20161.13771.11611.01941.05761.02331.03351.12741.09091.0334
University of Würzburg20131.12441.14980.97790.97040.96641.00411.12441.14980.9779
University of East Anglia20121.09771.11410.98530.97970.99340.98621.12031.10981.0095
VU University Amsterdam20161.18101.12201.05261.09101.07191.01781.11981.06431.0522
Erasmus Univ. Rotterdam20131.18991.10131.08050.98141.00000.98141.11641.00001.1164
University of Bonn20121.11561.07751.03531.00510.98551.01981.11561.07751.0353
University of Glasgow20121.11431.07411.03741.03451.00451.02991.11431.07411.0374
Karolinska Institute20121.07191.00001.07191.02391.00001.02391.10801.00001.1080
LMU Munich20121.10731.07571.02941.03411.02281.01111.10731.07571.0294
Stockholm University20140.95350.98320.96971.05291.02501.02720.95350.98320.9697
Uppsala University20130.97080.94351.02890.97450.97111.00350.95330.95550.9976
Bielefeld University20140.95290.99260.96001.05471.02501.02900.95290.99260.9600
Humboldt University of Berlin20140.89341.00000.89341.01251.00001.01250.95271.00000.9527
KTH Royal Institute of Tec.20160.93961.00000.93961.06891.06671.00200.95251.00000.9525
Lund University20140.92170.95560.96451.01080.97901.03250.94670.96430.9817
Durham University20140.94531.00000.94531.00450.92661.08410.94651.00000.9465
University of Glasgow20130.94110.95740.98300.96900.96171.00760.94190.95760.9836
University of Bergen20140.94280.94640.99611.04531.01681.02800.94130.94710.9939
University College Dublin20130.92580.96430.96011.06061.08030.98180.94110.98090.9594
University of York20160.93590.91271.02551.08381.08550.99850.93840.91841.0218
Aarhus University20140.93720.94620.99051.08141.04801.03180.93740.94670.9902
King’s College London20130.96300.96261.00050.98860.99700.99160.93660.96210.9736
University of Nottingham20130.98040.99310.98720.98720.99190.99530.93590.98090.9541
Lancaster University20120.94670.98360.96241.01330.98021.03390.93580.96120.9736
Stockholm University20160.92880.88811.04581.05001.03991.00970.92880.88811.0458
Bielefeld University20150.92470.92021.00501.02501.04080.98490.92470.92021.0050
Humboldt University of Berlin20130.96921.00000.96920.89131.00000.89130.92151.00000.9215
University of Konstanz20120.91181.00000.91180.92761.00000.92760.91181.00000.9118
University of Bristol20130.95370.97520.97800.84620.87990.96180.91030.95310.9551
University of Göttingen20120.90910.87531.03871.24551.18751.04880.90950.87531.0391
University of Dundee20130.91910.94000.97780.84340.86120.97940.90900.95210.9547
Tec. University of Munich20120.91050.88751.02591.05391.02351.02970.90770.88711.0233
Tec. University of Denmark20160.87860.87361.00571.06291.04111.02090.89180.88201.0112
University of St Andrews20140.97561.00000.97560.90391.00000.90390.88991.00000.8899
University of York20120.88350.89380.98851.00800.96531.04420.88900.89120.9974
University of Sussex20120.88211.00000.88211.03051.00001.03050.88211.00000.8821
Eindhoven University of Tec.20160.85490.84951.00631.12591.11591.00900.86700.86331.0042
Bielefeld University20120.80610.82520.97681.00550.99161.01400.80610.82520.9768
University of Göttingen20160.79600.77131.03211.03041.00851.02170.79600.77131.0321
Table 7. Arithmetic Mean Malmquist Index Values 2011–2016 per Institution.
Table 7. Arithmetic Mean Malmquist Index Values 2011–2016 per Institution.
Arithm. Mean per UniversityRUN I
2012–2016Malmquist IndexCatch-UpFrontier ShiftMalmquist IndexCatch-UpFrontier ShiftMalmquist Index *Catch-Up *Frontier Shift *
RWTH Aachen University1.07481.07331.00091.05541.02041.03411.07481.07331.0009
University of Copenhagen1.09761.07851.01581.06891.04491.02311.06811.04941.0173
University of Tübingen1.06731.05151.01511.01320.99741.01591.06671.05151.0146
University of Groningen1.06561.04441.01961.04781.02441.02291.06621.04841.0181
University of Exeter1.06571.05281.01241.06411.05421.01131.06451.05351.0107
University of Konstanz1.06511.00001.06510.99211.00000.99211.06321.00001.0632
Karlsruhe Institute of Tec.1.06521.06630.99891.07461.06671.00731.06321.06480.9985
LMU Munich1.05661.04651.00931.01790.98431.03441.05661.04651.0093
University of Bonn1.05621.04861.00761.02471.00641.01811.05621.04861.0076
Aarhus University1.05601.05461.00011.06171.04641.01511.05611.05461.0002
University of Würzburg1.05291.04721.00671.00370.98431.01971.05291.04721.0067
University of Liverpool1.05351.04771.00551.01560.99711.01851.05281.04671.0057
LSE London1.05421.00001.05421.02351.20680.91201.05241.00001.0524
University of Twente1.05651.03751.02021.02691.02221.00471.05091.03331.0198
University of Amsterdam1.07561.06911.00591.04701.02531.02121.04881.02571.0227
University of Freiburg1.04761.03831.00851.03301.01251.02021.04751.03831.0084
Heidelberg University1.04721.03581.01051.03681.00971.02701.04731.03581.0105
University of Leeds1.05351.05151.00211.02201.00181.01991.04711.04321.0035
Humboldt Univ. of Berlin1.04831.01091.03740.98931.00000.98931.03821.00001.0382
VU University Amsterdam1.03851.02081.01721.04561.02821.01721.03481.00821.0262
Tec. University of Munich1.03471.02971.00471.05361.01581.03721.03411.02961.0042
University of East Anglia1.03571.04150.99520.97320.96301.01121.03091.03510.9960
Lund University1.03381.02821.00451.00430.98421.02061.03031.01141.0182
University of Oxford1.00521.00001.00521.05321.00001.05321.03021.00001.0302
Utrecht University1.09651.08071.01341.02981.00501.02471.02801.00501.0229
University of Bergen1.02741.01461.01241.03991.02531.01431.02671.01431.0119
University of Sheffield1.03541.03371.00160.98740.97141.01641.02661.01881.0075
University of Southampton1.02621.02171.00451.02441.00521.01911.02661.02061.0058
Goethe University Frankfurt1.02591.01931.00631.02531.00671.01861.02611.01941.0064
University of Glasgow1.02521.01191.01321.01860.99941.01931.02561.01201.0136
Erasmus Univ. Rotterdam1.06321.02341.03801.01621.00001.01621.02521.00001.0252
King’s College London1.03131.02021.01101.05951.03611.02301.02501.01341.0110
University College London1.01201.00241.00951.02861.00001.02861.02461.00001.0246
University of Edinburgh1.02341.01401.00861.02261.00051.02211.02381.01401.0090
Wageningen University & R.1.01631.00001.01631.03901.03141.00761.02361.00001.0236
Karolinska Institute1.03171.00001.03111.01170.99861.01341.02361.00001.0236
University of Lausanne1.02211.01351.00831.00870.98841.02111.01981.01251.0069
Stockholm University1.01961.01301.00741.05551.03971.01541.01961.01301.0074
University of Cambridge1.00121.00001.00121.02531.00001.02531.01961.00001.0196
University of Birmingham1.02361.02760.99591.00180.98401.01821.01941.01371.0054
University of Basel1.01851.00321.01531.05711.03951.01741.01841.00281.0156
University of Manchester1.02211.01681.00511.00380.98631.01791.01701.00741.0092
University of Nottingham1.01501.01650.99821.02771.01131.01611.01631.00831.0075
University of Helsinki1.01751.01161.00621.01991.00661.01361.01551.00231.0142
Queen Mary U. of London1.01491.00091.01431.08741.07691.00911.01521.00211.0134
Bielefeld University1.01430.98951.02081.03701.04570.99351.01400.98951.0205
KU Leuven1.01511.00261.01240.99880.98481.01461.01311.00001.0131
University of Dundee1.01651.00551.01120.97290.95281.01981.01270.99871.0141
Uppsala University1.02501.00601.01881.00530.98701.01851.01270.99421.0179
Swedish University of A. S.1.01111.00001.01111.05571.04221.01331.01201.00001.0120
ETH Lausanne1.00951.00001.00951.10341.07991.02241.01141.00001.0114
Delft University of Tec.1.00911.00001.00911.06031.03481.02561.00971.00001.0097
Leiden University1.00400.98221.02241.03441.01931.01351.00790.98911.0192
University of Geneva1.00661.00870.99801.02430.99781.02661.00731.00401.0032
University of Bristol1.02031.01681.00300.96410.95831.00591.00670.99541.0109
University of Aberdeen1.01201.00571.00680.96810.95331.01621.00661.00051.0063
Newcastle University1.02061.01141.00991.00361.00001.00361.00241.00001.0025
University of St Andrews1.00921.00001.00920.98631.00000.98631.00191.00001.0019
University College Dublin0.99310.98411.00911.05701.04461.01250.99950.99041.0092
Durham University0.99840.98751.01160.99770.98631.01350.99930.99381.0058
Ghent University0.99430.98001.01471.02341.01091.01250.99840.98841.0101
Trinity College Dublin0.98680.98061.00681.08071.05471.02580.99470.99281.0021
Imperial College London0.98021.00000.98021.01401.00001.01400.99301.00000.9930
University of York0.99200.99201.00120.97760.96321.01520.99180.99271.0005
KTH Royal Institute of Tec.0.98821.00000.98821.02801.00731.02090.99091.00000.9909
Tec. University of Denmark0.98780.97731.01061.03601.03031.00630.99060.97921.0117
Lancaster University0.99211.01560.97670.94290.98260.95800.98861.01080.9778
University of Sussex0.97081.00000.97080.96690.98580.98010.97781.00000.9778
Eindhoven Univ. of Tec.0.97240.96991.00261.06081.05651.00460.97570.97271.0032
University of Göttingen0.95960.95391.00871.01700.98941.02820.95940.95391.0085
* Top three values in Italics.
Table 8. Malmquist Index Annual Average Values, Variances, and Standard Deviations (Run III).
Table 8. Malmquist Index Annual Average Values, Variances, and Standard Deviations (Run III).
MeasureYearMalmquist IndexCatchupFrontier Shift
Arithmetic Mean20121.03141.00521.0260
Standard Deviation20120.06860.05540.0364

© 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (
Educ. Sci. EISSN 2227-7102 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top