The Index Number Problem with DEA: Insights from European University Efficiency Data

Abstract

An increasing effort has been put into dealing with the question of time-series analysis regarding institutional efficiency, including in the area of higher education. Universities are important institutions for economies and societies and are expected to provide excellence as well as efficiency in their processes and outputs. This is reflected in the context of an increased global competitive environment by more refined international university rankings. Combining the two areas, this paper points towards a methodological challenge in comparing different ranking datasets for their use in a data envelopment analysis (DEA) Malmquist index time-series efficiency analysis, namely, index-based data compared to additive data. The problem is discussed in a theoretical framework and complemented with an empirical application: calculations for 70 European universities with budget and staff input data and different ranking output data for the timeframe of 2011–2016 show that there is no evidence for a specific index data problem. Important implications regarding university management and higher education policies are outlined. Efficiency improvements among the analyzed universities are significant but also unevenly distributed and not easy to obtain for individual institutions.

1. Introduction

University institutions play an important role in economic development, innovation, and internationalization, e.g., through their objectives of research, teaching, and third mission, and therefore for societies at large. Steering resources within university systems, as done by higher education politicians regarding public budgets, by university managers within the institutions themselves as well as by stakeholders, such as corporations as research partners, and students as study program participants, is an important task within the economic and management domain. To fulfil this task effectively, deciders have to rely on information regarding the performance of universities, recently, for example, provided by a growing number of national and international university rankings [1,2,3,4]. Rankings have evolved regarding their principal setup, incorporating criticism addressing indicators, institutional inclusion, and data quality, including the discourse on journal publication and the individual researcher level [5,6,7,8]. This also went hand in hand with an increased influence on policies and resource decisions in higher education [9,10,11,12]. Regarding the analysis and use of ranking data as well as for higher education efficiency analysis in general, increasing emphasis is put on the question of dynamic time-series developments. Specific calculation methods, such as data envelopment analysis (DEA) window analysis as well as DEA Malmquist index, are employed for such questions. As Parteka and Wolszczak-Delacz [13] (p. 68) outline, this may overcome shortcomings of former analysis perspectives which had focused mainly on static efficiency analysis results [14,15,16,17,18,19,20,21,22,23]. The underlying technique for efficiency measurement is the DEA introduced by Charnes, Cooper, and Rhodes [24] in the basic form with constant returns to scale (CRS) and extended by Banker, Charnes, and Cooper [25] with variable returns to scale (VRS). DEA applications within the higher education sector as a typical multi-input and multioutput production environment are numerous [26,27,28,29,30,31,32,33]. In addition, the Malmquist index for analysing longitudinal developments over time [34,35,36] has also been applied for universities, e.g., for the Philippines [37] and Australia [38].

This paper addresses the research question of if university ranking data is applicable for longitudinal efficiency analysis endeavors. The specific methodological question therein is, if ranking index data can be used. This is motivated by the potential problem that an increased university input volume could not be met by an increased university output volume if index numbers are used.

This is of importance, as many rankings, as for example the Times Higher Education (THE) ranking as well as the ARWU Shanghai ranking, use index numbers for comparing the performance of universities for any yearly publication [39,40]. In these cases, all performance and evaluation measures are indexed for a maximum value of 100. If such an index number problem would exist for dynamic efficiency analysis, it would restrict the analysis potential of using university ranking data as one of the largest and most comprehensive international datasets. To test for this specific problem, a DEA Malmquist index calculation is applied with three different datasets from (a) the THE ranking with index numbers; (b) the CWTS Leiden ranking without index numbers; as well as (c) the combined case with data from both rankings systems as output indicators. This methodological management science question may also be applied to other industries. For higher education, it is connected to the research discussion of university rankings being a “zero sum game”, as rankings depict only relative positions of institutions among themselves, not the overall (e.g., quality, productivity, excellence) development of the higher education sector (see for example [41] (pp. 195–196) or [42] (p. 45)). Additionally, many researchers also connect this question to the presumably necessary increasing input volumes (budget, staff, further tight resources) in order to stay in the same positions within university rankings, such as, for example, Hazelkorn argues [43] (p. 71). For efficiency analysis matters and methodology, this is connected to the question of industry or structural efficiency as introduced by Farrell [44] (p. 262), [45] (p. 165).

The remainder of this paper is structured as follows: Section 2 describes the characteristics of university rankings and their data, especially for the applied systems of the THE and CWTS university rankings. Section 3 provides the methodological background regarding DEA and the Malmquist index as well as a short insight into index number theory. Section 4 presents the calculation results for the ranking datasets. Section 5 lays down some discussion points before Section 6 closes with conclusions and possible further research questions.

2. Ranking Systems and Research Data

University rankings have been established as a part of the higher education information environment for stakeholders, such as students, university managers, corporations, as well as politicians. However, they are also informative for researchers interested in the performance and international or national comparison of university institutions [46,47,48,49,50]. The following two ranking systems have been used in order to gather longitudinal output data for several universities [51,52,53]:

(A) The Times Higher Education (THE) ranking is one of the most long-standing and acknowledged international university rankings, established in 2004. This ranking has incorporated several changes due to feedback and criticism during the last decade. The THE ranking establishes five evaluation areas, all individually indexed for a maximum of 100. For the 2015/16 ranking, for example, the leading California Institute of Technology received the evaluation results of 95.6 for teaching, 64.0 for international outlook, 97.6 for research, 99.8 for citations, and 97.8 for industry income. This altogether provided the total ranking evaluation of 95.2. Though there have been changes and adaptions in the underlying 13 evaluation indicators, the basic setup of this structure has been continued since the 2011 ranking. Therefore, data can be used in this timeline (2011–2016) in a sensible way. In a detailed breakdown, the 13 indicators are explained as follows in Table 1.

Table 1. Indicator Weights and Definitions for the THE Ranking [54].

Output Field	Weight	Indicator	Weight
THE Teaching *	30.00%	Academic reputation survey (THE)	15.00%
		Doctorates awarded-to-academic staff ratio **	6.00%
		Staff-to-student ratio	4.50%
		Doctorate-to-bachelor’s ratio	2.25%
		Institutional income ***	2.25%
THE International Outlook *	7.50%	International-to-domestic-student ratio	2.50%
		International-to-domestic-staff ratio	2.50%
		International collaboration (proportion of research journal publications with at least one international co-author) **	2.50%
THE Research *	30.00%	Academic reputation survey (THE)	18.00%
		Research income	6.00%
		Research productivity (publications in Scopus indexed academic journals per scholar) **	6.00%
THE Citations *	30.00%	Number of times a university’s published work is cited by scholars globally, compared with the number of citations a publication of similar type and subject is expected to have. (Bibliometric data supplier Elsevier examined more than 51 million citations to 11.3 million journal articles, published over five years. The data are drawn from the 23,000 academic journals indexed by Scopus and include all indexed journals published between 2010 and 2014. Only three types of publications are analysed: journal articles, conference proceedings and reviews—citations to these papers from 2010 to 2015 are collected.)	30.00%
THE Industry Income *	2.50%	Research income an institution earns from industry ***	2.50%

* Indexed value, maximum data value of 100.00. ** Discipline normalised. *** Scaled against staff numbers and normalised for purchasing-power parity.

The following thresholds and inclusion criteria are employed by THE, which play an important role in the question of which institutions are listed and which not: Universities are excluded from the ranking if they do not teach undergraduates or if their research output averaged fewer than 200 journal articles per year over the five-year period 2010–2014. In exceptional cases, institutions below the 200-paper threshold are included if they have a particular focus on disciplines with generally low-publication volumes. There are significant and elaborate processes in place regarding data gathering, also including a defined error management approach. This is connected to the “Berlin Principles on Rankings of Higher Education Institutions” (see [55] (pp. 51–53) and [56] (pp. 80–86)). Institutions provide and sign off their institutional data for use in the THE ranking. On the rare occasions when a particular data point is not provided, a low estimate between the average value and the lowest value reported by all institutions is entered (25th percentile of all data values). In addition, a standardization approach for each indicator is used based on the distribution of data within a particular indicator—a cumulative probability function using a version of Z-scoring (see [57] (pp. 91–93) and [58]). Within the applied dataset from THE, not all out of the five indicator values reached a maximum of 100 among the selected 70 European universities, as in some cases the 100 maximum value was attained by a non-European university (Australia, Canada, China, the United States, etc.).

(B) The CWTS Leiden ranking is seen as one of the international rankings featuring the highest quality standards, especially because of the high impact of research, publication, and citation data included [53]. In this analysis, size-independent data is used in order to maintain comparability. In addition, due to input data such as budget size, institutional size is already incorporated. Data has been available for this ranking since 2011 [59]. CWTS data is based on bibliometric statistics from the Web of Science (Thomson Reuters), where the universities in the 2016 edition of the Leiden Ranking are ranked according to their percentage of highly cited publications. A publication therein is considered highly cited if it belongs to the top 1%, 10%, or 50% most cited publications in its field as explained by [60]. This focused basis on publications and citations from one large database is a strength (in terms of comparability and data quality) and also a weakness, e.g., regarding disciplinary bias or quality evaluation (see [11] (pp. 13–14)). Compared to THE ranking data, for CWTS data, there is no index value used but additive data with no upper bound (e.g., citations numbers). From the CWTS dataset, P (publications, with partial share points for coauthors), TCS and TCNS (total citations and total citations normalized), as well as P_top1 and P_top50 (number of publications among the top 1% or 50% most frequently cited) are selected.

For input data gathering, the European ETER project was used, which provides large datasets (among others: budgets, staff, students, graduates, etc.) for the years 2011, 2012, and 2013 in the current version, accessible via the Internet [61]. Concurrent with the time series in the THE ranking datasets, input data total budget and total academic staff was used, connecting the input year 2011 with the output (ranking) data of 2011 and 2012, the 2012 input year with the output data of 2013 and 2014, as well as the 2013 input year with the output data of 2015 and 2016.

University selection for this analysis was established regarding the principle of selecting the European institutions from the 2011 THE ranking, which featured the top 200 universities worldwide; that amounted to 81 universities. Furthermore, 11 institutions had to be excluded due to missing data, some due to missing or inconsistent ranking data (THE or CWTS), the majority due to missing input budget or staff data (ETER), leaving 70 institutions. Data as described in Table 2 for 2 example universities has been used for all 70 universities. In this case, the research question can be highlighted: as the University of Oxford (UK) has increased budget and staff input for 2011–2016 by 20.54% and 13.87%, respectively, output numbers have risen in the indexed THE ranking by 5.80% on average, whereas in the nonindexed CWTS ranking, output has risen by 34.33%. On the other hand, for the University of Würzburg (Germany), numbers are different. With a budget increase by 2.06% and a staff increase by 4.24%, the output rise amounted to 17.51% with THE, but only to 14.08% with CWTS.

Table 2. Calculation Data Cut-out for Two Example Universities, Sources: [39,40,61].

U. Oxford (UK)	2011	2016	Change 2011–2016	Arith. Mean **	U. Würzburg (DE)	2011	2016	Change 2011–2016	Arith. Mean **
THE Teaching *	88.20	86.50	−1.93%			48.70	34.60	−28.95%
THE Int. Outlook *	77.20	94.40	22.28%			40.30	50.90	26.30%
THE Research *	93.90	98.90	5.32%			40.90	35.80	−12.47%
THE Citations *	95.10	98.80	3.89%			60.40	79.10	30.96%
THE Industry Income *	73.50	73.10	−0.54%	5.80%		27.90	47.90	71.68%	17.51%
CWTS_P	10,701.00	13,300.00	24.29%			3219.00	3349.00	4.04%
CWTS_TCS	89,149.00	127,888.00	43.45%			22,932.00	26,866.00	17.16%
CWTS_TNCS	15,464.00	20,373.00	31.74%			3748.00	3998.00	6.67%
CWTS_P_top1	215.00	311.00	44.65%			36.00	51.00	41.67%
CWTS_P_top50	6593.00	8408.00	27.53%	34.33%		1883.00	1899.00	0.85%	14.08%
Budget (Mil. €, Input)	1119.77	1349.76	20.54%		Budget (Mil. €, Input)	806.97	823.58	2.06%
Academic Staff (Input)	5375.00	6120.00	13.86%	17.20%	Academic Staff (Input)	3281.00	3420.00	4.24%	3.15%

* Indexed numbers; ** Arithmetic Mean of all changes for THE (indexed), CWTS (non-indexed) and input group data per university in italics.

Figure 1 highlights the input–output scheme for the analysis, in this case regarding the THE indicators as outputs. Some example correlations are included, especially the input indicator correlation between budget and staff of r = 0.86.

Figure 1. Input and Output Indicators and Correlations.

Further correlations between input and output indicators assessed from the whole dataset of 420 units (70 universities and 6 years of data) are outlined in Table 3. Interestingly, high correlation levels are not only obvious between the two input indicators budget and academic staff, but also among output indicators, mainly within the two used ranking systems, for example, between the THE indicators “Teaching” and “Research” (0.890), calling in mind the “Humboldt principle” regarding the unity of teaching and research [62] (p. 274), [63,64], and also among the CWTS indicators publications (P) and citations (TCS, 0.963, and TNCS, 0.980), which is obvious given the fact that the same database is used and publications are a requirement for receiving citations. The same holds true for the correlation of publications with being among the top 1% or top 50% of cited publications (0.921 and 0.995, respectively). Therefore, it can be concluded that especially the CWTS indicators are highly correlated as they stem from the same database and are all connected to initial publications.

Table 3. Total Input and Output Indicator Correlations (n = 420).

n = 420	Budget	Acad. Staff	THE Teaching	THE Int. Outlook	THE Research	THE Citations	THE Ind.Inc.	CWTS P	CWTS TCS	CWTS TNCS	CWTS P_top1	CWTS P_top50
Budget	1.000	0.860	0.533	−0.084	0.386	0.265	0.077	0.678	0.683	0.654	0.594	0.665
Acad. Staff		1.000	0.490	−0.130	0.378	0.172	0.186	0.681	0.631	0.639	0.579	0.656
THE Teaching			1.000	0.228	0.890	0.231	0.213	0.690	0.724	0.733	0.746	0.711
THE Internat. Outlook				1.000	0.173	0.375	−0.142	0.053	0.149	0.138	0.240	0.092
THE Research					1.000	0.198	0.273	0.700	0.709	0.733	0.734	0.718
THE Citations						1.000	−0.252	0.296	0.423	0.368	0.408	0.334
THE Industry Income							1.000	0.167	0.128	0.178	0.173	0.174
CWTS_P								1.000	0.963	0.980	0.921	0.995
CWTS_TCS									1.000	0.985	0.959	0.978
CWTS_TNCS										1.000	0.974	0.994
CWTS_P_top1											1.000	0.949
CWTS_P_top50												1.000
“Loading” (ari. mean r)	0.483	0.468	0.563	0.099	0.536	0.256	0.107	0.648	0.666	0.671	0.662	0.661

Furthermore, high correlations between the two rankings datasets can also be observed: THE indicators “Teaching” and “Research” feature high correlation levels with all CWTS (publication-based) indicators. For “Research”, this is not so much surprising, for “Teaching”, it surely is.

Finally, it can also be of interest to look into low levels of correlation. The THE indicators “International Outlook” as well as “Industry Income” show only weak to nonexistent correlations with the other indicators within THE as well as in CWTS. This can support the hypothesis that these fields are fairly independent and should be covered by separate indicators (as THE argues, for example). It also can be seen as proof for a supposition that these areas do not really belong to the core of academic and university objectives. At least, the negative correlation between citations and industry income can be seen in such a light of “estrangement” between academe and the corporate world. At least, it can be understood from an individual researcher’s perspective, who often enough faces the nontrivial trade-off between time invested in topics interesting from an academic perspective and derived publications on the one hand, and industry-affine questions with connected projects, income, and publications for those topics.

3. Research Methodology and Index Numbers

The data envelopment analysis (DEA) method is based on works of Koopmans regarding the activity analysis concept [65], Debreu [66] and Farrell in terms of the radial efficiency measure [44], as well as the works of Diewert [67]. This led to the specific DEA method suggestion by Charnes, Cooper, and Rhodes in 1978 [24]. Reasons for the increasing use of this efficiency analysis technique in higher education research are the fact that no a priori knowledge about a production function is required, only real-life data is used, and a multitude of inputs can be combined with a multitude of outputs, which is very typical for universities as “multi-product-organisations” [68,69,70,71,72,73].

DEA studies decision making units (DMU), which can be seen as the entities responsible for input, throughput, and output decision making [74] (p. 22). DMU such as, e.g., university institutions, departments, schools, or institutes and research groups, can be evaluated and compared, showing a specific level of decision-making success in terms of overall efficiency. DEA uses a nonparametric mathematical programming approach for the evaluation of DMU efficiency relative to each other. Further, it is assumed that there are several DMU and it is supposed that inputs and outputs comply with these requirements:

▪

For each input and output, there are numerical, positive data for all DMU.

▪

Selected values (inputs, outputs, and the chosen DMU) should depict the interest of decision-makers towards the relative efficiency evaluations.

▪

DMU are homogenous in terms of identical inputs and outputs.

▪

Input and output indicator units and scales are congruent.

Furthermore, two different models can be distinguished. The CCR model, named after the authors Charnes, Cooper, and Rhodes [24], with constant returns to scale, and the BCC model with variable returns to scale [25]. For both models and their efficiency measurement, the following specifications are made [75] (p. 239):

n   the number of DMU to be evaluated

DMU_j the jth DMU

m   the number of inputs to each DMU

s   the number of outputs to each DMU

x_ij   amount of the ith input consumed by DMU j

y_kj   amount of the kth output produced by DMU j

eff   abbreviation for efficiency

v_i   the weight assigned to the ith input

u_k   the weight assigned to the kth output.

$${\mathrm{eff}\mathrm{DMU}}_{\mathrm{jo}}=\frac{{{\displaystyle \sum }}_{\mathrm{k}=1}^{\mathrm{s}}{\mathsf{\mu }}_{\mathrm{k}}{\mathrm{y}}_{\mathrm{kjo}}}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{v}}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{ijo}}}$$

(1)

A basic characteristic of the CCR model is the reduction of a multioutput and multi-input setting to a single (weighted) input and output combination for each DMU. For a certain DMU, measuring its efficiency and comparison with other DMU in the system is enabled. Usually executed by a series of linear programming formulations, DMU performance comparison facilitates a ranking of the different analyzed DMU and scales their relative efficiency from low to high, whereby the latter is defined as efficient. The CCR model contains both mathematical maximization and minimization problems. Detecting DMU relative technical efficiency requires on the one hand detection of each DMU technical efficiency, and on the other hand, the comparison of all DMU efficiencies. These steps are executed in the DEA simultaneous arithmetic operation. The calculation of the DMU efficiency value results from the consideration of the weighted inputs and weighted outputs. With the help of quantified inputs and quantified outputs, DEA generates via a quotient one single efficiency ratio for each DMU. The weighting factors are endogenously determined and allow the pooling of heterogeneous inputs and outputs with different units of measurement in one efficiency ratio. Hereby, each DMU’s weights are considered optimally in order to maximize the efficiency value and determine only the definitely provable inefficiency. With the help of the following figure, the different scales of CCR and BCC are depicted in the case of a single input and single output situation. In Figure 2, H illustrates a scale-efficient DMU (on the border production function). Inspection of DMU K reveals that the distance XJ/XK stands for possible input savings regarding a decline of technical inefficiency, whereas XK/YL represents possible output enlargements regarding the decline of technical inefficiency. The distance XI/XK stands for the gross-scale efficiency and XI/XJ shows the pure-scale efficiency with a corrected input. YL/YM stands for the pure-scale efficiency with corrected output (in case of variable scales).

Figure 2. DEA Frontier Graph for CCR and BBC Models.

Calculating DMU efficiency, it can be observed that DMU H has the highest efficiency value. Building the border production function (“envelopment function”) under the assumption of constant returns to scale (CCR) therefore complies with a line through the origin. This function with the assumption of variable returns to scale (BCC) is built by combining the points J, H, and L. As this is the case, the area of production opportunities is limited by the set of convex combinations of realized productions belonging to the border production function. Therefore, in the case of variable returns to scale, DMU J, H, and L are efficient, whereas DMU K is inefficient. In a CCR model, usually only one DMU is found to be efficient, whereas in a BCC model several, DMU are expected to be efficient.

Based on early works of Malmquist [34], Caves, Christensen, and Diewert [12] proposed a calculation of a productivity index in order to shed light on efficiency changes over time. This longitudinal perspective is promising for the DEA method, as most efficiency measurement approaches are directed towards the question of efficiency improvement. To provide usable information for this objective, the index is therefore distinguishing between a technological progress for the whole set of DMU, as, for example, universities—individually adapted for each institution—on the one side, and the technological efficiency on the other side [35]. This second technological efficiency is caused by the organizational and process setup of an institution. The following formula depicts the mathematical distance function algorithm used for the Malmquist index [36,38]:

$${\mathrm{M}}_{\mathrm{O},\mathrm{CRS}}\left({\mathrm{x}}_{\mathrm{t}},{\mathrm{y}}_{\mathrm{t}};{\mathrm{x}}_{\mathrm{t}+1},{\mathrm{y}}_{\mathrm{t}+1}\right)=\frac{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}+1}\left({\mathrm{x}}_{\mathrm{t}+1,}{\mathrm{y}}_{\mathrm{t}+1}\right)}{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}},{\mathrm{y}}_{\mathrm{t}}\right)}*{\left[\frac{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}},{\mathrm{y}}_{\mathrm{t}}\right)}{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}+1}\left({\mathrm{x}}_{\mathrm{t}},{\mathrm{y}}_{\mathrm{t}}\right)}*\frac{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}+1},{\mathrm{y}}_{\mathrm{t}+1}\right)}{{\mathrm{D}}_{\mathrm{O},\mathrm{CRS}}^{\mathrm{t}+1}\left({\mathrm{x}}_{\mathrm{t}+1},{\mathrm{y}}_{\mathrm{t}+1}\right)}\right]}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$$

(2)

Regarding the use of index numbers, the basic discussion can be divided into three areas of economic application and discussion: (i) Based on mathematical theory, the first and still foremost application area for index numbers was the question of price and monetary value development over time, as, for example, outlined by Fisher (1911, 1922) [76,77], as well as Divisia [78]. This “basket index” field is still discussed and amended today as an intertemporal application, e.g., with questions regarding testing and quality evaluation of price indices [79,80]; (ii) The second area of index number application in economics is further intertemporal comparisons of organizational or industry quantities such as indexed economic cycle growth and development indices or forecasts [81]; (iii) The third area applies the use of index numbers for an interorganizational or geographical comparison of economic quantities such as production outputs between different corporations or commodity prices between different trading locations and stock markets [82,83]. This third application area has to be connected to the index number application in question here, which is the interorganizational comparison of university performance measures within ranking systems. Many problems have been identified in connection with the use of index numbers starting with, e.g., [84]. Mainly quality and stability problems allocated to the basket or definition problems are on record [85,86], but also problems of numerical comparison due to index number calculation are expressed [87,88].

4. Results

The following results have been obtained in a Malmquist index calculation for the timeframe 2011–2016 regarding the 70 selected European universities. The calculation was conducted with the software package BANXIA Frontier Analyst with an output maximization mode in a BCC model with variable returns to scale.

Three calculation runs are reported, all with the same input data (budget and staff data). (I) First, inputs were combined with five output indicators from the THE ranking (all indexed values); (II) Second, two input indicators were combined with five selected output indicators from the CWTS Leiden ranking (all nonindexed values); (III) Third, the inputs were combined in a DEA Malmquist calculation with 10 output indicators (five from THE as indexed values, five from CWTS ranking results as nonindexed values). No superefficiencies were calculated for these DEA runs, therefore all efficient units are showing a maximum efficiency score of 100.00. Table 4 presents the base efficiencies calculated for the initial year 2011 for all 70 universities.

Table 4. University Efficiency Scores and Returns to Scale 2011 (Base Year).

University	Run I—Eff. Score	Returns to Scale	Run II—Eff. Score	Returns to Scale	Run III—Eff. Score	Returns to Scale
Aarhus University	66.00%	Decrease	45.00%	Decrease	66.00%	Decrease
Bielefeld University	83.10%	Decrease	32.30%	Increase	83.10%	Decrease
Delft University of Technology	100.00%	Constant	54.00%	Decrease	100.00%	Constant
Durham University	99.20%	Decrease	66.70%	Increase	99.30%	Decrease
ETH Lausanne	100.00%	Constant	54.10%	Decrease	100.00%	Constant
Eindhoven University of Technology	100.00%	Constant	45.50%	Decrease	100.00%	Constant
Erasmus University Rotterdam	89.40%	Decrease	100.00%	Constant	100.00%	Constant
Ghent University	97.80%	Decrease	71.40%	Decrease	99.10%	Decrease
Goethe University Frankfurt	74.40%	Decrease	40.10%	Decrease	74.40%	Decrease
Heidelberg University	76.30%	Decrease	53.90%	Decrease	76.30%	Decrease
Humboldt University of Berlin	94.90%	Decrease	100.00%	Constant	100.00%	Constant
Imperial College London	100.00%	Constant	100.00%	Constant	100.00%	Constant
KTH Royal Institute of Technology	100.00%	Constant	58.70%	Increase	100.00%	Constant
KU Leuven	98.70%	Decrease	76.70%	Decrease	100.00%	Constant
Karlsruhe Institute of Technology	73.40%	Decrease	43.90%	Decrease	73.70%	Decrease
Karolinska Institute	100.00%	Constant	100.00%	Constant	100.00%	Constant
King’s College London	89.00%	Decrease	68.00%	Decrease	91.90%	Decrease
LMU Munich	80.30%	Decrease	59.30%	Decrease	80.30%	Decrease
LSE London	100.00%	Constant	47.20%	Increase	100.00%	Constant
Lancaster University	92.30%	Decrease	85.80%	Increase	94.70%	Decrease
Leiden University	100.00%	Constant	82.70%	Decrease	100.00%	Constant
Lund University	76.50%	Decrease	83.20%	Decrease	84.70%	Decrease
Newcastle University	90.00%	Decrease	100.00%	Constant	100.00%	Constant
Queen Mary University of London	98.50%	Decrease	33.60%	Increase	98.50%	Decrease
RWTH Aachen University	69.30%	Decrease	35.00%	Decrease	69.30%	Decrease
Stockholm University	82.50%	Decrease	36.70%	Decrease	82.50%	Decrease
Swedish U. of Agricultural Sciences	100.00%	Constant	38.50%	Increase	100.00%	Constant
Technical University of Denmark	98.30%	Decrease	45.20%	Decrease	98.30%	Decrease
Technical University of Munich	87.80%	Decrease	44.70%	Decrease	87.90%	Decrease
Trinity College Dublin	100.00%	Constant	65.50%	Increase	100.00%	Constant
University College Dublin	100.00%	Constant	50.60%	Increase	100.00%	Constant
University College London	97.80%	Decrease	100.00%	Constant	100.00%	Constant
University of Aberdeen	97.70%	Decrease	74.60%	Increase	100.00%	Constant
University of Amsterdam	67.00%	Decrease	88.10%	Decrease	88.10%	Decrease
University of Basel	98.40%	Decrease	49.50%	Decrease	98.60%	Decrease
University of Bergen	81.60%	Decrease	41.00%	Decrease	81.60%	Decrease
University of Birmingham	76.00%	Decrease	63.10%	Decrease	81.30%	Decrease
University of Bonn	69.50%	Decrease	39.90%	Decrease	69.50%	Decrease
University of Bristol	88.10%	Decrease	92.30%	Increase	98.30%	Decrease
University of Cambridge	100.00%	Constant	100.00%	Constant	100.00%	Constant
University of Copenhagen	61.60%	Decrease	71.90%	Decrease	71.90%	Decrease
University of Dundee	96.80%	Decrease	69.90%	Increase	100.00%	Constant
University of East Anglia	82.30%	Decrease	60.00%	Increase	84.80%	Decrease
University of Edinburgh	92.90%	Decrease	66.10%	Decrease	92.90%	Decrease
University of Exeter	77.50%	Decrease	44.70%	Increase	77.50%	Decrease
University of Freiburg	83.40%	Decrease	37.80%	Decrease	83.40%	Decrease
University of Geneva	95.80%	Decrease	54.20%	Decrease	98.00%	Decrease
University of Glasgow	81.30%	Decrease	59.40%	Decrease	81.30%	Decrease
University of Groningen	77.00%	Decrease	79.60%	Decrease	79.90%	Decrease
University of Göttingen	99.20%	Decrease	53.80%	Decrease	99.20%	Decrease
University of Helsinki	80.80%	Decrease	70.00%	Decrease	85.40%	Decrease
University of Konstanz	100.00%	Constant	100.00%	Constant	100.00%	Constant
University of Lausanne	87.20%	Decrease	54.80%	Decrease	88.10%	Decrease
University of Leeds	64.80%	Decrease	60.00%	Decrease	67.30%	Decrease
University of Liverpool	71.80%	Decrease	55.10%	Decrease	72.80%	Decrease
University of Manchester	81.90%	Decrease	83.80%	Decrease	87.00%	Decrease
University of Nottingham	76.60%	Decrease	63.50%	Decrease	82.30%	Decrease
University of Oxford	100.00%	Constant	100.00%	Constant	100.00%	Constant
University of Sheffield	71.80%	Decrease	69.70%	Decrease	79.20%	Decrease
University of Southampton	82.20%	Decrease	65.50%	Decrease	84.70%	Decrease
University of St Andrews	100.00%	Constant	100.00%	Constant	100.00%	Constant
University of Sussex	100.00%	Constant	100.00%	Constant	100.00%	Constant
University of Twente	83.60%	Decrease	58.50%	Decrease	85.50%	Decrease
University of Tübingen	64.70%	Decrease	44.70%	Decrease	64.70%	Decrease
University of Würzburg	65.70%	Decrease	36.00%	Decrease	65.70%	Decrease
University of York	94.00%	Decrease	67.00%	Increase	94.30%	Decrease
Uppsala University	82.20%	Decrease	73.30%	Decrease	87.20%	Decrease
Utrecht University	67.80%	Decrease	97.60%	Decrease	97.60%	Decrease
VU University Amsterdam	87.70%	Decrease	85.10%	Decrease	96.20%	Decrease
Wageningen University and Research	100.00%	Constant	85.10%	Increase	100.00%	Constant
Eff. DMU	17		2		24
Arithm. Mean	87.21%		66.20%		89.78%
Min	61.60%		32.30%		64.70%

The reason for applying three different calculation runs is to look into possible differences between efficiency level changes of indexed data and nonindexed ranking data in the two different cases. The third run (III) is a control case where both datasets are combined. This should enable a result regarding the question of if an index data problem is existing in using university ranking data for a DEA longitudinal efficiency analysis.

From Table 4, the following obvious recognitions from the data results have to be stated in order to support a further discussion in the next section. First, the 2011 efficiency values for the three calculation runs feature different values for the 70 universities. This can be expected, as different output indicators lead to different efficiency values in a relative efficiency measurement scheme. For some institutions, such as, e.g., Cambridge or the Karolinska Institute, identical efficiency scores are calculated, as such an institution is leading across a multitude of output indicators.

Second, several universities accomplish decreasing returns to scale or even increasing returns to scale, proving the assumption that the DEA BCC model for variable returns to scale is appropriate for university and higher education efficiency analysis settings.

Third, 17 (respectively 2 and 24) out of 70 universities are calculated as being efficient in regard to the selected two inputs and five (respectively 10 in run III) outputs. This is a comparatively high number, underlined by the average efficiency of all 70 universities of 87.21% (run I). For comparison, do Jiménez-Sáez et al. [89] (p. 235) report a mean efficiency of between 58.2% and 82.2% for Spanish research programs between 1988 and 1999. Therefore, a proposed problem with the index numbers (used in runs I and III) cannot be shown. This is supported by detailed data from Table 5 and Figure 3, indicating that there are differences in the efficiency development (measured by the DEA Malmquist index), as can be expected due to different output indicators. However, these differences do not point towards a significant structural disadvantage of index numbers (with the THE data in run I and run III). Especially, the 2012–2016 mean efficiency improvement is nearly identical (1.0266; 1.0251; 1.0240—bold numbers) for the three calculation runs.

Table 5. Mean Efficiency Improvement Scores in Three Calculation Runs I–III.

Arithm. Mean	I—MI	I—CU	I—FS	II—MI	II—CU	II—FS	III—MI	III—CU	III—FS
2012	1.0343	1.0148	1.0190	1.0368	1.0015	1.0353	1.0314	1.0052	1.0260
2013	1.0216	1.0291	0.9926	0.9984	1.0134	0.9860	1.0099	1.0221	0.9881
2014	0.9895	1.0086	0.9811	1.0350	1.0025	1.0328	1.0000	1.0087	0.9914
2015	1.0108	1.0007	1.0103	1.0174	1.0268	0.9965	1.0125	1.0040	1.0086
2016	1.0771	1.0318	1.0444	1.0377	1.0184	1.0191	1.0659	1.0222	1.0430
Total Arithm. Mean	1.0266	1.0170	1.0095	1.0251	1.0125	1.0139	1.0240	1.0124	1.0114

MI: Malmquist index; CU: Catch-up; FS: Frontier shift.

Figure 3. Mean Malmquist Index 2012–2016 for Three Datasets (Run I, II, III).

Table 6 reports some cut-outs for Malmquist index values per university in the calculated dataset for run III. From this, the following observations can be noted in order to support the discussion (Section 5). Leading institutions for annual efficiency improvements are, for example, the University of Konstanz (33.20% improvement 2016 on 2015, run III), Bielefeld University (30.06% improvement 2016 on 2015), and the Technical University of Munich (22.87% improvement 2016 on 2015). The fact that the top improvement numbers are reduced significantly in the first ranking positions as well as the fact that, for example, Bielefeld University is also listed among the bottom 30 institutions regarding annual efficiency changes (−19.39% for 2012 on 2011) hints to the assumption that data irregularities or changes in data gathering, e.g., with the ranking systems or within the universities themselves, have led to these high numbers in efficiency changes. Therefore, in the next table, the long-term average changes are also reported.

Table 6. Top 30 and Bottom 30 Annual Malmquist Index 2012–2016 (Cut-out, Sorted from Run III).

University	Year	RUN I (THE)			RUN II (CWTS)			RUN III (THE & CWTS)
		Malmquist Index	Catch-Up	Frontier Shift	Malmquist Index	Catch-Up	Frontier Shift	Malmquist Index	Catch-Up	Frontier Shift
University of Konstanz	2016	1.3417	1.0000	1.3417	1.0163	1.0000	1.0163	1.3320	1.0000	1.3320
Bielefeld University	2016	1.3023	1.0984	1.1856	1.0516	1.0312	1.0198	1.3006	1.0984	1.1841
Tec. University of Munich	2016	1.2287	1.2016	1.0225	1.0331	1.0024	1.0306	1.2287	1.2016	1.0225
Humboldt University of Berlin	2016	1.2664	1.0000	1.2664	1.0083	1.0000	1.0083	1.2122	1.0000	1.2122
RWTH Aachen University	2016	1.1986	1.1760	1.0192	1.0441	1.0129	1.0308	1.1986	1.1760	1.0192
University of Bergen	2016	1.1952	1.1739	1.0182	1.0330	1.0248	1.0080	1.1901	1.1637	1.0226
Karlsruhe Institute of Tec.	2016	1.1948	1.1865	1.0070	1.0734	1.0572	1.0153	1.1875	1.1827	1.0041
Aarhus University	2016	1.1883	1.1423	1.0403	1.0696	1.0425	1.0260	1.1870	1.1408	1.0405
University of Groningen	2013	1.2216	1.1876	1.0286	1.0497	1.0331	1.0161	1.1758	1.1930	0.9856
University of Freiburg	2016	1.1739	1.1458	1.0245	1.0250	0.9971	1.0279	1.1736	1.1458	1.0242
LMU Munich	2015	1.1686	1.1505	1.0157	1.0025	0.9567	1.0479	1.1683	1.1501	1.0158
University of Copenhagen	2012	1.3003	1.2325	1.0550	1.0775	1.0391	1.0370	1.1659	1.1227	1.0385
Stockholm University	2012	1.1557	1.1324	1.0206	1.0440	1.0191	1.0244	1.1557	1.1324	1.0206
University of Exeter	2016	1.1610	1.1233	1.0335	1.2387	1.2357	1.0025	1.1543	1.1255	1.0256
LSE London	2016	1.1503	1.0000	1.1503	1.1364	1.0000	1.1364	1.1503	1.0000	1.1503
Aarhus University	2012	1.1500	1.1335	1.0146	1.0367	0.9989	1.0378	1.1500	1.1335	1.0146
LSE London	2012	1.1377	1.0000	1.1377	1.0873	1.0371	1.0485	1.1377	1.0000	1.1377
University of Tübingen	2012	1.1367	1.0764	1.0560	1.0126	1.0014	1.0112	1.1367	1.0764	1.0560
Uppsala University	2016	1.1425	1.0734	1.0644	1.0445	1.0264	1.0177	1.1357	1.0667	1.0647
Karlsruhe Institute of Tec.	2013	1.1554	1.1597	0.9963	0.9848	1.0329	0.9534	1.1325	1.1455	0.9886
Lund University	2016	1.1367	1.0901	1.0428	1.0357	1.0190	1.0165	1.1320	1.0662	1.0617
University of Leeds	2016	1.1377	1.1161	1.0194	1.0576	1.0233	1.0335	1.1274	1.0909	1.0334
University of Würzburg	2013	1.1244	1.1498	0.9779	0.9704	0.9664	1.0041	1.1244	1.1498	0.9779
University of East Anglia	2012	1.0977	1.1141	0.9853	0.9797	0.9934	0.9862	1.1203	1.1098	1.0095
VU University Amsterdam	2016	1.1810	1.1220	1.0526	1.0910	1.0719	1.0178	1.1198	1.0643	1.0522
Erasmus Univ. Rotterdam	2013	1.1899	1.1013	1.0805	0.9814	1.0000	0.9814	1.1164	1.0000	1.1164
University of Bonn	2012	1.1156	1.0775	1.0353	1.0051	0.9855	1.0198	1.1156	1.0775	1.0353
University of Glasgow	2012	1.1143	1.0741	1.0374	1.0345	1.0045	1.0299	1.1143	1.0741	1.0374
Karolinska Institute	2012	1.0719	1.0000	1.0719	1.0239	1.0000	1.0239	1.1080	1.0000	1.1080
LMU Munich	2012	1.1073	1.0757	1.0294	1.0341	1.0228	1.0111	1.1073	1.0757	1.0294
Stockholm University	2014	0.9535	0.9832	0.9697	1.0529	1.0250	1.0272	0.9535	0.9832	0.9697
Uppsala University	2013	0.9708	0.9435	1.0289	0.9745	0.9711	1.0035	0.9533	0.9555	0.9976
Bielefeld University	2014	0.9529	0.9926	0.9600	1.0547	1.0250	1.0290	0.9529	0.9926	0.9600
Humboldt University of Berlin	2014	0.8934	1.0000	0.8934	1.0125	1.0000	1.0125	0.9527	1.0000	0.9527
KTH Royal Institute of Tec.	2016	0.9396	1.0000	0.9396	1.0689	1.0667	1.0020	0.9525	1.0000	0.9525
Lund University	2014	0.9217	0.9556	0.9645	1.0108	0.9790	1.0325	0.9467	0.9643	0.9817
Durham University	2014	0.9453	1.0000	0.9453	1.0045	0.9266	1.0841	0.9465	1.0000	0.9465
University of Glasgow	2013	0.9411	0.9574	0.9830	0.9690	0.9617	1.0076	0.9419	0.9576	0.9836
University of Bergen	2014	0.9428	0.9464	0.9961	1.0453	1.0168	1.0280	0.9413	0.9471	0.9939
University College Dublin	2013	0.9258	0.9643	0.9601	1.0606	1.0803	0.9818	0.9411	0.9809	0.9594
University of York	2016	0.9359	0.9127	1.0255	1.0838	1.0855	0.9985	0.9384	0.9184	1.0218
Aarhus University	2014	0.9372	0.9462	0.9905	1.0814	1.0480	1.0318	0.9374	0.9467	0.9902
King’s College London	2013	0.9630	0.9626	1.0005	0.9886	0.9970	0.9916	0.9366	0.9621	0.9736
University of Nottingham	2013	0.9804	0.9931	0.9872	0.9872	0.9919	0.9953	0.9359	0.9809	0.9541
Lancaster University	2012	0.9467	0.9836	0.9624	1.0133	0.9802	1.0339	0.9358	0.9612	0.9736
Stockholm University	2016	0.9288	0.8881	1.0458	1.0500	1.0399	1.0097	0.9288	0.8881	1.0458
Bielefeld University	2015	0.9247	0.9202	1.0050	1.0250	1.0408	0.9849	0.9247	0.9202	1.0050
Humboldt University of Berlin	2013	0.9692	1.0000	0.9692	0.8913	1.0000	0.8913	0.9215	1.0000	0.9215
University of Konstanz	2012	0.9118	1.0000	0.9118	0.9276	1.0000	0.9276	0.9118	1.0000	0.9118
University of Bristol	2013	0.9537	0.9752	0.9780	0.8462	0.8799	0.9618	0.9103	0.9531	0.9551
University of Göttingen	2012	0.9091	0.8753	1.0387	1.2455	1.1875	1.0488	0.9095	0.8753	1.0391
University of Dundee	2013	0.9191	0.9400	0.9778	0.8434	0.8612	0.9794	0.9090	0.9521	0.9547
Tec. University of Munich	2012	0.9105	0.8875	1.0259	1.0539	1.0235	1.0297	0.9077	0.8871	1.0233
Tec. University of Denmark	2016	0.8786	0.8736	1.0057	1.0629	1.0411	1.0209	0.8918	0.8820	1.0112
University of St Andrews	2014	0.9756	1.0000	0.9756	0.9039	1.0000	0.9039	0.8899	1.0000	0.8899
University of York	2012	0.8835	0.8938	0.9885	1.0080	0.9653	1.0442	0.8890	0.8912	0.9974
University of Sussex	2012	0.8821	1.0000	0.8821	1.0305	1.0000	1.0305	0.8821	1.0000	0.8821
Eindhoven University of Tec.	2016	0.8549	0.8495	1.0063	1.1259	1.1159	1.0090	0.8670	0.8633	1.0042
Bielefeld University	2012	0.8061	0.8252	0.9768	1.0055	0.9916	1.0140	0.8061	0.8252	0.9768
University of Göttingen	2016	0.7960	0.7713	1.0321	1.0304	1.0085	1.0217	0.7960	0.7713	1.0321

Further on, Table 7 depicts the average of all year-on-year changes in efficiency between 2011 and 2016 for all universities, ranked according to the average Malmquist index values (total average efficiency change). The following statements may be derived in a first analysis concerning run III (last three columns). The highest level of average improvement over the six-year period was attained by the German RWTH University Aachen (average 1.0748 or 7.48% annually), followed by the University of Copenhagen (Denmark), as well as the University of Tübingen (Germany). Obviously, the individual leaders of efficiency improvement in the categories of “Catch-up” (individual DMU improvement due to closing the gap to the border production frontier) and “Frontier shift” (collective improvement of the border production frontier, compare, e.g., [13] (p. 74) are different from the total efficiency improvement champions, which are the Universities of Exeter and Karlsruhe (Catch-up, besides RWTH), as well as the University of Konstanz, LSE London, and Humboldt-University of Berlin (Frontier shift).

Table 7. Arithmetic Mean Malmquist Index Values 2011–2016 per Institution.

Arithm. Mean per University	RUN I (THE)			RUN II (CWTS)			RUN III (THE & CWTS)
2012–2016	Malmquist Index	Catch-Up	Frontier Shift	Malmquist Index	Catch-Up	Frontier Shift	Malmquist Index *	Catch-Up *	Frontier Shift *
RWTH Aachen University	1.0748	1.0733	1.0009	1.0554	1.0204	1.0341	1.0748	1.0733	1.0009
University of Copenhagen	1.0976	1.0785	1.0158	1.0689	1.0449	1.0231	1.0681	1.0494	1.0173
University of Tübingen	1.0673	1.0515	1.0151	1.0132	0.9974	1.0159	1.0667	1.0515	1.0146
University of Groningen	1.0656	1.0444	1.0196	1.0478	1.0244	1.0229	1.0662	1.0484	1.0181
University of Exeter	1.0657	1.0528	1.0124	1.0641	1.0542	1.0113	1.0645	1.0535	1.0107
University of Konstanz	1.0651	1.0000	1.0651	0.9921	1.0000	0.9921	1.0632	1.0000	1.0632
Karlsruhe Institute of Tec.	1.0652	1.0663	0.9989	1.0746	1.0667	1.0073	1.0632	1.0648	0.9985
LMU Munich	1.0566	1.0465	1.0093	1.0179	0.9843	1.0344	1.0566	1.0465	1.0093
University of Bonn	1.0562	1.0486	1.0076	1.0247	1.0064	1.0181	1.0562	1.0486	1.0076
Aarhus University	1.0560	1.0546	1.0001	1.0617	1.0464	1.0151	1.0561	1.0546	1.0002
University of Würzburg	1.0529	1.0472	1.0067	1.0037	0.9843	1.0197	1.0529	1.0472	1.0067
University of Liverpool	1.0535	1.0477	1.0055	1.0156	0.9971	1.0185	1.0528	1.0467	1.0057
LSE London	1.0542	1.0000	1.0542	1.0235	1.2068	0.9120	1.0524	1.0000	1.0524
University of Twente	1.0565	1.0375	1.0202	1.0269	1.0222	1.0047	1.0509	1.0333	1.0198
University of Amsterdam	1.0756	1.0691	1.0059	1.0470	1.0253	1.0212	1.0488	1.0257	1.0227
University of Freiburg	1.0476	1.0383	1.0085	1.0330	1.0125	1.0202	1.0475	1.0383	1.0084
Heidelberg University	1.0472	1.0358	1.0105	1.0368	1.0097	1.0270	1.0473	1.0358	1.0105
University of Leeds	1.0535	1.0515	1.0021	1.0220	1.0018	1.0199	1.0471	1.0432	1.0035
Humboldt Univ. of Berlin	1.0483	1.0109	1.0374	0.9893	1.0000	0.9893	1.0382	1.0000	1.0382
VU University Amsterdam	1.0385	1.0208	1.0172	1.0456	1.0282	1.0172	1.0348	1.0082	1.0262
Tec. University of Munich	1.0347	1.0297	1.0047	1.0536	1.0158	1.0372	1.0341	1.0296	1.0042
University of East Anglia	1.0357	1.0415	0.9952	0.9732	0.9630	1.0112	1.0309	1.0351	0.9960
Lund University	1.0338	1.0282	1.0045	1.0043	0.9842	1.0206	1.0303	1.0114	1.0182
University of Oxford	1.0052	1.0000	1.0052	1.0532	1.0000	1.0532	1.0302	1.0000	1.0302
Utrecht University	1.0965	1.0807	1.0134	1.0298	1.0050	1.0247	1.0280	1.0050	1.0229
University of Bergen	1.0274	1.0146	1.0124	1.0399	1.0253	1.0143	1.0267	1.0143	1.0119
University of Sheffield	1.0354	1.0337	1.0016	0.9874	0.9714	1.0164	1.0266	1.0188	1.0075
University of Southampton	1.0262	1.0217	1.0045	1.0244	1.0052	1.0191	1.0266	1.0206	1.0058
Goethe University Frankfurt	1.0259	1.0193	1.0063	1.0253	1.0067	1.0186	1.0261	1.0194	1.0064
University of Glasgow	1.0252	1.0119	1.0132	1.0186	0.9994	1.0193	1.0256	1.0120	1.0136
Erasmus Univ. Rotterdam	1.0632	1.0234	1.0380	1.0162	1.0000	1.0162	1.0252	1.0000	1.0252
King’s College London	1.0313	1.0202	1.0110	1.0595	1.0361	1.0230	1.0250	1.0134	1.0110
University College London	1.0120	1.0024	1.0095	1.0286	1.0000	1.0286	1.0246	1.0000	1.0246
University of Edinburgh	1.0234	1.0140	1.0086	1.0226	1.0005	1.0221	1.0238	1.0140	1.0090
Wageningen University & R.	1.0163	1.0000	1.0163	1.0390	1.0314	1.0076	1.0236	1.0000	1.0236
Karolinska Institute	1.0317	1.0000	1.0311	1.0117	0.9986	1.0134	1.0236	1.0000	1.0236
University of Lausanne	1.0221	1.0135	1.0083	1.0087	0.9884	1.0211	1.0198	1.0125	1.0069
Stockholm University	1.0196	1.0130	1.0074	1.0555	1.0397	1.0154	1.0196	1.0130	1.0074
University of Cambridge	1.0012	1.0000	1.0012	1.0253	1.0000	1.0253	1.0196	1.0000	1.0196
University of Birmingham	1.0236	1.0276	0.9959	1.0018	0.9840	1.0182	1.0194	1.0137	1.0054
University of Basel	1.0185	1.0032	1.0153	1.0571	1.0395	1.0174	1.0184	1.0028	1.0156
University of Manchester	1.0221	1.0168	1.0051	1.0038	0.9863	1.0179	1.0170	1.0074	1.0092
University of Nottingham	1.0150	1.0165	0.9982	1.0277	1.0113	1.0161	1.0163	1.0083	1.0075
University of Helsinki	1.0175	1.0116	1.0062	1.0199	1.0066	1.0136	1.0155	1.0023	1.0142
Queen Mary U. of London	1.0149	1.0009	1.0143	1.0874	1.0769	1.0091	1.0152	1.0021	1.0134
Bielefeld University	1.0143	0.9895	1.0208	1.0370	1.0457	0.9935	1.0140	0.9895	1.0205
KU Leuven	1.0151	1.0026	1.0124	0.9988	0.9848	1.0146	1.0131	1.0000	1.0131
University of Dundee	1.0165	1.0055	1.0112	0.9729	0.9528	1.0198	1.0127	0.9987	1.0141
Uppsala University	1.0250	1.0060	1.0188	1.0053	0.9870	1.0185	1.0127	0.9942	1.0179
Swedish University of A. S.	1.0111	1.0000	1.0111	1.0557	1.0422	1.0133	1.0120	1.0000	1.0120
ETH Lausanne	1.0095	1.0000	1.0095	1.1034	1.0799	1.0224	1.0114	1.0000	1.0114
Delft University of Tec.	1.0091	1.0000	1.0091	1.0603	1.0348	1.0256	1.0097	1.0000	1.0097
Leiden University	1.0040	0.9822	1.0224	1.0344	1.0193	1.0135	1.0079	0.9891	1.0192
University of Geneva	1.0066	1.0087	0.9980	1.0243	0.9978	1.0266	1.0073	1.0040	1.0032
University of Bristol	1.0203	1.0168	1.0030	0.9641	0.9583	1.0059	1.0067	0.9954	1.0109
University of Aberdeen	1.0120	1.0057	1.0068	0.9681	0.9533	1.0162	1.0066	1.0005	1.0063
Newcastle University	1.0206	1.0114	1.0099	1.0036	1.0000	1.0036	1.0024	1.0000	1.0025
University of St Andrews	1.0092	1.0000	1.0092	0.9863	1.0000	0.9863	1.0019	1.0000	1.0019
University College Dublin	0.9931	0.9841	1.0091	1.0570	1.0446	1.0125	0.9995	0.9904	1.0092
Durham University	0.9984	0.9875	1.0116	0.9977	0.9863	1.0135	0.9993	0.9938	1.0058
Ghent University	0.9943	0.9800	1.0147	1.0234	1.0109	1.0125	0.9984	0.9884	1.0101
Trinity College Dublin	0.9868	0.9806	1.0068	1.0807	1.0547	1.0258	0.9947	0.9928	1.0021
Imperial College London	0.9802	1.0000	0.9802	1.0140	1.0000	1.0140	0.9930	1.0000	0.9930
University of York	0.9920	0.9920	1.0012	0.9776	0.9632	1.0152	0.9918	0.9927	1.0005
KTH Royal Institute of Tec.	0.9882	1.0000	0.9882	1.0280	1.0073	1.0209	0.9909	1.0000	0.9909
Tec. University of Denmark	0.9878	0.9773	1.0106	1.0360	1.0303	1.0063	0.9906	0.9792	1.0117
Lancaster University	0.9921	1.0156	0.9767	0.9429	0.9826	0.9580	0.9886	1.0108	0.9778
University of Sussex	0.9708	1.0000	0.9708	0.9669	0.9858	0.9801	0.9778	1.0000	0.9778
Eindhoven Univ. of Tec.	0.9724	0.9699	1.0026	1.0608	1.0565	1.0046	0.9757	0.9727	1.0032
University of Göttingen	0.9596	0.9539	1.0087	1.0170	0.9894	1.0282	0.9594	0.9539	1.0085

* Top three values in Italics.

In addition, the leading institutions, regarding average improvement from this perspective (Table 7), are different from the institutions leading on yearly improvements (Table 6). This connects to the fact known widely in management science, and especially with accounting and controlling, that achieving short-term improvements is a totally different game than sustaining long-term enhancements, in this case a six-year period [90,91,92]. The success of German institutions in the sample may be connected to the experience that these institutions had to adapt to the international ranking system, regarding the specific data gathering in this arena, within the analyzed timeframe. It also may fall into the same period when, e.g., due to the German excellence initiative, the first results in increasing international visibility and competitiveness were achieved [4,93].

From Table 8, depicting the statistical characteristics of all Malmquist index values by year (2012–2016) on the previous year for run III, some further descriptions can be outlined (highest values in bold numbers). First, high average levels of improvement are shown for the years 2012 and 2016, but not so much for the years in between. As an absolute value, the overall efficiency improvement of 6.59% for the last reported year of 2016 is high, also compared to other industries. For example, Örkcü et al. [94] (p. 101) report an efficiency improvement of 3.28% for Turkish airports on average from 2009 to 2014, Sueyoshi and Goto [95] (p. 342) describe a 1.10% efficiency improvement for 12 national petroleum companies from 2005 to 2009, and Emrouznejad and Yang [96] (p. 853) calculate a 2.89% efficiency improvement for the Chinese metal industry on average from 2004 to 2012. Again, it has to be emphasized that while these are exceptional values, there are also years, such as 2013 and 2014, with only minimal or zero improvement on efficiency regarding the inputs and outputs reported here. Second, variance and standard deviation with an annual sample among the 70 universities show remarkably high values, especially again in the first (2012) and last (2016) analyzed year.

Table 8. Malmquist Index Annual Average Values, Variances, and Standard Deviations (Run III).

Measure	Year	Malmquist Index	Catchup	Frontier Shift
Arithmetic Mean	2012	1.0314	1.0052	1.0260
	2013	1.0099	1.0221	0.9881
	2014	1.0000	1.0087	0.9914
	2015	1.0125	1.0040	1.0086
	2016	1.0659	1.0222	1.0430
Variance	2012	0.0046	0.0030	0.0013
	2013	0.0032	0.0023	0.0010
	2014	0.0014	0.0008	0.0008
	2015	0.0011	0.0010	0.0003
	2016	0.0076	0.0050	0.0027
Standard Deviation	2012	0.0686	0.0554	0.0364
	2013	0.0565	0.0486	0.0319
	2014	0.0379	0.0282	0.0278
	2015	0.0333	0.0312	0.0188
	2016	0.0878	0.0714	0.0520

Specific Malmquist index values for each university per year and for all three calculation runs I–III, as well as detailed numbers on the Malmquist catch-up and frontier shift breakdown, are reported in Appendix A (Table A1).

5. Discussion

The described results can be connected to existing research regarding university efficiency and is discussed as follows. The overall range of efficiency scores for 2011, as outlined in Table 4, is quite small between (17 institutions with) 100.00% and a minimum value of 61.60% (University of Copenhagen, run I, mean 87.21%). Such high levels of efficiency are not necessarily usual, but also not unexpected with 70 decision making units and 7, respectively 12, indicators. For example, the results of Fandel [97] (2007, p. 527), detailing 10 out of 15 German universities within North Rhine-Westphalia as efficient, with a mean efficiency of 92.77%, by using two inputs and three outputs for the 15 DMU. Also, Johnes [98] (p. 281) reports a mean efficiency of 92.51% regarding 130 UK universities with six inputs and three outputs, also with the lowest individual university efficiency levels around 60%. See also the comprehensive review of DEA studies in education by Fuentes, Fuster, and Lillo-Banuls [99] (pp. 91–93) regarding the applied inputs and outputs which are highly influential for the calculated efficiency results (in comparison with the number of observed DMU). This baseline is further analyzed regarding time-series changes in the following years until 2016.

Out of 350 dynamic data points (5 years of change from 2011 for 70 universities, run III), altogether 118 institutions with annual negative changes of efficiency are reported, whereas for 232 instances, a positive improvement of efficiency is recognized. This shows that universities are working hard to improve their efficiency as demanded from stakeholders and taxpayers regarding the inflow of public money into the higher education system. However, it also highlights that efficiency improvement is not an easy task but has to be earned with hard work. There is no “automatism” in efficiency improvement as is, for example, sometimes made to be believed due to technological change, as, e.g., for e-learning [100,101]. The results regarding productivity increases (individual and overall) are within range of existing results, for example, by Parteka and Wolszczak-Derlacz [13] (p. 73), who report an average 4.1% annual increase of productivity for 266 public universities in 7 European countries between 2001 and 2005. In their dataset, universities from Germany, Italy, and Switzerland provided the highest efficiency improvements. However, it has to be recognized that in this case, three inputs (staff, students, budget) were compared only to two outputs (publications and graduates). As also in this research, German universities are reported to have above-average efficiency improvements, which seems to be a stable result. Obviously, the reported analysis results and differences regarding the distinction between general technological progress (“frontier shift”) and its use on the one hand and the individual organizational reasons for efficiency changes (“catch-up”) on the other hand for universities are interesting and should be studied further. In addition, in-depth analysis is required in terms of resource and organizational consequences of such efficiency development results for universities as done, e.g., in the health care or service sector [102,103]. For example, it can be questioned if an institution or a department should receive unequivocal research or teaching funding when long-term negative developments of efficiency are recognized. This is superior to the question applied mainly today of, if due to an existing (low) efficiency status compared to others, restrictions in terms of funding shall be implemented. Whereas an efficiency development can largely be attributed to the institutional responsibility (given stable and comparable circumstances), the static efficiency position compared to other institutions may have a multitude of (external) influences and institutional responsibility is not a given.

As a policy implication, this would hint at an adjusted resource distribution scheme where allocation is connected to the longitudinal efficiency development of institutions. For example, institutions with decreasing efficiencies over time would also receive a reduced amount of resources, whereas institutions increasing their efficiency (change, not absolute level, potentially above a threshold level) would receive in increase in funding, e.g., by state budgets or also competitively distributed research funding and other sources.

Requirements for efficiency analysis with the DEA technique and for ranking endeavors have to be recalled into the academic and public discourse. Mainly, it has to be ensured that the analyzed DMU are actually comparable. This notion can be discussed in different perspectives: (i) From a production theory perspective, the border production function has to be identical or at least the same technical production possibilities have to be available to all compared units and institutions [65]; (ii) From an economic price and market perspective, the used factor prices, as, e.g., for academic personnel, have to be identical or at least comparable; whereas wages for academics are not identical, not even within one country, the argument may be stated that even in a global perspective wages for research and teaching assistants as well as professors as core personnel for universities do not feature too much deviation; (iii) From a higher education research perspective, the main notion is the question of comparable objectives, missions, and profiles. This may be tested mainly against the following research hypothesis. The subject mix and homogeneity within and in between institutions has to be taken into account. Either only broad university institutions calling upon the “universitas” principle are included into comparative analyzes, or the analysis is broken down into subject fields, as is done by most rankings today (i.e., THE and CWTS, but also others such as the Shanghai ARWU ranking).

Altogether, it can be argued that the contribution of this paper in finding no proof for an index data problem for the application of ranking data in DEA efficiency analysis endevors for universities has the following implication for the knowledge and future research directions. Ranking data can be used unequivocally for efficiency analysis projects, independent of the fact if the rankings contain indexed data or aggregate data. This alleviates the application of ranking data for higher education efficiency research and therefore provides an important potential for further analysis, as the data realm of international university rankings is growing every year.

It has to be stressed that efficiency questions and the interest in analyzing and improving the performance, excellence, and output (given more or less fixed inputs) of universities in higher education is not new (see, for example, [104]). However, today the available techniques, such as DEA windows analysis and DEA Malmquist index for a dynamic time-series analysis, the available data due to information technology, as well as chances to compare these data and analyses internationally have improved the level of analysis significantly. To put these available instruments to a good use, this article wants to contribute to the methodological discussion regarding efficiency analyses for higher education.

6. Conclusions and Outlook

This paper has made it obvious that no specific index number problem can be found for longitudinal efficiency analysis calculations for universities based on ranking data. This holds true for the indexed ranking publications of Times Higher Education, as this data (for the timeframe 2011–2016) was analyzed herein in comparison with the CWTS (nonindexed) data. Research and policy implications for the presented and analyzed data include, among others, the following points for further discourse. Ranking data, also within an indexed form, are assumed to be feasible and can acts as a quality output indicator basis for efficiency analysis endeavors, also in a longitudinal time-series analysis with such instruments as, for example, the DEA window analysis or the DEA Malmquist index analysis. As described above, the time-series analysis of efficiency development per institution may be an interesting field for academic analysis as well as a decision basis for university managers, politicians, as well as stakeholders and partners of universities. For example, if a company is thinking about a long-term research cooperation with a university, a look into past long-term efficiency developments at this institution may be well advised and informative in order to protect such a strategic investment. Further, it is has been shown that long-term efficiency improvement is a different playing field from yearly improvements. In the long-term perspective, German universities in particular were faring well within the analyzed timeframe of 2011–2016 and the applied dataset of 70 European universities. From the input and output correlation analysis, interesting results are that larger budgets and staff numbers correlate with higher ranking evaluations in the fields of teaching and research, but less so for citations. Institutions with larger input volumes therefore have at least a larger chance to reach higher output levels in these fields. However, this is definitively not true for the evaluation fields of international outlook and industry income as measured by THE rankings. This may be connected to the fact that achieving productive international as well as industry cooperation may not so much depend on the size of the budget and staff numbers but more on a mind-set within the university regarding industry income and especially a form of flexibility. This is hypothetical, as there is no causal analysis at hand, but at least the correlation numbers for the observed 420 cases may provide some interesting basis for creating and further on testing hypotheses regarding such productivity connections in universities.

Efficiency analysis may provide an elaborate form of quality check towards ranking systems, especially in the proposed time-series form, as with constant inputs for the same institutions and timeframe, rankings are supposed to present similar results. If not, the quality of the ranking performance measurement can be doubted. This has to be enlarged with datasets from other rankings in a timeline perspective, and with that, the problem of establishing a “reference set” or “baseline” performance dataset may arise. Which ranking or dataset would other rankings have to be measures “against”? Such avenues of inquiry are connected to the quality debate regarding evaluations and rankings for universities, as, for example, stated by Bornmann [105], Osterloh and Frey [106], and Harmann [107] for an international perspective. For the national German context regarding the VHB JOURQUAL3 journal rankings in economics and business administration, Eisend [108] and Schrader and Hennig-Thurau [109] discuss this as well as Lorenz and Löffler [110] for the Handelsblatt ranking of business economists. Policy implications point towards the discourse regarding conflicts of interest for university management in pursuing efficiency goals compared to other objectives such as excellence, reputation, or cooperation, as Blackmore points out [111].

Further research is warranted to address, for example, the following five areas regarding university efficiency, rankings, and overall sector performance development: (a) Further ranking data for indexed and nonindex values should be tested in order to enlarge the database for the falsification of an index number problem for university ranking data in longitudinal efficiency analysis projects; (b) The further eligibility of such a dynamic efficiency analysis, e.g., with the DEA Malmquist index for a metaevaluation of rankings, could be tested. This can be an important contribution to ranking system quality; (c) Institutional management implications do earn a further look into possible steering and efficiency improvement measures based on longitudinal efficiency results for individual universities. Sideways comparisons with other knowledge-intensive service industries such as health care, finance and insurance, accounting and consulting, or logistics could be promising; (d) In addition, the possible implications and measures on a policy-systems level of higher education are of high interest given the fact that, e.g., for Germany, the public resources spent within university institutions totals about 48.2 billion Euros or 1.3% of total GDP for 2014 [112] (p. 71). Any fact-based research implying possible changes and improvements in the setup for public funding distribution in the sector might be of high value in an economic perspective; (e) Finally, also a look into the organizational level of research groups and individual researchers regarding the long-term dynamic efficiency development may be very interesting. Much research work already does exist for this in static as well as dynamic output perspectives, e.g., [113,114,115], but little yet regarding a longitudinal efficiency perspective taking inputs into account. This could, for example, connect to the long-standing question of if outside and additional resources are able to improve the efficiency of an individual researcher or if there is no or a marginal effect, as, for example, found by Fedderke and Goldschmidt [116] (p. 479).

Altogether, the question of ranking data as an output database for time-series efficiency analysis in higher education has been proven to be a worthwhile and interesting field of inquiry for higher education research and management.