Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach
2. Literature Review
3.1. Institutional Context
- The Andalusian university system has 1 private and 10 public universities (one of them being distance-learning). The Andalusian Public University System has historical and traditional universities, which are larger, in general (such as the Universities of Granada and Seville) and very modern universities that are smaller (such as the Universities of Almería, Huelva and Jaén). It is the largest Spanish university system by number of faculty members and researchers, and it has more than 230,000 students.The Andalusian region has been one of the pioneers in Spain in introducing some performance mechanisms to articulate a university funding model that rewards for better performance and quality of teaching and research. In this way, the first model for the Andalusian public universities (2002–2006), signed in November 2001, constitutes a first attempt to establish a mix cost-oriented and performance-oriented funding formula, to allocate around 7.5% of the university budget linking to outputs. The subsequent university funding models correlate regional funds with (quantitative) measures of institutions’ past activity (10%), in addition to basic financing that is based primarily on costs (approximately 90% of the regional support). The small amount of public resources that are distributed with a performance criteria causes that, conform to reality, the funding system has remained as a formula funding model based on provider (supplier) in Andalusia in the analyzed period (2012–2017). Please note that this funding model is not maintained from 2017.Furthermore, the Andalusian universities are increasingly depending on public funding from the Regional Government, which leads to a greater equalization among them in terms of financial effort per student. This is reflected in a more similar pattern in terms of teaching and research results funding (“intra” universities), although there are different research performance behaviours as shown, for instance, in ARWU (Academic Ranking of World University) and .As a result of the landscape of research in Andalusia, the regional government launched in 2015 an international talent recruitment program, called TALENTIA, so that Andalusian universities could attract and bring back highly regarded research professors, although it has resulted in few professors and highly cited researchers received by universities, because of the scarce resources available for the program.
- The Catalan university system is also one of the largest in Spain, with approximately 200,000 students. It comprises 12 universities, seven of which are public research universities. The Catalan public university system offers a certain differentiation or institutional profiling: first, several universities with prominent positions in the international rankings regarding research, such as the University of Barcelona, the Pompeu Fabra University or the Autonomous University of Barcelona; second, universities that are able to attract industry income, such as the Universitat Politecnica de Catalonia and Rovira i Virgili; and third, several peripherical and smaller universities with excellence levels in teaching activities, such as Universities of Girona and Lleida.The Catalan university funding model based on a formula base began in 2002. It is a funding model based on inputs (cost-oriented), but with a clear performance-oriented intention (the output weight is around 40%). The financing mechanism consists of four different grants: fixed, basic, derivative, and strategic subsidies. In the funding model implemented in 2006, the strategic subsidy integrated the contract agreements that the Catalan government had been establishing with universities in the old periods (a supplementary funding program was created with the UPC in 1997). In addition, in 2008, a target funding program was created, complementary to the model, which represents just over 5% of the total funding that universities received.Moreover, Catalonia Region has promoted the excellence in research activities through CERCAS (Catalan Centers for Research) but also through initiatives such as ICREA (Catalan Institution for Research and Advanced Studies) that attract and retain talent all over the world. ICREA research professors choose their host institutions within the Catalan research system, some of which are universities, and act as research leaders and attractors of additional talent. This initiative has influenced the high levels of research in many Catalan universities because it has generated synergies.
- The Madrid university system is the largest one in Spain by number of students and the second in number of faculty members and researchers. It comprises six on-site public universities, which offer formation to approximately 220,000 students, and eight private universities that train approximately another 100,000 students. Due to the variety of academic programs and the R&D possibilities that the region offers, the Madrid university system attracts students and researchers from all over Spain and other countries.Moreover, it is worth examining the Madrid university system, since it has not followed a performance-based funding. However, an attempt was made in 2006 with the introduction of new policy instruments in the performance agreement 2006–2010, to achieve specific types of results, but its implementation was quickly truncated by the immediate economic crisis of 2008. It can be stated that there is no funding model anytime in Madrid region and, therefore, an annual negotiation together with an incremental distribution model, maintains financial dependency from regional funds, although universities set their own strategies in terms of teaching and research. Specifically, in the absence of a funding model, the resource-sharing system has been discretionary and based on political decisions with bilateral negotiations between the regional government and the six public universities in Madrid. This fact leads to many differences among the six universities within Madrid region.Furthermore, the Madrid university system is characterized by: (i) being one of the systems with a greater basic and applied research, given the prominent position of some Madrid universities, such as the Autonomous University of Madrid and the Complutense University; (ii) its ability to raise private funds, especially by the Polytechnic University of Madrid; (iii) and its joint research with the IMDEA Institutes (Research centres of excellence based in the Region of Madrid: https://www.imdea.org/en/ (accessed on 16 June 2021)) and the Mixed Centers of the Spanish National Research Council (CSIC) (https://www.csic.es/en (accessed on 16 June 2021)) that work jointly with researchers from the six Madrid public universities.
3.2. System of Indicators for the Spanish Public University System
- Teaching: in order to measure the teaching mission of the universities, the quantity and quality of the results obtained and the international projection of the teaching activities are taken into account. Therefore, two sub-blocks have been considered:
- The teaching Results are measured using the indicators available about the students’ performance. They measure how many students submit to evaluation, how many pass their courses, how many graduate and how many drop out. Additionally, the number of post-graduate students has been considered.
- The External projection is linked to the bilingual official masters degrees, the percentage of foreign students and the percentage of students in exchange programs.
- Research: this mission includes the resources available for research, and the corresponding outputs, in terms of their quantity and quality. Namely, the following sub-blocks have been considered:
- Given their importance in all the rankings, the Publications have been considered separately from the rest of the research activity. The amount of publications per doctor, and the quality of the publications (in terms of the relative position of the journals and the citations) have been included.
- The Other research activities consider the international collaborations, the number of doctoral theses defended, the participation in projects and the overall official recognition of research, provided by the National Agency of Evaluation.
- Finally, the Projects approved for research, taking into account the research grants and contracts and the success rate in National and European research projects have been considered.
- Reference levels: For each indicator i, four reference levels (minimum, ; reservation, ; aspiration, ; and maximum, ) are established. Please note that is a level regarded as acceptable, i.e., values worse than the reservation level are regarded as poor performance values, and is a level regarded as desirable, i.e., values better than the aspiration level are regarded as good or desirable.According to , these reference levels can be relative (statically set) or absolute (given by a decision maker). In this paper, the four reference levels have been statistically established, considering data of all the Spanish public universities for the entire period analyzed (from 2012–2013 to 2016–2017). Namely, when the indicator is of type “the more, the better”, the has been set as the average value between the mean () and the of each indicator for all Spanish public universities for the entire period. On the other hand, the is the average value of () and the (for further details about the reference levels of each indicator, see Figure A1 in Appendix A.1). These reference levels have two consequences in the process. First, the relative position of each university is analyzed with respect to the whole Spanish public university system. Second, since the reference levels are fixed for the entire period analyzed, any university improvement or worsening relates only to its own performance.Please note that an outlier detection, through the interquartile range () method, is carried out . According to this test, two different outliers can be detected: mild outliers (; ) and extreme outliers (; ). In this case, no extreme outliers were detected, while the mild outliers are assigned, for the corresponding achievement function, the minimum or maximum value (depending on its relative position) of the scale.
- Achievement functions: The purpose of calculating the achievement functions () is twofold. First, they allow the normalization of the indicators, i.e., all the indicators are brought to a common measurement scale. Second, they inform about the relative position of each university with respect to the reference levels given (see more details about the construction of the achievement function, , in Appendix A.2).Specifically, in this paper, all the indicators are translated to a common scale from 0 to 3. This way, for each indicator i, different performance levels are defined (See Figure 1):
- Poor ⇒ The university performs worse than the corresponding (values between 0 and 1).
- Fair ⇒ The university performs better than the level, but worse than the level (values between 1 and 2).
- Good ⇒ The university performs better than the corresponding (values between 2 and 3).
- Aggregation: A particular feature of the MRP-WSCI method is the possibility of constructing two different composite indicators, by considering the compensation degree among the indicators. In this paper, in order to analyze the universities overall performance, the Weak Composite Indicator (WCI) is used, which allows for full compensation. Moreover, the possible improvement areas of each university will be detected through the Strong Composite Indicator (SCI), which does not allow for any compensation. In particular, as can be observed in Figure 2, the WCI and SCI are provided for each sub-block and each university mission through two aggregation steps, i.e., in the first step the individual indicators are aggregated in order to construct the WCI and SCI by sub-block; and, in the second step, the corresponding sub-blocks are aggregated in order to obtain the WCI and SCI by mission. Therefore, the WCI by sub-block (and mission) gives an idea about the overall performance of each university within the corresponding sub-block (and mission), while the SCI measures the worst performance of each university within the corresponding sub-block (and mission). Specifically, the SCI provides a measure of the worst individual indicator (and sub-block) of each university, and therefore, it gives an idea about the improvement areas. The details about the construction of the MRP-WSCI are displayed in Appendix A.3.
5.1. Teaching Mission
5.2. Research Mission
6.1. Dynamic Evolution by Regions
- The behavior of each region is represented by a line in the graph.
- Each point of the line represents the teaching (x component) and research (y component) performance for each academic year. In particular, the point with the arrow represents the teaching WWCI and research WWCI values for the given region in the final academic year 2016–2017, while the other extreme point of the line corresponds to the initial year, 2012–2013.
- A good region performance in both missions means being located further right and on the top.
6.2. Policy Message and Funding Models
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
|UCO||University of Cordoba|
|UAL||University of Almeria|
|UCA||University of Cadiz|
|UGR||University of Granada|
|UHU||University of Huelva|
|UJAEN||University of Jaen|
|UMA||University of Malaga|
|US||University of Sevilla|
|UPO||University Pablo de Olavide|
|UAB||University Autonoma de Barcelona|
|UB||University of Barcelona|
|UDG||University of Girona|
|UDL||University of Lleida|
|UPC||University Politecnica de Catalonia|
|UPF||University Pompeu Fabra|
|URV||University Rovira I Virgili|
|UAM||University Autonoma de Madrid|
|UC3M||University Carlos III de Madrid|
|UCM||University Complutense de Madrid|
|UAH||University Alcala de Henares|
|UPM||University Politecnica de Madrid|
|URJC||University Rey Juan Carlos|
|t-th value of the common measurement scale|
|I||Number of indicators (i is the corresponding index)|
|J||Number of units (j is the corresponding index)|
|Weight assigned to indicator i|
|n||Number of reference levels (t is the corresponding index)|
|Normalized weight of indicator i|
|t-th reference level of indicator i|
|Value of the scalarizing achievement function of indicator i for unit j|
|Value of indicator i for unit j|
Appendix A. The MRP-WSCI Approach
Appendix A.1. Weights and Reference Levels
Appendix A.2. Achievement Function
Appendix A.3. Aggregation: WCI and SCI
Appendix A.4. Hypothetical Example
Appendix B. Results: WCI and SCI Values
Appendix C. Results: WWCI Values
- EUA. Financially Sustainable Universities II. European Universities Diversifying Income Streams; Technical Report; European University Association: Brussels, Belgium, 2011. [Google Scholar]
- Pruvot, E.B.; Claeys-Kulik, A.L.; Estermann, T. Developments in resource allocation policies for universities. A comparative European overview. J. Eur. High. Educ. Area 2015, 4, 1–17. [Google Scholar]
- De la Torre, E.M.; Perez-Esparrells, C. Reforms in the Spanish Higher Education System since Democracy and future Challenges. In Higher Education System Reform: An International Comparison after Twenty Years of Bologna; Broucker, B., Ed.; Brill Publishers: Leiden, The Netherlands, 2019; pp. 51–71. [Google Scholar]
- Johnes, J.; Taylor, J. Performance Indicators in Higher Education: UK Universities; Open University Press and the Society for Research into Higher Education: Maidenhead, UK, 1990. [Google Scholar]
- De la Torre, E.M.; Casani, F.; Perez-Esparrells, C. Measuring universities’ engagement: A revision of the European research projects and the actual use of the so-called ‘third mission’ indicators. Rev. Estud. Reg. 2020, 1, 97–128. [Google Scholar]
- Jefatura del Estado «Boletín Oficial del Estado». Real Decreto-ley 14/2012, de 20 de Abril, de Medidas Urgentes de Racionalización del Gasto Público en el áMbito Educativo. núm. 96, de 21 de abril de 2012. Referencia: BOE-A-2012-5337. 2012. Available online: https://www.boe.es/buscar/pdf/2012/BOE-A-2012-5337-consolidado.pdf (accessed on 16 June 2021).
- Dehon, C.; Jacobs, D.; Vermandele, C. Ranking and research assessment in higher education: Current and future challenges. In Ranking Universities; Dehon, C., Jacobs, D., Vermandele, C., Eds.; Editions de l’Université de Bruxelles: Brussels, Belgium, 2009; pp. 1–10. [Google Scholar]
- Rauhvargers, A. Global University Ranking and Their Impact: EUA Report on Rankings; Technical Report; European University Association: Brussels, Belgium, 2011. [Google Scholar]
- Heitor, M.V.; Horta, H. Democratizing Higher Education and Access to Science: The Portuguese Reform 2006–2010. High. Educ. Policy 2013, 27, 1–19. [Google Scholar] [CrossRef]
- Marope, P.T.M.; Wells, P.J.; Hazelkorn, E. Rankings and Accountability in Higher Education: Uses and Misuses; Technical Report; UNESCO: Paris, France, 2013. [Google Scholar]
- OECD. Education at a Glance 2015: OECD Indicators; Technical Report; OECD Publishing: Paris, France, 2015. [Google Scholar] [CrossRef]
- El Gibari, S.; Gómez, T.; Ruiz, F. Evaluating university performance using reference point based composite indicators. J. Inf. 2018, 12, 1235–1250. [Google Scholar] [CrossRef]
- Zhang, Q.Z.; Jiang, S.; Liu, R.; Liu, H.C. An Integrated Decision-Making Model for Analyzing Key Performance Indicators in University Performance Management. Mathematics 2020, 8, 1729. [Google Scholar] [CrossRef]
- JICA. Japan International Cooperation Agency Annual Report; Technical Report; JICA: Tokyo, Japan, 2020.
- González-Garay, A.; Pozo, C.; Galán-Martín, A.; Brechtelsbauer, C.; Chachuat, B.; Chadha, D.; Hale, C.; Hellgardt, K.; Kogelbauer, A.; Matar, O.K.; et al. Assessing the performance of UK universities in the field of chemical engineering using data envelopment analysis. Educ. Chem. Eng. 2019, 29, 29–41. [Google Scholar] [CrossRef]
- Cave, M.; Hanney, S.; Henkel, M. Performance measurement in higher education—Revisited. Public Money Manag. 2009, 15, 17–23. [Google Scholar] [CrossRef]
- De Boer, H.B.; Jongbloed, B.; Benneworth, P.; Cremonini, L.; Kolster, R.; Kottmann, A.; Lemmens-Krug, K.; Vossensteyn, H. Performance-Based Funding and Performance Agreements in Fourteen Higher Education Systems; Technical Report; The Ministry of Education, Culture and Science; CHEPS: Enschede, The Netherlands, 2015.
- Jongbloed, B.; De Boer, H.B. Performance Agreements in Denmark, Ontario and the Netherlands Report for the Project Evaluation of Development Contracts in Norwegian Higher Education; Technical Report; Center for Higher Education Policy Studies (Cheps) and Nordisk Intitutt for Studier av Innovasjon, Forskning of Utdanning (NIFU): Oslo, Noruega, 2020. [Google Scholar]
- Marginson, S. Global trends in higher education financing: The United Kingdom. Int. J. Educ. Dev. 2018, 58, 26–36. [Google Scholar] [CrossRef]
- Teichler, U. Recent changes of financing higher education in Germany and their intended and unintended consequences. Int. J. Educ. Dev. 2018, 58, 37–46. [Google Scholar] [CrossRef]
- Van Vught, F.A.; File, J.M.; Kaiser, F.; Jongbloed, B.; Faber, M. U-Map: A university Profiling Tool; Updated Report; CHEPS: Enschede, The Netherlands, 2011. [Google Scholar]
- Van Vught, F.; Ziegele, F. Multidimensional Ranking—The Making of U-Multirank; Springer: Cham, The Netherland, 2012. [Google Scholar]
- De la Torre, E.M. Typologies of Universities: Role of the Third Mission in University Policies and Strategies in the New Framework of Higher Education. Ph.D. Thesis, Universidad Autónoma de Madrid, Madrid, Spain, 2016. [Google Scholar]
- Hazelkorn, E. How Rankings are Reshaping Higher Education. In Los Rankings Universitarios: Mitos y Realidades; Climent, V., Michavila, F., Ripollés, M., Eds.; Tecnos (ANAYA): Madrid, Spain, 2013; pp. 49–60. [Google Scholar]
- Aguillo, I.F.; Ortega, J.L.; Fernández, M. Webometric Ranking of World Universities: Introduction, Methodology, and Future Developments. High. Educ. Eur. 2008, 33, 233–244. [Google Scholar] [CrossRef]
- Murias, P.; de Miguel, J.C.; Rodríguez, D. A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System. Soc. Indic. Res. 2008, 89, 129–146. [Google Scholar] [CrossRef]
- Giannoulis, C.; Ishizaka, A. A Web-based decision support system with ELECTRE III for a personalised ranking of British universities. Decis. Support Syst. 2010, 48, 488–497. [Google Scholar] [CrossRef][Green Version]
- Torres-Salinas, D.; Moreno-Torres, J.G.; Delgado-López-Cózar, E.; Herrera, F. A methodology for Institution-Field ranking based on a bidimensional analysis: The IFQ2A index. Scientometrics 2011, 88, 771–786. [Google Scholar] [CrossRef]
- Alasehir, O.; Perit-Cakir, M.; Acarturk, C.; Baykal, N.; Akbulut, U. URAP-TR: A national ranking for Turkish universities based on academic performance. Scientometrics 2014, 101, 159–178. [Google Scholar] [CrossRef]
- El Gibari, S.; Cabello, J.M.; Gómez, T.; Ruiz, F. Composite Indicators as Decision Making Tools: The Joint Use of Compensatory and Noncompensatory Schemes. Int. J. Inf. Technol. Decis. Mak. 2021, 20, 847–879. [Google Scholar] [CrossRef]
- Freudenberg, M. Composite Indicators of Country Performance: A Critical Assessment; OECD Publishing: Paris, France, 2003. [Google Scholar] [CrossRef]
- Jacobs, R.; Smith, P.; Goddard, M. Measuring Performance: An Examination of Composite Performance Indicators; Technical Report; The University of York and Centre for Health Economics: York, UK, 2004. [Google Scholar]
- Becker, W.; Paruolo, P.; Saisana, M.; Saltelli, A. Weights and Importance in Composite Indicators: Mind the Gap. In Handbook of Uncertainty Quantification; Ghanem, R., Higdon, D., Owhadi, H., Eds.; Springer International Publishing: Berlin/Heidelberg, Germany, 2016; pp. 1–30. [Google Scholar] [CrossRef]
- El Gibari, S.; Gómez, T.; Ruiz, F. Building composite indicators using multicriteria methods: A review. J. Bus. Econ. 2019, 89, 1–24. [Google Scholar] [CrossRef]
- Laguna-Sánchez, P.; Palomo, J.; de la Fuente-Cabrero, C.; de Castro-Pardo, M. A Multiple Criteria Decision Making Approach to Designing Teaching Plans in Higher Education Institutions. Mathematics 2021, 9, 9. [Google Scholar] [CrossRef]
- Blancas, F.J.; Caballero, R.; González, M.; Lozano-Oyola, M.; Pérez, F. Goal Programming synthetic indicators: An application for sustainable tourism in Andalusian coastal counties. Ecol. Econ. 2010, 69, 2158–2172. [Google Scholar] [CrossRef]
- Ijiri, Y. Management Goals and Accounting for Control; North-Holland: Amsterdam, The Netherlands, 1965. [Google Scholar]
- Ignizio, J.P. Goal Programming and Extensions; Lexington Books: Lexington, MA, USA, 1976. [Google Scholar]
- Ruiz, F.; El Gibari, S.; Cabello, J.M.; Gómez, T. MRP-WSCI: Multiple Reference Point based Weak and Strong Composite Indicators. Omega 2020, 95, 102060. [Google Scholar] [CrossRef]
- Ruiz, F.; Cabello, J.M. MRP-PCI: A Multiple Reference Point Based Partially Composite Indicators for Sustainability Assessment. Sustainability 2021, 13, 1261. [Google Scholar] [CrossRef]
- Wierzbicki, A.P. The Use of Reference Objectives in Multiobjective Optimization. In Multiple Criteria Decision Making Theory and Application; Fandel, G., Gal, T., Eds.; Lecture Notes in Economics and Mathematical Systems; Springer: Berlin/Heidelberg, Germany, 1980; Volume 177, pp. 468–486. [Google Scholar]
- Wierzbicki, A.P.; Makowski, M.; Wessels, J. (Eds.) Model-Based Decision Support Methodology with Environmental Applications; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2000. [Google Scholar]
- Ruiz, F.; Cabello, J.M.; Luque, M. An application of reference point techniques to the calculation of synthetic sustainability indicators. J. Oper. Res. Soc. 2011, 62, 189–197. [Google Scholar] [CrossRef]
- European Commission. Delivering on the Modernisation Agenda for Universities: Education, Research and Innovation; Technical Report; Communication from the Commission to the Council and the European Parliament: Brussels, Belgium, 2006.
- Daumiller, M.; Dresel, M. Teaching and research: Specificity and congruence of university faculty achievement goals. Int. J. Educ. Res. 2020, 99, 101460. [Google Scholar] [CrossRef]
- European Commission. Science, Research and Innovation Performance of the EU 2018: Strengthening the Foundations for Europe’s Future; Technical Report; Publications Office of the European Union: Luxembourg, 2018. [Google Scholar]
- European Commission. Research and Innovation Performance in Spain: Country Profile 2014; Technical Report; Publications Office of the European Union: Luxembourg, 2014. [Google Scholar]
- Hernández Armenteros, J.; Pérez García, J.A. La Universidad Española en Cifras—2017/2018; Technical Report; CRUE Universidades Españolas: Madrid, Spain, 2019; ISBN 978-84-09-18182-7. [Google Scholar]
- Cabello, J.M.; Ruiz, F.; Pérez-Gladish, B.; Méndez-Rodríguez, P. Synthetic indicators of mutual funds’ environmental responsibility: An application of the Reference Point Method. Eur. J. Oper. Res. 2014, 236, 313–325. [Google Scholar] [CrossRef]
- Tukey, W.J. Exploratory Data Analysis; Addison-Wesley: Boston, MA, USA, 1977; ISBN 9780201076165. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
El Gibari, S.; Perez-Esparrells, C.; Gomez, T.; Ruiz, F. Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach. Mathematics 2021, 9, 1626. https://doi.org/10.3390/math9141626
El Gibari S, Perez-Esparrells C, Gomez T, Ruiz F. Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach. Mathematics. 2021; 9(14):1626. https://doi.org/10.3390/math9141626Chicago/Turabian Style
El Gibari, Samira, Carmen Perez-Esparrells, Trinidad Gomez, and Francisco Ruiz. 2021. "Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach" Mathematics 9, no. 14: 1626. https://doi.org/10.3390/math9141626