- freely available
Societies 2013, 3(3), 266-279; doi:10.3390/soc3030266
Abstract: In this paper, we review the current official methodology of scientific research output evaluation in the Czech Republic and present a case study on twenty-one Czech public universities. We analyze the results of four successive official research assessment reports from 2008 to 2011 and draw the following main conclusions: (a) the overall research production of the universities more than doubled in the period under investigation, with virtually all universities increasing their absolute research output each year, (b) the total research production growth is slowing down and (c) Charles University in Prague is still the top research university in the Czech Republic in both absolute and relative terms, but its relative share in the total research performance is decreasing in favor of some smaller universities. We also show that the rankings of universities based on the current methodology are quite strongly correlated with established indicators of scientific productivity. This is the first time ever that the official present-day Czech science policy and evaluation methodology along with the results for the Czech university system has been communicated to the international public.
1. Introduction and Related Work
The evaluation of scientific research output has become crucial in recent years, as the budgets of science funding bodies (governments, foundations, etc.) have become tight, but the need for research and innovations has been ongoing or even growing. Therefore, it has become clear that it is absolutely necessary to identify high quality research that should be prioritized in receiving funding and also poor quality research whose funding is no more effective. The key concept here is to promote the advancement of science as efficiently as possible, i.e., to maximally increase the effort/award rate from the point of view of financing science. This is why many countries have introduced various research performance evaluation systems (especially for institutions), some of which are the well-known Research Assessment Exercise (RAE) in the United Kingdom or Excellence in Research for Australia (ERA) in Australia. Science evaluation has also been a hot topic in the Czech Republic in recent years. The Czech government (or more precisely, the Research, Development and Innovation Council—an advisory body to the government) published an official methodology of research output evaluation that later changed several times within a few years. We will review the current methodology (from May 2011) in the following sections and show the results of the last four official research evaluation reports based on this methodology in the context of twenty-one Czech public universities. Although the official methodology should only serve as an input into the process of research budget creation, its application inevitably leads to university rankings, which are part of this paper’s results section. (There are no official university rankings in the Czech Republic.)
The Czech Republic is little covered in science and technology literature. Some of the few studies devoted exclusively to the Czech Republic include bibliometric analyses of Czech research publications , patents  or European framework program results . Other scientometric studies usually observe the Czech Republic in the context of a larger group of (Central) European countries, e.g.,  or . As far as the official evaluation of scientific research output in the Czech Republic is concerned, it seems that the Czech research evaluation system is (almost) unknown to the rest of the world: neither  nor, more recently,  make an explicit mention of the Czech Republic in their comprehensive overviews of university research evaluation and funding systems in different countries. Country-specific research evaluation at the university level is currently a lively topic for scientometricians, as is well documented by the recent studies for Colombian , Spanish , Chinese , South African  or Taiwanese  universities. Many papers (e.g., [13,14,15,16]) are also concerned with the use of peer review and bibliometric indicators in national university research evaluation and funding systems and argue why the former or latter approach is better, but this is not the intent of this article. We solely present the currently used research evaluation methodology and a case study for universities.
2. Data and Methods
In this study, we concentrated on a set of twenty-one public universities (see Table 1) run by the Ministry of Education, Youth and Sports of the Czech Republic and, in one case, by the Ministry of Defense of the Czech Republic (University of Defense). These universities are also the most highly ranked in the 2011 Research Evaluation Report (the most recent evaluation). Other public universities in the Czech Republic do not conduct research in the fields of science and technology (such as colleges of arts or police academies) and are discarded from this study.
|University name in English||Acronym|
|University of South Bohemia in České Budějovice||Budějovice|
|Czech Technical University in Prague||ČVUT|
|Czech University of Life Sciences Prague||ČZU|
|University of Hradec Králové||Hradec|
|Technical University of Liberec||Liberec|
|Mendel University in Brno||MZLU|
|Palacký University, Olomouc||Olomouc|
|Silesian University in Opava||Opava|
|University of Ostrava||Ostrava|
|University of Pardubice||Pardubice|
|University of West Bohemia||Plzeň|
|Charles University in Prague||UK|
|University of Defense||UO|
|Jan Evangelista Purkyně University in Ústí nad Labem||Ústí|
|University of Veterinary and Pharmaceutical Sciences Brno||VFU|
|VŠB-Technical University of Ostrava||VŠB-TUO|
|University of Economics, Prague||VŠE|
|Institute of Chemical Technology, Prague||VŠCHT|
|Brno University of Technology||VUT|
|Tomas Bata University in Zlín||Zlín|
The official methodology for the evaluation of research output has been slightly modified a few times since 2008, the first year in a series of successive comparable research evaluation reports. (There were research evaluation methodologies and reports before 2008, but they differed from the current methodology to such extent that it would make no sense to compare those evaluations to the current ones. For instance, the reports only considered research results related to completed grant projects, etc. In contrast, the current methodology considers all results.) In the following sections, we will present a short summary of the current methodology (available in Czech ) defined by the Czech government in May 2011. In general, the methodology is based on assessing scientific production, i.e., it counts publications and other research results produced and only indirectly (in some cases) on assessing the quality of research output. No citations are counted, but, in the case of journal articles, the journal impact factor is taken into account, which is a de facto cheap estimate of potential citation counts. In this methodology, all research results yielded in the five years preceding the evaluation year are assigned the scores shown in Table 2. For instance, all journal articles indexed in the Web of Science (WoS) database by Thomson Reuters that were published in journals with a nonzero impact factor in the Journal Citation Reports (JCR, edited in the publication year) from 2006 to 2010 will be assigned a score between ten and 305 in the 2011 Evaluation. The score is computed according to the following formula:
|Result category||“National” fields||Other fields|
|Jimp||impacted journal article||10–305|
|Nature or Science article||500|
|Jnoimp||refereed journal article||Scopus||12|
|Jref||Czech refereed journal article||list of refereed journals||10||4|
|B||book or book chapter||world language||40||40|
|D||conference proceedings paper||8|
|P||patent||EPO, USA, Japan||500|
|license-exploited Czech or national patent||200|
|Z||pilot plant, certified technology, variety, breed||100|
|G||prototype, functional sample||40|
|H||results implemented by funding body||40|
|N||certified methodologies and procedures, specialized maps||40|
|V||research report with confidential information||50|
The other result categories in Table 2 comprise applied research results such as patents (P), pilot plants, certified technologies, varieties, and breeds (Z), utility models and industrial designs (F), prototypes and functional samples (G), results implemented by funding body (H, e.g., results implemented in legal documents), certified methodologies and procedures and specialized maps (N), software (R), and research reports with confidential information (V). The highest score here (500) can be assigned to a patent granted by the European Patent Office or by the US or Japanese patent offices. The second highest score (200) is achieved by a national patent (granted by patent offices other than the three above offices), provided the patent is commercially exploited based on a valid license. All other patents receive a unified score of forty. The other applied research results equally obtain forty points each, except for categories Z (100) and V (fifty). The result categories, H and N, are further split into subcategories (with the same score) whose descriptions are not shown in Table 2.
The scores in Table 2 are given for a full research result—they are further distributed to individual universities (or, more generally speaking, to research institutes) according to their share in the result. In principle, outputs are fractionally allocated to universities based on their share of authors. However, domestic and foreign affiliations are weighted differently. Finally, the current methodology employs a score renormalization process, whose goals are the following: (a) prevent excessive growth of results whose existence and quality is difficult to verify, (b) retain the funding proportion between basic and applied research and (c) retain the funding proportion among various disciplinary research areas. The renormalization steps must be taken exactly in the following order:
115% reduction of excessive growth of results of a certain type. Let X2009 be the total score of results of type X yielded in 2009 and X2010 be the total score of results of type X yielded in 2010. If X2010/X2009 > 1.15, then the scores of all results of type X from 2010 shall be multiplied by factor cx:cx = 1.15(X2009/X2010). This step does not concern Jimp results.
Correction of the proportion between basic and applied research results to eighty-five: fifteen. Let SB = J + B + D be the total score of basic research results and SA = P + Z + F + G + H + N + R + V be the total score of applied research results. (Previous methodologies also included result categories, C—basic research—and L, S, and T—applied research.) Let a85 = 0.85(SB + SA)/SB be the correction factor for basic research results and a15 = 0.15(SB + SA)/SA be the correction factor for applied research results. Then, all results of categories J, B and D shall be multiplied by factor, a85, and all results of categories P, Z, F, G, H, N, R, and V shall be multiplied by factor, a15.
Setting of the proportion among various disciplinary research areas. Let ax = px(SB + SA)/X be the correction factor of research area, X, where SB and SA are defined above, X is the total score of results in research area, X, after the corrections described in the two previous steps, and px is the (desired) research area share from Table 3. The results in each research area shall be multiplied by the corresponding correction factor.
The final scores achieved by universities after renormalization are used by the Czech government in the creation of the budget for the support of research institutions. Officially, the scores are not used to rank research institutions in any way.
|3.||mathematics and computer science||5.16|
|10.||arts and humanities||7.75|
3. Results and Discussion
From 2008 to 2011, the universities under investigation more than doubled their overall research output, achieving a total score of 0.73, 1.20, 1.56 and 1.75 million points in the respective years (see Table 4). Thus, there is an increase of 140% in scientific productivity between 2008 and 2011. This can be documented by the year-by-year growth in 2009, 2010 and 2011, which is 65%, 30% and 12%, respectively. Therefore, research productivity is still growing, but the growth is slowing down. As far as the absolute scores of the individual universities are concerned, all of the universities (but two) managed to increase their research output compared to the previous year, sometimes quite remarkably, e.g., Hradec by 131% in 2009 and by 114% in 2010 or Ostrava by 101% in 2009; other times, only modestly, e.g., MU by 3% in 2011, VŠCHT by 5% in 2010 or Charles University (UK) by 5% in 2011. The only exceptions to the “ever-growing” research productivity are VŠE, dropping by 6% in 2011, and the University of Defense (UO) in 2011, which declined by 2%. Note, however, that because of some methodological changes in the research assessment between 2008 and 2011, a 100% score growth does not necessarily mean a twofold productivity.
Now, let us have a look at how the relative shares of universities in the overall research output (produced by twenty-one public science and technology universities) changed between 2008 and 2011. In Figure 1, we can see that Charles University (UK) was the leading institute, with 34% in 2008, followed by ČVUT and MU (other “big” universities), with 12% and 11%, respectively. In 2011 the top three universities remained the same, but UK’s share dropped by five percentage points (see bottom chart in Figure 1). On the other hand, some “small” universities managed to raise their shares, e.g., Olomouc, Budějovice or Plzeň. In Figure 2, the pie charts are quite similar, even though they are based on the number of publications indexed in Web of Science in 2003–2007 (for 2008) and in 2006–2010 (for 2011) that were affiliated with the Czech universities under study. (The publication counts were retrieved in April 2013 using the “Organization-Enhanced” advanced search feature, including all document types from the five main citation databases of the Web of Science by Thomson Reuters.)
The difference between the absolute and relative research output can be seen by comparing the two charts in Figure 3. In the top chart, all universities improve their absolute research performance (except VŠE and UO in 2011), but in the bottom chart, only some of them increase their relative research output, while others decline it. Speaking in relative terms, Charles University (UK) is still the top research university, but its lead is diminishing, other big universities stagnate (ČVUT and MU) and small universities are catching up (the trend is definitely positive for Olomouc and Budějovice). As for the rankings themselves, they are very highly correlated with Spearman’s rho, varying from 0.961 between 2008 and 2011 to 0.992 between 2008 and 2009 (both statistically significant at the 0.01 level two-tailed). However, let us underline again that the scores we are comparing here are not officially meant to be used to create university rankings—they are merely input into the process of research budget creation in the Czech Republic. As for the scientific production of Czech universities as measured by their publication counts in Web of Science in the five years preceding the census years, let us have a look at Figure 4. The growth of absolute publication output is still quite evident (see top chart) and so is (to a smaller extent) the relative production increase of some smaller universities (see bottom chart). Furthermore, the relative decline of Charles University (UK) is less steep. Nevertheless, the rankings of universities based on the methodology described in this paper and those grounded in the productivity indicators from Web of Science in a particular year are very highly positively correlated with Spearman’s correlation coefficients between 0.884 in 2008 and 0.935 in 2011 (always significant at the 0.01 level, two-tailed). For complete information on WoS-indexed publication output, see Table 5, in which we can see that productivity increased by about 49% between 2008 and 2011 and grew by only 13% in the last year.
4. Conclusions and Future Work
The evaluation of scientific research output at the level of institutions has become extremely important in recent years, due to the increasing effort of national governments (and other research funding bodies) to support research, development and innovations as efficiently as possible. In this study, we concentrate on the science evaluation policy in the Czech Republic (which is hardly known in science and technology literature) and present the results of the most recent official assessments (2008–2011) of the research output of twenty-one Czech public universities. The key findings are the following:
The overall research output of the universities under study more than doubled from 2008 and 2011, with virtually all universities increasing their absolute research production each year.
The production growth seems to be slowing down.
Charles University in Prague is still the leading research university in both absolute and relative terms, but its relative share in the total research production is decreasing in favor of smaller universities.
In addition, we have shown that although the current evaluation methodology places some emphasis on applied research, the rankings of universities that can be generated using these assessment reports are very strongly correlated with the rankings based on publication counts from Web of Science. Even if the total production increase between 2008 and 2011 was 240% based on the official methodology and only 49% based on Web of Science publication data, the trends of university research output remained similar. The difference in the overall production growth may be caused by taking into account also non-WoS publications and applied research results, such as patents or prototypes by the official methodology, as well as by the way the points for research results are normalized and distributed to individual institutions in the national assessment. In spite of this, university rankings grounded in Web of Science publication data seem to be a good approximation to the national assessment results. However, there are no official university rankings in the Czech Republic, and even the results of the annual research evaluations are only used to help allocate research funds. Therefore, the rankings presented in this article should be considered “unofficial”, even though they are based on an analysis of official and publicly available data. In our future work, we would like to focus on the updates and modifications of the official science assessment methodology, as well as on other types of research institutions, as well, such as the institutes of the Academy of Sciences of the Czech Republic and on the comparison of the research evaluation systems and university performance in Central European countries.
This work was supported by the European Regional Development Fund (ERDF), project “NTIS—New Technologies for Information Society”, European Centre of Excellence, CZ.1.05/1.1.00/02.0090. Many thanks are due to the anonymous reviewers for their useful comments.
Conflict of Interest
The author declares no conflict of interest.
- Vaněček, J. Bibliometric analysis of the Czech research publications from 1994 to 2005. Scientometrics 2008, 77, 345–360. [Google Scholar] [CrossRef]
- Vaněček, J. Patenting propensity in the Czech Republic. Scientometrics 2008, 75, 381–394. [Google Scholar] [CrossRef]
- Vaněček, J.; Fatun, M.; Albrecht, V. Bibliometric evaluation of the FP-5 and FP-6 results in the Czech Republic. Scientometrics 2010, 83, 103–114. [Google Scholar] [CrossRef]
- Gorraiz, J.; Reimann, R.; Gumpenberger, C. Key factors and considerations in the assessment of international collaboration: A case study for Austria and six countries. Scientometrics 2011, 91, 417–433. [Google Scholar]
- Radosevic, S.; Auriol, L. Patterns of restructuring in research, development and innovation activities in Central and Eastern European countries: An analysis based on S&T indicators. Res. Pol. 1999, 28, 351–376. [Google Scholar] [CrossRef]
- Geuna, A.; Martin, B.R. University research evaluation and funding: An international comparison. Minerva 2003, 41, 277–304. [Google Scholar] [CrossRef]
- Hicks, D. Performance-based university research funding systems. Res. Pol. 2012, 41, 251–261. [Google Scholar] [CrossRef]
- Bucheli, V.; Díaz, A.; Calderón, J.P.; Lemoine, P.; Valdivia, J.A.; Villaveces, J.L.; Zarama, R. Growth of scientific production in Colombian universities: An intellectual capital-based approach. Scientometrics 2012, 91, 369–382. [Google Scholar] [CrossRef]
- Buela-Casal, G.; Paz Bermúdez, M.; Sierra, J.C.; Quevedo-Blasco, R.; Castro, A.; Guillén-Riquelme, A. Ranking 2010 in production and research productivity in Spanish public universities. Psicothema 2011, 23, 527–536. [Google Scholar]
- Li, F.; Yi, Y.; Guo, X.; Qi, W. Performance evaluation of research universities in mainland China, Hong Kong and Taiwan: Based on a two-dimensional approach. Scientometrics 2012, 90, 531–542. [Google Scholar] [CrossRef]
- Matthews, A.P. South African universities in world rankings. Scientometrics 2012, 92, 675–695. [Google Scholar] [CrossRef]
- Wu, H.Y.; Chen, J.K.; Chen, I.S.; Zhuo, H.H. Ranking universities based on performance evaluation by a hybrid MCDM model. Measurement 2012, 45, 856–880. [Google Scholar] [CrossRef]
- Abramo, G.; Cicero, T.; D'Angelo, C.A. A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation. J. Informetrics 2012, 6, 298–306. [Google Scholar] [CrossRef]
- Abramo, G.; D'Angelo, C.A.; Costa, F.D. National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics 2011, 89, 929–941. [Google Scholar] [CrossRef]
- Franceschet, M.; Costantini, A. The first Italian research assessment exercise: A bibliometric perspective. J. Informetrics 2011, 5, 275–291. [Google Scholar] [CrossRef]
- Vanclay, J.K.; Bornmann, L. Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative. Scientometrics 2012, 91, 751–771. [Google Scholar] [CrossRef]
- Research and Development in the Czech Republic. Available online: http://www.vyzkum.cz/ (accessed on 1 April 2013).
© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).