Science Evaluation in the Czech Republic: The Case of Universities

Abstract

In this paper, we review the current official methodology of scientific research output evaluation in the Czech Republic and present a case study on twenty-one Czech public universities. We analyze the results of four successive official research assessment reports from 2008 to 2011 and draw the following main conclusions: (a) the overall research production of the universities more than doubled in the period under investigation, with virtually all universities increasing their absolute research output each year, (b) the total research production growth is slowing down and (c) Charles University in Prague is still the top research university in the Czech Republic in both absolute and relative terms, but its relative share in the total research performance is decreasing in favor of some smaller universities. We also show that the rankings of universities based on the current methodology are quite strongly correlated with established indicators of scientific productivity. This is the first time ever that the official present-day Czech science policy and evaluation methodology along with the results for the Czech university system has been communicated to the international public.

1. Introduction and Related Work

The evaluation of scientific research output has become crucial in recent years, as the budgets of science funding bodies (governments, foundations, etc.) have become tight, but the need for research and innovations has been ongoing or even growing. Therefore, it has become clear that it is absolutely necessary to identify high quality research that should be prioritized in receiving funding and also poor quality research whose funding is no more effective. The key concept here is to promote the advancement of science as efficiently as possible, i.e., to maximally increase the effort/award rate from the point of view of financing science. This is why many countries have introduced various research performance evaluation systems (especially for institutions), some of which are the well-known Research Assessment Exercise (RAE) in the United Kingdom or Excellence in Research for Australia (ERA) in Australia. Science evaluation has also been a hot topic in the Czech Republic in recent years. The Czech government (or more precisely, the Research, Development and Innovation Council—an advisory body to the government) published an official methodology of research output evaluation that later changed several times within a few years. We will review the current methodology (from May 2011) in the following sections and show the results of the last four official research evaluation reports based on this methodology in the context of twenty-one Czech public universities. Although the official methodology should only serve as an input into the process of research budget creation, its application inevitably leads to university rankings, which are part of this paper’s results section. (There are no official university rankings in the Czech Republic.)

The Czech Republic is little covered in science and technology literature. Some of the few studies devoted exclusively to the Czech Republic include bibliometric analyses of Czech research publications [1], patents [2] or European framework program results [3]. Other scientometric studies usually observe the Czech Republic in the context of a larger group of (Central) European countries, e.g., [4] or [5]. As far as the official evaluation of scientific research output in the Czech Republic is concerned, it seems that the Czech research evaluation system is (almost) unknown to the rest of the world: neither [6] nor, more recently, [7] make an explicit mention of the Czech Republic in their comprehensive overviews of university research evaluation and funding systems in different countries. Country-specific research evaluation at the university level is currently a lively topic for scientometricians, as is well documented by the recent studies for Colombian [8], Spanish [9], Chinese [10], South African [11] or Taiwanese [12] universities. Many papers (e.g., [13,14,15,16]) are also concerned with the use of peer review and bibliometric indicators in national university research evaluation and funding systems and argue why the former or latter approach is better, but this is not the intent of this article. We solely present the currently used research evaluation methodology and a case study for universities.

2. Data and Methods

In this study, we concentrated on a set of twenty-one public universities (see Table 1) run by the Ministry of Education, Youth and Sports of the Czech Republic and, in one case, by the Ministry of Defense of the Czech Republic (University of Defense). These universities are also the most highly ranked in the 2011 Research Evaluation Report (the most recent evaluation). Other public universities in the Czech Republic do not conduct research in the fields of science and technology (such as colleges of arts or police academies) and are discarded from this study.

Table 1. List of universities and their acronyms.

University name in English	Acronym
University of South Bohemia in České Budějovice	Budějovice
Czech Technical University in Prague	ČVUT
Czech University of Life Sciences Prague	ČZU
University of Hradec Králové	Hradec
Technical University of Liberec	Liberec
Masaryk University	MU
Mendel University in Brno	MZLU
Palacký University, Olomouc	Olomouc
Silesian University in Opava	Opava
University of Ostrava	Ostrava
University of Pardubice	Pardubice
University of West Bohemia	Plzeň
Charles University in Prague	UK
University of Defense	UO
Jan Evangelista Purkyně University in Ústí nad Labem	Ústí
University of Veterinary and Pharmaceutical Sciences Brno	VFU
VŠB-Technical University of Ostrava	VŠB-TUO
University of Economics, Prague	VŠE
Institute of Chemical Technology, Prague	VŠCHT
Brno University of Technology	VUT
Tomas Bata University in Zlín	Zlín

Table 1. List of universities and their acronyms.

University name in English	Acronym
University of South Bohemia in České Budějovice	Budějovice
Czech Technical University in Prague	ČVUT
Czech University of Life Sciences Prague	ČZU
University of Hradec Králové	Hradec
Technical University of Liberec	Liberec
Masaryk University	MU
Mendel University in Brno	MZLU
Palacký University, Olomouc	Olomouc
Silesian University in Opava	Opava
University of Ostrava	Ostrava
University of Pardubice	Pardubice
University of West Bohemia	Plzeň
Charles University in Prague	UK
University of Defense	UO
Jan Evangelista Purkyně University in Ústí nad Labem	Ústí
University of Veterinary and Pharmaceutical Sciences Brno	VFU
VŠB-Technical University of Ostrava	VŠB-TUO
University of Economics, Prague	VŠE
Institute of Chemical Technology, Prague	VŠCHT
Brno University of Technology	VUT
Tomas Bata University in Zlín	Zlín

2.1. Scores

The official methodology for the evaluation of research output has been slightly modified a few times since 2008, the first year in a series of successive comparable research evaluation reports. (There were research evaluation methodologies and reports before 2008, but they differed from the current methodology to such extent that it would make no sense to compare those evaluations to the current ones. For instance, the reports only considered research results related to completed grant projects, etc. In contrast, the current methodology considers all results.) In the following sections, we will present a short summary of the current methodology (available in Czech [17]) defined by the Czech government in May 2011. In general, the methodology is based on assessing scientific production, i.e., it counts publications and other research results produced and only indirectly (in some cases) on assessing the quality of research output. No citations are counted, but, in the case of journal articles, the journal impact factor is taken into account, which is a de facto cheap estimate of potential citation counts. In this methodology, all research results yielded in the five years preceding the evaluation year are assigned the scores shown in Table 2. For instance, all journal articles indexed in the Web of Science (WoS) database by Thomson Reuters that were published in journals with a nonzero impact factor in the Journal Citation Reports (JCR, edited in the publication year) from 2006 to 2010 will be assigned a score between ten and 305 in the 2011 Evaluation. The score is computed according to the following formula:

J_imp = 10 + 295((1 – N) / (1 + (N/0.057)),

where N is the normalized journal rank obtained from JCR when the journals in its category are sorted by their impact factor (IF) in descending order: N = (P – 1)/(P_max – 1), where P is the journal rank and P_max is the number of journals in the category. If the journal belongs to two or more categories, N is the average normalized rank from all categories. However, there are two cases in which this formula is not needed: if an article is published in the prestigious multidisciplinary journals, Nature or Science, it is assigned a score of 500 without any computation. Articles published in refereed journals without IF (J_noimp) can also get scores, provided they are indexed by the well-known databases, Scopus and/or ERIH (European Reference Index for the Humanities—categories A, B, C). For Scopus, there is a unique score of twelve, whereas for ERIH, there is a distinct score for each journal category, and in addition, articles in journals on “nation-specific” topics, such as history or linguistics, have more weight than articles in other journals. There is also a category for articles that appear in Czech refereed journals (J_ref), whose list is pre-defined and which can also be classified into “national” fields and other fields subcategory. In the case a journal article happens to belong to two or more categories (or subcategories), the highest possible score is considered for that article. Books (B) are rewarded with scores of forty or twenty, depending on the publication language (English, Chinese, French, German, Russian and Spanish are considered “world” languages) and scientific field. Book chapters receive scores proportional to the score of the entire book based on the chapter’s scope within the book. The last result category in basic research are conference proceedings papers (D) indexed in WoS that score eight points each. In addition, any of the above results whose presence in WoS is required must be one of the following document types: article, review, proceedings paper or letter.

Table 2. Research result categories and their scores. ERIH, European Reference Index for the Humanities. EPO, European Patent Office.

Result category				“National” fields	Other fields
Jimp	impacted journal article			10–305
	Nature or Science article			500
Jnoimp	refereed journal article	Scopus		12
		ERIH	A	30	12
			B	20	11
			C	10	10
Jref	Czech refereed journal article	list of refereed journals		10	4
B	book or book chapter	world language		40	40
		other languages			20
D	conference proceedings paper			8
P	patent	EPO, USA, Japan		500
		license-exploited Czech or national patent		200
		other patents		40
Z	pilot plant, certified technology, variety, breed			100
F	utility model			40
	industrial design			40
G	prototype, functional sample			40
H	results implemented by funding body			40
N	certified methodologies and procedures, specialized maps			40
R	software			40
V	research report with confidential information			50

Table 2. Research result categories and their scores. ERIH, European Reference Index for the Humanities. EPO, European Patent Office.

Result category				“National” fields	Other fields
Jimp	impacted journal article			10–305
	Nature or Science article			500
Jnoimp	refereed journal article	Scopus		12
		ERIH	A	30	12
			B	20	11
			C	10	10
Jref	Czech refereed journal article	list of refereed journals		10	4
B	book or book chapter	world language		40	40
		other languages			20
D	conference proceedings paper			8
P	patent	EPO, USA, Japan		500
		license-exploited Czech or national patent		200
		other patents		40
Z	pilot plant, certified technology, variety, breed			100
F	utility model			40
	industrial design			40
G	prototype, functional sample			40
H	results implemented by funding body			40
N	certified methodologies and procedures, specialized maps			40
R	software			40
V	research report with confidential information			50

The other result categories in Table 2 comprise applied research results such as patents (P), pilot plants, certified technologies, varieties, and breeds (Z), utility models and industrial designs (F), prototypes and functional samples (G), results implemented by funding body (H, e.g., results implemented in legal documents), certified methodologies and procedures and specialized maps (N), software (R), and research reports with confidential information (V). The highest score here (500) can be assigned to a patent granted by the European Patent Office or by the US or Japanese patent offices. The second highest score (200) is achieved by a national patent (granted by patent offices other than the three above offices), provided the patent is commercially exploited based on a valid license. All other patents receive a unified score of forty. The other applied research results equally obtain forty points each, except for categories Z (100) and V (fifty). The result categories, H and N, are further split into subcategories (with the same score) whose descriptions are not shown in Table 2.

2.2. Renormalization

The scores in Table 2 are given for a full research result—they are further distributed to individual universities (or, more generally speaking, to research institutes) according to their share in the result. In principle, outputs are fractionally allocated to universities based on their share of authors. However, domestic and foreign affiliations are weighted differently. Finally, the current methodology employs a score renormalization process, whose goals are the following: (a) prevent excessive growth of results whose existence and quality is difficult to verify, (b) retain the funding proportion between basic and applied research and (c) retain the funding proportion among various disciplinary research areas. The renormalization steps must be taken exactly in the following order:

(a)

115% reduction of excessive growth of results of a certain type. Let X₂₀₀₉ be the total score of results of type X yielded in 2009 and X₂₀₁₀ be the total score of results of type X yielded in 2010. If X₂₀₁₀/X₂₀₀₉ > 1.15, then the scores of all results of type X from 2010 shall be multiplied by factor c_x:c_x = 1.15(X₂₀₀₉/X₂₀₁₀). This step does not concern J_imp results.

(b)

Correction of the proportion between basic and applied research results to eighty-five: fifteen. Let SB = J + B + D be the total score of basic research results and SA = P + Z + F + G + H + N + R + V be the total score of applied research results. (Previous methodologies also included result categories, C—basic research—and L, S, and T—applied research.) Let a₈₅ = 0.85(SB + SA)/SB be the correction factor for basic research results and a₁₅ = 0.15(SB + SA)/SA be the correction factor for applied research results. Then, all results of categories J, B and D shall be multiplied by factor, a₈₅, and all results of categories P, Z, F, G, H, N, R, and V shall be multiplied by factor, a₁₅.

(c)

Setting of the proportion among various disciplinary research areas. Let a_x = p_x(SB + SA)/X be the correction factor of research area, X, where SB and SA are defined above, X is the total score of results in research area, X, after the corrections described in the two previous steps, and p_x is the (desired) research area share from Table 3. The results in each research area shall be multiplied by the corresponding correction factor.

The final scores achieved by universities after renormalization are used by the Czech government in the creation of the budget for the support of research institutions. Officially, the scores are not used to rank research institutions in any way.

Table 3. Disciplinary areas and their desired shares.

Disciplinary area		px
1.	social sciences	7.85
2.	engineering	15.60
3.	mathematics and computer science	5.16
4.	physics	15.08
5.	chemical sciences	15.80
6.	Earth sciences	5.06
7.	biological sciences	12.00
8.	agriculture	4.96
9.	medicine	10.74
10.	arts and humanities	7.75
		100.00

Table 3. Disciplinary areas and their desired shares.

Disciplinary area		px
1.	social sciences	7.85
2.	engineering	15.60
3.	mathematics and computer science	5.16
4.	physics	15.08
5.	chemical sciences	15.80
6.	Earth sciences	5.06
7.	biological sciences	12.00
8.	agriculture	4.96
9.	medicine	10.74
10.	arts and humanities	7.75
		100.00

3. Results and Discussion

From 2008 to 2011, the universities under investigation more than doubled their overall research output, achieving a total score of 0.73, 1.20, 1.56 and 1.75 million points in the respective years (see Table 4). Thus, there is an increase of 140% in scientific productivity between 2008 and 2011. This can be documented by the year-by-year growth in 2009, 2010 and 2011, which is 65%, 30% and 12%, respectively. Therefore, research productivity is still growing, but the growth is slowing down. As far as the absolute scores of the individual universities are concerned, all of the universities (but two) managed to increase their research output compared to the previous year, sometimes quite remarkably, e.g., Hradec by 131% in 2009 and by 114% in 2010 or Ostrava by 101% in 2009; other times, only modestly, e.g., MU by 3% in 2011, VŠCHT by 5% in 2010 or Charles University (UK) by 5% in 2011. The only exceptions to the “ever-growing” research productivity are VŠE, dropping by 6% in 2011, and the University of Defense (UO) in 2011, which declined by 2%. Note, however, that because of some methodological changes in the research assessment between 2008 and 2011, a 100% score growth does not necessarily mean a twofold productivity.

Now, let us have a look at how the relative shares of universities in the overall research output (produced by twenty-one public science and technology universities) changed between 2008 and 2011. In Figure 1, we can see that Charles University (UK) was the leading institute, with 34% in 2008, followed by ČVUT and MU (other “big” universities), with 12% and 11%, respectively. In 2011 the top three universities remained the same, but UK’s share dropped by five percentage points (see bottom chart in Figure 1). On the other hand, some “small” universities managed to raise their shares, e.g., Olomouc, Budějovice or Plzeň. In Figure 2, the pie charts are quite similar, even though they are based on the number of publications indexed in Web of Science in 2003–2007 (for 2008) and in 2006–2010 (for 2011) that were affiliated with the Czech universities under study. (The publication counts were retrieved in April 2013 using the “Organization-Enhanced” advanced search feature, including all document types from the five main citation databases of the Web of Science by Thomson Reuters.)

Table 4. Absolute and relative university scores in 2008–2011.

University	2008	%	2009	%	2010	%	2011	%	Δ09	Δ10	Δ11
Budějovice	21,440	2.95	39,082	3.26	55,586	3.55	65,244	3.74	82%	42%	17%
ČVUT	87,631	12.06	155,587	12.96	194,547	13.20	211,796	12.13	78%	25%	9%
ČZU	11,561	1.59	19,023	1.58	30,097	1.86	39,261	2.25	65%	58%	30%
Hradec	1,567	0.22	3,623	0.30	7,739	0.42	10,506	0.60	131%	114%	36%
Liberec	10,200	1.40	14,149	1.18	21,218	1.45	25,653	1.47	39%	50%	21%
MU	78,608	10.82	122,392	10.20	191,667	11.76	197,256	11.30	56%	57%	3%
MZLU	17,024	2.34	23,058	1.92	30,722	1.85	37,076	2.12	35%	33%	21%
Olomouc	40,332	5.55	72,485	6.04	101,708	6.44	122,835	7.04	80%	40%	21%
Opava	4,065	0.56	7,062	0.59	11,649	0.65	12,796	0.73	74%	65%	10%
Ostrava	5,135	0.71	10,318	0.86	18,683	1.08	23,417	1.34	101%	81%	25%
Pardubice	21,670	2.98	39,524	3.29	49,098	3.04	56,925	3.26	82%	24%	16%
Plzeň	20,956	2.88	29,495	2.46	49,036	3.30	62,430	3.58	41%	66%	27%
UK	246,366	33.90	429,261	35.77	487,227	31.25	513,338	29.41	74%	14%	5%
UO	11,870	1.63	18,033	1.50	21,426	1.43	20,993	1.20	52%	19%	−2%
Ústí	5,113	0.70	7,753	0.65	10,794	0.65	13,999	0.80	52%	39%	30%
VFU	8,080	1.11	13,423	1.12	16,599	1.09	18,838	1.08	66%	24%	13%
VŠB-TUO	12,912	1.78	20,670	1.72	35,287	2.27	52,308	3.00	60%	71%	48%
VŠE	12,126	1.67	14,750	1.23	25,529	1.31	24,030	1.38	22%	73%	−6%
VŠCHT	41,734	5.74	62,164	5.18	65,174	4.22	79,556	4.56	49%	5%	22%
VUT	62,100	8.55	88,667	7.39	115,882	8.10	134,934	7.73	43%	31%	16%
Zlín	61,69	0.85	9,701	0.81	17,823	1.06	22,529	1.29	57%	84%	26%
	726,658	100	1,200,220	100	1,557,490	100	1,745,720	100	65%	30%	12%

Table 4. Absolute and relative university scores in 2008–2011.

University	2008	%	2009	%	2010	%	2011	%	Δ09	Δ10	Δ11
Budějovice	21,440	2.95	39,082	3.26	55,586	3.55	65,244	3.74	82%	42%	17%
ČVUT	87,631	12.06	155,587	12.96	194,547	13.20	211,796	12.13	78%	25%	9%
ČZU	11,561	1.59	19,023	1.58	30,097	1.86	39,261	2.25	65%	58%	30%
Hradec	1,567	0.22	3,623	0.30	7,739	0.42	10,506	0.60	131%	114%	36%
Liberec	10,200	1.40	14,149	1.18	21,218	1.45	25,653	1.47	39%	50%	21%
MU	78,608	10.82	122,392	10.20	191,667	11.76	197,256	11.30	56%	57%	3%
MZLU	17,024	2.34	23,058	1.92	30,722	1.85	37,076	2.12	35%	33%	21%
Olomouc	40,332	5.55	72,485	6.04	101,708	6.44	122,835	7.04	80%	40%	21%
Opava	4,065	0.56	7,062	0.59	11,649	0.65	12,796	0.73	74%	65%	10%
Ostrava	5,135	0.71	10,318	0.86	18,683	1.08	23,417	1.34	101%	81%	25%
Pardubice	21,670	2.98	39,524	3.29	49,098	3.04	56,925	3.26	82%	24%	16%
Plzeň	20,956	2.88	29,495	2.46	49,036	3.30	62,430	3.58	41%	66%	27%
UK	246,366	33.90	429,261	35.77	487,227	31.25	513,338	29.41	74%	14%	5%
UO	11,870	1.63	18,033	1.50	21,426	1.43	20,993	1.20	52%	19%	−2%
Ústí	5,113	0.70	7,753	0.65	10,794	0.65	13,999	0.80	52%	39%	30%
VFU	8,080	1.11	13,423	1.12	16,599	1.09	18,838	1.08	66%	24%	13%
VŠB-TUO	12,912	1.78	20,670	1.72	35,287	2.27	52,308	3.00	60%	71%	48%
VŠE	12,126	1.67	14,750	1.23	25,529	1.31	24,030	1.38	22%	73%	−6%
VŠCHT	41,734	5.74	62,164	5.18	65,174	4.22	79,556	4.56	49%	5%	22%
VUT	62,100	8.55	88,667	7.39	115,882	8.10	134,934	7.73	43%	31%	16%
Zlín	61,69	0.85	9,701	0.81	17,823	1.06	22,529	1.29	57%	84%	26%
	726,658	100	1,200,220	100	1,557,490	100	1,745,720	100	65%	30%	12%

Figure 1. Relative university scores in 2008 and 2011.

Figure 1. Relative university scores in 2008 and 2011.

Figure 2. Relative university publication output in 2008 and 2011 by Web of Science (WoS).

Figure 2. Relative university publication output in 2008 and 2011 by Web of Science (WoS).

The difference between the absolute and relative research output can be seen by comparing the two charts in Figure 3. In the top chart, all universities improve their absolute research performance (except VŠE and UO in 2011), but in the bottom chart, only some of them increase their relative research output, while others decline it. Speaking in relative terms, Charles University (UK) is still the top research university, but its lead is diminishing, other big universities stagnate (ČVUT and MU) and small universities are catching up (the trend is definitely positive for Olomouc and Budějovice). As for the rankings themselves, they are very highly correlated with Spearman’s rho, varying from 0.961 between 2008 and 2011 to 0.992 between 2008 and 2009 (both statistically significant at the 0.01 level two-tailed). However, let us underline again that the scores we are comparing here are not officially meant to be used to create university rankings—they are merely input into the process of research budget creation in the Czech Republic. As for the scientific production of Czech universities as measured by their publication counts in Web of Science in the five years preceding the census years, let us have a look at Figure 4. The growth of absolute publication output is still quite evident (see top chart) and so is (to a smaller extent) the relative production increase of some smaller universities (see bottom chart). Furthermore, the relative decline of Charles University (UK) is less steep. Nevertheless, the rankings of universities based on the methodology described in this paper and those grounded in the productivity indicators from Web of Science in a particular year are very highly positively correlated with Spearman’s correlation coefficients between 0.884 in 2008 and 0.935 in 2011 (always significant at the 0.01 level, two-tailed). For complete information on WoS-indexed publication output, see Table 5, in which we can see that productivity increased by about 49% between 2008 and 2011 and grew by only 13% in the last year.

Figure 3. Absolute and relative university scores in 2008–2011.

Figure 3. Absolute and relative university scores in 2008–2011.

Figure 4. Absolute and relative university publication output in 2008–2011 by WoS.

Figure 4. Absolute and relative university publication output in 2008–2011 by WoS.

Table 5. University publication output in 2008–2011 by WoS.

University	2008	%	2009	%	2010	%	2011	%	Δ09	Δ10	Δ11
Budějovice	1,216	4.17	1,417	4.21	1,715	4.47	1,929	4.44	17%	21%	12%
ČVUT	2,846	9.76	3,299	9.81	3,697	9.63	4,111	9.45	16%	12%	11%
ČZU	600	2.06	756	2.25	932	2.43	1,186	2.73	26%	23%	27%
Hradec	157	0.54	172	0.51	229	0.60	306	0.70	10%	33%	34%
Liberec	340	1.17	431	1.28	507	1.32	572	1.32	27%	18%	13%
MU	3,457	11.85	3,902	11.61	4,347	11.32	4,883	11.23	13%	11%	12%
MZLU	613	2.10	697	2.07	816	2.13	968	2.23	14%	17%	19%
Olomouc	1,642	5.63	1,926	5.73	2,227	5.80	2,643	6.08	17%	16%	19%
Opava	183	0.63	180	0.54	206	0.54	223	0.51	−2%	14%	8%
Ostrava	363	1.24	422	1.26	501	1.31	608	1.40	16%	19%	21%
Pardubice	865	2.97	983	2.92	1,136	2.96	1,252	2.88	14%	16%	10%
Plzeň	663	2.27	782	2.33	962	2.51	1,152	2.65	18%	23%	20%
UK	10,787	36.98	12,242	36.41	13,571	35.35	14,909	34.29	13%	11%	10%
UO	7	0.02	7	0.02	7	0.02	6	0.01	0%	0%	−14%
Ústí	129	0.44	150	0.45	191	0.50	252	0.58	16%	27%	32%
VFU	681	2.33	780	2.32	895	2.33	994	2.29	15%	15%	11%
VŠB-TUO	714	2.45	891	2.65	1,149	2.99	1,498	3.45	25%	29%	30%
VŠE	275	0.94	343	1.02	373	0.97	450	1.03	25%	9%	21%
VŠCHT	1,805	6.19	1,945	5.78	2,063	5.37	2,220	5.11	8%	6%	8%
VUT	1,476	5.06	1,855	5.52	2,316	6.03	2,649	6.09	26%	25%	14%
Zlín	353	1.21	442	1.31	546	1.42	671	1.54	25%	24%	23%
	29,172	100	33,622	100	38,386	100	43,482	100	15%	14%	13%

Table 5. University publication output in 2008–2011 by WoS.

University	2008	%	2009	%	2010	%	2011	%	Δ09	Δ10	Δ11
Budějovice	1,216	4.17	1,417	4.21	1,715	4.47	1,929	4.44	17%	21%	12%
ČVUT	2,846	9.76	3,299	9.81	3,697	9.63	4,111	9.45	16%	12%	11%
ČZU	600	2.06	756	2.25	932	2.43	1,186	2.73	26%	23%	27%
Hradec	157	0.54	172	0.51	229	0.60	306	0.70	10%	33%	34%
Liberec	340	1.17	431	1.28	507	1.32	572	1.32	27%	18%	13%
MU	3,457	11.85	3,902	11.61	4,347	11.32	4,883	11.23	13%	11%	12%
MZLU	613	2.10	697	2.07	816	2.13	968	2.23	14%	17%	19%
Olomouc	1,642	5.63	1,926	5.73	2,227	5.80	2,643	6.08	17%	16%	19%
Opava	183	0.63	180	0.54	206	0.54	223	0.51	−2%	14%	8%
Ostrava	363	1.24	422	1.26	501	1.31	608	1.40	16%	19%	21%
Pardubice	865	2.97	983	2.92	1,136	2.96	1,252	2.88	14%	16%	10%
Plzeň	663	2.27	782	2.33	962	2.51	1,152	2.65	18%	23%	20%
UK	10,787	36.98	12,242	36.41	13,571	35.35	14,909	34.29	13%	11%	10%
UO	7	0.02	7	0.02	7	0.02	6	0.01	0%	0%	−14%
Ústí	129	0.44	150	0.45	191	0.50	252	0.58	16%	27%	32%
VFU	681	2.33	780	2.32	895	2.33	994	2.29	15%	15%	11%
VŠB-TUO	714	2.45	891	2.65	1,149	2.99	1,498	3.45	25%	29%	30%
VŠE	275	0.94	343	1.02	373	0.97	450	1.03	25%	9%	21%
VŠCHT	1,805	6.19	1,945	5.78	2,063	5.37	2,220	5.11	8%	6%	8%
VUT	1,476	5.06	1,855	5.52	2,316	6.03	2,649	6.09	26%	25%	14%
Zlín	353	1.21	442	1.31	546	1.42	671	1.54	25%	24%	23%
	29,172	100	33,622	100	38,386	100	43,482	100	15%	14%	13%

4. Conclusions and Future Work

The evaluation of scientific research output at the level of institutions has become extremely important in recent years, due to the increasing effort of national governments (and other research funding bodies) to support research, development and innovations as efficiently as possible. In this study, we concentrate on the science evaluation policy in the Czech Republic (which is hardly known in science and technology literature) and present the results of the most recent official assessments (2008–2011) of the research output of twenty-one Czech public universities. The key findings are the following:

–

The overall research output of the universities under study more than doubled from 2008 and 2011, with virtually all universities increasing their absolute research production each year.

–

The production growth seems to be slowing down.

–

Charles University in Prague is still the leading research university in both absolute and relative terms, but its relative share in the total research production is decreasing in favor of smaller universities.

In addition, we have shown that although the current evaluation methodology places some emphasis on applied research, the rankings of universities that can be generated using these assessment reports are very strongly correlated with the rankings based on publication counts from Web of Science. Even if the total production increase between 2008 and 2011 was 240% based on the official methodology and only 49% based on Web of Science publication data, the trends of university research output remained similar. The difference in the overall production growth may be caused by taking into account also non-WoS publications and applied research results, such as patents or prototypes by the official methodology, as well as by the way the points for research results are normalized and distributed to individual institutions in the national assessment. In spite of this, university rankings grounded in Web of Science publication data seem to be a good approximation to the national assessment results. However, there are no official university rankings in the Czech Republic, and even the results of the annual research evaluations are only used to help allocate research funds. Therefore, the rankings presented in this article should be considered “unofficial”, even though they are based on an analysis of official and publicly available data. In our future work, we would like to focus on the updates and modifications of the official science assessment methodology, as well as on other types of research institutions, as well, such as the institutes of the Academy of Sciences of the Czech Republic and on the comparison of the research evaluation systems and university performance in Central European countries.