Research Production and International Visibility in Higher Education: The Evolution of Romanian Universities from 2011 to 2019

Abstract

The international visibility of Romanian universities is relatively weak, even when compared with neighboring countries. This is explained by the historical heritage of the former communist regime and the lack of a clear vision for a post-communist strategic policy that could guide the research enterprise. This study examined whether the higher education reforms launched in 2011 had an effect on the research output, impact and international visibility of Romanian universities. The researchers used bibliometric data and university ranking data from ARWU, THE and QS Rankings. Trends in aggregate bibliometric indicators were analyzed for 38 universities as a whole, and by type of institution. Additionally, the Theil index was utilized to measure the degree of concentration of the research output across three institutional types. The findings reveal that there was significant growth in research output during the first years following the 2011 Education Law. However, this growth could not be sustained beyond 2014, except for Teaching & Research Universities and medical universities. This suggests that the reforms are producing the expected results at institutions that implemented and enforced new evaluation systems heavily oriented toward research.

1. Introduction

Both peer-reviewed articles and national and international reports abound with examples of the impact of structural reforms in education and research on the development of universities. For instance, past studies have examined the impact of reforms aimed at increasing institutional diversity and differentiation [1], redefinition of the state role in public management reform [2,3] and mergers of universities [4]. Multiple studies have also focused on the transition from centralized to decentralized education and research systems in Central and East European countries [5], China [6] and Iran [7], showing positive effects in terms of the research output and international visibility of universities.

Following the 1989 Revolution, Romania began to reintegrate its political, economic, social and cultural systems into those of the larger European family. In this transition process, the higher education system was considered to have a key role in developing a research enterprise on par with those of other European nations. The Romanian research system is heavily concentrated in public universities; historically, these universities have had relatively poor performance in international rankings. Additionally, their overall international visibility has suffered because the number of publications authored jointly by Romanians and scholars from other countries has remained one of the lowest in Europe.

The main drivers of this poor performance lie both in the historical heritage of the communist regime and in the lack of a contemporary strategic policy vision embracing the economic importance of the research enterprise, which was seriously underfinanced relative to its real needs. It took more than 20 years for the Ministry of Education to write legislation that would prioritize research and innovation as key instruments of economic development. The 2011 Education Law promoted autonomy, transparency, and university mission diversification, as well as the diversification of sources of funding.

Although these reforms were ambitious in scope, there is limited literature about the results, especially the impact on research performance and international visibility of Romanian universities.

In this paper, we analyze how Romanian universities performed in terms of research output, research impact, and international visibility following the enactment of the 2011 Education Law. We use Percent Growth Rate for Articles to measure the change in research output and Percent Growth Rate for Citations to measure the change in research impact. The international visibility of the Romanian universities is examined for the years following the reforms using two indicators: percentage of articles published with international collaborators and number of Romanian universities listed in international rankings.

2. International Ranking of Universities and the Relationship to Research Production

The global rankings of universities have become both popular with the public and increasingly important for higher education institutions [8]. At the same time, rankings “have become successful as an agenda-setting device for both politicians and for the higher education sector” [9].

The international brain race has also intensified the pressure to perform well in global rankings, which has become more visible and begun to influence government policies in higher education [10,11,12].

In particular, policymakers in Asian countries have focused on improving the performance of their higher education systems as measured by international university rankings, such as the Times Higher Education World University Rankings (THE), the QS World University Rankings (QS), and the Academic Rankings of World Universities (ARWU). By benchmarking the performance of top-tier research universities in the US, these countries made significant strategic investments in higher education with STEM (science, technology, engineering and mathematics) priorities, with the goal of building or adding more of their own world-class universities [13,14,15]. These investments proved highly effective. For example, between 2004 and 2020, the number of Chinese universities listed in the ARWU top 500 increased from 16 to 81. At the same time, the number of universities in the top 500 dropped from 170 to 133 in the US, and from 209 to 185 in Europe. While the dominance of American and European universities has weakened, the representation of Asian universities, particularly from China, has increased rapidly.

3. Drivers of Research Productivity

Research productivity is the key element in the establishment of ranking and rating systems in the Higher Education sector [16]. Research productivity is measured by the number of publications: journal articles, papers in conference proceedings, books, book chapters, awarded research grants and patents [17,18] and differs by impact, which is measured by citations [19].

Journal impact ranking, top percentiles journals (top 1%/10%), collaboration networks (the type and extent of research collaborations—international and industry—in publications), are other bibliometric measures that can be used in research output evaluations at individual, department, institutional, national, or international levels [20].

A study of the correlates and determinants of publication productivity [21] identified three major categories of factors influencing research productivity: individual-level characteristics (motivation, self-direction, attitudes, work habits, cognitive style, ability, work habits, age), environmental variables (the caliber of graduate school training, the prestige of the scientists’ institution affiliation, degree of collegiality within the unit and the organizational freedom in the institutional location) and feedback processes (with focus on reinforcement and cumulative advantage perspectives).

Bland and Ruffin [17] focused on the environmental factors that can stimulate and maintain research productivity, and conducted extensive research that revealed a consistent set of 12 characteristics that can be found in research-conducive environments: (1) clear goals that serve a coordinating function, (2) research emphasis, (3) distinctive culture, (4) positive group climate, (5) assertive participative governance, (6) decentralized organization, (7) frequent communication, (8) accessible resources, particularly human, (9) sufficient size, age and diversity of the research group, (10) appropriate rewards, (11) concentration on recruitment and selection and (12) leadership with research expertise and skill in both initiating appropriate organizational structure and using participatory management practices.

Dunbar [22] organized the factors that impact scientific productivity into two categories: individual (IQ, personality, gender, age, culture of graduate training and culture of employing department) and organizational attributes (institutional structure and leadership, department culture and working conditions).

Based on Fox’s [21] literature review, Teodorescu [23], in a cross-national (US) examination of publication productivity, clustered the variables that have an impact on research productivity into three major categories: individual ascriptive, individual achievement, and institutional characteristics. These variables are considered to have an impact on the publication productivity measured as the self-reported number of journal articles and chapters in academic books that the individual faculty member had published in the three years prior to the survey. Without a doubt, of the three blocks of variables, the institutional category is the most important for policy purposes (faculty development) since it can be manipulated—more or less successfully—by academic leaders [23].

4. Structural Reforms in Romania for Improving International Visibility and Research Performance

Romania is a small country, with approximately 19 m people, belonging to the Central and Eastern Europe (CEE) region. As in other CEE countries, the Romanian Higher education system shares common communist-era characteristics that, despite the variety of independent socio-economic and political paths, explain common patterns in higher education reforms. Under communist regimes, higher education was subject to a high degree of centralization, ideological shaping of the curriculum, and the separation between teaching and research, where the latter was carried out in research institutes outside universities [24].

The expansion of higher education in this region has been accomplished mainly through a significant increase in the number of HEIs. After 30 years, many of the newly created universities are small in size, narrowly specialized, with modest scientific research activity. Chronic underfunding, fragmentation of the financial resources, and “brain drain” of researchers and academic staff, are all possible explanations of the poor performance in international rankings [25].

According to the Ministry of Education [26], the higher education sector in Romania includes 47 public civil universities, seven military institutes and 38 private accredited universities that follow the Bologna structure: a three-year bachelor’s degree followed by a two-year master’s degree and a three-year doctoral degree [27]. Nevertheless, some programs take longer to complete, for example, those in engineering fields (four-year programs), and some combine bachelor’s and master’s degrees into an integrated six-year program (e.g., in medicine and architecture) or four-year programs for Ph.D. studies in medicine.

After the 1989 Revolution, the state gradually abandoned its role as a “system designer” and introduced a series of performance-based mechanisms, which increased university leaders’ autonomy [28]. In 2007, a Presidential Commission released a highly critical report on the state of education and research, concluding that higher education and research systems were in need of far-reaching and substantial reforms and modernization [29]. Following feedback from public debates, a political document, “National Pact for Education”, was adopted and endorsed by all political parties and key stakeholders. Based on the analysis of feedback and the pact, a reform strategy, “Education and Research for the Knowledge Society”, was developed to guide pivotal changes in higher education. These changes included curricular reform, improvements in the management of HEIs, full university autonomy, a classification of universities by their mission statements, rankings of study programs (connected with the financing system) and improved equity in higher education and lifelong learning programs [29].

This strategy would form the foundation for new legislation. The 2011 Education Law brought radical changes to higher education by requiring that all universities be evaluated in order to obtain provisional authorization and accreditation. It also called for a ranking of degree programs based on quality and classification of universities. An evaluation methodology for classifying universities and ranking study programs was launched in 2011 to provide context for the diversification of HEIs and study programs and consistent and transparent data and information on universities and their curricula.

The ranking methodology included more than 60 indicators clustered in four broader dimensions: research, teaching, community engagement and institutional capacity. Based on these dimensions, universities were classified into three categories: Teaching Universities, Teaching and Research Universities and Research-Intensive Universities. This first classification exercise showed that in 2012, there were 12 Research-Intensive Universities, 30 Teaching and Research Universities together with Universities of Teaching and Performing Arts and 48 Teaching Universities in the Romanian higher education system, including both private and public universities. Furthermore, all study programs within universities were ranked into five classes (from A to E) based on their academic quality. These two classification processes were aimed not merely at producing a ranking of universities and programs but also at supporting universities in defining their unique missions and developing strategies for differentiation [27].

These methodologies had an immediate effect on higher education finance, as budget allocations to public universities underwent a sudden and dramatic change when a new mechanism for funding HEIs was implemented. The National Council for Financing Higher Education methodology for allocating core and additional funds began placing a greater emphasis on scientific production. As a result of the 2011 evaluation project, the Research-Intensive Universities were allocated significantly more budgeted spots for their master’s and doctoral programs. Teaching and Research Universities received approximately the same number of spots in master’s programs but fewer in doctoral programs and only for areas in which their programs were included in highly ranked classes. However, Teaching Universities received a drastically reduced number of spots in both master’s and doctoral programs. The new budget allocations immediately translated into much lower core funding for this group [30].

The 2011 Education Law also required that the university classification exercise be followed by an objective institutional evaluation of all universities by an outside international quality agency recognized by the European Quality Assurance Register in Higher Education. The European University Association (EUA) was selected to conduct the evaluation. Among the 30 recommendations advanced in the EUA report, three pointed to the need to create an adequate research infrastructure, provide incentives for fostering institutional alliances and networking and increase international visibility by joining international research projects, encouraging and supporting researchers publishing in foreign languages, inviting international researchers to visit and sending staff abroad [31].

In parallel with the reforms for research evaluation and funding, the doctoral studies were also reorganized as the new “habilitation” process was launched, which imposed higher standards for the scientific activity of the Ph.D. advisors (compared with the previous requirements) in all academic fields.

Lastly, the commissions within the National Council for Attesting University Titles, Diplomas and Certificates were completely revamped after the 2011 Education Law, as more rigorous criteria for research were introduced for candidates who were seeking teaching and research positions and for those who were seeking doctorate degrees.

5. How the 2011 Education Law Impacted the Research Performance and International Visibility of Romanian Universities?

The 2011 Education Law triggered a wave of changes intended to reorganize HEIs through meritocracy by pushing the research priority through income diversification, incentives and performance criteria [32]. First, in 2011–2012, almost all Romanian universities changed their management teams, including rectors, vice-rectors, deans and department directors. Additionally, new structures were created to manage doctoral studies: The Council for Doctorate Studies (the director is equivalent with vice-rector as managerial position) and Doctoral Schools. The methodology used at the national level to fund universities introduced a series of performance-based mechanisms that increased university leaders’ capacity for autonomous decisions.

Given the significant transformations at the national and institutional levels, there has been limited analysis of the impact the 2011 Education Law had on the research performance of Romanian universities. Studies examining the impact of these reforms on the international visibility of Romanian universities are equally scarce. Additionally, there have been few follow-up studies discussing the results of the EUA evaluation and whether the measures taken by universities, especially in the area of faculty evaluation systems, have led to an increase in scientific output and an improvement in international visibility, as measured by their presence in international rankings.

In a study assessing the impact of these reforms, Vîiu, Păunescu and Miroiu [33] noted that the Romanian classification of universities was implicitly hierarchical and produced hierarchical results due to its close association with the ranking of study programs and its heavy reliance on research outputs. The authors argued that a polarized landscape emerged, in which HEIs classified as Teaching Universities included the large majority of poorly performing programs, while those classified as Research-Intensive Universities included the better part of the top-performing programs.

Other studies have suggested that national reforms have not affected universities significantly enough to improve their research productivity [34].

Given the negative views of the recent reforms expressed in the literature, there is a need to use objective data to examine the evolution of key bibliometric research indicators for Romanian universities. In addition, data on international collaborations must be reviewed to assess the impact of recent reforms on the degree of international collaboration via co-authorships for Romanian faculty. Last, data on the presence of Romanian universities in international rankings should be scrutinized to evaluate changes in the international visibility of Romanian universities since the 2011 Education Law.

6. Research Questions

Our analysis sought to examine how the research production and international visibility of Romanian universities changed after the adoption of the 2011 Education Law. Specifically, we sought to answer the following research questions (RQs):

• To what extent has the research output of Romanian universities, as measured by the total number of articles, changed after the 2011 Education Law?
• To what extent has the research impact of Romanian universities, as measured by citations per article, changed after the 2011 Education Law?
• How has the international visibility of Romanian universities, as measured by the percentage of articles published with international collaborators and presence in key international rankings, evolved after the 2011 Education Law?
• How has the degree of concentration in research output across universities changed since the 2011 Education Law?
• Which group of universities have experienced the largest growth since the reforms, in terms of research output, research impact and university rankings?

We hypothesized that after the 2011 Education Law, Romanian universities experienced changes in four key areas: (a) higher research output, (b) improved research impact, (c) increased international visibility and (d) decreased degree of concentration of research production across HEIs.

7. Methods

We started our analysis with the assumption that the earliest year, when the results of the implementation of the 2011–2012 educational reforms could have been observed in the higher education system, was 2013. Therefore, we examined the evolution of research output and impact before and after 2013.

A total of 38 public universities were included in the final analysis based on the following criteria: (a) falls into one of the Research-Intensive Universities, Teaching and Research Universities or Teaching Universities categories and (b) is listed in the Ministry of Education budget for core and additional funding. All metrics were disaggregated by university type according to the 2011 classification. The universities selected from each category are listed in Figure 1.

Searches were performed in the WoS and InCites databases based on the names of the authors’ organizations (organization enhanced), and we selected only the documents directly associated with those affiliation profiles.

Research Output was defined as the number of articles and reviews published in journals that are indexed in Science Citation Expanded, Social Sciences Citation and Arts & Humanities from the WoS database. When counting publications for the examined universities, articles were credited on a whole-count basis (i.e., articles with authors from more than one institution received full credit for each institution on the basis of each author’s institutional address) using the definition provided in the InCites Indicators Handbook [35]. The data were extracted from WoS and InCites between November and December 2020; therefore, some variation in the live databases is possible if the same search criteria were applied after this period.

A Percent Growth Rate for articles (PGRa) was used to calculate the growth in the total number of articles and reviews published during 2013–2018 (A_2013–2018) compared with 2007–2012 (A_2007–2012) (1):

$$PGRa=\left(A_{2013\text{–}2018}-A_{2007\text{–}2012}\right)/A_{2007\text{–}2012}\times 100$$

(1)

Research Impact was evaluated using the Percent Growth Rate for citations (PGRc), measured as the growth in the total number of citations from 2013–2019 (C_2013–2019) for articles published during 2013–2018 (A_2013–2018) compared with the total number of citations from 2007–2013 (C_2007–2013) for articles published during 2007–2012 (A_2007–2012) (2):

$$PGRc=\left(C_{2013\text{–}2019} for A_{2013\text{–}2018}-C_{2007\text{–}2013 }for A_{2007\text{–}2012}\right)/C_{2007\text{–}2013} for A_{2007\text{–}2012}\times 100$$

(2)

Research Impact was measured using the following three indicators:

• average percentile, which represents the average performance of the articles in the dataset having been normalized for field, year and document type; it describes the relative position of an article compared to similar papers in terms of number of citations;
• % publications in top 1% based on citations by category, year and document type; and
• % publications in top 10% based on citations by category, year and document type.

International Visibility was measured with two indicators:

• % publications that have international co-authors; and
• number of universities listed in the major international rankings of universities.

To assess the evolution of Romanian universities in international rankings, we chose the following rankings: ARWU, THE and QS Rankings.

Additionally, a Theil index was calculated for each of the three groups of universities to assess the degree of concentration of research production before and after the 2011 reforms.

8. Findings

8.1. How Has the Research Output of Romanian Universities Changed after the 2011 Education Law?

To benchmark the research production of Romania as a whole, we compared (

Table 1. Bibliometric Indicators for Romania and Benchmark Groups (Data Collected from Clarivate InCites).

Country/Group of Countries	Number of Articles (WoS)		Growth Rate (%) of Number of Articles	Times Cited		Average Percentile for Citations		Citation Impact for Articles from:		% in Top 1% Cited Articles		% in Top 10% Cited Articles		% International Collaborations
	2007–2012	2013–2018		2007–2012	2013–2018	2007–2012	2013–2018	2007–2012	2013–2018	2007–2012	2013–2018	2007–2012	2013–2018	2007–2012	2013–2018
Romania	37,586	48,935	30.19	542,322	598,652	63.63	58.54	14.43	12.23	0.63	1.44	5.97	8.54	35.17	43.72
Poland	118,229	167,645	41.80	2,277,810	2,106,177	58.95	55.25	19.27	12.56	0.76	1.11	6.18	8.12	33.18	36.64
Hungary	34,436	44,245	28.43	976,894	742,712	51.20	51.31	28.37	16.79	1.34	2.02	10.46	11.95	53.78	58.42
EU−28	2,636,255	3,249,104	23.21	83,438,723	52,483,768	46.11	46.76	31.65	16.15	1.36	1.45	12.72	12.87	37.68	45.58
World	7,940,298	10,323,036	31.73	212,119,039	139,239,568	51.41	52.72	26.71	13.49	1.15	1.13	10.74	10.52	19.88	24.19
China	784,538	1800,177	129.45	19,919,786	27,743,876	50.36	48.69	25.39	15.41	1.09	1.42	10.76	12.53	22.85	25.43

) their research productivity to the averages for the world, and of the EU28 group (to which Romania belongs, from 2007). We also selected three countries for benchmarking—Poland, Hungary and China—based on the rationale provided below.

Hungary and Poland are countries belonging to the CEE region that joined the European Union in 2004. Poland’s population is almost double when compared with Romania (approx. 38 million people), while Hungary’s is half (approx. 10 mil. people). All three countries are grouped in the so-called Moderate Innovators category with performance well below the EU average, according to the European Innovation Scoreboard [36].

China was also included in the benchmark group, even though we are aware of differences in size, culture and system. The main reason is related to the rapid growth of research performance in China: while in 1990, slightly over 1% of research papers worldwide had authors from China, this proportion increased to 13% in 2011. In comparison, the United States had authors on 35% of all papers in 1990, while this proportion fell to 26% in 2011. This increase is due to the reforms undertaken in the field of higher education and research, of which the ‘‘211 Project’’ and ‘‘985 Project’’ were most notable, which mainly build high-level national universities by improving their capacity in teaching, research and public service [37]. As a result, many Chinese universities improved their research performance and their position in international rankings, pushing down universities from other regions.

Romanian researchers published 48,935 WoS-indexed articles during 2013–2018, with 30.19% growth compared to the 37,586 articles published in 2007–2012. While this growth rate is higher than the corresponding average for the EU28 group (23.21%), it is lower than the growth experienced by Poland (41.80%) or China (129.45%) (Table 1).

During 2007–2018, Romania ranked 43rd worldwide in terms of the total number of articles indexed in WoS and 48th in terms of citations. Within the EU28 group, Romania ranked 19th in terms of the number of published articles and 21st in terms of citations. If we examine separately the two established periods, we note that during the 2007–2012 period, Romania ranked 40th in the world in terms of the number of articles published and 47th in terms of citations of published articles. During the next period (2013–2018), Romania dropped to the 45th position in terms of the number of articles and 48th in terms of citations. Within the EU28 group, Romania maintained its position for published articles (19th) and citations (21st).

An in-depth examination of the Romanian higher education system and of the universities selected for the analysis reveals major differences in the evolution of research output by university type (Figure 1).

For most Research-Intensive Universities, there was a peak in research output around 2013–2015, followed by modest growth or stagnation. The top three producers in this category were the Polytechnic University of Bucharest, the Babes-Bolyai University of Cluj-Napoca and the Carol Davila University of Medicine and Pharmacy in Bucharest (Figure 1). One university that sustained relatively high growth rates after 2013 was Carol Davila University of Medicine and Pharmacy in Bucharest.

Consistent upward trends after the implementation of the reforms were also observed at two other medical academic centers: Iuliu Hațeganu University of Medicine and Pharmacy in Cluj-Napoca and Grigore T. Popa University of Medicine and Pharmacy in Iași. In general, medical universities were more successful at publishing scholarly articles and maintaining consistent growth over the years.

Sustained growth can be identified within the Teaching and Research Universities group, where 10 of the 15 institutions improved the research outputs constantly after 2013 (Figure 1). The top three producers in 2019 were the Transylvania University of Brașov, the Victor Babeș University of Medicine and Pharmacy of Timișoara and the University of Medicine and Pharmacy of Craiova. As with the Research-Intensive group, the medical academic centers saw higher growth rates than the rest of the universities.

Lastly, the trend data for Teaching Universities indicate that there was no clear pattern of growth in research output before or after 2013 (Figure 1); a notable exception is Ștefan cel Mare University of Suceava, which registered high growth rates after 2013.

The selected Romanian universities experienced a 45.71% growth in the number of articles published between the two selected periods (Figure 2). On average, Teaching and Research Universities witnessed the highest growth rate for articles published (81.57%), followed by Research-Intensive Universities (36.37%).

These results corroborate the findings of the Theil index, which indicate a declining degree of concentration of research output across Romanian universities since 2007.

As shown in Figure 3, the annual growth rate for the number of articles was close to 40% in 2007 and 2008 but decreased in 2009–2011, reaching only 2.1% in 2011. After two years of growth, 2014 was the first year with a decrease in the number of articles published compared to the previous year. In the last two years of the examined period, the selected Romanian universities again showed an increase in scientific production, yielding annual growth rates of 3.5% in 2018 and 14.1% in 2019.

8.2. How Has the Research Impact of Romanian Universities Changed after the 2011 Education Law?

The impact of published research, measured as the number of citations generated in 2013–2020 for articles published in 2013–2018, shows that Hungary has the highest citation rate/article (16.79), whereas the EU28 group has a rate of 16.15 (Table 1). The average for Romania (12.23) was comparable to the average for Poland (12.56) and lower if compared with China (15.41).

Another indicator that provides a representative picture of the quality of research is the average percentile for all publications (measured in terms of citations) (Table 1). The results suggest that researchers in the EU28 countries and China are oriented toward publications positioned mainly in Q1 (Quartile 1, top 25% of the Impact Factor distribution) and Q2 (Quartile 2, between top 50% and top 25%) journals, while authors from Romania and Poland tend to publish an important amount of their work in Q3 and Q4 journals. However, one positive trend for Romanian research is that the average percentile improved during the 2013–2018 period (from 63.63 to 58.54). This positive evolution was also reflected in the higher percentage of articles published in the top 1% and 10% in 2013–2018 compared to 2007–2012.

The growth in citations during the selected periods varies widely within the Romanian higher education system (Figure 2), with the highest growth belonging to Teaching Universities (134.01%), followed by that of the Teaching and Research Universities (114.51%). On average, Research-Intensive Universities witnessed the lowest growth rate for citations (102.02%), if compared to the average for all selected universities (105.26%).

8.3. How Has the International Visibility of Romanian Universities Evolved after the 2011 Education Law?

To assess how Romanian universities fared in the international university rankings following the 2011 Education Law and the ensuing reforms, we selected three representative years for the analysis: 2013, 2016 and 2020 (ARWU) respectively 2021 (the year when THE Rankings and QS Rankings announced the results for 2020). For benchmarking purposes, we included universities ranked in each of the countries analyzed: Romania, Hungary, Poland and China.

Table 2. Number of Universities Included in International Rankings.

Country	Ranking Period	QS Rankings				THE Rankings				ARWU Rankings
		Top 500	Top 700	Top 800	Top 1000	Top 500	Top 800	Top 1000	Top 1001+	Top 500	Top 1000
Romania	2013	-	-	4	n.r. **	-	n.r. **	n.r. **	n.r. **	-	n.r. **
	2016	-	-	4	n.r. **	-	4	n.r. **	n.r. **	-	n.r. **
	2020/2021 *	-	-	-	2	-	1	2	13	-	1
China (Mainland)	2013	17	25	25	n.r. **	9	n.r. **	n.r. **	n.r. **	42	n.r. **
	2016	24	32	33	n.r. **	11	37	n.r. **	n.r. **	54	n.r. **
	2020/2021 *	26	39	45	51	22	50	75	91	81	168
Hungary	2013	-	4	4	n.r. **	-	n.r. **	n.r. **	n.r. **	2	n.r. **
	2016	-	3	6	n.r. **	-	6	n.r. **	n.r. **	-	n.r. **
	2020/2021 *	-	4	4	8	1	3	5	9	-	5
Poland	2013	2	3	6	n.r. **	2	n.r. **	n.r. **	n.r. **	2	n.r. **
	2016	2	3	6	n.r. **	-	7	n.r. **	n.r. **	2	n.r. **
	2020/2021 *	2	3	3	15	-	1	3	19	2	8

* The 2021 rankings for QS and THE Rankings announced in 2020. ** No Ranking in this year.

indicates that Romania, Hungary and Poland are losing ground in international rankings, while more universities in China occupy important positions in the top 500, 800 and 1000.

These results can be explained in part by the modest growth in scientific output in Romania, Poland and Hungary, as the progress made in recent years in these countries was slower than that registered by universities from China or other countries.

While there is stagnation for Hungary and Poland in the rankings, there is a decline for Romanian universities. Specifically, the number of Romanian universities ranked by QS Rankings in the top 800 declined from 4 in 2013 to 0 in the 2021 rankings, and two universities moved down to the top 1000 category. Regarding the Rankings, we identified four Romanian universities ranked in the top 800 in 2016, but only one maintained its position in the 2021 edition, while two universities moved down to the top 1000 category. Only one Romanian university was listed in the ARWU (in the top 1000).

The analysis of the international rankings by university category indicates, as expected, a decline for both Research-Intensive Universities, and Teaching and Research Universities (

Table 3. Romanian Universities in THE, QS and ARWU Rankings.

	University	Year/ Rank	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021
Research Intensive Universities	Polytechnic University of Bucharest	THE									#1001+	#1001+
		QS								#801–1000	#1001+	#1001+
	University of Bucharest	THE					#601–800	#801+	#801–1000	#801–1000	#1001+	#1001+
		QS	#601+	#701+	#651–700	#651–700	#701+	#701+	#701–750	#801–1000	#801–1000	#801–1000
		ARWU							#801–900	#901–1000
	Bucharest University of Economic Studies	THE								#1001+	#801–1000	#601–800
	Technical University of Cluj-Napoca	THE									#1001+	#1001+
		QS										#1001+
	Babeş-Bolyai University of Cluj-Napoca	THE					#501–600	#601–800	#601–800	#801–1000	#801–1000	#1001+
		QS	#601+	#701+	#701+	#701+	#701+	#701+	#801–1000	#801–1000	#801–1000	#801–1000
		ARWU						#601–700	#601–700	#701–800	#701–800
	Iuliu Hatieganu University of Medicine and Pharmacy of Cluj-Napoca	THE								#801–1000		#801–1000
	Alexandru Ioan Cuza University of Iasi	THE					#601–800	#801+	#801–1000	#1001+	#1001+	#1001+
		QS	#601+	#701+	#701+	#701+	#701+	#701 +	#801–1000	#801–1000	#1001+	#1001+
	Polytechnic University of Timişoara	THE									#1001+	#1001+
	Gheorghe Asachi Technical University of Iasi	QS										#1001+
	Grigore T. Popa University of Medicine and Pharmacy of Iaşi	THE							#801–1000	#1001+	#1001+	#1001+
Teaching and Research Universities	West University of Timisoara	THE					#601–800	#601–800	#801–1000	#1001+	#1001+
		QS	#601+	#701+	#701+	#701+	#701+	#701+	#801–1000	#801–1000	#1001+	#1001+
	George Emil Palade University of Medicine, Pharmacy, Science and Technology of Târgu Mureş	THE										#1001+
	Dunărea de Jos University of Galaţi	THE										#1001+
	Transylvania University of Braşov	THE										#1001+

).

8.4. How Has the Degree of Concentration in Research Output across Universities Changed since the 2011 Education Law?

In order to assess the degree of concentration of research production before and after the 2011 reforms, the global Theil index (for all Romanian universities selected for analysis) was calculated for each year:

$$T=\frac{1}{n}{\displaystyle \sum }_{i=1}^n\frac{x_i}{\overline{x}}\log \left(\frac{x_i}{\overline{x}}\right)$$

(3)

where $x_i$ is the number of articles published by a university, $\overline{x}$ is the average number of articles published by the university and $n$ is the number of universities.

The results summarized in Figure 4 indicate a sudden decrease in the degree of concentration, from 0.36 in 2006 to 0.24 in 2011, followed by a moderate decrease down to 0.20 in 2019.

For the analysis by university category, the Theil index was calculated with the formula:

$$T={\displaystyle \sum }_{j=1}^m{y}_j{T}_j+{\displaystyle \sum }_{i=1}^m{y}_j\log \left(\frac{{\overline{x}}_j}{\overline{x}}\right)$$

(4)

where:

• $T_j$ is the Theil index calculated for the universities included in a category;
• $y_j={{\displaystyle \sum }}_{i=1}^{m_j}{x}_i/{{\displaystyle \sum }}_{i=1}^n{x}_i$ represents the share of the number of articles published by universities in a given category, in the total number of articles published by all universities;
• ${\overline{x}}_j$ is the average number of articles published by a university; and
• $m$ is the number of categories used in the university classification (4).

Figure 4 shows a gradual reduction in the degree of concentration for the number of articles published in WoS journals within the Research-Intensive Universities group (D3). At the same time, the level of concentration grew slightly within the Teaching and Research Universities (D2) and the Teaching Universities (D1). This suggests that the production of articles was concentrated at a lower number of institutions for these two groups. There were more variations from one year to the next in the degree of concentration for these two groups, with the highest variability for Teaching Universities.

9. Discussion

Romanian universities registered significant growth in research output during the first years following the enactment of the 2011 Education Law. However, this growth could not be sustained over time, as research output leveled off after 2014 or even declined at some universities. Although the growth rate for the number of citations per article was higher than the growth rate for the number of articles, it remained inferior to the average growth experienced by universities in other countries, which explains the decline of Romanian universities in international rankings, particularly after 2013.

The Teaching and Research Universities experienced the highest growth rate after the adoption of the 2011 Education Law in terms of research output, impact and international visibility. However, although they were well above the national average, the growth rates for these universities lagged behind the rates achieved in other countries, which explains the modest presence of this group in international rankings in recent years. The relatively small number of faculty employed at these universities also contributed to this decline, since these universities tend to be medium-sized.

A more granular analysis shows that a few universities in each category have performed extremely well compared to their institutional type average. What factors led to higher productivity in some universities compared to others? Since the number of faculty employed by most Romanian universities remained approximately the same in both the 2007–2013 and 2013–2018 periods [38], it is unlikely that determinants such as individual attributes [23] and individual-level variables [21] would have affected the evolution of scientific output in 2013–2018.

First, it is clear that the academic medical centers managed to sustain consistent growth in their research output. This positive evolution might explain the relative improvement in international rankings for these institutions. One factor that is linked to the higher publication productivity of faculty in medical universities is the high co-authorship rate that is specific to health sciences, where there are often several authors listed on one publication, thus increasing their productivity [39]. While social sciences and humanities have experienced an upward trend in co-authorship, they continue to be dominated by sole-authored publications and this is often a requirement for promotion and tenure in these disciplines despite a rise in interdisciplinarity [40]. Other factors that explain the higher publication productivity rate among health scientists are the time spent on research activities and the availability of grants and industrial funding. Grants often lead to publishable research [41], which further improves the research productivity of faculty in these disciplines.

Second, outside the group of academic medical centers, the institutions that sustained growth in their research output were those that implemented rigorous faculty evaluation systems and established meritocratic promotion criteria that were heavily weighted toward research, especially toward publications in quality journals and publishing houses. The success obtained by these universities can be attributed to the university managers (rectors), who were able to not only implement such systems but enforce them, often despite significant faculty resistance. One example of an institution that implemented such a system is the Babeș-Bolyai University. As the author of [42] noted, the university’s evaluation system differentiates according to the level of prestige of the journals: Web of Science, IDB publications (international databases other than Web of Science), national databases and other Romanian publications. Conference proceedings are also weighted using a similar ranking system as journal publications. Published books are ranked according to four categories, which distinguish between prestigious international publishing houses (i.e., Cambridge University Press, Blackwell or Sage) and other international publishing houses.

The findings of this study have broader regional implications for other universities in Central and Eastern Europe (CEE) that are working toward improving their international visibility. As a result of the 2011 Education Law, faculty performance in the area of research became the main factor for the evaluation of the internal quality in Romanian universities. The first implication is that by implementing and enforcing rigorous faculty evaluation systems and additional funding systems that are heavily weighted toward research, and in particular toward publications in quality journals, universities can increase and sustain growth in their research output, research impact and international visibility as measured by the extent of international collaborations. Although such systems might be seen by much faculty in the region as neo-liberal, market-oriented policies meant to instigate academic institutions against each other and to increase competition inside academic departments in order to make faculty more productive [42], these reforms do produce results. They also provide an avenue for productive faculty to supplement their wages, while increasing the prestige of their institution. Yet, there is a concern that an overreliance on research will have a negative effect on professors’ teaching and engagement with students. Additionally, the additional funding based on research activity is viewed by some with skepticism, especially when total budget allocations to universities are declining, while faculty have to work outside the 40-h schedule on external grants [41].

It is important to note that university rankings by themselves do not capture well the progress that CEE universities made in key bibliometric indicators such as research output, research impact and extent of international collaborations. As such, these rankings should be used with caution in assessing the research enterprise of universities in this region. Part of the distortion introduced in the CEE university rankings data over the last 10 years can be explained by the accession in rankings of universities in Asia, and especially Chinese universities. Therefore, any analysis of the university rankings should be conducted relative to the other universities in the CEE region, not only globally.

10. Limitations

The analysis at university level was based on two main indicators: number of publications and citations. Although these indicators are offering a realistic image of how research performance evolute after the application of Education Law, more analyses at university level based on “quality” indicators (articles in the top 1% and top 10% journals) could be useful to offer a broader image of this evolution.

In our analysis, we did not disaggregate the universities based on their scientific fields, and this can lead to advantages for some universities, having fundamental sciences and other fields better represented in bibliometric databases, and disadvantages for universities from social sciences and humanities field, where the books and book chapters are representing important research outputs.

In order to improve the robustness of the method, the bibliometric analysis in WoS could be complemented by an examination of Scopus data. Additionally, the analysis of international visibility could be improved by expanding the number of rankings.

The last limitation is how the articles and citations with multiple authors are counted. For instance, if there is an article with two authors from university A and university B, the article is counted twice at both universities. A fractional counting, where each university receives the corresponding proportion (1/2 article for each university in our case), is also possible to influence the final results.

11. Conclusions

Although the higher education reforms in Romania initiated by the 2011 Education Law were highly ambitious in scope, the overall international visibility of Romanian universities has not improved following the reforms. The growth in research output for many Romanian universities was not enough for them to keep up with other universities internationally. As such, most universities experienced stagnation or declined in the rankings. Since a similar trend emerged for other CEE countries (e.g., Poland and Hungary), this trajectory might be explained by the rise of the academic profile of universities in China and a more competitive academic market at the international level.

Despite the decline in international rankings, the trends observed in the bibliometric indicators are promising. Both research production and research impact, as well as international collaborations, improved considerably after the adoption of the 2011 Education Law, particularly during the first three years of implementation. The percentage of articles co-authored internationally grew to a level that is quickly approaching the EU28 average. Although still lagging behind the EU28 average, the number of citations per article more than doubled between 2007–2012 and 2013–2018. Similarly, Romanian scholars witnessed improvements in the average percentile of their published articles and the percentage of articles in the top 1% and top 10% in terms of citations.

Overall, it is evident that the 2011 Education Law had a positive effect on the research enterprise, especially at Teaching and Research Universities. Most universities within this group witnessed sustained annual growth in research output after the 2011 education reform, as standards for faculty evaluation and promotion were raised across the board. This trend suggests that the reforms following the 2011 Education Law are producing the expected results, particularly for this group of institutions. The reforms also had a visible impact on medical universities. All academic medical centers maintained growth in research output for each year following the implementation of the 2011 Education Law.

In parallel with the growth in publications and research impact, there was a gradual decline in the degree of concentration of research publications across the higher education system. Compared to other groups, Research-Intensive Universities experienced the largest decline in the degree of concentration.

The triangulation of the analyzed indicators in the areas of scientific production, impact and visibility confirms that the methodology proposed is appropriate, the period window chosen for the analysis is reasonable, and the methodology can be used for similar analyses in other education systems.