Have Academics’ Citation Patterns Changed in Response to the Rise of World University Rankings? A Test Using First-Citation Speeds
Round 1
Reviewer 1 Report
The study is well structured and very interesting. I recommend:
- The abstract must contain the practical implications of the results found and the added value (novelty) of this study
- The introduction should be more in-depth in terms of the context of the study and should include the originality and parts that make up the study.
- Literature review should be updated with more recent references.
Author Response
Many thanks for your valuable comments. We added sentences to address the practical implications of the results in abstract. We also added more recent references as follows:
Uslu, B. A path for ranking success: what does the expanded indicator-set of international university rankings suggest? High Educ 2020, 80, 949-972, doi:10.1007/s10734-020-00527-0.
Luque-Martínez, T.; Faraoni, N. Meta-ranking to position world universities. Stud High Educ 2020, 45, 819-833.
Selten, F.; Neylon, C.; Huang, C.-K.; Groth, P. A longitudinal analysis of university rankings. Quantitative Science Studies 2020, 1, 1109-1135.
Huang, Y.; Bu, Y.; Ding, Y.; Lu, W. Exploring direct citations between citing publications. J Inform Sci 2020, doi:Artn 0165551520917654
De la Poza, E.; Merello, P.; Barbera, A.; Celani, A. Universities' Reporting on SDGs: Using THE Impact Rankings to Model and Measure Their Contribution to Sustainability. Sustainability 2021, 13, doi:10.3390/su13042038.
Moshtagh, M.; Sotudeh, H. Correlation between universities' Altmetric Attention Scores and their performance scores in Nature Index, Leiden, Times Higher Education and Quacquarelli Symonds ranking systems. J Inform Sci, doi:10.1177/01655515211030868.
We agree with the reviewer’s comments that the introduction should be more in-depth in terms of the context of the study and should include the originality and parts that make up the study. At first, we tried to address the aim of this study in the introduction and explain contexts related to this study in the literature parts, and so it might be shown that the introduction part did not address the context of this study well. Therefore, we tried to revise the introduction based on the reviewer’s comments and added several sentences to address the aim of this study in the introduction part.
“There have been many studies on the impact of WUR on higher educational institutions, but little is known about the citation behavior of academics in response to the rise of the evaluation environment under WURs [1-4]. To investigate the “trickle-down” effects [5] and impact of WURs on academics, this empirical study examines how academics’ citation patterns have changed in response to the rise of an evaluation environment focusing on first-citation speed and probability.”
Reviewer 2 Report
Thank you for the opportunity to review such an interesting research paper, examining citation patterns as influenced by specific ways of academic work evaluation.
The paper presentation is of high quality, with well-developed methodology and clearly presented results. While reading, I only felt need to find two more elements within the paper:
(1) As I was reading literature review, I observed that no reference to other study on citation patterns appeared, nor of studies focusing on WUR, the factor you have chosen as determinant for changes in citation patterns. If your research is the first of its kind, as far as you are concerned, maybe you should better highlight the novelty of it.
(2) In the Discussion section, a summarizing table could be included, to comprise all tested hypothesis and the result (accepted or rejected) for Minerva and SHE. The authors did discuss all cases within the fifth section, however, a centralizing of all situations that could be found more easily, is needed.
Good luck with your future research!
Author Response
Many thanks for your thoughtful and valuable comments. As much as we known, we did not find out the right literature to directly address the impact of WURs on academic’s citation patterns, but we are also cautious to mention directly that this study is the first empirical research to examine the impact of WURs on academic’s citation patterns. This study explored how academic’s citation patterns have changed in the rise of WURs instead of directly analyzing the effect of WURs on academic’s citation behaviors.
But we also agree with the reviewer’s comments that we should better highlight the novelty of this study. We tried to add some sentences about it as follows:
[Introduction parts]
“There have been many studies on the impact of WUR on higher educational institutions, but little is known about the citation behavior of academics in response to the rise of the evaluation environment under WURs.”
[Literature parts]
“However, aside from general assumptions about the impacts of WURs on higher education institutions, little is known about academics’ behaviors in response to the rise of WURs [1-4]. To address this research gap, we focus on the micro-level “reactivity” [6] of academics by studying academics’ citation behaviors. That is, aside from the various reasons for citing the research of others [7], a citation by an academic can be interpreted as an explicit social act [8], which can be assumed to be subject to environmental changes and organizational and professional pressures [9,10]. Considering Goldratt’s [11] insights into measurement-driven behavior (“tell me how you will measure me, and I will tell you how I behave”), we test how citations and, specifically, first-citation speeds have changed amid the rise of WURs.”
We also agree with the reviewer’s comments on the discussion section. We added some sentences to point out the main results of this study as follows:
“As shown in this study, WURs influence the academics’ citation patterns as well as competitive behaviors of higher education institutions. We found that not only has first-citation speed become faster, but first-citation probability has also increased after the WURs appearance. We identified journal impact factors and third-party funding as factors influencing first-citation probability, and we also found that the general effects of influence factors on first citation probability have changed with the emergence of the WURs. We surmised that academics are likely to cite papers granted by journal impact factor and third-party funding agencies in the rise of the evaluation environment under WURs. These citation patterns of academics might strengthen the evaluation social mechanisms of academia and influence the sustainability of science. The findings broaden our understanding of the citation patterns of academics in the rise of WURs and provide practical grounds for research policy as well as higher education policy.”
