Peer Review of Reviewers : The Author ’ s Perspective

The aim of this study was to investigate the opinion of authors on the overall quality and effectiveness of reviewers’ contributions to reviewed papers. We employed an on-line survey of thirteen journals which publish articles in the field of life, social or technological sciences. Responses received from 193 authors were analysed using a mixed-effects model in order to determine factors deemed the most important in the authors’ evaluation of the reviewers. Qualitative content analysis of the responses to open questions was performed as well. The mixed-effects model revealed that the authors’ assessment of the competence of referees strongly depended on the final editorial decision and that the speed of the review process was influential as well. In Ordinary Least Squares (OLS) analysis on seven questions detailing authors’ opinions, perception of review speed remained a significant predictor of the assessment. In addition, both the perceived competence and helpfulness of the reviewers significantly and positively affected the authors’ evaluation. New models were used to re-check the value of these two factors and it was confirmed that the assessment of the competence of reviewers strongly depended on the final editorial decision.


Introduction
Peer review is a cornerstone of science and its quality and efficiency depend on a complex, large-scale collaborative process between authors, reviewers, and editors.The efficiency and effectiveness of peer-review cannot be easily shown [1] but both are under permanent examination and re-evaluation.Attempts at improvement have spread in many directions: scoring of reviewers by editors, checklists, and surveys created for reviewers, anonymity of authors for reviewers, educational training for reviewers, post-peer review surveys, etc. [2][3][4][5].Feedback opinion from authors of the submitted articles about the review process may be another very useful source of information.This should contribute to evaluation of the process and creation of guidelines for improvement, but has received less attention than other approaches [4,[6][7][8].On the other hand, the number of authors who are not satisfied with reviewers' reports may be considerable, as shown in a survey of journals dealing with chemistry and related subjects published in Serbia [9].
Recently, the Publons website [10] investigated several aspects of reviewer activity.According to their data, 300,000 individuals each spent about 5 h reviewing, but with large differences depending on the topic and the experience of the reviewer.Less time was dedicated to papers that were very bad or very good.According to Publons [10], reviewers from countries that are English-speaking or have a significant proportion of English speakers wrote longer reports than those from Asia and the Middle East.Review reports in scientific fields where articles are more discussion-based, such as in the social sciences, arts, and humanities, were the longest.When the quantity, variety and distribution of words were examined, it was observed that peer review reports were very different from usual texts [11].A clear shift towards longer reviews was seen as the journal impact factor increased [10].It was postulated that reviewers for high-quality journals considered their task as an investment in their own reputation.The effect of bias in peer review has been examined as well [12][13][14][15].Authors suspected an influence of non-scientific factors on the decision-making process, such as institution of origin, country of origin, different conceptual approach, gender, personal reasons, competitiveness, religion, ethnicity, etc.It was noted that bias most often has negative consequences for the paper and very rarely positive.Thus, many factors influence the work of reviewers.
The aim of this study was to assess the opinion of authors on the overall quality and effectiveness of the peer review process and the reviewers' contribution to their paper.Authors who submitted manuscripts to scientific journals were asked to complete a survey offering their perception of the reviewers' work.

Methods
To gain insight into authors' opinions about reviewers and their reports, an on-line survey (Supplementary Materials) was conducted in thirteen journals (after agreement with their editors).An invitation to participate in the survey was sent to more than 100 journals, aiming to cover different scientific disciplines and places of origin, but positive responses were received from just thirteen.Those enrolled in the study publish in the fields of life, social or technological sciences (Table 1).The survey started in April 2016 and lasted up to October 2017.Corresponding authors were requested to fill in the questionnaire when they received a final decision on their paper.Responses were received from 193 authors.No exact data could be retrieved on the number of invitations sent, as we could not obtain such information from individual journals.Answers to most questions had a semantic differential structure with 1 conveying the most negative opinion on the reviewers' work and 5 the most positive one (Table 2) and were considered as interval-scale variables in the analysis below.
Authors also had the opportunity to add their impressions in the form of open answers, as additional comments.Qualitative data were examined by qualitative content analysis.Authors graded each reviewer's report separately.

General Data
The majority of authors who answered the survey received the final decision "Accepted" for their papers (92.7%).More than half of the responses (54.9%) were received from one journal (Table 1).This result was consistent with the survey on authors' perception of peer review in one leading medical journal [4].Knowing that in most cases the rate of article acceptance is lower than the rate of article rejection [16][17][18], it may be concluded that after receiving a negative decision, authors are not willing to have anything else to do with the journal (including responding to the survey).Although, a reviewer's failure to send a report on time is a major problem in the peer review process-as stated by both editors and authors [6,9,[19][20][21]-in this survey the majority of authors responded that they received reports on time (57.5%) or even fast (29.0%).On average, review reports were received 8.8 weeks after submission.However, the average period for notifying rejection was 16.1 weeks.Almost half of the surveyed authors (45.6%) received two reports, 35.3% had only one, 16.0% received three, 2.6% had four reports, and 0.5% (a single author) five.Most authors stated that the editorial decision was coherent with the opinion of the reviewers (96.8%) and that they will submit another article to the same journal (95.8%).
Most authors believed that reviewers fully understood their work, were competent in the subject, thoroughly read the manuscript, dedicated sufficient time to review and expressed clear comments, which helped the authors to improve their paper (Table 2).The average overall assessment grade for reviewers was 8.7 (on a 1-10 scale) and the distribution of grades is given in Figure 1.A larger range of grades was offered to the authors for overall assessment of the reviewers in order to enable more precise evaluation.Authors' opinions on the review process may be influenced by many factors, such as the outcome (accepted or rejected paper), the journal impact factor and differences in expectations between scientific fields [6].The first factor seemed to be much more influential than the others, as grades given by authors of rejected papers were lower than those given by authors of accepted manuscripts (Table 2).The data show that authors of rejected papers have less confidence in their reviewers.Authors of accepted papers, on the other hand, reacted positively as reviewers "agreed" with them [6,22,23].Our experimental results confirm those of Garcia et al. [22] who employed a model in which authors were seen as Bayesian-rational individuals.They conducted a survey among 156 individual authors of papers published in Scientometrics in the period 2000-2012.The main conclusion was that authors tend to attribute acceptance of their manuscript to their own ability to write a good quality paper, whereas they blame negative bias in peer review as a cause of rejection, displaying a self-serving attributional bias.Since rejection undermines self-confidence and self-worth, authors tend to see external, situational factors as true causes for the negative decision.The same group of researchers created an online tool that can help authors to examine their beliefs about the causes of rejection, as they found that confirmatory bias induces overconfidence whereby one believes more strongly in a preferred hypotheses than one should [23].Due to this confirmatory bias, authors may misinterpret review signals that do not agree with their current ideas.Bias can be defined as a psychological strategy employed to protect self-esteem [24] or as a behavioural tendency to take credit for personal success and deny responsibility for failure [25].Therefore, the results obtained in this study can be explained from the more general aspect, as a protective attitude against self-threat.
Publications 2019, 7, x FOR PEER REVIEW 4 of 10 manuscript to their own ability to write a good quality paper, whereas they blame negative bias in peer review as a cause of rejection, displaying a self-serving attributional bias.Since rejection undermines self-confidence and self-worth, authors tend to see external, situational factors as true causes for the negative decision.The same group of researchers created an online tool that can help authors to examine their beliefs about the causes of rejection, as they found that confirmatory bias induces overconfidence whereby one believes more strongly in a preferred hypotheses than one should [23].Due to this confirmatory bias, authors may misinterpret review signals that do not agree with their current ideas.Bias can be defined as a psychological strategy employed to protect selfesteem [24] or as a behavioural tendency to take credit for personal success and deny responsibility for failure [25].Therefore, the results obtained in this study can be explained from the more general aspect, as a protective attitude against self-threat.
Another difference between authors of accepted and rejected papers was seen in the time invested to fill-in the questionnaire (Table 2).Authors of rejected manuscripts took more time to complete the survey, which can be interpreted as a longer period needed for thought about their answers.

Data Analysis-Mixed-Effects Model
To determine which factors influenced the authors' assessment of reviewers and their work, a mixed-effects model [26] was employed taking into account the overall assessment grade as outcome and including the review time (both in weeks and as authors' perception), the number of referee reports and the editor's final decision as predictors (fixed effects).To take into account that an author could have made more than one submission to these journals and that the papers themselves are grouped into journals sharing the same editorial team, publisher, etc., random effects (random intercepts) were included for both authors and journals.This relatively simple statistical technique allowed us to deal with the problem of non-independent observations in our data [26] (because of the repeated submissions to the same journal and, in some cases, by the same author) which prevented the use of the most common binary tests in the analysis.Moreover, it led to a more clear and comprehensive picture of the factors affecting the authors' assessment (Table 3).Everything else being equal, authors of rejected papers expressed significantly worse opinions on the reviewers' work, while the speed of the review process had a positive effect.Increases in the number of reviewers/reports negatively affected the assessment grade awarded to reviewers, although this Another difference between authors of accepted and rejected papers was seen in the time invested to fill-in the questionnaire (Table 2).Authors of rejected manuscripts took more time to complete the survey, which can be interpreted as a longer period needed for thought about their answers.

Data Analysis-Mixed-Effects Model
To determine which factors influenced the authors' assessment of reviewers and their work, a mixed-effects model [26] was employed taking into account the overall assessment grade as outcome and including the review time (both in weeks and as authors' perception), the number of referee reports and the editor's final decision as predictors (fixed effects).To take into account that an author could have made more than one submission to these journals and that the papers themselves are grouped into journals sharing the same editorial team, publisher, etc., random effects (random intercepts) were included for both authors and journals.This relatively simple statistical technique allowed us to deal with the problem of non-independent observations in our data [26] (because of the repeated submissions to the same journal and, in some cases, by the same author) which prevented the use of the most common binary tests in the analysis.Moreover, it led to a more clear and comprehensive picture of the factors affecting the authors' assessment (Table 3).Everything else being equal, authors of rejected papers expressed significantly worse opinions on the reviewers' work, while the speed of the review process had a positive effect.Increases in the number of reviewers/reports negatively affected the assessment grade awarded to reviewers, although this effect was comparatively weak.
Table 3. Mixed-effects model on the authors' assessment of reviewers' work with random effects (intercepts) for authors and journals.Reference classes were "accepted" for the final decision and "slow" for the speed of review.All t-tests used Satterthwaite's approximation to compute the degrees of freedom.To deepen the analysis of factors leading to the authors' assessment of their reviewers, we ran a factor analysis on seven questions detailing the authors' opinions.Note, to avoid repeated observations of the same individual, we averaged the evaluation of reviewers given by authors, if more than one.This led to the definition of two latent constructs (factors) explaining a large part (81%) of the total variance.The most relevant (MR2, eigenvalue 5.26) was linked to the reviewers' proficiency.More specifically, loadings were especially high for the reviewers' competence, the fact that they carefully read and understood the paper, and the clarity of their comments (Figure 2).The second factor (MR1, eigenvalue 0.39) focused on the helpfulness of reviewers' suggestions.

Group
Publications 2019, 7, x FOR PEER REVIEW 5 of 10 To deepen the analysis of factors leading to the authors' assessment of their reviewers, we ran a factor analysis on seven questions detailing the authors' opinions.Note, to avoid repeated observations of the same individual, we averaged the evaluation of reviewers given by authors, if more than one.This led to the definition of two latent constructs (factors) explaining a large part (81%) of the total variance.The most relevant (MR2, eigenvalue 5.26) was linked to the reviewers' proficiency.More specifically, loadings were especially high for the reviewers' competence, the fact that they carefully read and understood the paper, and the clarity of their comments (Figure 2).The second factor (MR1, eigenvalue 0.39) focused on the helpfulness of reviewers' suggestions.We used these two new factors as predictors, along with the variables already considered in Table 3, in a second model of the overall assessment of reviewers' work as an outcome (Table 4).The review speed perception remained a significant predictor of the assessment.In addition, both the perceived competence and helpfulness of the reviewers significantly and positively affected authors' assessments.It is also worth noting that the final decision was no longer significant in the new model.We used these two new factors as predictors, along with the variables already considered in Table 3, in a second model of the overall assessment of reviewers' work as an outcome (Table 4).The review speed perception remained a significant predictor of the assessment.In addition, both the perceived competence and helpfulness of the reviewers significantly and positively affected authors' assessments.It is also worth noting that the final decision was no longer significant in the new model.Although most of the predictors highlighted in Table 3 turned out not to be significant in the new analysis, it is still possible that they affected the competence and helpfulness factors.
To check that, we employed two new models using these two factors as outcomes.Besides the variables mentioned above, in the new models we also included the author's opinion on the coherence of the editorial decision with the reviewers' reports as a predictor (Tables 5 and 6).
Table 5. OLS estimation of factors predicting authors' assessments of reviewers' competence.Reference classes were "accepted" for the final decision, "slow" for the speed of review, and "yes" for the coherence of the editorial decision.The model on the competence factor was able to explain only 37% of the observed variance (Table 5).In particular, the assessment of the competence of reviewers strongly depended on the final editorial decision.In addition, receiving more reports significantly reduced the assessment grade, possibly because of an unobserved disagreement among reviewers.On the other hand, the model on the helpfulness factor was much weaker, as only the final decision remained significant and the model as a whole explained only a small proportion (9%) of the observed variance (Table 6).

The Content Analysis of Textual Answers
The qualitative content analysis, which enabled systematic overview of textual information by its frequencies in specific categories, was performed using the responses given to the questions listed in Table 7, as well as the additional comments given by authors in free form.The content analysis of the first question extracted three categories of responses: Authors' satisfaction with reviewers' comments (n = 34), Authors dissatisfaction with reviewers' comments (n = 8) and Comments directed to proof reading (n = 8).Authors' satisfaction was further sub-categorized as shown in Table 8.Approximately one third of the responses confirmed that reviewers' input contributed to the quality of the scientific part of the paper.Authors who expressed dissatisfaction with the reviewers' comments named four reasons for such a statement: lack of contribution in general (n = 3), lack of feedback (n = 2), lack of attention (n = 2), and lack of clarification (n = 1).Responses from authors of rejected papers were all within the category of author dissatisfaction with reviewers' comments and suggestions.As can be noted by simply summing authors' statements (n = 50), there were more statements than responses (n = 40), since some authors expressed satisfaction with one aspect of reviewing and dissatisfaction with another.The second open question offered several options of choice in order to identify the possible influence of non-scientific factors on the review and the reviewer's final suggestion, so the participants

Figure 1 .
Figure 1.Distribution of grades given for the overall assessment of reviewers by authors.

Figure 1 .
Figure 1.Distribution of grades given for the overall assessment of reviewers by authors.

Figure 2 .
Figure 2. Factor loadings in the analysis on the variables reflecting the authors' assessment of the reviewers.

Figure 2 .
Figure 2. Factor loadings in the analysis on the variables reflecting the authors' assessment of the reviewers.

Table 1 .
Journals involved in the survey and the number of responses obtained.

Table 2 .
Average grades which authors gave as a response to specific questions.

Table 3 .
Mixed-effects model on the authors' assessment of reviewers' work with random effects (intercepts) for authors and journals.Reference classes were "accepted" for the final decision and "slow" for the speed of review.All t-tests used Satterthwaite's approximation to compute the degrees of freedom.

Table 4 .
Ordinary Least Squares (OLS) estimation of factors predicting authors' assessment of reviewers' work.Reference classes were "accepted" for the final decision and "slow" for the speed of review.

Table 6 .
OLS estimation of factors predicting authors' assessment of reviewers' helpfulness.Reference classes were "accepted" for the final decision, "slow" for the speed of review, and "yes" for the coherence of the editorial decision.

Table 7 .
Average grades which authors gave as a response to open questions.Rej.-authors of rejected papers (number of responses).* If the answer was YES, the author could choose one of the options: institution of origin, country of origin, different conceptual approach, gender, personal reasons, competitiveness, religion, ethnicity, other, _________, and explain.
Acc.-authors of accepted papers,

Table 8 .
Subcategories of authors' satisfaction with reviewers' comments.