Using Peer Review for Student Performance Enhancement: Experiences in a Multidisciplinary Higher Education Setting
, Alicia Tocino 2, Sergio Fortes 3,†, Cristian Martín 4,†, Pere Mercadé-Melé 5,*,†, Rafael Moreno-Sáez 6,†, Antonio Muñoz 4,†, Sara Palomo-Hierro 5,† and Antoni Torres 7,†
Round 1
Reviewer 1 Report
- Establish a weighting. The assessment of students has a direct impact on scores, and the role of students will change. The implementation process may encounter difficulties, and students may even have doubts. However, due to the mistakes of peers or self-assessment discovered during the assessment process, increase their confidence in the skills they have learned.
- It is stated in the article that knowing the person who will be evaluated may result in a biased score, depending on the affinity with the person being evaluated, and it may even be that the rating affects personal relationships, so in order to avoid this situation, clearly define and define Evaluation criteria have become very necessary.
- Anonymous evaluation can be used to alleviate friendship bias. For example, a double-blind review process can be ensured by random assignment. This can be used for courses that are graded in writing. In this case, provide teachers with valuable information resources to support their evaluation more acurrate.
- The traditional passive teaching mode is transformed into a two-way communication of knowledge dissemination. In this process, the transfer of knowledge is mutual, triggering students’ critical reasoning and self-evaluation, increasing their participation, and students can evaluate other students and provide Your own feedback not only helps learning, but also helps teachers improve teaching.
- The article mentions some factors that restrict the use of this technology, such as: no necessary assessment maturity, not taking assessment seriously, and a negative attitude towards the evaluation of these characteristics. Therefore, students need to absorb the content and know what they want to evaluate. It's not just judging scores.
Author Response
REVIEWER 1:
|
Comments |
Responses |
|
1. Establish a weighting. The assessment of students has a direct impact on scores, and the role of students will change. The implementation process may encounter difficulties, and students may even have doubts. However, due to the mistakes of peers or self-assessment discovered during the assessment process, increase their confidence in the skills they have learned. |
We really appreciate your comments. Actually your remark has provided us the chance to add some new clarification in Methodology:
“Also, it is convenient that there is an established weighting, where the qualification of the students influences the final grade of the task, but it is not determinant. In this line, we have not considered students weighting as we are on pilot experiences, but it is considered as ongoing work. The fact that student assessments have a direct impact on the score, is a change in the usual role of students. It is important that students are aware that this activity prepares them, among other benefits, for the relevant critical thinking development in their upcoming professional life, so that without their involvement, the activity is meaningless.” |
|
|
|
|
2. It is stated in the article that knowing the person who will be evaluated may result in a biased score, depending on the affinity with the person being evaluated, and it may even be that the rating affects personal relationships, so in order to avoid this situation, clearly define and define Evaluation criteria have become very necessary.
|
Thank you for your comments. We have taken advantage of your point to clarify description about evaluation criteria and process:
Experiences type I. We have added next part: “The tasks that are being evaluated by peer review consist of solving exercises in which students apply the concepts learned in class following a theoretical approach. For example, the instructor explains the concept of the derivative and its applications and students have to solve a concrete problem that involves the derivative and that has not been solved in class. It means that the proposed tasks are very similar to the exercises in the exam. …. In these experiences, on-site activities have been carried out. Peer review was not implemented in all exercises as we considered that its application only on selected parts of the syllabus is a more effective way to preserve the appealing character of such a new methodology. However, we have relied on the background and expertise of the instructor to choose those exercises that deal with fundamental concepts of the course and which, after previous years' experience, students find difficult to acquire. Between 2-3 activities have been carried out throughout the course, covering around 30-35% of the syllabus so as not to overload the students. Also, in these experiences, each student involved in the peer review has a period of time to carry out the exercises under the guidance of the instructor. These guidelines serve as a reference to accomplish the review of the exercises (of 2-3 peers) that takes place during the second stage. The number of exercises in each activity will be between 3-5. During this time, the student reinforces the knowledge that he/she has acquired after facing the proposed exercises that he/she knew how to solve. Also he/she has solutions to those exercises unavoidable for him/her and in some cases new possible solutions appear. After the correction of the exercises, the instructor supervises all the scores, in order to avoid deviations by students who mistakenly give wrong solutions.”
Experiences type II. We added next additional explanation: “In these experiences, we selected activities in the form of projects to be presented by the students, at all times under the instructor guidance. These projects focused on the fundamental topic of the subject while reinforcing other transverse skills such as public speaking, synthesis, slide preparation, etc. Also, in these experiences, the students had a detailed rubric prepared by the instructor before defending each of their projects, so that they were aware of the items to be assessed, both by their classmates and by the instructor himself. Before starting, it was proposed that the students review the rubric in depth in order to know how it should be applied and resolve any relevant doubt of the process. From this point on, students began to prepare the project under the supervision of the instructor. When the time came to the presentations, the students were in charge of assessing following both the rubrics and the exhibitions of their classmates' projects. It was the instructor's task to make their own evaluations of the presentations, as well as to review each student's.” |
|
3. Anonymous evaluation can be used to alleviate friendship bias. For example, a double-blind review process can be ensured by random assignment. This can be used for courses that are graded in writing. In this case, provide teachers with valuable information resources to support their evaluation more acurrate.
|
We really appreciate your comments and we have enhanced the methodology description section to improve the reproducibility of our research. We have included additional information regarding the performed activities, the evaluation of the results by the teachers, and many other relevant information in this concern. Also, we consider, as ongoing work, to study an anonymous peer review process in order to analyze how the evaluation bias condition results. Current paper was not initially intended to address this issue. |
|
4. The traditional passive teaching mode is transformed into a two-way communication of knowledge dissemination. In this process, the transfer of knowledge is mutual, triggering students’ critical reasoning and self-evaluation, increasing their participation, and students can evaluate other students and provide Your own feedback not only helps learning, but also helps teachers improve teaching.
|
Thank you for your encouraging remark. We believe that our approach can improve student motivation. |
|
5. The article mentions some factors that restrict the use of this technology, such as: no necessary assessment maturity, not taking assessment seriously, and a negative attitude towards the evaluation of these characteristics. Therefore, students need to absorb the content and know what they want to evaluate. It's not just judging scores |
Thanks for this comment, as we commented in a previous answer, criteria evaluation has been clearly defined for students before they start each activity.
We agree that many factors need to be considered since only scoring does not guarantee a successful learning process and this has been included in the text:
“Nevertheless, some factors need to be considered since only scoring does not guarantee a successful learning process. Among them we are aware of no necessary assessment maturity, not taking assessment seriously and a negative attitude towards the evaluation of these characteristics that can restrain the success of this methodology. For this reason, activities are intentionally prepared by the instructor so that they cannot be solved without having a wide understanding of the resolution process. In other words, judging peers' solutions are not about comparing numerical results, but following and reviewing the reasoning process. Instructors have also reported that this has ever encouraged students to boost their learning process.” |
Reviewer 2 Report
The manuscript is of great interest.
The authors carried out many important experiments related to students' peer review. Both the courses studied and the setting of the problem during the experiment are varied. The experimental part is wonderful and very rich. The article can be useful in practical educational activities.
Вut the theoretical part is somewhat inferior. There is a lack of analytical work identifying the key points that influenced the results of the experiments.
small remarks - figure 5 are not very illustrative for presenting key results, perhaps a better display can be found. The description of it is also insufficient.
Author Response
REVIEWER 2:
|
Comments |
Responses |
|
The manuscript is of great interest. |
We would like to show our gratitude for your comment. |
|
The authors carried out many important experiments related to students' peer review. Both the courses studied and the setting of the problem during the experiment are varied. The experimental part is wonderful and very rich. The article can be useful in practical educational activities. |
We appreciate the reviewer's comments for our work and we believe the drawn conclusions can be useful in future experiences in the higher education context. |
|
Вut the theoretical part is somewhat inferior. There is a lack of analytical work identifying the key points that influenced the results of the experiments. |
Thank you for your enriching comment. We have substantially improved sections 2 and 3 where the results and theoretical foundations of the experiences are explained in greater detail. In addition, we have strengthened the analysis and conclusions of the key questions in section 4. |
|
small remarks - figure 5 are not very illustrative for presenting key results, perhaps a better display can be found. The description of it is also insufficient. |
Thank you for the remarks. We really appreciate it. As far as authors have noticed, no figure 5 has been found in the submitted draft. This confusion may have been a result of the already mentioned flaw in the lay out process described in the note to all reviewers.
Authors guess that, in line with other reviewers comments, it may refer to Figures 1 and 2 described in section 3.1. For this reason tags of both figures, as well as section 3.1, have been completely reformulated in order to ease its understanding. |
Reviewer 3 Report
This article examines the concept of peer review in higher education. Research questions include the impact of peer review on academic performance in the form of the final exam and the extent to which peer review is a valid measure of student performance. A total of 6 learning scenarios with over 400 participants are considered. Test scores and variance to instructor evaluations are used as data. The study confirms a high positive influence of peer reviews on academic performance and sees the results of peer review as a valid assessment of student performance.
The article is well structured and examines a very relevant topic. However, the text still has some drawbacks.
- In particular, I have concerns about reproducibility. The learning activities are not described in such a way that they can be reproduced. What exactly do the tasks look like? How are the learning activities structured? Which activities were performed only to ensure the study results, e.g., the evaluation of the results by the teachers, and which activities would also be present in a peer review activity in practice conditions. The description of the study method needs to be more precise. Did all learning scenarios ("Experiences") have the same instructors? How did the learning scenarios relate to each other? Why did they find their way into the study?
- The text provides quite a lot of information, not all of which seems necessary, e.g. the diagrams (see below). The text should be simplified and only the relevant details should be mentioned. At the moment the text is in parts confusing.
- Instuctional Design: what are the guidelines by which the scenarios presented were designed? Were they designed this way by chance? Were they planned together for this study? If so, surely there was a framework underlying the sceneries? Such a framework would be good for practitioners to implement peer review-based learning activities themselves.
- If learning scenarios are described more precisely as well as instructional design, then the text can easily become too long for a single article. From my point of view, it would be good for the comprehensibility of the article to be limited to one research question and to answer the other research questions in a second article. It is difficult for the reader to absorb all the details.
- Selection bias: It is not clearly stated what the sampling is for the treatment group and the control group. If students could self-assign to a group, then there is probably a strong selection bias that could explain the high differences in the final exam between the groups.
- The high differences between the control and treatment group in the final test seem very implausible to me. Just because a peer review took place once in a teaching activity, the final test should be 10-20 percent better? It would be good if the authors could discuss this convincingly, for example, whether there is a selection bias.
Further remarks:
- What tasks / results are being assessed by peers? What are the requirements for a task, which is appropriate for being peer reviewed?
- The experiences lack a detailed description resp. the description is not clear, e.g. Experience 1:
- What is the size of a team?
- “the copies of the activities solved correctly were also handed out so that students had a correction guide”
- “The most complicated part was to prepare the possible resolutions of the proposed activities due to the fact that there are different ways to solve them.” What does this mean? What are “possible resolutions”? It is always possible to solve a task in different ways, isn’t it?!
- “41 of them performed the peer review task while another 40 students did not.” Why were students able to choose? What did those students who decided not take part in the review activity?
- What is the difference between an “academic group” and “course”? What is the implication for this study?
- What is the meaning of the 10-point scale? Where does stem from?f
- The introduction could be structured in subsections, so that the text flow would be improved and information is given only once. In the current form, the introduction reads verbose and long-winded.
- Section 3.1
- This section reads very confusing to me. It is too much information explained too unconcise.
- Why are there different categories of results (NP, 0-5, 5-7, 9-10). Are the categories created by you? Or do they have a deeper meaning? I suggest providing just mean value and standard deviation. Why is the distribution of importance? Why do you provide the diagram in the variants of total numbers and percentage? In my eyes, percentage would be sufficient – if this diagram is essential at all.
- Test 1 and test 2 have to be explained. Why is test 2 printed in bold?
- How was the selection of group A and group B done? By the students themselves? Is there an alternative learning activity for students not taking part in the review activity?
- Research question 2 might be addressed in the conclusions more directly, as it is done for research question 1.
- This section reads very confusing to me. It is too much information explained too unconcise.
Formal comments:
- Is this a new document template? I am missing the line numbers, which help reviewing.
- If group B is the control group, I suggest to name it control group in diagrams as well (and not use group B, as it is harder to understand.
- Please make sure that you use consistently “peer review” as you defined it. There are still occurrences of other terms: “Peer evaluation has been successfully applied and analyzed in the literature. In [14], peer assessment at university is analysed, highlighting”
- „For example, one of the first documented experiences was conducted by [32] in an Education Science” Why not: by Sánchez Rodríguez et al. [32]” ?
- “Finally, our conclusions and future work are presented in Section 4.” -> Section 4 is named Discussion but not conclusion
- Table 2: Column Cat. Needs to be explained better (I guess: C: Content; F: Formal, like a few pages below explained.)
- Professional proofreading required. The language requires improvements as in parts the meaning of the sentences is hardly to understand. In the following some examples:
- “It is also noteworthy the fact”
- “so that student knows at all times which items are going to be scored”
- “It is important to mention the fact that data has been treated at all times in an anonymous way, according to the privacy regulation.“ -> “All data collected have been anonymized”?
- “so that students are able to manage all relevant issues under two units out of four” -> What does this mean?
- “the teacher creates activities for two units out of four.” What does this mean?
Author Response
We thank to reviewer for the effort and valuable reviewer's comments. Please find attached to this message a .pdf file with a point-by-point response to the reviewer's comments.
Author Response File: 
Author Response.pdf
Round 2
Reviewer 3 Report
Dear authors, thank you for the serious and much appreciated improvements of this article on an important teaching approach! Thanks to your explanation, I could understand the study much better.
I spotted only one small issue:
L284: „(see Tables 2,3,and4)“ -missing spaces
