On the Spot Presentation-Based Assessment (OTSPA): Student Perception and Predictive Value of a Novel Summative Assessment with a Formative Assessment Flavour

Round 1

Reviewer 1 Report

see attached file

Comments for author File: Comments.pdf

Quite a few typos and some need to generally tighten up the language/ readability in some areas

Author Response

Dear Reviewer 1, 

Thank you for your input and comments to improve the quality of this paper. We have made changes and provided responses to your comments in the PDF document attached. To improve the standard or the paper after incorporating changes, we have had to revise the paper to ensure it aligns and reads well. We thank you again. 

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear Authors, I find your paper really interesting and valuable to be published. It  is in many ways having high quality, and all the new, inspiring ways of assessments are welcome to higher education, and specifically to veterinary medicine which is a stressful place to study. As you noted. The best and valuable aspect in this study is the one concerning how OTSPA can be used to help assessing and learning the communication skills/general skills/professional skills.  And the authentic assessment aspect.

I have some questions and I point out the aspects that in my mind need clarifications in your paper. I hope to see your paper soon published, BR xxxxxx

------------------------------------------------  

The title: On the Spot Presentation Based Assessments (OTSPA): Student 2 Perception and Predictive Value of a Novel Summative 3 Assessment with a Formative Assessment Flavour

Question 1. Are the participants students? Were they students when they answered the questionnaires? If not, you need to change the title it does not tell students’ perceptions but a set of academics. You need also check the text and replace  ‘student’ by participant etc.  

‘’Participants (N=98) comprised of undergraduate vet- 158 erinary students (years 3-5), recently graduated veterinarians alumni and academics from 159 the School of Veterinary Medicine, University of that had participated in the OTSPA as- 160 sessment from years 2017 to 2020.’’

Question 2. You point out the formative aspect but where is it really? Each assessment has both aspects, summative and formative, if one thinks that summative assessment helps the students to prepare to next assessment task. Based on the literature, it is really difficult to find out or understand that this OTSPA is having any specific formative aspect as it is summative. Or you need to clarify this more. Why do you define it as low-stake and why do you define it formative in comparison to any other assessment during vet education.  

Feedback was given after the assessment and grades were given…so there is not really formative aspect. The predictive value may be just because the students who did well in OTSPA do well in all other assessments also? Maybe they are the students with good study skills etc. They prepare to assessment well.  

The introduction

is otherwise fine but the description of the OPSTA with all details should be removed to Material and Methods, and only mention in brief in introduction.

It should be clearly stated out how the OTSPA is ‘low-stake’ assessment: it is mandatory? It is summative, it is graded, and students need to pass it? And you point out e.g.  that the main point is to test: ‘The main purpose of OTSPA is to test depth of knowledge, knowledge assimilation, 134 knowledge communication, presentation skills, and professionalism’

I would not say this strongly as there are lots of  stress/well-being  literature dealing with how stress can be decreased, such as interventions of mindfulness, self-compassion etc.: ‘’The impact of stress on learning is discussed in literature but there is little discussion 56 on how to better prepare students to manage this stress

Maybe just focus on stress on assessment and how you try to help the students for preparing to future (see e.g. Richardson’s interesting paper on how important predictor is  test anxiety ) Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students' academic performance: a systematic review and meta-analysis. Psychological bulletin, 138(2), 353.

The Materials and Results

I recommend to delete the whole ‘enjoyable’ part. There are many weaknesses. 1) Asking students if they enjoy assessment that is evaluating their competences and grading them, is not a valid way to explore the usefulness or either the appreciation that student show for the specific assessment method. It is clearly easy to know that the answer is mostly no. ‘Do you enjoy ..’ I would delete that part of discussion also. It does not refer literature but is only kind of common chatting. This goes to methods of educational sciences: how to make the questions and I will not go deeper here.

2) Next point is the group work. The OTSPA, based on the description, did not include any group work, or working in groups as in general understood. Further, group working was not assessed. So, the questions 1 and 2 do not seem sensible and it is easy to know that the participants answer like that. These kinds of questions are showing weak method and does not give any valuable new scientific information.

‘’on the survey explored student perceptions to professional 304 skills development during the OTSPAs. Firstly, in response to “I feel that OTSPAs help to 305 develop student participation in group work”, 16.3% of respondents were in agreement 306 and 64.3 % disagree. Secondly, in response to “I feel that OTSPAs are a good way of as- 307 sessing student group work skills”, majority of the respondents disagreed to this (78.5%) 308 while only 8.2% were in agreement,’’

Maybe you are trying to describe if the group is safe it tends to make a good atmosphere in the assessment situation. That part of  theory and methods and discussion need to be revised in line: clarifythe role of group in this assessment situation, or in learning in VET. .  

3) There is no need to repeat the results in text that is already shown by figures. So, combine and tell for example are there differences how the older or younger academics answered. How the student still studying for example know what are the skills needed in work? That was unclear.

Discussion

The interesting and scientifically valid results concerns the authenticity and the communication skills/general skills/professional skills concerning OTSPA. I recommend to focus on those and that brings new to your readers, delete the weak parts (enjoyment, group working).

This is really good and focus on those aspects:

4.3. Future use of OTSPA 474 There are currently limited ways to test communication skills and most are summa- 475 tive high stake assessment like Objective Structured Clinical Examinations (OSCE)s 54 476 which are usually associated with high stress levels. Findings from this study recognise 477 OTSPA as a novel low weightage summative assessment that can be integrated in clinical 478 and non-clinical subjects where better communication skills is an expected outcome as 479 well as promoting a deeper learning of the subject matter among students. The ‘on the 480 spot’ testing, is a unique feature in this assessment although suspected to be the cause of 481 students not enjoying the assessment however, arguably this aspect of the assessment 482 mimics the real-life questions that may come from clients during a consultation. In a con- 483 sultation room, clients are in a position to request explanations for any broad range of 484 animal related topics. A competent veterinarian must be able to assimilate pre-existing 485 knowledge and to communicate to the client in a manner that is easily comprehended. 486 To address the lack of effective communication skills within the work force, the au- 487 thors propose that undergraduates be given the opportunity to practice communication 488 skills at various points within the curriculum by incorporating OTSPAs at various mod- 489 ules.

Author Response

Dear Reviewer 2, 

Thank you for your input and comments to improve the quality of this paper. We have made changes and provided responses to your comments in the PDF document attached. To improve the standard or the paper after incorporating changes, we have had to revise the paper quite significantly to ensure it aligns and reads well. We thank you again. 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments for author File: Comments.pdf

Suggest running this past an education expert in the school just to help with editing and consistency etc. I have tried to give some examples but this is not exhaustive

Author Response

Please see the attachment.

Author Response File: Author Response.pdf