Taking the Big Leap: A Case Study on Implementing Programmatic Assessment in an Undergraduate Medical Program

Round 1

Reviewer 1 Report

I enjoyed this well-written case study about the implementation of programmatic assessment (PA) in a new course. The drivers for PA are well laid out and some of the barriers are considered honestly, but could perhaps have been expanded on further. When other institutions are considering major changes, they want to have a realistic understanding of the potential difficulties and how they may or may not be overcome. The assessment system is described well on the whole, but I think the reader would be helped by giving some concrete examples regarding the stakes of different assessments. For example, could a student fail a medium stakes OSCE yet still be promoted at the end of the year? Is every assessment at least low stakes or are some "no stakes"? While the remediation after a promotion decision is well described, I wasn't clear whether there is any remedial intervention at a much earlier stage, well before the promotion decision, as is the case in Cleveland. I really liked the "importance of the narrative" section. However, I found the sentence that begins on line 434 hard to follow.

I wasn't convinced that the quotes in italics at the start of various subsections really added anything, but that may just be my personal preference.

Author Response

Thank you for your thoughtful suggestions and insights for improving our manuscript.

We had the manuscript checked by a professional editor. Those changes are not made visible in the review mode to keep the manuscript readable. Only content-related changes are made visible.

1. The drivers for PA are well laid out and some of the barriers are considered honestly, but could perhaps have been expanded on further. When other institutions are considering major changes, they want to have a realistic understanding of the potential difficulties and how they may or may not be overcome.

We expanded with some further consideration on the hindrances we faced (lines 107-116, 121-132).

2. The assessment system is described well on the whole, but I think the reader would be helped by giving some concrete examples regarding the stakes of different assessments. For example, could a student fail a medium stakes OSCE yet still be promoted at the end of the year? Is every assessment at least low stakes or are some "no stakes"?

Thank you for your comment. We added a description to clarify those aspects (lines 209-213).

3. While the remediation after a promotion decision is well described, I wasn't clear whether there is any remedial intervention at a much earlier stage, well before the promotion decision, as is the case in Cleveland.

Thank you for pointing this out. We do have an alternative path to trigger remediation. We added a brief of this alternative process (lines 360-364).

4. I really liked the "importance of the narrative" section. However, I found the sentence that begins on line 434 hard to follow.

Thank you for mentioning this. We rephrased this section to clarify the analogy (lines 464-468).

5. I wasn't convinced that the quotes in italics at the start of various subsections really added anything, but that may just be my personal preference.

We would appreciate keeping the quotes. They illustrate the overall mindset of the different chapters and seem to appeal to some readers.

Reviewer 2 Report

This is a generally well written descriptive paper providing a detailed case study of PA implementation in a small cohort medical program at a University in Switzerland. The contextual background is helpful to understanding the contextual forces in play (PROFILES, Accreditation and the Swiss Federal Act), and the rationale for choice of PA in the program is effectively argued. The four key concepts, and the nine program elements, used to describe the assessment system provide a helpful framework for understanding both common and unique aspects of the system, and the figures provide helpful illustration of the concepts and ideas discussed.  The reflection on implementation section (and particularly the insight provided thorough section 4.4 focused on narratives) provided additional illumination of the transformative changes required for this kind of new implementation of PA. The points raised in the discussion highlight lessons learned and future direction for the paper.

There are a number of minor issues which impact on the flow and readability of the paper which should be addressed. There is reference made to “avoid exhausting faculty by creating high-stakes MCQs….” at line 84/85 which is not clearly expressed (and again later at lines 374-376).  I’m left unsure how the burden of summative exams was decreased and what alternative to MCQ items was used.

The meaning of +, ++, +++ & (+) should be clarified in Table 1.

There are some minor typographical and grammatical errors throughout the paper which would likely be corrected through and English language editing service.  For example, “changes behavioural” (line 134), “Reflectiv” (line 268), use of period (“.”) after Student Progress Committee instead of colon (“:”) in line 279 (please note this is not an exhaustive list of typographical and grammatical errors).

The sentences in lines 434-436 require further clarification and/or correction.

There’s also a number of instances where the authors assume understanding of abbreviations used (for example use of “MC” and “OSCE” in line 51), or do not provide full detail of the abbreviation with its first use (for example the abbreviation SFA - line 193, and 4C/ID in line 340) (please not this is not an exhaustive list of these instances).

Author Response

Thank you for your thoughtful suggestions and insights for improving our manuscript.

We had the manuscript checked by a professional editor. Those changes are not made visible in the review mode to keep the manuscript readable. Only content-related changes are made visible.

1. There is reference made to “avoid exhausting faculty by creating high-stakes MCQs….” at line 84/85 which is not clearly expressed (and again later at lines 374-376).

I’m left unsure how the burden of summative exams was decreased and what alternative to MCQ items was used.

Thank you for your comments. We reworded the whole paragraph to clarify our point (lines 87-92).
We left lines 418-420 (“We were able to reduce the burden of summative exams…”) unchanged, as the explanation under 2.1.3 should have clarified the point.

2. The meaning of +, ++, +++ & (+) should be clarified in Table 1.

Thank you for noting this. We added a clarifying explanation on use of “+” in the caption of Table 1.

3. The sentences in lines 434-436 require further clarification and/or correction.

Thank you for pointing this out. As some line references did not reference the submitted pdf version of the manuscript, we assumed this comment refers to the “The Accreditation narrative” part.
We rephrased the lines 464-468 and the lines 479-483.

4. There are some minor typographical and grammatical errors throughout the paper which would likely be corrected through and English language editing service. For example, “changes behavioural” (line 134), “Reflectiv” (line 268), use of period (“.”) after Student Progress Committee instead of colon (“:”) in line 279 (please note this is not an exhaustive list of typographical and grammatical errors).

Thank you for pointing out those typos and errors in the manuscript. We apologize for the oversight.

We have corrected the errors and had the manuscript checked by a professional editor. We hope that this has resolved any language issues.

5. There’s also a number of instances where the authors assume understanding of abbreviations used (for example use of “MC” and “OSCE” in line 51), or do not provide full detail of the abbreviation with its first use (for example the abbreviation SFA - line 193, and 4C/ID in line 340) (please note this is not an exhaustive list of these instances).

Thank you for pointing this out. We have checked the manuscript for any instances of undefined abbreviation. We added definitions for MCQ, OSCE, 4C/ID, POCUS, ACGME, USMLE. We also changed all occurrences of MC to MCQ for a unified appellation.

SFA was a typo – it was corrected to ASF.