Student Reactions to Just-in-Time Formative and Summative Feedback in a Tablet-Based Family Medicine MCQ Exam

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Comments to the authors. 

 

Overall the paper is interesting and the research design is situated in a real context which offers limitations and possibilities. 

 

Regarding the abstract: Rephrase  the first sentence of  the abstract. Perhaps “Regarding digital assessment…. “ or “Digital assessment potential  benefits for universities and educators have been widely discussed, but little is known about the students’ perspective.” 

Regarding the design: The paper mentioned it examines the impact of formative feedback on learning behaviors. However, It is hard to see an examination of learning behaviour since the research does not explore learning tasks after the feedback was received. I would consider changing the learning behavior category to learning attitudes throughout the paper. 

 

One of the sentences reads: “In a digital format, KFQs can be locked, preventing students from returning to previous questions after receiving additional information or feedback. These techniques facilitate testing at higher cognitive levels [9]”  Please explain the reasoning behind this statement. How does not returning the previous questions promote Higher Cognitive Levels? Consider defining Higher Order Thinking levels. HOT requires transferring concepts, abstractions, generalizations, etc. That is, requires addressing situations to apply  and combine concepts. It is the design of the task that would promote HOT or LOT (Lower Order thinking). 

 

Another sentence reads: “Just-in-time FF and SF were incorporated into the tablet-based MCQ assessment at the end of the Family Medicine curriculum in the fifth year of medical school. This integration was undertaken to delineate the capabilities and limitations of MCQs.” Please make explicit how the addition of SF and FF contributes to identifying the limitations of MCQ. Also, there are different dimensions of MCQ  to analyze such as the  validity  (evaluation of the tested  concept), the relation to performance (prediction of medical practice) etc. or perhaps you are interested in understanding  how MCQ  promotes learning…. This statement is unclear. 

 

The paper can be strengthened if you further define “assessment for learning”. Following Dylan Williams, formative feedback can be considered formative or assessment for learning when there is an opportunity to change your learning path, either because a teaching strategy is revised (if the teacher receives the feedback) or a learning strategy is revised. In the context of MCQ, I don't see how the feedback can contribute to teaching and learning since the assessment focus is summative and it is the final exam. Please state the team reasoning behind including FF in a MCQ format where there is no opportunity to change the learning path. 

 

The paper states “The SF categories show that students still want a straightforward conclusion without always seeking a deeper understanding of their mistakes” Please explain how the data leads to this conclusion and how it relates to the positive learning attitudes of students you also find.

 

The discussion section  needs to be separated by paragraph to improve the reading. 

 

The paper states “Although only 2.2% of students explicitly stated that they perceived FF as a learning experience in SS, these categories indicate that a greater number of students unconsciously gained, reflected on, or consolidated knowledge. To further support this, the results in WS 23/24 showed that most students believed the just-in-time FF would have a positive impact on their learning behavior” This statement is confusing. The conditional expresses a possible attitude towards learning (would have a positive impact). Furthermore, it is not clear the methods used to access the unconscious. This needs a lot of clarification. 

 

The paper states: “Follow-up research should further investigate how this type of FF affects the learning behavior of students.” Again, the paper mentioned that learning behaviour is investigated but this statement implies it is not. What I think is being investigated are learning attitudes. 

 

At the end the article mentions “For educators, it may mean a possibility of combining learning activity and reflection elements with assessment”  I think this idea needs to be further developed as this is the core of the paper. It seems complicated to include formative assessment on a final exam when the student does not have an opportunity to use the feedback in a formative way. However, including formative feedback in other learning activities such as practice tests has more sense. This idea is particularly supported with the evidence of this study where students positively valued SF and FF. 

 

Finally, this phrase needs explanation “Autonomy affects the impact of feedback during highly sensitive phases, leaving them empowered in high-stake university examinations”. 

 

Also the article mentions that “overall exam performances were high”, could you provide some data?

Author Response

Dear reviewers,

 

Thank you for your structured and insightful work on our paper. It has been instrumental in revising and presenting our work to the best possible extent. Changes to text passages in the manuscript have been marked in red. On behalf of all authors, I would like to thank you for your support.

Enclosed you will find our statements (bold font) and comments on the concrete steps that were taken to incorporate your feedback into the revision.

 

Reviewer 1; Comments to the Author:

 

Comment 1: Overall the paper is interesting and the research design is situated in a real context which offers limitations and possibilities.

 

Response 1: Thank you very much for the encouraging comments.

 

Comment 2: Regarding the abstract: Rephrase the first sentence of the abstract. Perhaps “Regarding digital assessment…. “or “Digital assessment potential benefits for universities and educators have been widely discussed, but little is known about the students’ perspective.” 

 

Response 2: Thank you for this helpful comment. We agree that the language of the first sentence needs to be enhanced. We have changed the sentenced taking both reviewers comments into account (see lines 9-10).

Comment 3: Regarding the design: The paper mentioned it examines the impact of formative feedback on learning behaviors. However, It is hard to see an examination of learning behaviour since the research does not explore learning tasks after the feedback was received. I would consider changing the learning behavior category to learning attitudes throughout the paper. 

Response 3: We appreciate the constructive and insightful feedback. We agree, that the examination of learning behavior was not conducted; rather, the focus was on subjective perception. In light of this, we have revised the manuscript to replace "learning behavior" with "learning attitudes."

Comment 4: One of the sentences reads: “In a digital format, KFQs can be locked, preventing students from returning to previous questions after receiving additional information or feedback. These techniques facilitate testing at higher cognitive levels [9]”  Please explain the reasoning behind this statement. How does not returning the previous questions promote Higher Cognitive Levels? Consider defining Higher Order Thinking levels. HOT requires transferring concepts, abstractions, generalizations, etc. That is, requires addressing situations to apply  and combine concepts. It is the design of the task that would promote HOT or LOT (Lower Order thinking). 

Response 4: Thank you for pointing this out. Our objective was to highlight the digital characteristic of "locked questions" that we utilize for our tablet-based assessments. This feature enables the precise analysis of students' responses, as they are not influenced by subsequent questions. As you mentioned, students must apply and combine concepts, a necessity that is facilitated by this feature. A paragraph has been incorporated to define HOT levels, and it has been noted that the task's design promotes HOT) (see lines 48-54).

Comment 5: Another sentence reads: “Just-in-time FF and SF were incorporated into the tablet-based MCQ assessment at the end of the Family Medicine curriculum in the fifth year of medical school. This integration was undertaken to delineate the capabilities and limitations of MCQs.” Please make explicit how the addition of SF and FF contributes to identifying the limitations of MCQ. Also, there are different dimensions of MCQ to analyze such as the validity (evaluation of the tested concept), the relation to performance (prediction of medical practice) etc. or perhaps you are interested in understanding how MCQ promotes learning…. This statement is unclear. 

 

Response 5: Thank you for this important objection. MCQs are a widely researched assessment tool. We deliberately focused on the students’ perspectives on feedback and emotions evoked by it. We wanted to gain new information about how MCQs can promote learning and how effective feedback in an assessment setting could be. The inclusion of limitations is misleading. We rephrased the sentence accordingly (see lines 89-94).

 

Comment 6: The paper can be strengthened if you further define “assessment for learning”. Following Dylan Williams, formative feedback can be considered formative or assessment for learning when there is an opportunity to change your learning path, either because a teaching strategy is revised (if the teacher receives the feedback) or a learning strategy is revised. In the context of MCQ, I don't see how the feedback can contribute to teaching and learning since the assessment focus is summative and it is the final exam. Please state the team reasoning behind including FF in a MCQ format where there is no opportunity to change the learning path. 

Response 6: Thank you for this helpful comment. The reasoning behind including FF in a MCQ format was our desire to improve digital assessments. In Germany, as well as in many parts of the world, MCQs are more widespread due to the MCQ-based state exams. Our aim was to try out new feedback methods within an MCQ-based system to explore the student perspective on these changes. Our hypothesis is that by continuously including digital feedback in an MCQ exam, it will be better absorbed, and the exam will become a learning activity. We are currently developing a follow-up study to show these effects over time. We incorporated a paragraph explaining this in the introduction (see lines 58-63).

 

Comment 7: The paper states “The SF categories show that students still want a straightforward conclusion without always seeking a deeper understanding of their mistakes” Please explain how the data leads to this conclusion and how it relates to the positive learning attitudes of students you also find.

Response 7: Thank you for this helpful comment. Indeed, it is important to explain the relation of the contrasting results. Removing negative emotions, planning security, and closure exemplify students who conclude with the assessments without seeking a comprehensive understanding of their errors or the subject matter assessed. The discrepancy in results regarding the positive learning attitudes may be due to the nature of the two feedback methods. SF could lack the impetus to engage with it further. In contrast, FF provides a structured framework and motivation for students to reflect on their errors. We have revised the discussion to emphasize this point (see lines 338-341 and 357-362)

 

Comment 8: The discussion section needs to be separated by paragraph to improve the reading. 

 

Response 8: You are correct, thank you for this notice. We have separated the paragraphs accordingly.

 

Comment 9: The paper states “Although only 2.2% of students explicitly stated that they perceived FF as a learning experience in SS, these categories indicate that a greater number of students unconsciously gained, reflected on, or consolidated knowledge. To further support this, the results in WS 23/24 showed that most students believed the just-in-time FF would have a positive impact on their learning behavior” This statement is confusing. The conditional expresses a possible attitude towards learning (would have a positive impact). Furthermore, it is not clear the methods used to access the unconscious. This needs a lot of clarification. 

 

Response 9: Thank you for pointing this out. We wanted to illustrate the discrepancy that only 2.2% of the students themselves referred to the FF as a learning activity, but in the other categories of the FF a conscious engagement with various aspects of learning was evident (intuition, thought process, existing knowledge, and learning method). In our opinion, it is therefore still clear that FF leads to a deeper examination of the topics surveyed. In addition, the data from WS 23/24 helped to support this, as a subjective change in learning attitude is perceived by the students. This also shows an examination of the feedback regarding possible changes in the learning strategy. We have removed the reference to “the unconscious” as it was misleading and edited the paragraph (see lines 343-352).

 

Comment 10: The paper states: “Follow-up research should further investigate how this type of FF affects the learning behavior of students.” Again, the paper mentioned that learning behaviour is investigated but this statement implies it is not. What I think is being investigated are learning attitudes. 

Response 10: Thank you for this helpful comment. We have revised the manuscript to replace "learning behavior" with "learning attitudes."

Comment 11: At the end the article mentions “For educators, it may mean a possibility of combining learning activity and reflection elements with assessment”  I think this idea needs to be further developed as this is the core of the paper. It seems complicated to include formative assessment on a final exam when the student does not have an opportunity to use the feedback in a formative way. However, including formative feedback in other learning activities such as practice tests has more sense. This idea is particularly supported with the evidence of this study where students positively valued SF and FF. 

Response 11: Thank you for this important comment. We agree in this being the core of the paper. We have suggested two approaches to implicate our suggestion into the curriculum (see lines 365-371).

 

Comment 12: Finally, this phrase needs explanation “Autonomy affects the impact of feedback during highly sensitive phases, leaving them empowered in high-stake university examinations”. 

Response 12: Thank you for this helpful comment. According to Harrison, the receptivity to feedback can be enhanced by assessment cultures that promote students' agency. Optional feedback has the potential to elevate the significance of high-stakes university assessments. Taking both reviewers comments into account, we removed the paragraph as it did not discuss the main findings. We did this to improve the lengthy structure of the discussion.

Comment 13: Also the article mentions that “overall exam performances were high”, could you provide some data?

Response 13: Of course. We mentioned at the beginning of the results, that “the average performance of the exam was comparable for the three semesters, with average scores of 80-85%.” We adjusted the discussion and mentioned the average performance there as well (see line 397).

 

Reviewer 2; Comments to the Author:

 

Comment 1: Thank you for the opportunity to review this manuscript titled "Feedback in Key Feature MCQ: Students’ Reactions to Providing Just-In-Time (Formative and Summative) Feedback During a Tablet-Based MCQ Undergraduate Family Medicine Exam". This study provides an interesting and relevant contribution to International Medical Education, and the authors have presented their findings in a structured manner. However, there are some suggestions to improve the manuscript as follows:

 

Response 1: We thank you very much for the encouraging comments.

 

Comment 2: Title: Consider slightly simplifying for conciseness. E.g., “Student Reactions to Just-In-Time Formative and Summative Feedback in a Tablet-Based Family Medicine MCQ Exam.”

 

Response 2: Thank you for this helpful comment. We agree with this simplification and adjusted the title accordingly (see lines 2-3).

 

Comment 3: Abstract: Minor rewording for fluency: “While the benefits of digital assessments for universities and educators are well-documented, students' perspectives remain underexplored.”

Response 3: Thank you for this helpful comment. We have included this suggestion in our manuscript to improve the quality and fluency (see lines 9-10).

 

Comment 4: Introduction: Line 63-74: Research gap and research question not clearly stated to the reader I suggest to revise them as follows: Lack of Clear Transition Between Concepts: The paragraph shifts abruptly from the importance of post-assessment feedback to the implementation of digital tools in German medical education. While both are relevant, the connection is not clearly articulated.

 

Response 4: Thank you for pointing this out. We agree that the transition is abruptly. Therefore, we have updated the manuscript and integrated a transition (see lines 70-81).

 

Comment 5: Insufficient Elaboration on Prior Work: The paragraph asserts that post-assessment feedback is effective but does not sufficiently review or reference existing studies that support this claim, especially in the context of CBME or digital formats

 

Response 5: thank you for pointing this out. We agree and have included references to the statement mentioned (see line 75).

 

Comment 6: Line 66: Check that between brackets …………test [that] would teach as ……

 

Response 6: Thank you for pointing this out. The quote from Epstein is “the test would teach as well as assess.” Regarding an MCQ assessment, which also corrects mistakes. We therefore included this quote and added “[that]” for fluency.

 

Comment 7: The research question is not clearly stated. I suggest adding, for instance, ‘How does the implementation of tablet-based, just-in-time feedback influence medical students’ motivation, emotional engagement, and learning outcomes within a competency-based medical education framework in Germany?’’

 

Response 7: Thank you for this helpful suggestion. We agree that the addition of the research question elevates the quality of the introduction. Therefore, we added this suggestion (see lines 92-94).

 

Comment 8: 75-76: The objective of the study was mentioned, but it is not clearly defined.

 

Response 8: Thank you for pointing this out. We have revised the paragraph and taken suggestions from both reviewers into account to improve it. We elaborated on the objective and clearly stated a research question (see lines 89-94).

 

Comment 9: Line 89-90: An exploratory mixed-methods study was conducted, including qualitative and quantitative components in the data collection and analysis, which is more suitable than data analysis. Also, elaborate more on which part of data collection is qualitative and which one is quantitative.

 

Response 9: Thank you for your helpful comment. The qualitative component of the study included research questions exploring students’ reactions to FF and SF and emotions evoked by it. The quantitative component descriptively analyzed the categories to see a quantity distribution. We have incorporated a more detailed explanation in the manuscript (see lines 102-105)

 

Comment 10: Line 90-93: Needs to be reworded to be more academic tone.

 

Response 10: Thank you for pointing this out. We have adjusted the wording accordingly and integrated a more academic tone (see lines 105-106).

 

Comment 11: Line 99-100: Rephrase these sentences using combined or complex sentences: (WS 22/23, 97 out of 99 students participated. In SS 23, 115 out of 115 students 99 participated. In WS 23/24, 95 out of 99 students participated).

 

Response 11: Thank you for this helpful suggestion.  We agree that the use of complex sentences elevates the manuscript. We have rephrased the paragraph (see 106-109).

 

Comment 12: Discussion: Break the content into smaller, theme-based paragraphs.

 

Response 12: You are correct, thank you for this notice. We have separated the paragraphs accordingly.

 

Comment 13: Unnecessarily lengthy structure: This paragraph is too long and is trying too hard to encompass multiple topics—student attitudes, motivation, learning behaviors, emotional reactions, pedagogy, evaluation procedures, and digital technology—in one unit. It does not have a distinct internal organization, so it is hard for readers to keep track of the key findings and points.

 

Response 13: Thank you for pointing this out. We agree that the discussion had a lengthy structure. We have revised the discussion and removed the paragraph on autonomy, as it was not among the primary findings. We anticipate that these modifications, which include enhancing the clarity and structure of the discussion, will address the perceived lengthiness.

 

Comment 14: Inconsistent thematic emphasis: There is no logical progression. For instance, thematic discussion of affective responses (e.g., motivation, self-confirmation) is alternated with institutional critique (e.g., Germany’s focus on summative feedback), resulting in fragmented narration.

 

Response 14: Thank you for pointing this out. We modified the discussion according to a new structure: First, discussing the three main findings, including controversies. Second, transitioning to digital assessment and how it is a key element in integrating feedback in assessments. Third, strength and weaknesses of the study.

 

Comment 15: English needs to be revised throughout the entire manuscript; for instance, use clearer structure and transitions, some times Ideas are confused due to long sentences.

 

Response 15: Thank you for pointing this out. We revised the whole manuscript, and a native speaker has proofread the text. We focused on shortening the sentences to give the manuscript a clearer structure.

 

 

 

 

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this manuscript titled "Feedback in Key Feature MCQ: Students’ Reactions to Providing Just-In-Time (Formative and Summative) Feedback During a Tablet-Based MCQ Undergraduate Family Medicine Exam". This study provides an interesting and relevant contribution to International Medical Education, and the authors have presented their findings in a structured manner. However, there are some suggestions to improve the manuscript as follows:

• Title: Consider slightly simplifying for conciseness. E.g., “Student Reactions to Just-In-Time Formative and Summative Feedback in a Tablet-Based Family Medicine MCQ Exam.”
• Abstract: Minor rewording for fluency: “While the benefits of digital assessments for universities and educators are well-documented, students' perspectives remain underexplored.”
• Introduction:
• Line 63-74: Research gap and research question not clearly stated to the reader> I suggest to  revise them as follows:
• Lack of Clear Transition Between Concepts: The paragraph shifts abruptly from the importance of post-assessment feedback to the implementation of digital tools in German medical education. While both are relevant, the connection is not clearly articulated.
• Insufficient Elaboration on Prior Work: The paragraph asserts that post-assessment feedback is effective but does not sufficiently review or reference existing studies that support this claim, especially in the context of CBME or digital formats
• Line 66: Check that between brackets …………test [that] would teach as ……
• The research question is not clearly stated. I suggest adding, for instance, ‘How does the implementation of tablet-based, just-in-time feedback influence medical students’ motivation, emotional engagement, and learning outcomes within a competency-based medical education framework in Germany?’’
• 75-76: The objective of the study was mentioned, but it is not clearly defined.

 

• Method:
• Line 89-90: An exploratory mixed-methods study was conducted, including qualitative and quantitative components in the data collection and analysis, which is more suitable than data analysis. Also, elaborate more on which part of data collection is qualitative and which one is quantitative.
• Line 90-93: Needs to be reworded to be more academic tone.
• Line 99-100: Rephrase these sentences using combined or complex sentences: (WS 22/23, 97 out of 99 students participated. In SS 23, 115 out of 115 students 99 participated. In WS 23/24, 95 out of 99 students participated).

 

• Discussion: Break the content into smaller, theme-based paragraphs.
•  Unnecessarily lengthy structure: This paragraph is too long and is trying too hard to encompass multiple topics—student attitudes, motivation, learning behaviors, emotional reactions, pedagogy, evaluation procedures, and digital technology—in one unit. It does not have a distinct internal organization, so it is hard for readers to keep track of the key findings and points.
• Inconsistent thematic emphasis: There is no logical progression. For instance, thematic discussion of affective responses (e.g., motivation, self-confirmation) is alternated with institutional critique (e.g., Germany’s focus on summative feedback), resulting in fragmented narration.

Comments on the Quality of English Language

English needs to be revised throughout the entire manuscript; for instance, use clearer structure and transitions, some times Ideas are confused due to long sentences.

Author Response

Dear reviewers,

 

Thank you for your structured and insightful work on our paper. It has been instrumental in revising and presenting our work to the best possible extent. Changes to text passages in the manuscript have been marked in red. On behalf of all authors, I would like to thank you for your support.

Enclosed you will find our statements (bold font) and comments on the concrete steps that were taken to incorporate your feedback into the revision.

 

 

Reviewer 2; Comments to the Author:

 

Comment 1: Thank you for the opportunity to review this manuscript titled "Feedback in Key Feature MCQ: Students’ Reactions to Providing Just-In-Time (Formative and Summative) Feedback During a Tablet-Based MCQ Undergraduate Family Medicine Exam". This study provides an interesting and relevant contribution to International Medical Education, and the authors have presented their findings in a structured manner. However, there are some suggestions to improve the manuscript as follows:

 

Response 1: We thank you very much for the encouraging comments.

 

Comment 2: Title: Consider slightly simplifying for conciseness. E.g., “Student Reactions to Just-In-Time Formative and Summative Feedback in a Tablet-Based Family Medicine MCQ Exam.”

 

Response 2: Thank you for this helpful comment. We agree with this simplification and adjusted the title accordingly (see lines 2-3).

 

Comment 3: Abstract: Minor rewording for fluency: “While the benefits of digital assessments for universities and educators are well-documented, students' perspectives remain underexplored.”

Response 3: Thank you for this helpful comment. We have included this suggestion in our manuscript to improve the quality and fluency (see lines 9-10).

 

Comment 4: Introduction: Line 63-74: Research gap and research question not clearly stated to the reader I suggest to revise them as follows: Lack of Clear Transition Between Concepts: The paragraph shifts abruptly from the importance of post-assessment feedback to the implementation of digital tools in German medical education. While both are relevant, the connection is not clearly articulated.

 

Response 4: Thank you for pointing this out. We agree that the transition is abruptly. Therefore, we have updated the manuscript and integrated a transition (see lines 70-81).

 

Comment 5: Insufficient Elaboration on Prior Work: The paragraph asserts that post-assessment feedback is effective but does not sufficiently review or reference existing studies that support this claim, especially in the context of CBME or digital formats

 

Response 5: thank you for pointing this out. We agree and have included references to the statement mentioned (see line 75).

 

Comment 6: Line 66: Check that between brackets …………test [that] would teach as ……

 

Response 6: Thank you for pointing this out. The quote from Epstein is “the test would teach as well as assess.” Regarding an MCQ assessment, which also corrects mistakes. We therefore included this quote and added “[that]” for fluency.

 

Comment 7: The research question is not clearly stated. I suggest adding, for instance, ‘How does the implementation of tablet-based, just-in-time feedback influence medical students’ motivation, emotional engagement, and learning outcomes within a competency-based medical education framework in Germany?’’

 

Response 7: Thank you for this helpful suggestion. We agree that the addition of the research question elevates the quality of the introduction. Therefore, we added this suggestion (see lines 92-94).

 

Comment 8: 75-76: The objective of the study was mentioned, but it is not clearly defined.

 

Response 8: Thank you for pointing this out. We have revised the paragraph and taken suggestions from both reviewers into account to improve it. We elaborated on the objective and clearly stated a research question (see lines 89-94).

 

Comment 9: Line 89-90: An exploratory mixed-methods study was conducted, including qualitative and quantitative components in the data collection and analysis, which is more suitable than data analysis. Also, elaborate more on which part of data collection is qualitative and which one is quantitative.

 

Response 9: Thank you for your helpful comment. The qualitative component of the study included research questions exploring students’ reactions to FF and SF and emotions evoked by it. The quantitative component descriptively analyzed the categories to see a quantity distribution. We have incorporated a more detailed explanation in the manuscript (see lines 102-105)

 

Comment 10: Line 90-93: Needs to be reworded to be more academic tone.

 

Response 10: Thank you for pointing this out. We have adjusted the wording accordingly and integrated a more academic tone (see lines 105-106).

 

Comment 11: Line 99-100: Rephrase these sentences using combined or complex sentences: (WS 22/23, 97 out of 99 students participated. In SS 23, 115 out of 115 students 99 participated. In WS 23/24, 95 out of 99 students participated).

 

Response 11: Thank you for this helpful suggestion.  We agree that the use of complex sentences elevates the manuscript. We have rephrased the paragraph (see 106-109).

 

Comment 12: Discussion: Break the content into smaller, theme-based paragraphs.

 

Response 12: You are correct, thank you for this notice. We have separated the paragraphs accordingly.

 

Comment 13: Unnecessarily lengthy structure: This paragraph is too long and is trying too hard to encompass multiple topics—student attitudes, motivation, learning behaviors, emotional reactions, pedagogy, evaluation procedures, and digital technology—in one unit. It does not have a distinct internal organization, so it is hard for readers to keep track of the key findings and points.

 

Response 13: Thank you for pointing this out. We agree that the discussion had a lengthy structure. We have revised the discussion and removed the paragraph on autonomy, as it was not among the primary findings. We anticipate that these modifications, which include enhancing the clarity and structure of the discussion, will address the perceived lengthiness.

 

Comment 14: Inconsistent thematic emphasis: There is no logical progression. For instance, thematic discussion of affective responses (e.g., motivation, self-confirmation) is alternated with institutional critique (e.g., Germany’s focus on summative feedback), resulting in fragmented narration.

 

Response 14: Thank you for pointing this out. We modified the discussion according to a new structure: First, discussing the three main findings, including controversies. Second, transitioning to digital assessment and how it is a key element in integrating feedback in assessments. Third, strength and weaknesses of the study.

 

Comment 15: English needs to be revised throughout the entire manuscript; for instance, use clearer structure and transitions, some times Ideas are confused due to long sentences.

 

Response 15: Thank you for pointing this out. We revised the whole manuscript, and a native speaker has proofread the text. We focused on shortening the sentences to give the manuscript a clearer structure.

 

Reviewer 1; Comments to the Author:

 

Comment 1: Overall the paper is interesting and the research design is situated in a real context which offers limitations and possibilities.

 

Response 1: Thank you very much for the encouraging comments.

 

Comment 2: Regarding the abstract: Rephrase the first sentence of the abstract. Perhaps “Regarding digital assessment…. “or “Digital assessment potential benefits for universities and educators have been widely discussed, but little is known about the students’ perspective.” 

 

Response 2: Thank you for this helpful comment. We agree that the language of the first sentence needs to be enhanced. We have changed the sentenced taking both reviewers comments into account (see lines 9-10).

Comment 3: Regarding the design: The paper mentioned it examines the impact of formative feedback on learning behaviors. However, It is hard to see an examination of learning behaviour since the research does not explore learning tasks after the feedback was received. I would consider changing the learning behavior category to learning attitudes throughout the paper. 

Response 3: We appreciate the constructive and insightful feedback. We agree, that the examination of learning behavior was not conducted; rather, the focus was on subjective perception. In light of this, we have revised the manuscript to replace "learning behavior" with "learning attitudes."

Comment 4: One of the sentences reads: “In a digital format, KFQs can be locked, preventing students from returning to previous questions after receiving additional information or feedback. These techniques facilitate testing at higher cognitive levels [9]”  Please explain the reasoning behind this statement. How does not returning the previous questions promote Higher Cognitive Levels? Consider defining Higher Order Thinking levels. HOT requires transferring concepts, abstractions, generalizations, etc. That is, requires addressing situations to apply  and combine concepts. It is the design of the task that would promote HOT or LOT (Lower Order thinking). 

Response 4: Thank you for pointing this out. Our objective was to highlight the digital characteristic of "locked questions" that we utilize for our tablet-based assessments. This feature enables the precise analysis of students' responses, as they are not influenced by subsequent questions. As you mentioned, students must apply and combine concepts, a necessity that is facilitated by this feature. A paragraph has been incorporated to define HOT levels, and it has been noted that the task's design promotes HOT) (see lines 48-54).

Comment 5: Another sentence reads: “Just-in-time FF and SF were incorporated into the tablet-based MCQ assessment at the end of the Family Medicine curriculum in the fifth year of medical school. This integration was undertaken to delineate the capabilities and limitations of MCQs.” Please make explicit how the addition of SF and FF contributes to identifying the limitations of MCQ. Also, there are different dimensions of MCQ to analyze such as the validity (evaluation of the tested concept), the relation to performance (prediction of medical practice) etc. or perhaps you are interested in understanding how MCQ promotes learning…. This statement is unclear. 

 

Response 5: Thank you for this important objection. MCQs are a widely researched assessment tool. We deliberately focused on the students’ perspectives on feedback and emotions evoked by it. We wanted to gain new information about how MCQs can promote learning and how effective feedback in an assessment setting could be. The inclusion of limitations is misleading. We rephrased the sentence accordingly (see lines 89-94).

 

Comment 6: The paper can be strengthened if you further define “assessment for learning”. Following Dylan Williams, formative feedback can be considered formative or assessment for learning when there is an opportunity to change your learning path, either because a teaching strategy is revised (if the teacher receives the feedback) or a learning strategy is revised. In the context of MCQ, I don't see how the feedback can contribute to teaching and learning since the assessment focus is summative and it is the final exam. Please state the team reasoning behind including FF in a MCQ format where there is no opportunity to change the learning path. 

Response 6: Thank you for this helpful comment. The reasoning behind including FF in a MCQ format was our desire to improve digital assessments. In Germany, as well as in many parts of the world, MCQs are more widespread due to the MCQ-based state exams. Our aim was to try out new feedback methods within an MCQ-based system to explore the student perspective on these changes. Our hypothesis is that by continuously including digital feedback in an MCQ exam, it will be better absorbed, and the exam will become a learning activity. We are currently developing a follow-up study to show these effects over time. We incorporated a paragraph explaining this in the introduction (see lines 58-63).

 

Comment 7: The paper states “The SF categories show that students still want a straightforward conclusion without always seeking a deeper understanding of their mistakes” Please explain how the data leads to this conclusion and how it relates to the positive learning attitudes of students you also find.

Response 7: Thank you for this helpful comment. Indeed, it is important to explain the relation of the contrasting results. Removing negative emotions, planning security, and closure exemplify students who conclude with the assessments without seeking a comprehensive understanding of their errors or the subject matter assessed. The discrepancy in results regarding the positive learning attitudes may be due to the nature of the two feedback methods. SF could lack the impetus to engage with it further. In contrast, FF provides a structured framework and motivation for students to reflect on their errors. We have revised the discussion to emphasize this point (see lines 338-341 and 357-362)

 

Comment 8: The discussion section needs to be separated by paragraph to improve the reading. 

 

Response 8: You are correct, thank you for this notice. We have separated the paragraphs accordingly.

 

Comment 9: The paper states “Although only 2.2% of students explicitly stated that they perceived FF as a learning experience in SS, these categories indicate that a greater number of students unconsciously gained, reflected on, or consolidated knowledge. To further support this, the results in WS 23/24 showed that most students believed the just-in-time FF would have a positive impact on their learning behavior” This statement is confusing. The conditional expresses a possible attitude towards learning (would have a positive impact). Furthermore, it is not clear the methods used to access the unconscious. This needs a lot of clarification. 

 

Response 9: Thank you for pointing this out. We wanted to illustrate the discrepancy that only 2.2% of the students themselves referred to the FF as a learning activity, but in the other categories of the FF a conscious engagement with various aspects of learning was evident (intuition, thought process, existing knowledge, and learning method). In our opinion, it is therefore still clear that FF leads to a deeper examination of the topics surveyed. In addition, the data from WS 23/24 helped to support this, as a subjective change in learning attitude is perceived by the students. This also shows an examination of the feedback regarding possible changes in the learning strategy. We have removed the reference to “the unconscious” as it was misleading and edited the paragraph (see lines 343-352).

 

Comment 10: The paper states: “Follow-up research should further investigate how this type of FF affects the learning behavior of students.” Again, the paper mentioned that learning behaviour is investigated but this statement implies it is not. What I think is being investigated are learning attitudes. 

Response 10: Thank you for this helpful comment. We have revised the manuscript to replace "learning behavior" with "learning attitudes."

Comment 11: At the end the article mentions “For educators, it may mean a possibility of combining learning activity and reflection elements with assessment”  I think this idea needs to be further developed as this is the core of the paper. It seems complicated to include formative assessment on a final exam when the student does not have an opportunity to use the feedback in a formative way. However, including formative feedback in other learning activities such as practice tests has more sense. This idea is particularly supported with the evidence of this study where students positively valued SF and FF. 

Response 11: Thank you for this important comment. We agree in this being the core of the paper. We have suggested two approaches to implicate our suggestion into the curriculum (see lines 365-371).

 

Comment 12: Finally, this phrase needs explanation “Autonomy affects the impact of feedback during highly sensitive phases, leaving them empowered in high-stake university examinations”. 

Response 12: Thank you for this helpful comment. According to Harrison, the receptivity to feedback can be enhanced by assessment cultures that promote students' agency. Optional feedback has the potential to elevate the significance of high-stakes university assessments. Taking both reviewers comments into account, we removed the paragraph as it did not discuss the main findings. We did this to improve the lengthy structure of the discussion.

Comment 13: Also the article mentions that “overall exam performances were high”, could you provide some data?

Response 13: Of course. We mentioned at the beginning of the results, that “the average performance of the exam was comparable for the three semesters, with average scores of 80-85%.” We adjusted the discussion and mentioned the average performance there as well (see line 397).

 

Author Response File: Author Response.docx