An Exploratory Study of Small-Group Learning Interactions in Pre-Clerkship Medical Education: Uncovering a Mismatch Between Student Perceptions and Real-Time Observations
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsAn Exploratory Study of Small Group Learning Interactions in 2 Pre-Clerkship Medical Education: Uncovering a Mismatch Between Student Perceptions and Real-Time Observations is an interesting paper It used real-time observations of PBL with undergraduate medical students to explore patterns of verbal interaction in small groups, and to examine the relationships between interactivity, tutor involvement, tutor content expertise, and participants’ perceptions of the learning experience. In addition, to exploring patterns of group interaction, the links between these factors and student performance on course-based exams were also explored. This involved using a complex and robust methodology guided by the authors' extensive knowledge and understanding of the literature.
Abstract: Sound
Introduction Sound
Methods Sound
Results Sound
Discussion Sound
References: An extensive range of references is included.
Overall, this paper is well-written and easily understood.
Minor Points
Lines 162 and 165: Please flip around use of SNP abbreviation to be written in full the first time used
Figure 2 Please clarify the meaning of this sentence: Data points with shaded circles indicate observations with content expert tutors and those with open circles indicate those with non-content expert tutors. Are you referring to the observations within the circles or external to the circles?
Author Response
Comments 1: Lines 162 and 165: Please flip around use of SNP abbreviation to be written in full the first time used
Response 1: Done. See tracked changes edits.
Comments 2: Figure 2 Please clarify the meaning of this sentence: Data points with shaded circles indicate observations with content expert tutors and those with open circles indicate those with non-content expert tutors. Are you referring to the observations within the circles or external to the circles?
Response 2: Clarification added to figure legend to indicate that the shaded versus open circles refer to the data points on the central graph. See tracked changes.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe manuscript takes an interesting approach to determine the impact of interactivity on learning, which provides an alternative light on PBL. Research on PBL is, I often find, contentious given the impact of implementation efficiency on the outcome, rather than the pure and simple modality itself. The manuscript is worth publishing with some minor comments below.
Line 205: Is this the average number of students per group, or within a session? How many students were typically aimed for per group, and how homogenous were the group compositions? Were these predominantly dictated by social circles, the facilitator of the session, and so forth?
Interaction, as mentioned in the discussion, can be seen in different ways. Is it possible that a difference in ontology of quality interaction is present? Can observers provide information on what was being interacted with, for example, critical elements of the brief, questions related to the theoretical work, experiences, and so forth? Students may see interaction with a clear 'learning element' more valuable than pure and simple social interactions on experiences, with the same being said for interactions with a tutor as part of the group rather than a facilitator thereof.
Assessment practice often brings a difficult discussion in - scores were not different between groups, but how aligned was the assessment to the PBL activities itself? In other words, might the assessment be less affected by the interaction within the PBL sessions, and more with the material the have access to via consolidation or alternative sources? Does the assessment delve into elements of problem-solving which may be arguable (and assumingly) higher within the interactive groups?
Author Response
Comments 1: Line 205: Is this the average number of students per group, or within a session? How many students were typically aimed for per group, and how homogenous were the group compositions? Were these predominantly dictated by social circles, the facilitator of the session, and so forth?
Response 1: This is the average number of students per group. The text has been edited to clarify. Students and facilitators were assigned to groups by the course coordinator at the start of the course. This clarification has been added in the Materials and Methods section (context), lines 106-107.
Comment 2: Interaction, as mentioned in the discussion, can be seen in different ways. Is it possible that a difference in ontology of quality interaction is present? Can observers provide information on what was being interacted with, for example, critical elements of the brief, questions related to the theoretical work, experiences, and so forth? Students may see interaction with a clear 'learning element' more valuable than pure and simple social interactions on experiences, with the same being said for interactions with a tutor as part of the group rather than a facilitator thereof.
Response 2: The reviewer makes an excellent point. This study was not designed to look at the quality of interactions, nor to explore students' and tutors' perceptions of interactions with social versus learning elements. These would be interesting areas for future study.
Comment 3: Assessment practice often brings a difficult discussion in - scores were not different between groups, but how aligned was the assessment to the PBL activities itself? In other words, might the assessment be less affected by the interaction within the PBL sessions, and more with the material the have access to via consolidation or alternative sources? Does the assessment delve into elements of problem-solving which may be arguable (and assumingly) higher within the interactive groups?
Response 3: This is another excellent point. As we have clarified in the manuscript (lines 379-382), the overall course assignments may have been variably aligned to the PBL activities.
Reviewer 3 Report
Comments and Suggestions for Authors1. Abstract: It has to be structured. Objective of the study is missing
2. Materials and Methods
A) CONTEXT
-Indicate the number of students per batch
-Number of students who are involved in PBL per group
- A hypothetical doubt (Context of paper) - What if the student arrives at satisfactory solution of the given problem in just 2 sessions, will you still do the third one?
Likewise, if suppose, students don't arrive at a satisfactory resolution, will the duration of third session be extended or another session will happen??
- Also, no mention has been made about whether the participants ever had exposure to PBL before inclusion in the study?? Both scenarios will have different results.
B) Numbering is not correct... Data collection has to be 2.2, and the 3 methods should be 2.2.1, 2.2.2, 2.2.3, etc.
C) Direct Observation - Not including participants observation in the first 4 steps of the classical 7 step PBL cycle makes it a limitation of the study. Is there any specific reason, why authors have not included first small group discussion?? Justify
- For the purpose of this study, we defined interactivity as the frequency and sequence of verbal interactions between group mem-128 bers. = Was this criteria operational or you have some references for the same (better option) - If yes, please cite
- Who carried out Observation? Their Qualification? Were they researcher themselves? How many? Were they trained? Reflexivity?
D) Survey - The kind of interaction that happens in PBL is somewhat different from the discussions reported in other collaborative methods. This survey asked students perception about small group experience in general. Is it right??
E) Course Marks - Are these marks of only PBL or something else. Detail has to be provided. Also, add, what all types of assessments were carried out to arrive at these scores. Elaborate
F) Authors must elaborate about the research experience of authors in Qualitative research
G) Add what all was done as a precautionary measure to ascertain trustworthiness of the qualitative data
H) Was it a mixed methods study design? Please justify
3. Discussion: The findings of the study should be discussed in the same order as the results were presented.
Moreover, also add possible reasons for obtaining similar or contrasting results from your study
Add limitations of the study
Recommendations need to be removed from the text
4. Conclusion: Should highlight only those things that were specified in Aim. Do not go beyond the scope of the study
Author Response
Comment 1: Abstract: It has to be structured. Objective of the study is missing
Response 1: Thank you. We have reworded line 10 of the abstract to make it more explicit that this is our objective.
Comment 2: Materials and Methods, A) CONTEXT, -Indicate the number of students per batch, -Number of students who are involved in PBL per group
Response 2: This information is included in line 107 and as per our response to reviewer 2, we have also added language to line 208-209 to clarify that there was a mean of 10.3 students per group with a range of 8-11 students per group. We have added information to lines 106-108 to indicate that all students are required to participate in the PBL component in each of the courses.
Comment 3: A hypothetical doubt (Context of paper) - What if the student arrives at satisfactory solution of the given problem in just 2 sessions, will you still do the third one?
Likewise, if suppose, students don't arrive at a satisfactory resolution, will the duration of third session be extended or another session will happen??
Response 3: As per our description in lines 100-102, each PBL cycle consists of 3 sessions. As described in lines 102-105, the PBL exercises were meant to teach students to develop an approach to core problems in the course (e.g. approach to monoarthritis). While a secondary objective is for students to apply their approach to the case at hand and come to a solution, finding a solution to the case (e.g. the patient has gout as their cause of monoarthritis) is neither sufficient nor necessary. The facilitators are guided by the facilitator materials in regard to the pacing of the sessions to ensure the groups complete the learning objectives in the available time. We do not feel this level of detailed explanation of facilitator instruction adds to the manuscript, and so we have not added it but we are willing to add this information if the editor would like this information added.
Comment 4: Also, no mention has been made about whether the participants ever had exposure to PBL before inclusion in the study?? Both scenarios will have different results.
Response 4: As outlined in lines 99-100, every course in this curriculum has a PBL component and so unless it is the 1st course of the 1st year, in all courses students would have prior experience with PBL.
Comment 5: B) Numbering is not correct... Data collection has to be 2.2, and the 3 methods should be 2.2.1, 2.2.2, 2.2.3, etc.
Response 5: We thank the reviewer for catching this numbering error and we have corrected it using track changes in the revised manuscript.
Comment 6: C) Direct Observation - Not including participants observation in the first 4 steps of the classical 7 step PBL cycle makes it a limitation of the study. Is there any specific reason, why authors have not included first small group discussion?? Justify
Response 6: We have provided justification for the selection of the 2nd session in lines 119-130.
Comment 7: For the purpose of this study, we defined interactivity as the frequency and sequence of verbal interactions between group mem-128 bers. = Was this criteria operational or you have some references for the same (better option) - If yes, please cite
Response 7: We have added the citation for this definition of an utterance to the paper.
Comment 8: Who carried out Observation? Their Qualification? Were they researcher themselves? How many? Were they trained? Reflexivity?
Response 8: We have added information in lines 132-134 to clarify that there were two trained observers with one being a trained research assistant and the other being a trained PI. We have not included a reflexivity statement as one of the observers is a paid research assistant and the other a key member of the study. In addition, this is not a qualitative interview or focus group study where the background or lens of the person collecting data could influence the questions asked and prompts produced, but rather the observers use a mobile data collection instrument. We have provided inter-rater reliability statistics to address potential differences in use of this data collection instrument in lines 220-226.
Comment 9: D) Survey - The kind of interaction that happens in PBL is somewhat different from the discussions reported in other collaborative methods. This survey asked students perception about small group experience in general. Is it right??
Response 9: We have added clarifying language to line 140-141 to explain that the survey questions were specific to the PBL session.
Comment 11: F) Authors must elaborate about the research experience of authors in Qualitative research
Response 11: This is not a qualitative research study given we used social network analysis, survey methodology and exploratory analysis of quantitative overall course marks and so we have not added this requested text about our qualitative research experience. Should the editors wish, we can provide our research training information, but we find it not to be a standard practice to add this to a manuscript.
Comment 12: G) Add what all was done as a precautionary measure to ascertain trustworthiness of the qualitative data
Response 12: We thank the reviewer for this comment but reiterate again that this is not a qualitative research study. We used social network analysis, survey methodology and exploratory analysis of quantitative overall course marks and so we have not added this requested section on checking for trustworthiness of qualitative data to the manuscript.
Comment 13: H) Was it a mixed methods study design? Please justify
Response 13: No, this was not a mixed methods study but rather was a pilot social network analysis study with a supplementary survey and exploratory analysis of aggregate quantitative student course marks. There was no qualitative integrative component to the study.
Comment 14: Discussion: The findings of the study should be discussed in the same order as the results were presented.
Response 14: We thank the reviewer for this comment. However, we feel this request pertains to writing style. We have chosen to highlight the most interesting and relevant findings first in the discussion and then move to more minor discussion points rather than following the order that the findings were presented in the results section. Should the editor prefer that we reorder our discussion, we are willing to do this, but we don’t this this is a standard requirement and worry it may cause our key messages in the discussion to be buried.
Comment 15: Moreover, also add possible reasons for obtaining similar or contrasting results from your study
Response 15: We thank the reviewer for this comment and point them to lines 306-317 and lines 340-366 of the discussion, where we have compared our results to the relevant literature and have provided hypothesized reasons for the similarities and differences.
Comment 16: Add limitations of the study
Response 16: We are somewhat puzzled by this comment as we have a limitations paragraph in our discussion on lines 367-385. We wonder if the reviewer prefers that we have a sub-heading for limitations within our Discussion section? If the editors would like us to insert a sub-heading we are willing to do this. Please let us know.
Comment 17: Recommendations need to be removed from the text
Response 17: We would request to leave our recommendations in the manuscript. We feel that many of our readers are educators who are trying to improve their PBL and active small group curricular experiences at their own centres and that they would likely appreciate and potentially benefit from recommendations based on the findings of our study. We feel this provides a “value add” for our readers. However, should the editor wish for these to be removed please let us know and we will comply.
Comment 18: Conclusion: Should highlight only those things that were specified in Aim. Do not go beyond the scope of the study
Response 18: We thank the reviewer for the comment. We feel that our conclusions marry the key findings from the aim of our study with learnings for readers who are trying to optimize their own active small group learner experiences and add value for the reader. If the editors wish for us to remove the recommendations based on this study and the possible strategies to address our findings we would be willing to do so.
Round 2
Reviewer 3 Report
Comments and Suggestions for AuthorsI am satisfied with the corrections made by the authors and the responses to the raised comments.
