Lifting the Gate: Evaluation of Supplemental Instruction Program in Chemistry

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this paper for Education Sciences

Overall, I consider that the paper deals with an interesting and worthwhile topic and the research findings have shown the value of including Supplemental Instruction is college-level introductory chemistry courses.

Overall, the paper is well-written, and the overall message is reasonably clear. Below I comment on the parts of the paper and identify with ** where I consider additional work is needed.

** I am not familiar with the terms ‘underserved’ and ‘better-served’ students.

If the former term refers to socially disadvantaged, then use that term.

I note that none of the 52 references uses that term.

Please provide a reference for the use of this term.

Aim 

The research investigated (a) the impact of the revamped Supplemental Instruction as part of college-level introductory chemistry courses on students’ academic success, (b) the extent to which Supplemental Instruction closed equity gaps, and (c) any longitudinal impact on subsequent studies in Chemistry.

Literature 

The review of literature about gateway chemistry course is rather brief citing only three papers. 

**More could be presented to show the extent and range of researchers being interested in this topic to create different chemistry courses to different target groups.

The discussion of Supplemental Instruction in chemistry and other programs is extensive.

** What does FG non-FG stand for? Also DFW is not described but I can easily guess what this stands for. 

Methodology

The sample, the Supplemental Instruction program study design, data collection and analyses are clearly presented.  and limitations are clearly laid out and easy to follow. 

**Limitations are discussed in the Conclusions but would be helpful to provide some indication in the methods also.

**The research design needs to be explicitly stated and referenced.

**A sample of items should be provided for readers to assess and critique

** For ethical issues, please provide reference to the documentation so that it can be validated if necessary. This issue is becoming an increased requirement in educational research.

Results 

The results are presented under the three research questions. 

**In table 2, the text refers to Difference by SI (between a and c). I could find no reference 

to a and b.

The comparison of the equity gaps between the student groups is clearly presented in Table 4.

** For the comparison data in table 6, both courses had SI. I think that the Spring 2022 did not have SI. Otherwise the text and table do not match.

** Tables 4 and 5 only present statistical analyses – please include the numeral data used to do the analyses in the tables. This way the reader understands better what you did and the results.

** For table 6 for the first column, provide a heading Introductory Chemistry Course, otherwise the table is hard to read.

** Some comments are warranted to explain that despite all the effort by the researchers that (on page 14 The results show..) the increase in final exam and pass rate are low. Reading the paper, I had anticipated that these increases would be higher.

Discussion

The discussion returned to the group equity issues and describes the outcomes of the Supplemental Instruction

** When referring to the data in section 6: Conclusions and Discussion, be sure to cross check the data in discussion with the numerical data in the tables. For example, Line 4, page 13 has GPA of 0.89 – I did not find that number in the tabulated data

** Also, page 13 second paragraph, ‘lowest ability’ should be ‘lowest achievement’ - there has been no discussion of ability. 

** In describing the redesigned program under Conclusions and Discussion in the third paragraph beginning Our studies…, this may be better as new paragraph, and the three points made as recommendations rather than only describing the program and the main participants. 

Author Response

Dear Reviewer,

Thank you very much for taking the time to review our manuscript. We believe we addressed your comments and please find the detailed responses below to your comments and the corresponding revisions in track changes in the re-submitted files. We look forward to hearing back from you.

Overall, I consider that the paper deals with an interesting and worthwhile topic and the research findings have shown the value of including Supplemental Instruction is college-level introductory chemistry courses.

Overall, the paper is well-written, and the overall message is reasonably clear. Below I comment on the parts of the paper and identify with ** where I consider additional work is needed.

–Thank you for your kind comments and we appreciate that you marked the additional work needed areas with ** so they are easy to identify for us!

** I am not familiar with the terms ‘underserved’ and ‘better-served’ students.

If the former term refers to socially disadvantaged, then use that term.

I note that none of the 52 references uses that term.

Please provide a reference for the use of this term.

–Thank you for pointing this out. The ‘underserved’ and ‘better-served’ terms are used in the California State University (CSU) system, our research institution is one of the CSU campus;  Compared to other terms refers to socially disadvantage that are specificaly linked to a certain student’s social class such as underrepresented racial/ethnic population, the term ‘underserved’ are defined broadly as those who do not receive equitable resources as other students in the academic pipeline, which could include racial/ethnic minorities, low income, first-generation students, etc. We believe these terms are more accurate to represent the student population we study in this research project; Also, we try to use the language aligned with our university system to make better communication with our colleagues, administrators, and stakeholders in our community within the CSU University system. We added a citation entitled “Diversity/Inclusivity Style Guide” from the California State University system website to address the origins of these terms.  Please see reference number [9] in the first paragraph on page 2,  and the list of references at the end was updated with this article.

“9. Bevly, D.; Maguire, J. Diversity/Inclusivity Style Guide Available online:    https://www.calstate.edu/csu-system/csu-branding-standards/editorial-style-guide/Pages/diversity-style-guide.aspx#low-income (accessed on 11 October 2024)”.

Aim 

The research investigated (a) the impact of the revamped Supplemental Instruction as part of college-level introductory chemistry courses on students’ academic success, (b) the extent to which Supplemental Instruction closed equity gaps, and (c) any longitudinal impact on subsequent studies in Chemistry.

Literature 

The review of literature about gateway chemistry course is rather brief citing only three papers. 

**More could be presented to show the extent and range of researchers being interested in this topic to create different chemistry courses to different target groups.

–Agree. We added a more detailed review about the gateway chemistry courses and alternative approaches to improve these courses with reference. Please see the first paragraph on the first paragraph on page 1 as below:

“Various strategies have been reported to improve student learning outcomes in these gateway courses, including bridging and corequisite support courses targeting at-risk students [3-4], innovative pedagogical practices emphasizing conceptual understanding and collaborative learning [5-6], and adaptive online preparatory modules or intelligent tutoring systems [7-8].”

The list of references at the end was updated to reflect these literature reviews.

       “2.   Hatfield, N.; Brown, N.; Topaz, C.M. Do Introductory Courses Disproportionately Drive Minoritized Students Out 

              of STEM Pathways? PANS Nexus 2022, 1, 1-10.

3. Reid, S.A. Design, Evolution, and Evaluation of a General Chemistry-Bridging Course. Educ. Sci. (Basel) 2023, 13,

     891.

4. Shah, L.; Butler Basner, E.; Ferraro, K.; Sajan, A.; Fatima, A.; Rushton, G.T. Diversifying Undergraduate Chemistry

      Course Pathways to Improve Outcomes for At-Risk Students. J. Chem. Educ. 2020, 97, 1822–1831.

5. Talbert, L.E.; Bonner, J.; Mortezaei, K.; Guregyan, C.; Henbest, G.; Eichler, J.F. Revisiting the Use of Concept Maps

     in a Large Enrollment General Chemistry Course: Implementation and Assessment. Chem. Educ. Res. Pract. 2019,

     doi:10.1039/C9RP00059C.

6. Ye, L.; Eichler, J.F.; Gilewski, A.; Talbert, L.E.; Mallory, E.; Litvak, M.; M. Rigsby, E.; Henbest, G.; Mortezaei, K.;

      Guregyan, C. The Impact of Coupling Assessments on Conceptual Understanding and Connection-Making in 

      Chemical Equilibrium and Acid–Base Chemistry. Chem. Educ. Res. Pr. 2020, 21, 1000–1012.

7. Reid, S.A.; MacBride, L.; Nobile, L.; Fiedler, A.T.; Gardinier, J.R. Implementation and Evaluation of an Adaptive

     Online Summer Preparatory Course for General Chemistry: Whom Does It Benefit? Chem. Educ. Res. Pr. 2021, 22, 

     303–311.

8. Hickey, D.T.; Robinson, J.; Fiorini, S.; Feng, Y. Internet-Based Alternatives for Equitable Preparation, Access, and

     Success in Gateway Courses. Internet High. Educ. 2020, 44, 100693.”

The discussion of Supplemental Instruction in chemistry and other programs is extensive.

** What does FG non-FG stand for? Also DFW is not described but I can easily guess what this stands for. 

– FG stands for First-generation (FG) college students; non-FG stands for students who are not First-generation (FG) college students. Please see the definition of FG on the first paragraph on page 2 as “First-generation (FG) college students are ones whose parents did not obtain a degree from a four-year institution”; We added the definition of DFW on the first paragraph on page 7 as “...DFW rate (students who received D and F grades and Withdraw from the course).”.

Methodology

The sample, the Supplemental Instruction program study design, data collection and analyses are clearly presented.  and limitations are clearly laid out and easy to follow.

**Limitations are discussed in the Conclusions but would be helpful to provide some indication in the methods also.

–Agree. We stated this limitation in the methods in the newly added section entitled “2.1. Research Design”, please see our detailed responses for the next point below.

**The research design needs to be explicitly stated and referenced.

--We added a new section entitled “2.1. Research Design” to explain our research design before the “2.2. Study Setting and Sample” and cited a reference book on page 5. The texts are listed below:

“2.1. Research Design

                                       The research study employed a quasi-experimental research design that the SI program was considered an intervention applied to the experimental group while the control group did have SI. It is worth noting that the students who enrolled into SI sessions were not randomly assigned into the program, students chosen to stay might be more motivated or they believe that they need more support outside of the lecture classes. Although the participants were not randomly assigned to the two groups, certain confounding variables were matching, including participants who were enrolled in the same semester and used common textbook, online homework assignments, assessments, and grading schemes. Student demographics and backgrounds between the two study groups were also compared to ensure they were comparable samples. Multiple student performance outcome variables such as course GPA, final exam scores, and pass rates were chosen to evaluate the effectiveness of the SI program. In addition to the matching-only design, factorial designs were also used to study the interactions between the independent variables (e.g., URM status, FG status, gender) with the outcome variable of course GPA [53].”

The list of references at the end was updated with this article.

“53. Fraenkel, Jack R., Wallen, Norman E., Hyun, Helen H. How to Design and Evaluate Research in Education, 10th ed.; New York: McGraw Hill Education, 2019.”

**A sample of items should be provided for readers to assess and critique

–We provided a sample of the common final exam questions and its learning objectives in the supplemental document. We also indicated this at the end of the first paragraph of in the “2.2 Study Setting and Sample” section on Page 5 as “A list of sample final exam questions can be found in the supplemental documents.”

** For ethical issues, please provide reference to the documentation so that it can be validated if necessary. This issue is becoming an increased requirement in educational research.

–We referred to the Institutional Review Board approval number at the research institution in the third paragraph on page 63 as “This study was reviewed and approved by the Institutional Review Board at the research institution (IRB-FY20-411) .” We also sent a copy of the approval to the journal’s editor office already for validation purposes. If you need any more evidence, please let us know.

Results 

The results are presented under the three research questions. 

**In table 2, the text refers to Difference by SI (between a and c). I could find no reference 

to a and b.

— In Table 2, a,b,c were listed as superscripts on each of the semesters, we now moved them to the front of each semester on the first column so they are more obvious to see. Please see Table 2 on page 8. Same changes were made in Table 1 on page 7.

The comparison of the equity gaps between the student groups is clearly presented in Table 4.

** For the comparison data in table 6, both courses had SI. I think that the Spring 2022 did not have SI. Otherwise the text and table do not match.

— In Spring 2022, about half of the students were in SI, the other half of the students were not in SI. Table 4 compared the students who were in SI and not in SI in the same semester of Spring 2022. To avoid confusion, we moved “Spring 2022” to the top row to indicate the semester was the same for the samples. Please see Table 4. Same changes were naked in Tables 1 & 2.

** Tables 4 and 5 only present statistical analyses – please include the numeral data used to do the analyses in the tables. This way the reader understands better what you did and the results.

 —We modified Tables 4 and 5 and added more numeral data used to do the data analysis and we also now added more details of the statistical results in the text before Table 4 to explain the results of these analyses on pages 9-10.

** For table 6 for the first column, provide a heading Introductory Chemistry Course, otherwise the table is hard to read.

 — Changes were made. Please see Table 6 on page 14.

** Some comments are warranted to explain that despite all the effort by the researchers that (on page 14 The results show..) the increase in final exam and pass rate are low. Reading the paper, I had anticipated that these increases would be higher.

 —The statements of “The results show that the full-scale virtual mandatory SI program increased students’ course GPA by 0.30 (~half a letter grade), final exam by 3%, pass rate by 8%, and decreased the DFW rate by 7%” were referred to a prior study we conducted before this study during Covid-19 time; We added the following sentences on the first paragraph on page 16 to address the reasons why the increases might not look as impressive as a whole for the spring 2022 semester.

“These results might not seem that significant because of the sudden changes of the instruction from in-person to virtual for both lecture and SI classes.” and “In Spring 2022, the main findings of this study showed the students with SI improved students’ course GPA by 0.89 (~half a letter grade), the final exam by 5.4%, pass rate by 25%, and decreased the DFW rate by 24% compared to those without SI in the same semester. Because this is the first semester that students came back to in-person instruction from the pandemic disruption, adjustment was certainly needed for students and the overall improvement would have to take into account the transition back from virtual to in-person instruction.”

Discussion

The discussion returned to the group equity issues and describes the outcomes of the Supplemental Instruction

** When referring to the data in section 6: Conclusions and Discussion, be sure to cross check the data in discussion with the numerical data in the tables. For example, Line 4, page 13 has GPA of 0.89 – I did not find that number in the tabulated data

 —This number is listed in Table 2 on page 8.

** Also, page 13 second paragraph, ‘lowest ability’ should be ‘lowest achievement’ - there has been no discussion of ability. 

 —We used “student ability” to represent students who obtained different grades in the introductory chemistry course. After seeing this comment, we realized “student achievement” might be a better term for it. So we modified all the “student ability” to “student achievement” throughout the manuscript and Figure 4(b). And we changed “lowest ability’”to “lowest achievement” in the last paragraph on page 14.

** In describing the redesigned program under Conclusions and Discussion in the third paragraph beginning Our studies…, this may be better as new paragraph, and the three points made as recommendations rather than only describing the program and the main participants. 

 —We made this change, thanks!

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript "Lifting the Gate: Evaluation of Supplemental Instruction Program in Chemistry" investigates the impact of a revamped SI program on student success in introductory chemistry, giving particular focus to underserved students at a Hispanic-serving institution. The study shows improved academic performance for students who participated in the mandatory SI program using several quantitative metrics. The authors provide detailed statistical analysis exploring the disaggregated student performance data and examine performance in a subsequent general chemistry course. While the SI program has been studied extensively, the results of this work make a valuable addition to the literature, and I recommend publication, assuming the following items are addressed.

General comments:

I would be curious if attempts were made to standardize grading/assessment across the different sections. For example, final exam performance provided a crucial point of comparison throughout the work. Was a common final exam administered across the different sections? Was an item analysis performed on the final exam to ensure ambiguous or misleading items did not impact the final results? 

Similarly, GPA is cited as another crucial piece of evidence. Were efforts made to standardize grading across the different sections? Also, how many instructors were involved in the study?  I would like to see descriptive statistics for the different chemistry lecture sections. E.g. were students enrolled in SI and not enrolled in SI randomly distributed across the different lecture sections? What is the relative student performance between different lecture sections? 

Specific comments:

Section 1.1 Paragraph 2: "...students from the traditionally underserved communities are disproportionately held back in those gateway courses and weeded out early on."  Consider revising the wording to avoid the phrase "weeded out"

Figure 1: The caption mentions error bars. Was this a mistake? Incorporating a brief description of the box plots would help. For example, is the black line the median performance? Is the colored box the middle 50th percentile, and the whiskers the range of scores? Additionally, increase the font size within the figures to match that of the rest of the text. 

Section 3.1 Paragraph 1: 

The text states, " Interestingly, the students who participated in the SI but did not pass with the credit performed much worse than those without the SI (students with SI who got no credit course GPA = 1.07 versus students without SI course GPA = 1.71)"

At first read, this result seems expected. If credit for the SI sections is obtained through attendance, then it is unsurprising that students who do not meet the minimum requirements to receive credit for SI also fail to meet the requirements of a more demanding lecture course. However, the wording of the text suggests I may be missing something.  Perhaps the authors could draw out the interesting aspect of this finding. 

Figure 3: Again, increase the font size for labels to match the font size in the text. 

Section 3.3:  The text states SI improved GPA regardless of who the intro chemistry instructor was. Was an ANOVA performed to compare performance between instructors as the text states it was to compare different SI leaders? 

An additional note regarding differences between instructors: The text discusses the SI program in more detail. Would it be fair to say the chemistry lecture instructors all engaged with their associated SI leaders to a similar extent? I could imagine greater student benefit if the SI and instructor regularly coordinate content delivery. 

Section 4: paragraph 3 "They found that students performed better in eight of the chemistry courses with SI than those without SI in every course." Consider rewording for clarity.  

Comments on the Quality of English Language

Section 1.2 Paragraph 2:   "The advantages of SI are twofold," not "two folded" 

Section 4, Paragraph 3: Reword for clarity and capitalize chemistry: 

 "chemistry instructors are communicated with best practices for supporting SI leaders." 

Author Response

Dear Reviewer,

Thank you very much for taking the time to review our manuscript. We believe we addressed your comments and please find the detailed responses below to your comments and the corresponding revisions in track changes in the re-submitted files. We look forward to hearing back from you.

The manuscript "Lifting the Gate: Evaluation of Supplemental Instruction Program in Chemistry" investigates the impact of a revamped SI program on student success in introductory chemistry, giving particular focus to underserved students at a Hispanic-serving institution. The study shows improved academic performance for students who participated in the mandatory SI program using several quantitative metrics. The authors provide detailed statistical analysis exploring the disaggregated student performance data and examine performance in a subsequent general chemistry course. While the SI program has been studied extensively, the results of this work make a valuable addition to the literature, and I recommend publication, assuming the following items are addressed.

 –Thank you for your kind comments!

General comments:

I would be curious if attempts were made to standardize grading/assessment across the different sections. For example, final exam performance provided a crucial point of comparison throughout the work. Was a common final exam administered across the different sections? Was an item analysis performed on the final exam to ensure ambiguous or misleading items did not impact the final results? 

–The final exam was common across all the introductory chemistry lecture course sections. We have mentioned this in the second paragraph on page 5 as “All lecture classes used a common textbook, learning objectives, online homework assignments, and a common final exam written by all the instructors who taught the course.”

We did not perform the item analysis on the final exam because there was no such data available. The final exam questions were multiple choice questions, and students used paper scantrons to record their answers. The instructors collected the scantrons and used a scantron machine to score student performance as a whole without the information of each individual question on the final exam.

Similarly, GPA is cited as another crucial piece of evidence. Were efforts made to standardize grading across the different sections?

–The course GPA was based on student letter grades in the course, and all the instructors used the same grading scheme for assigning grades.

Also, how many instructors were involved in the study?  I would like to see descriptive statistics for the different chemistry lecture sections. E.g. were students enrolled in SI and not enrolled in SI randomly distributed across the different lecture sections? What is the relative student performance between different lecture sections? 

–There were four instructors teaching the course. We mentioned this in the second paragraph on page 5 as “There were nine classes taught by four instructors in Spring 2022 with the revamped SI program.”

–The data were separated by the lecture classes taught by different instructors, please see Figure 4 (c);This figure not only showed the student performance between different classes taught by the same instructors, but also showed how SI made a difference for the students taught by the same instructors.

––Additionally, the students were not randomly distributed in SI, we added a new section entitled “2.1. Research Design” under the “Methods” section on page 5 to explain our study design and the limitation of the non-randomized design.

Specific comments:

Section 1.1 Paragraph 2: "...students from the traditionally underserved communities are disproportionately held back in those gateway courses and weeded out early on."  Consider revising the wording to avoid the phrase "weeded out"

––We removed “and weeded out early on” in this sentence.  Now it is changed to“...students from the traditionally underserved communities are disproportionately held back in those gateway courses.”

Figure 1: The caption mentions error bars. Was this a mistake? Incorporating a brief description of the box plots would help. For example, is the black line the median performance? Is the colored box the middle 50th percentile, and the whiskers the range of scores? Additionally, increase the font size within the figures to match that of the rest of the text. 

––We removed the ”error bars: 95% confidence interval” from the figure 1 title, thanks for catching that! We added a few sentences to explain the boxplots in the first paragrah on page 8 to answer those questions:

“More specifically, Figure 1 (a) indicated that the students who did not participate in SI had a lower median course GPA (around 2.0) and a wider range with more variability in their course GPAs. The 75% quartile reached close to 3.0 while the 25% quartile extended to nearly 1.0. Instead, the students participated in SI obtained a higher median GPA closed to 3.0 with a more compact range from 2.0 to almost 4.0 course GPA; Figure 2 (b) showed that students with SI had median final exam scores around 60% in the range between 17% to 97% while those without SI were about 50% in similar range of scores. These boxplots suggested a positive association between SI participation and higher course GPAs and final exam scores.”

— The font size within all the figures in the manuscript were changed to match the text.

Section 3.1 Paragraph 1: 

The text states, " Interestingly, the students who participated in the SI but did not pass with the credit performed much worse than those without the SI (students with SI who got no credit course GPA = 1.07 versus students without SI course GPA = 1.71)"

At first read, this result seems expected. If credit for the SI sections is obtained through attendance, then it is unsurprising that students who do not meet the minimum requirements to receive credit for SI also fail to meet the requirements of a more demanding lecture course. However, the wording of the text suggests I may be missing something.  Perhaps the authors could draw out the interesting aspect of this finding. 

––We modified this sentence from “Interestingly, the students ..." to " As expected, the students …"

Figure 3: Again, increase the font size for labels to match the font size in the text. 

— The font size within all figures in the manuscript were changed to match the text.

Section 3.3:  The text states SI improved GPA regardless of who the intro chemistry instructor was. Was an ANOVA performed to compare performance between instructors as the text states it was to compare different SI leaders? 

–Two different ANOVA were performed, one was to compare performance between lecture instructors, the other was to compare performance between SI leaders.

We modified this section to make it clearer as below:

“One-way ANOVA results indicated that having SI classes significantly improved students' course GPA regardless of who the introductory chemistry instructor was. Additionally, another one-way ANOVA was conducted to explore the impact of SI leaders on introductory chemistry course GPA. The results indicated no statistically significant differences in introductory chemistry course GPA between students taught by different SI leaders.”

An additional note regarding differences between instructors: The text discusses the SI program in more detail. Would it be fair to say the chemistry lecture instructors all engaged with their associated SI leaders to a similar extent? I could imagine greater student benefit if the SI and instructor regularly coordinate content delivery. 

–Great points! We added this point to the future studies on the last paragraph of the manuscript on page 16 as below:

“Qualitative studies investigating stakeholders’ perspectives are also needed to gain more insights and comprehensive understandings about the holistic impacts of similar supplemental academic programs. These might include but not limited to how the SI program benefits the students for their sense of belonging in the science community, SI leaders’ leadership and employable skills, and chemistry instructors’ instructional approaches, and the mechanism and consistency of coordination between instructors and SI leaders.”

Section 4: paragraph 3 "They found that students performed better in eight of the chemistry courses with SI than those without SI in every course." Consider rewording for clarity.  

–We modified this sentence to “They found that students performed better in all of the eight chemistry courses with SI than those without SI”. 

Comments on the Quality of English Language

Section 1.2 Paragraph 2:   "The advantages of SI are twofold," not "two folded" 

–Fixed!

Section 4, Paragraph 3: Reword for clarity and capitalize chemistry: 

 "chemistry instructors are communicated with best practices for supporting SI leaders." 

–Fixed!