Analysis of the Effectiveness of Open-Ended Style Activities on Students’ Performance in an Engineering Admission Test

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This study has a promising premise - are there better ways to help prepare students for engineering entrance exams, and to enter the engineering field overall? Further, how can we encourage more women to enter engineering majors and professions? Given the promising results for women in this study, it would be helpful to lean into this finding more.

The paper overall is good, and highlights some important work in pre-college engineering education. It is just lacking some detail about study participants and the course that was employed with the goal of increasing TOL scores. Additionally, it would be enriched by a post-course survey of participants.

Introduction

This section described the impetus and goals of the study well, including some of the procedure even. The literature related to the study was included in the introduction, but it would be helpful to include more of a foundation for this work in past studies. I would prefer to see a sub-section that included more literature, followed by a brief(er) section that describes the study at hand - focusing mainly on the research questions and hypotheses and less on the "how" of the research in this section.

With regard to what literature to include, a bit more about STEM bridge programs and problem-based learning would be informative. Also, a discussion of the low representation of women in engineering, and how pre-college programs and problem-based learning helps to improve representation would provide a strong background for the study and its findings.

Materials & Methods

The methods were mostly well described, but the paper could benefit from more detail. First, how were students recruited into thecourse, "Introduzione metodologica alla preparazione al test di Ingegneria"? Were they incentivized to attend? Was the program during school time or during a school break? Was there a cost to attend? The authors mention that their students were from middle and upper income families. Was there a reason to recruit from schools with this makeup of students, versus lower-income schools that may have been more diverse and also have students who were more likely to need assistance with the TOL?

Were the appropriate ethical human subjects research protocols followed? I did not see these described or mentioned in the paper or at the end.

Second, given its success and importance to this paper, a description of the hands-on curricula that was employed would be very helpful. This could be included as an appendix or extra resource. 

Analysis 

These analyses appear to be appropriate for the type of data collected. It is helpful to analyze the data from the women and men separately. It would be even more helpful if there were more students in the G1 group. The authors mention near the end of the paper that more students have completed the course - would it be possible for the authors to include this group in this study?

Results

The box and whisker plots, and tables with data are well organized and helpful. The sentence that ends on Line 302 is the only one in its paragraph. It may make more sense to combine it with the following paragraph. 

Discussion

The discussion was helpful in detailing some possible explanations for the results, and the lack of impact of the new training on the TOL results. With a more detailed literature review, it would help to couch these findings with respect to similar studies. Ideally, rather than guessing about some of these reasons, it would be (or have been) helpful to reach out to the students to learn more about the training they received and what they felt helped them (or did not help them). Was a survey conducted? Similarly, a survey would help to explain why some students chose NOT to take the TOL after all of the work in the course. Were some students (women versus men, for example) more likely to opt out of the TOL? 

 

Author Response

Dear reviewer, thanks for your attention and your help with our paper. I upload here a PDF with all our responses to your comments and suggestions.

Kind regards

All the Authors

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Analysis of the Effectiveness of Open-Ended Style Activities 2 on Students' Performance in an Engineering Admission Test

 

First of all, I would like to thank you for presenting such a valuable study based on innovative pedagogical approaches that aims to increase students' academic performance and self-efficacy during the transition to engineering education. Such research is crucial for understanding how theoretical knowledge can be blended with practical application and exam success.

 

The main objective of the study is to examine the effectiveness of an orientation course organized for high school students preparing for the engineering entrance exam (TOL) at the Politecnico di Milano. In this process, which is designed within the framework of constructivist educational philosophy and Problem Based Learning (PBL), the role of "open-ended" activities, especially in the physics laboratory, on students' exam scores, motivations and perceptions of scientific self-efficacy is analyzed. The research aims to reveal how such active learning methods not only transfer knowledge but also support students' ability to manage their own learning processes, and how this effect varies according to the gender factor.

 

I can say the following about the strengths of your article. The introduction is based on Constructivism theory and Problem Based Learning (PBL) approach and is based on a solid theoretical foundation. In the literature, the difference between traditional "cookbook" style laboratory approaches and innovative student-centered methods has been clearly revealed and the low efficiency of traditional methods has been supported by the literature. The three key research questions (RQ1, RQ2, RQ3) presented at the end of the introduction clearly illustrate the scope of the study and what it aims at (effectiveness, the role of open-ended activities, and the gender factor). The objective of the study is not only to present a theoretical knowledge, but is based on a concrete and practice-oriented objective of improving students' real performance in the engineering entrance exam (TOL) at the Politecnico di Milano.

 

Regarding the weaknesses of the study, I can say the following. We encounter the ambiguity of methodological limitations in the introduction. In the introduction, it is stated that the course lasts only 15 hours and only 6 hours are devoted to the physics part. I think that such a short-term intervention is ambitious, even though the arguments presented (on the basis of self-efficacy) about how it can permanently change the student's overall learning skills are theoretically logical. In addition, in the introduction and method transition of the article, they stated that the G2 group was not a real control group, but only a baseline. I think this undermines the scientific certainty of the "effectiveness analysis" promised in the introduction of the article. Again, while the importance of physics laboratory activities is emphasized in the introduction, I think that the lack of a significant increase in physics scores in the following sections (and as mentioned in the discussion section) creates a disconnect between the expectations established in the introduction and the results. In addition, the fact that only 25 of the 113 students who attended the course were included in the analysis (G1 group) raises questions about the statistical power of the large-scale effectiveness analysis targeted in the introduction. I suggest that it be examined again in this section.

In conclusion, the introduction of the article provides a very persuasive and well-referenced background from a pedagogical point of view. However, the gap between the targeted research questions and the current constrained sample and lack of control group leaves the study's claims at the level of "observation and tendency" rather than "conclusive evidence". Re-examining these parts will be very good for the quality of the study.

For the method part, it is generally well-structured, transparent and prepared in a way that is largely in line with the research standards accepted in the field. The chapter presents in a systematic form how the study is designed, how data is collected and analyzed. However, there are some methodological weaknesses and points that need to be clarified.

First of all, since the aim of the study is to measure the "effectiveness of open-ended activities", it is a good choice to use a quasi-experimental pre-test-posttest design. How the groups (G1: course participants, G2: non-participants) are formed is explained step by step and clearly. In particular, selecting the comparison group G2 from the same high school types as the students in G1 and checking the homogeneity of the groups using INVALSI test results and Eduscopio rankings is an important and rigorous approach that strengthens the internal validity of the study. This makes it more likely that differences between groups are largely due to course attendance. Using the Shapiro-Wilk test to check the normality distribution of the data and using nonparametric or "robust" statistical methods (such as Yuen's truncated means t-test) is an extremely powerful methodological aspect in cases where the assumption of normality is not satisfied. This enhances the validity and reliability of the findings. Additionally, reporting not only significance (p-value) but also effect size (AKP) is valuable for assessing the practical significance of the findings.

In addition to these strengths, it also has some weaknesses. The most important limitation of the study was that the experimental group G1 was very small (n=25). Although there were 113 participants at the beginning, only 25 were both pre- and post-course TOL trials, severely narrowing the analyzable sample. This reduces the statistical power, especially in gender-based subgroup analyses (7 female students), and limits the generalizability of the findings. In addition, the G1 group consists of students who voluntarily participate in the course, which creates the possibility that these students are already more motivated, curious or more open to self-improvement (selection bias). Therefore, I think the observed development may be a result of these pre-existing traits, not just the course.

As the authors point out, the G2 group is not a randomly assigned control group. G2 is a comparison group consisting only of students who did not attend the course. The absence of randomization means that factors that cannot be measured and controlled (e.g., intrinsic motivation, family support, additional tutoring) may differ between groups. This makes it difficult to link the development in G1 entirely to the course; I think it will weaken the claims of causality.

In addition, although the concept of "open-ended physics laboratory" has been introduced, what exactly the students do during this 4-hour laboratory process has not been sufficiently concretized. What physical phenomena have been studied? What "basic ingredients" were given? What was the expected output from the students? In what format and how often was the "feedback" given? The lack of such details makes it difficult for the reader to fully understand the nature of the intervention and assess its applicability in another context.

The theoretical framework that forms the basis of the study is that the course will indirectly increase TOL success by increasing self-efficacy and motivation. However, these intermediate variables (self-efficacy and motivation) were not directly measured. From the increase in the TOL score, it is inferred that there is an improvement in these psychological constructs. If a short, valid self-efficacy or motivation questionnaire were used before and after the course, it would be possible to test this theoretical mechanism and further strengthen the claims.

The conclusion and discussion sections exhibit a cohesion that provides clear answers to research questions, interprets findings in a balanced manner, and honestly acknowledges limitations. The complex statistical outputs obtained from the method are presented in understandable language and in a logical flow.

Here, in the direct association of the findings with the research questions;

   - RQ1 (Effectiveness of the course): A positive response was given by stating that the improvement in G1 had an effect size 18.6% higher than G2. 

   - RQ2 (Contribution of open-ended activities): Although there was no significant improvement in the physics section, it is a rational argument to explain the increase in the overall TOL score with "autonomous preparation ability through self-efficacy and motivation". 

   - RQ3 (Gender effect): The effect size of female students was higher than that of both men in G1 and women in G2, which was linked to a balanced conclusion that the course was more beneficial for female students in particular.

The findings are generally presented in a conservative manner ("seems", "we think"), but claims of causality (especially for the gender gap) still remain somewhat speculative. An in-depth theoretical discussion of why female students benefit more is lacking

Why there is no significant improvement in the physics department has not been sufficiently investigated. Is it that there are only 5 physics questions, or that the open-ended lab doesn't transfer directly to the testing skills?

The discussion section ends with a general statement such as "further studies can confirm the data". More concrete proposals are missing: 

   - Working with larger and randomized samples, 

   - Adding questionnaires that directly measure self-efficacy and motivation, 

   - Conducting long-term follow-up studies (impact on university success).

This study examines the impact of open-ended laboratories on students' autonomous preparation skills. The difference of the study from the studies in the literature should be revealed. In particular, it is necessary to compare a different study that focuses on a concrete physical event, focuses on the active exploration and interpretation processes of the students, as well as a contextual and student-centered pedagogy instead of traditional narration, and to reveal the difference between them and discuss them in the literature. In this sense, I think the XXXX (2023) article may be suitable for this part. With the relevant (XXXX (2023)) study, it will strongly strengthen the place of the study you are conducting in the physics education literature.

XXXX (2023). Students’ contextualizing knowledge on the mirage incident, reflection and refraction: a case study. Physics Education, 58(6), 065004.

 

 

Author Response

Dear reviewer, thanks for your attention and your help with our paper. I upload here a PDF with all our responses to your comments and suggestions.

Kind regards

All the Authors

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The necessary corrections have been made to the article. It is suitable for publication.

Author Response

Dear Reviewer,
Thank you for your comments on our paper. As requested by the Editor, please find attached a PDF file with our point-by-point response to the Editor’s comments.

Author Response File: Author Response.pdf