Secondary School Students’ Engagement in Learning Activities: Validation of a Short Scale

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Learning engagement is a very current and important topic. As the authors have described and collected, there are many measuring tools available to measure and examine it. It is not clear to the reviewer what the present questionnaire offers more than the existing ones, what is the validity of the agent area. In my opinion, it is necessary to write about the specificities of the Portuguese education system so that it is clear to the reader.

The authors' goal was brevity, which was achieved. The acceptable reliability of the subscales may result from the fact that the statements are very similar. Furthermore, if a questionnaire is too short, it is not able to examine deeper contents.

I suggest that on the one hand, they write about the education system and its specificities. On the other hand, they thoroughly justify what the questionnaire offers new compared to what is available so far. Third, thoroughly support why a very short measuring instrument is good.

Author Response

Please see attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for letting me read this.

The manuscript Secondary School Students’ Engagement in Learning Activities: 2 Validation of a Short Scale addresses a relevant measurement gap in the student engagement literature by validating a brief 12-item instrument focused specifically on engagement in learning activities (SELA) and explicitly including agentic engagement. The study reports good CFA fit for the four-factor model and provides multiple sources of validity evidence (convergent/discriminant, concurrent, predictive, and invariance).

At the same time, several conceptual and methodological/reporting issues currently limit interpretability and generalizability - especially around sampling description, handling of outliers, ordinal data modeling choices, and measurement invariance depth.

Major comments

1) Please, clarify the sampling frame and resolve an apparent inconsistency (geography vs. Lisbon / public vs. private)

You describe the sample as drawn from “various geographic regions” of the country, yet later specify that 354/566 students attended private schools in Lisbon (62.5%) and 212 attended public schools in Lisbon (37.5%). This reads as inconsistent: is the sample national, or largely Lisbon-based? Please clarify (e.g., where schools were located (city/region); how many schools participated and how they were selected; whether “Lisbon” refers to a broader metropolitan area, and the implications of the public/private imbalance for representativeness).

2) “Random subset within a convenience sample” needs transparent operationalization

The manuscript states that, to reduce bias, you performed a “random selection of a subset of participants within the convenience sample.” This is not a standard remedy unless the sampling frame is clearly defined (i.e., what exactly was randomized - schools, classrooms, students, survey links?). Please, report: the unit of randomization; the procedure (software, seed, inclusion/exclusion criteria); what proportion of the convenience pool was retained.

3) Outlier removal (n = 58) is substantial and must be justified in detail

You removed 58 “severe” outliers before the final analysis (624 → 566). This is a large deletion (~9%). Please, specify: the outlier detection method (e.g., Mahalanobis distance, leverage, z-scores); the cutoff used and whether it was preregistered or post hoc; whether outliers were multivariate on the engagement items only or also on outcomes; sensitivity checks (e.g., CFA/regressions with vs. without outliers, or robust methods instead of deletion).

4) Ordinal Likert data modeled with ML in AMOS: strengthen the justification or consider robust/ordinal estimators

You run CFA in AMOS using maximum likelihood with 5-point Likert items. While ML is often used, the manuscript should justify treating items as continuous and address potential bias in standard errors/fit when data are ordinal. Please, consider: reporting robust corrections (if available), or replicating CFA using an ordinal estimator (e.g., WLSMV/DWLS) as a sensitivity analysis.

This is particularly relevant because you already emphasize fit indices and residual diagnostics.

5) Cognitive dimension shows weak convergent validity. I suggest treating this as a key result, not a side note. You report CR = .63 and AVE = .37 for the cognitive dimension (below conventional AVE thresholds), acknowledging it is weaker than the other dimensions. Currently, this is explained briefly and largely deferred to “future research.”

Please deepen this: Identify which item(s) have the lowest loadings (you report loadings as low as .51) and whether they are conceptually aligned with your definition of cognitive engagement.

Consider whether the cognitive items capture strategy use / self-regulation versus deep learning / critical thinking; your discussion hints at this, but it should be anchored to the actual item content (Table S2).

If you retain the current 3-item cognitive subscale, be explicit about limitations for individual diagnosis and about the recommended use (for example research vs screening).

6) Concurrent validity: clarify the SEQ item selection and correct possible numbering overlap

You state that you selected 3 items per SEQ dimension; however, the numbering shown suggests overlap (e.g., cognitive uses items 3, 9, 12 and behavioral uses 8, 9, 12). This may be a reporting error, but it needs correction because it affects interpretation of concurrent validity.

Also, the SEQ cognitive reliability is quite low (ω = .52), which likely attenuates correlations and should be discussed as a limitation of the concurrent validity test.

7) Predictive validity: provide full measurement details for competence/relatedness and reduce causal language

You regress outcomes on the four SELA dimensions and interpret results through SDT (competence/relatedness), which is reasonable.  However, the manuscript needs (in the Methods) basic psychometrics for the competence/relatedness measures (i.e., number of items, example items, response format, reliability indices, and whether they are validated Portuguese measures).

Additionally, because the design is cross-sectional and largely self-report, tone down causal wording (e.g., “predicted” can be retained statistically, but avoid implying developmental causation).

8) Measurement invariance: go beyond metric invariance (or explicitly limit claims)

You test configural and metric invariance across gender and school year, and conclude the scale can be used “across genders and school years.”

That is acceptable for comparing associations, but if you want to support comparisons of latent means (common in engagement research), you should test at least scalar invariance (and ideally strict). If you do not test scalar invariance, explicitly state that mean comparisons are not yet supported.

 

Minor comments

In the paragraph where you review existing engagement measures, you may also cite a recent scale-validation paper as a comparative benchmark for how engagement dimensions are operationalized and validated via CFA, reliability (CR/AVE/ω), and invariance testing. For example: Sulla, F., Harrad, R., Tontodimamma, A., Limone, P., & Aquino, A. (2023). Italian validation of the Online Student Engagement Scale (OSE) in higher education. Behavioral Sciences, 13(4), 324. https://doi.org/10.3390/bs13040324

This might help you (i) contextualize your four-dimensional structure (behavioral/emotional/cognitive/agentic) within adjacent engagement measurement work and (ii) enrich the discussion on dimensionality and psychometric reporting choices.

Terminology consistency: You switch between “agency” and “agentic” in places (e.g., “agency scoring the lowest”). Standardize terminology to “agentic engagement” throughout for clarity.

Model comparison reporting: You report that the second-order model fit decreases and the difference is statistically significant. Consider adding practical interpretation (how much ΔCFI/ΔRMSEA) and justify why you still discuss the higher-order model despite preferring the four-factor structure.

Ethics for minors: Since participants are aged 11–18, explicitly state procedures for parental consent/assent and how confidentiality was ensured (especially with classroom administration and required responses in Google Forms).

Author Response

Please see attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I accept the modifications and propose the study for publication.