Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

Thank you for an interesting manuscript on self-regulated learning and cognitive load in learner-controlled environments. The manuscript reads well but the structure and depth could be improved. Especially the structure of the introduction needs some revisions. Also, the use of SRL theory and work on combining SRL and CLT could be much stronger. I have made some suggestions to improve the manuscript.

 

Major:

-            As self-regulation of behavior in general refer to a large field with many different type of studies, I would recommend the authors to specifically refer to self-regulated learning throughout the manuscript.

-            The first part of the introduction seems to long and detailed to fit the purpose. One the one hand information is missing to fully understand what they authors mean on the other hand, it is too detailed for just introducing the aims of the study. Please adjust this first part, the foreshadowing of the study.

-            The different parts of the introduction are not very well connected. It would be good to make sure the part are connected in such a way it becomes clear why based on cognitive load theory and learning in a learner-controlled environment, it is interesting, perhaps crucial even, to look at self-regulated learning. Or perhaps the authors can reconsider the structure of the literature review part and start with self-regulated learning in learner-controlled environments and why this would be challenging. That would create an almost natural bridge to the cognitive load theory.

-            In relation to the previous point, there seems to be bits of cognitive theory throughout all the sections in the literature review. It would be easier to read if the literature review section was made up of coherent sub section covering cognitive load theory and SRL subsequently and only after those have a section in which these are integrated. For example, it would be good if 2.4 was already part of 2.1.

-            A proper definition of SRL, based on one of the models available is lacking. Please add this and connect it to the argumentation about self-regulated learning, cognitive load in learner-controlled environments.

-             When discussing self-regulated learning and cognitive load, the authors state: ‘Learners’ level of self-regulation serves as a source of diagnostic insight when confronted with high intrinsic complexity of the material, allowing them to overcome challenges’. It would be good to explain this further as it seems crucial to the argumentation in the manuscript. Also: ‘create low levels of intrinsic load by breaking up 158 element interactivity’, please elaborate and explain how this works. See comment about the hypotheses as well.

-            About the hypotheses: It is unclear why self-regulated learning skills would have a negative relation with intrinsic cognitive load. It seems unlikely as intrinsic load refers to the number of interactive elements in the task itself, which is not related to the SRL skills a learner has. I think this hypotheses is not valid. Please also adjust the discussion of the findings for this hypothesis if you agree. Include the cognitive load theory and SRL theory in this discussion.

-            The discussion of the results could be improved. Especially by including some of the recent articles that have proposed connection between SRL and CLT. In the discussion, it would be insightful to integrate the work by

o   de Bruin et al. (2020) see de Bruin, A. B., Roelle, J., Carpenter, S. K., Baars, M., & Efg-Mre. (2020). Synthesizing cognitive load and self-regulation theory: A theoretical framework and research agenda. Educational Psychology Review, 32(4), 903-915.

o   Baars, M., Wijnia, L., de Bruin, A., & Paas, F. (2020). The relation between students’ effort and monitoring judgments during learning: A meta-analysis. Educational Psychology Review, 32, 979-1002.

-            And for the discussion the S2D2 framework could be of interest as well, how do the results relate to this: de Bruin, A. B., Biwer, F., Hui, L., Onan, E., David, L., & Wiradhany, W. (2023). Worth the effort: the start and stick to desirable difficulties (S2D2) framework. Educational Psychology Review, 35(2), 41.

Minor:

-            The SEM results in te graph are hard to read

Author Response

 

See attached document 

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript uses a learner-control environment to examine relationships between prior-knowledge (PK) and self-regulated learning (SRL) with different types of cognitive load and test performance. Equation modelling was used to show how these factors interacted. This is an appealing study and of interest to the field and potentially publishable. However, there are some main issues that need to be resolved; mainly

1.     Lack of detail on the study materials

2.     Missing literature on previous studies that have investigated such relationships

3.     Potential misinterpretation of the results and implications.

The following outlines these issues

Issue 1. 

A)   To measure the 3 cognitive loads (intrinsic, extraneous, germane) the survey of Leppink et al. (2013) was adapted (see Line 247). Because of the importance of this survey (see below) each item should be reported along with a justification for the modifications. Furthermore, it is stated that a 10-point Likert scales (0-10) was used (line 246)- is this a 11 point scale if it ranges from 0 to 10?

B)   Line 258 states that the study materials were the same as used by the ‘Authors’. I find this annoying as no detail is given without referencing other work. I’m confused here as use of Authors is usually to keep the manuscript blind to reviewers. However, they are listed in the reference list. Either way, I expect most if not all details of the experiment to be described in the manuscript, for easy reading. The detail is very important as described below.

Issue 2.

A)   There are many studies that have used different types of cognitive load measures- perhaps not so many that have included SRL. Surely there is literature that have reported correlations between the 3 loads and other factors that could help frame the hypotheses and be used in the general discussion? 

Issue 3.

The mean scores of the 3 cognitive loads are informative. Extraneous was very low (2.13 out of 10), intrinsic below average, and germane very high. The various correlations (or lack of) found need further discussion

A)   Were the materials designed to generate extraneous load? A mentioned above, as they were not described directly, I do not know whether they were designed to create such load. If not is there any point in measuring it, other than to investigate a low extraneous load environment. This may be a limitation.

B)   If extraneous load is low, would we expect any correlation with intrinsic load which may vary differently? Aren’t they independent of each other? However, there is a significant correlation. Why? Is there other literature that supports this relationship? Is this correlation just dependent upon these materials?

C)   There is no correlation between intrinsic and germane load. Is this expected? More recent definitions of germane load (Sweller et al., 2019; Sweller et al., 2011) have linked it more directly with intrinsic load, rather than an independent construct. Do your results support such assumptions?

D)   The germane load is very high (8.37 out of 10), whereas the other loads are low. If germane load is supposed to be the mental load generated in schema acquisition, why is it so high when the negative loads were low. Surely low extraneous and intrinsic does not require much learning effort? Germane load is not an outcome, but load directed at learning.

E)    There is line of thinking which may explain/ question some of the interpretations in this study: does the Leppink survey actually measure the loads, as assumed? This is why it is particularly important to describe the survey used in this study. The wording of the original survey items features content complexity (representing intrinsic load), instructional/ explanations (representing extraneous load) and understanding (representing germane load). In particular, the link between germane load and understanding is not so clear as the other items. For example, the Zheng book chapter by Ayres (2018) [https://doi.org/10.4324/9781315296258-2] argues that this construct may simply be a factor that represents understanding. Leppink et al (2014) in later times questioned whether this construct was actually germane load. Given the recent changes to the definition of germane load since its original conception by Sweller et al. (2018), there is reasonable doubt over the measurement of germane load. 

F)    If this factor does represent understanding, then there is a very simple explanation for some of the results: that is both high PK and SRL should lead to better understanding. Furthermore, understanding is high because the extraneous load is very low and the content is not difficult either. Therefore, high levels of understanding are expected.

G)   Perhaps the findings of the current study can be used to question the use of this survey to measure germane load. In my opinion the results indicate that germane load is not measured directly, only understanding.

H)   There is no correlational data recorded between PK and SRL. Should this be reported? Does PK underpin SRL?

I)     One problem with learner control experiments is that participants can take their time and thus some have more time for learning than others, and thus this factor cannot be controlled. Was time measured? Are there any data on rewinds etc.? Did subjects have access to other resources?

In summary, the study has merit but can be improved by a deeper analysis of the results and broader conclusions. I think there is evidence that the Leppink survey does not necessarily measure (germane load) what it is supposed to, even though 3 distinct factors are usually identified. I think understanding is more an outcome of germane processes and not necessarily the same thing. More detail of the method is required to understand the conditions more thoroughly. There is very low extraneous load (on purpose?) and therefore it may be difficult to conclude too much about extraneous load. Further experiments that manipulate this load directly would be helpful. 

Author Response

See attached file 

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript investigates the relationships between prior knowledge, self-regulated learning skills, and cognitive load in learner-controlled environments. This is a valuable and promising area of research because (a) self-regulated learning environments constitute a significant portion of learning contexts, especially with the integration of technology, and (b) our understanding of the role of cognitive load in self-regulated environments needs further development. The conceptual exploration of the relationships between prior knowledge and the various forms of cognitive load, according to the learner’s control over their learning, is particularly interesting. The results did not confirm expectations regarding extraneous cognitive load (ECL) and intrinsic cognitive load (ICL). Only a mediation effect with germane cognitive load (GCL) was observed, but this finding should be interpreted with great caution due to a lack of data about self-control behaviors and a debatable measure of GCL. This measure can be considered a self-assessment of the learner's performance, requiring metacognitive skills and prior knowledge. Therefore, the results can be interpreted through the perspective of judgment of learning and this judgement is usually better among high-knowledge and high-metacognition learners. 
From my perspective, the results do not provide clear information about the role of different forms of cognitive load in learner-controlled environments, and the measure remains debatable. However, the manuscript could be significantly improved by thoroughly discussing the study's limitations and the conceptualization of cognitive load in relation to the results in self-regulated environments. I suggest substantial revisions to enhance the manuscript and make a meaningful conceptual contribution to research on cognitive load in learning environments. 

Please find below different points to improve the manuscript: 

- I question the informativeness of the section “2.4. Cognitive Load and Performance” in its current form. The points discussed regarding control, cognitive load, and performance could be integrated into the previous sections. 

- The hypotheses are clear, each stating the causal relationship between two variables. A model displaying an overall view of the hypotheses could be more interesting. A single model should show how the relationships between variables are structured (prior knowledge, different types of cognitive load, academic performance). Are the forms of cognitive load mediating variables between learners’ characteristics (prior knowledge and self-regulation skills) and performance? Given that structural equation modeling was used in the results, the hypotheses should reflect the tested model. 

- Examples of the measuring items should be presented (for cognitive load forms and self-regulated learning skills). 
- I am unsure if the measure of self-regulated skills assesses actual skills. The MSLQ assesses self-reported strategies. Therefore, the current study examined the effects of habits of self-regulated strategies rather than real and objective skills. That conceptual point is essential. 

- Was the prior knowledge test reliable, with a good Cronbach’s alpha? 

- Performance was assessed through three dimensions of cases: solution, argumentation, and terminology. How were the scores coded? Who did the scoring? Did the authors use double assessment with different judges checking Cohen’s kappa? This point is essential to ensure reliable data. 

- I believe that no comments should be made about relationships that are not significant. The authors state, “the negative relationship between students' self-regulated learning skills and extraneous load is consistent with previous literature [2]. This suggests that although the statistical analysis did not yield significant results, the theoretical basis and prior research provide support for the expected direction of these relationships.” If the authors defend this non-significant result, they should also report that ECL is positively related to performance almost at the 0.10 level, but this relationship is the reverse of the expectation. In sum, non-significant results should not be reported as consistent with expectations. 

- A main limitation of the study concerns the lack of recording self-control behaviors. It is difficult to conclude that self-regulation explains why high prior knowledge learners and habits of self-regulation predict better performance. Observations of objective self-control behaviors, expected to be higher or more efficient for such learners, would have been necessary to conclude this point. It is crucial to remain very cautious in discussing the role of self-control. 

- In line with the previous point, the measure of germane cognitive load might be considered a self-evaluation by learners of their own comprehension. If the items were, “The activity really enhanced my understanding of the topic(s) covered,” “The activity really enhanced my knowledge and understanding of statistics,” “The activity really enhanced my understanding of the formulas covered,” and “The activity really enhanced my understanding of concepts and definitions,” then it is not surprising that prior knowledge and self-regulation habits positively predict these ratings of GCL. Both factors contribute to better self-assessment. Finally, because the ratings of GCL reflect the level of self-assessment of their performance (understanding), it is somewhat obvious that the ratings predict outcomes. 

- It is very surprising that prior knowledge did not reduce intrinsic cognitive load. 

- The main limitations of the study should be thoroughly discussed. There are no substantial reflections on them. 

Author Response

See attached file

The manuscript investigates the relationships between prior knowledge, self-regulated learning skills, and cognitive load in learner-controlled environments. This is a valuable and promising area of research because (a) self-regulated learning environments constitute a significant portion of learning contexts, especially with the integration of technology, and (b) our understanding of the role of cognitive load in self-regulated environments needs further development. The conceptual exploration of the relationships between prior knowledge and the various forms of cognitive load, according to the learner’s control over their learning, is particularly interesting. The results did not confirm expectations regarding extraneous cognitive load (ECL) and intrinsic cognitive load (ICL). Only a mediation effect with germane cognitive load (GCL) was observed, but this finding should be interpreted with great caution due to a lack of data about self-control behaviors and a debatable measure of GCL. This measure can be considered a self-assessment of the learner's performance, requiring metacognitive skills and prior knowledge. Therefore, the results can be interpreted through the perspective of judgment of learning and this judgment is usually better among high-knowledge and high-metacognition learners. From my perspective, the results do not provide clear information about the role of different forms of cognitive load in learner-controlled environments, and the measure remains debatable. However, the manuscript could be significantly improved by thoroughly discussing the study's limitations and the conceptualization of cognitive load in relation to the results in self-regulated environments. I suggest substantial revisions to enhance the manuscript and make a meaningful conceptual contribution to research on cognitive load in learning environments. gates the relationships between prior knowledge, self-regulated learning skills, and cognitive load in learner-controlled environments. This is a valuable and promising area of research because (a) self-regulated learning environments constitute a significant portion of learning contexts, especially with the integration of technology, and (b) our understanding of the role of cognitive load in self-regulated environments needs further development. The conceptual exploration of the relationships between prior knowledge and the various forms of cognitive load, according to the learner’s control over their learning, is particularly interesting. The results did not confirm expectations regarding extraneous cognitive load (ECL) and intrinsic cognitive load (ICL). Only a mediation effect with germane cognitive load (GCL) was observed, but this finding should be interpreted with great caution due to a lack of data about self-control behaviors and a debatable measure of GCL. This measure can be considered a self-assessment of the learner's performance, requiring metacognitive skills and prior knowledge. Therefore, the results can be interpreted through the perspective of judgment of learning and this judgment is usually better among high-knowledge and high-metacognition learners. From my perspective, the results do not provide clear information about the role of different forms of cognitive load in learner-controlled environments, and the measure remains debatable. However, the manuscript could be significantly improved by thoroughly discussing the study's limitations and the conceptualization of cognitive load in relation to the results in self-regulated environments. I suggest substantial revisions to enhance the manuscript and make a meaningful conceptual contribution to research on cognitive load in learning environments.	Thank you for your valuable feedback, the manuscript has been substantially revised following yours and other reviewers’ comments.
I question the informativeness of the section “2.4. Cognitive Load and Performance” in its current form. The points discussed regarding control, cognitive load, and performance could be integrated into the previous sections.	Thank you for this suggestion, the literature review was significantly restructured, this section is not part of the structure anymore.
The hypotheses are clear, each stating the causal relationship between two variables. A model displaying an overall view of the hypotheses could be more interesting. A single model should show how the relationships between variables are structured (prior knowledge, different types of cognitive load, academic performance). Are the forms of cognitive load mediating variables between learners’ characteristics (prior knowledge and self-regulation skills) and performance? Given that structural equation modeling was used in the results, the hypotheses should reflect the tested model.	Thank you for this comment, indeed the terminology used is not quite appropriate. We did not run mediation analysis, rather we looked at direct and indirect relationships among variables. It is now more clearly stated in the manuscript.
Examples of the measuring items should be presented (for cognitive load forms and self-regulated learning skills).	Thank you for this suggestions, the items are added in the Appendix A and Appendix B
I am unsure if the measure of self-regulated skills assesses actual skills. The MSLQ assesses self-reported strategies. Therefore, the current study examined the effects of habits of self-regulated strategies rather than real and objective skills. That conceptual point is essential.	Thank you for this comment. We talk about self-regulated learning in this study, however we only used one scale from the MSLQ questionnaire, which is labeled “Self-regulation”. This scale includes both metacognitive and effort management items and is a better predictor of performance (Pintrich & de Groot, 1990). We have explained it in the manuscript more clearly, also including previous research suggesting that self-report surveys (like MSLQ) have certain limitations, as well as explaining what is assumed to be measured by this questionnaire (overall metacognitive skills, rather than the use of specific strategies). We hope it is now more clear.
Was the prior knowledge test reliable, with a good Cronbach’s alpha?	Thank you for this question, Cronbach’s alpha for prior knowledge was low (0.42) with mean inter-item-correlation equals 0.082. But that is a common problem of  wide-ranging conceptual knowledge tests. This is now included in the manuscript along with the reference to previous literature.
Performance was assessed through three dimensions of cases: solution, argumentation, and terminology. How were the scores coded? Who did the scoring? Did the authors use double assessment with different judges checking Cohen’s kappa? This point is essential to ensure reliable data.	Thank you for this suggestion, the assessment was scored by the instructor and his teaching assistant, it is now stated in the manuscript.
I believe that no comments should be made about relationships that are not significant. The authors state, “the negative relationship between students' self-regulated learning skills and extraneous load is consistent with previous literature [2]. This suggests that although the statistical analysis did not yield significant results, the theoretical basis and prior research provide support for the expected direction of these relationships.” If the authors defend this non-significant result, they should also report that ECL is positively related to performance almost at the 0.10 level, but this relationship is the reverse of the expectation. In sum, non-significant results should not be reported as consistent with expectations.	Thank you, we agree and have deleted the paragraph mentioning insignificant relationships.
A main limitation of the study concerns the lack of recording self-control behaviors. It is difficult to conclude that self-regulation explains why high prior knowledge learners and habits of self-regulation predict better performance. Observations of objective self-control behaviors, expected to be higher or more efficient for such learners, would have been necessary to conclude this point. It is crucial to remain very cautious in discussing the role of self-control.	Thank you for your comment, self-control construct is out of the scope of this research, although we do talk about self-management of cognitive load which might be connected and is definitely of interest for future research.
In line with the previous point, the measure of germane cognitive load might be considered a self-evaluation by learners of their own comprehension. If the items were, “The activity really enhanced my understanding of the topic(s) covered,” “The activity really enhanced my knowledge and understanding of statistics,” “The activity really enhanced my understanding of the formulas covered,” and “The activity really enhanced my understanding of concepts and definitions,” then it is not surprising that prior knowledge and self-regulation habits positively predict these ratings of GCL. Both factors contribute to better self-assessment. Finally, because the ratings of GCL reflect the level of self-assessment of their performance (understanding), it is somewhat obvious that the ratings predict outcomes.	Thank you, it is an important point and we have significantly revised the part about conceptualization of germane load and what is being measured by Leppink et al’s instrument.
It is very surprising that prior knowledge did not reduce intrinsic cognitive load.	We cannot agree more, however we attempt to provide an explanation for that in the discussion section.
The main limitations of the study should be thoroughly discussed. There are no substantial reflections on them.	Thank you, the limitations section is significantly expanded.

 

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Thank you very much for giving me the opportunity to read and review this manuscript. I think that the general topic (combining CLT with SRL) is very relevant. Especially because the study took place in law. However, I am a bit critical because although the general topic is very relevant I doubt that this study presents new insights. I have some major suggestions that need to be fixed before considering this manuscript to be published:

1)     Your study is correlational: please do not give any suggestions towards how to design learning environments. Your results only count for learners-controlled environments.

2)     In the abstract it should be mentioned that this is a correlational study in law and that 97 law students took part etc.

3)     You name the types of load quite early, but they are defined much later. Could this be fixed?

4)     P. 2 l 45 self-regulated learning skills or self-regulation skills? What do you mean?

5)     P. 2 l. 69 I think that CLT represents more than this.

6)     You define CL with three loads. Please, consider more recent work, because recent work is critical with the old view on the three types of load.

7)     P. 2 l. 84 What is THE cognitive load effect? Never heard of it. Do you mean effects?

8)     P. 3 l.  127 comma

9)     Chapter 2.4 is random. It should be incorporated in the other chapters.

10)  Students from which country?

11)  Should the correlations be presented in the results section or in the methods section to show what each factor measured. This again could be compared to other studies because I am quite sceptical – I do not really trust the Leppink scale totally.

12)  Can you give more detailed information on how you analysed the students’ answers to measurr peformance?

13)  In Table 1: please present means etc. for the three loads also

14)  Follow APA to present p values etc.

15)  There is a paper by Rovers et al. on granularity of SRL. You should consider it (at least within the discussion). How relevant was SRL when only learning a few minutes/hours.

16)  The start of the conclusion is another chapter (limitations). This could be put in the discussion. The conclusion should be a conclusion starting like your second paragraph of the conclusion.

17)  Your SRL instrument measures the quantity of strategy use, not the quality. This could have an effect. Wirth and Leutner (2008) wrote about it.

18)  Last paragraph: You did not explore the absence of guidance. You explored how different constructs correlate WHEN learning in an environment without guidance.

19)  The most critical aspect is: What’s new? Why is this work/study relevant? This is not clear to me. Especially because much work that was already done combining (and criticizing) SRL*CLT is not considered in your manuscript.

I wish you the best!

Comments on the Quality of English Language

good, but check APA

Author Response

See attached file 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors, 

Thank you for addressing my comments. I think the revisions have improved the manuscript. 

Author Response

Thank you for all your help. 

 

 

Reviewer 2 Report

Comments and Suggestions for Authors

My issues have been addressed and I recommend publication. My main issue about measuring the construct germane load has been mostly dealt with, although I think the authors could have gone a bit harder on it- in the sense that it really might not measure germane load directly, despite new definitions. I believe that it  purely measures understanding, and a result easily explained  by "with sufficient prior knowledge and self-regulation skills subjects found the topic easy to understand". However, I do not really object to the present discussion and hope it might stimulate this group and other researchers to examine measurement of germane load further. I would like to have seen the wording of the items (in English) within the document to stress this point further, but leave that to the editors. In summary, a good article that should lead to further ideas.

Author Response

Thank you for all your help. 

Reviewer 4 Report

Comments and Suggestions for Authors

Dear Authors,

I think that the manuscript has definitely profited from all the comments and revisions. However, there are still a few minor things that have to be addresses before the manuscript can be published.

1.      Page 2, line 70: I would write: “Originally, researchers classify…” Later you describe that GCL has another role.

2.      Page 2, lines 70 following: Are the loads represented by… or are the loads imposed by …?

3.      Chapter 2.1: Why suddenly methods in the theoretical background? This seems random to me.

4.      Page 12, line 542 following: your explanation on learners with low amount of intrinsic load would end in a correlation, or? But you do not fin this correlation?! Please clarify (maybe I am wrong).

5. Still I miss the unique aspect of your study. What is special and what makes it different to all existent studies?

Comments on the Quality of English Language

good.

Author Response

Thank you for all your help. 

Author Response File: Author Response.pdf