Review Reports

Round 1

Reviewer 1 Report

Thank you for your contribution—fascinating in-depth analysis. Many suggestions will be helpful for Academics. 

Author Response

Thank you very much for the positive feedback.

We have revised the manuscript based on the comments in reviews 2 and 3. We have marked the change in the manuscript.

Reviewer 2 Report

The article provides an exhaustive review of the subject. It is first a descriptive and then a critical analysis of the possible relationships between research on school effectiveness and school improvement. After a brief description of both traditions, the author/s expose the possible links from the analysis of the CF-ESI and DMEE/DASI models, focusing on epistemological and methodological aspects.

The article has an adequate development in the argumentation, although we consider that the first sections of the article do not bring novelty to what has been previously published, in the fourth section of the article is where more novel aspects are shown and with a more personal and critical vision. In this section, special emphasis is placed on the shortcomings of the possible link between research on school effectiveness and school improvement and on the ways in which we are trying to overcome these shortcomings.

The conclusions leave quite a few unresolved doubts that are raised by the authors as new implications. However, we consider that a greater effort could be made to show clearer conclusions.

Figures and tables, although clear, do not have an author, are they self-made? If so, this should be indicated, and if not, the author and year should be indicated.

The bibliography of the last ten years is almost non-existent. It should be added and detailed in the text some works related to these years, especially the last five years.

Author Response

A: Thank you for the helpful comment. We now explained in more detail in the first section of the manuscript what we mean by technological deficit and why this is crucial from a theoretical and methodological perspective for linking school effectiveness and school improvement.

Page 1, line 32- page 2, line 82

“The recent linking models consider the multi-level structure of organised teaching and learning. However, our further considerations problematise that those linking models are obviously not able to take into account the complexity of organisational and interactional processes in a methodologically appropriate way. Following a relevant educational research tradition, teaching can be conceptualised as a complex interaction process constantly generating unforeseeable and unpredictable events that are potentially able to effectively change the directions and aims of classroom communication. In particular Dreeben’s (1970) initial research on how classroom teaching has to get ‘working’ based on weak ‘technologies’ highlighted the complex ‘nature’ of concrete educational interaction. E.g., a didactical or instructional 'technique' processed by the same teacher may work in one class at one time while failing in another class and/or at another time. Thus, the application of such teaching or instructional techniques should not be conceptualised as an independent variable since the same causes that would explain why it works would also explain why it does not. One common methodological way out of these processual causality issues is to construct a person-related causal chain linked by social background or classroom composition variables embedded in a control-group-design: if students do react differently on the same teaching technology applied by the same teacher, then the discriminatory effect can be assumed to be caused by the students. However statistical likelihood based on person variables overlooks the complexity of the process. This may be one reason for the low effect sizes commonly measured within school improvement and school effectiveness research.

Following Dreeben’s approach, Luhmann & Schorr (1988) theorised classroom education itself as a non-linear, acausal and context-dependant interaction process characterised by a structural and in this perspective insurmountable ‘technology deficit’. This deficit makes the realisation of expected or desired learning effects through classroom interaction quite unlikely. Luhmann and Schorr further argue that the education system compensates for the ‘technology deficit’ by building up organisation-based replacement technologies such as ability grouping, rating, tracking and sorting students. It is the organisational dimension that provides output-related expectability, as well as accountability, by using its ability to make decisions.

Rethinking the ‘technological deficit’ by using Weick’s (1976) work on schools as loosely coupled systems forces the idea of a de facto given re-technologisation of education through organisation to be rejected. According to Weick, the decision-based process of organising is primarily grounded in a loose coupling of actions, interpretations and persons rather than in any decision-making technology. Weick’s considerations imply that schools endemically produce uncertainties about whether or not they did, do and will do the ‘right’ things; whether or not the decisions that have been made were goal-attaining – and if they were not, how and what should be decided next. Modern organisation theory claims a more ‘realistic’ and epistemologically reflected view on organisations as complex social systems. School improvement concepts and strategies apparently depend on a realistic or at least multi-perspective picture of the system they intend to change. Weick’s approach may help to avoid unrealistic ‘technological’ notions of what a school organisation is and how it works.

Bringing the interaction-level back in, school system processes appear to be determined by a twofold technological deficit: on the interaction and the organisation level and, as a 'causal' consequence, on the level of 'linking' both as well. How could a loosely coupled organisation system provide causality for a loosely coupled interaction system operating in the classroom? This is the question we think is crucial for any school improvement/school effectiveness linking approach. Regarding methodological implications, we need to consider both process dimensions as dependent variables varying contingently over time and space.”

In addition, we added or made more explicit references to the technological deficit throughout the manuscript in order to make the argument clearer and not to lose the focus.

In section 5 we added a passage to give guidance on how, taking into account the technological deficit a link between school effectiveness and school improvement can be further developed.

 Page 13, line 537-545

“Hence, the question is still whether or not the linking models are based on a ‘realistic’ picture of what occurs in everyday schooling. One apparent problem is that these models are confronted with at least two different process dimensions characterised through incommensurable processual ‘natures’: classroom interaction and organisational decision making. Both dimensions do not operationally ‘link’ in a linear or ‘technical’ manner in reality, nor does each dimension feature processual components that provide causal linkages between process and product. One suggestion our discussion directly implies is the necessity to establish a more advanced and reflexive theory base in a first step and to redesign the processual complexity of education methodologically.

 

R: Figures and tables, although clear, do not have an author, are they self-made? If so, this should be indicated, and if not, the author and year should be indicated.”

A: Thank you very much for the hint! In the process of anonymising the article, the sources of the tables and figures were lost. We added the references again.

 

R: The bibliography of the last ten years is almost non-existent. It should be added and detailed in the text some works related to these years, especially the last five years.

A: We want to thank the reviewer for the call for more recent work, which we understand very well. We did again a systematic extensive literature search, looking for literature covering the primary theoretical discussion of linking school effectiveness and school improvement research. We used Web of Science with the search strings “linking school effectiveness and school improvement” (83 results) and “linking educational effectiveness and school improvement” (64 results) for publication dates between 2012-2022 in the two categories “Education Educational Research” and “Education Scientific Disciplines”. Furthermore, we used backwards search for citations of the relevant core publications. But we did not find much newer literature covering the specific topic of linking school effectiveness research and school improvement, while we found a lot of empirical work which did not add to the rather theoretical discussion of designing models of linking school effectiveness research and school improvement. A full bibliography of educational effectiveness and/or school improvement research is beyond the scope of our paper, because we focus on the rather theoretical discussion of linking both research traditions. Two of the authors' anonymised sources deal specifically with the issue of linking school effectiveness research and school improvement; they are both from 2017. The most actual reference is to our knowledge the slightly revised DMEE by Kyriakides et al. (2020), which we cited already.

Nevertheless, to respond to the call, we added some new state of the art references. And we would be grateful for any suggestions of relevant literature we are not aware of yet!

We added the following references, which focus on the idea of research utilization of the knowledge base of educational effectiveness research for school improvement:

Reynolds, D.; Sammons, P.; Fraine, B.; van Damme, J.; Townsend, T.; Teddlie, C.; Stringfield, S. Educational effectiveness research (EER): a state-of-the-art review. School Effectiveness and School Improvement, 2014, 25, 197–230.

Scheerens, J. (2016). Recapitulation and Application to School Improvement. In J. Scheerens (Ed.), Educational Effectiveness and Ineffectiveness: A Critical Review of the Knowledge Base (p. 291–332). Springer Netherlands. https://doi.org/10.1007/978-94-017-7459-8_12

Scherer, R., & Nilsen, T. (2019). Closing the gaps? Differential effectiveness and accountability as a road to school improvement. School Effectiveness and School Improvement, 30(3), 255–260. https://doi.org/10.1080/09243453.2019.1623450.

Reynolds, D.; Neeleman, A. School Improvement Capacity. In: Concept and Design Developments in School Improvement Research. Oude Groote Beverborg, A., Feldhoff, T., Maag Merki, K., Radisch, F., Eds.; Springer International Publishing: Cham, 2021.

 

Furthermore we added new empirical papers quantitative school improvement research:

Heck, R. H., & Chang, J. (2017). Examining the Timing of Educational Changes Among Elementary Schools After the Implementation of NCLB. Educational Administration Quarterly, 53(4), 649–694. https://doi.org/10.1177/0013161X17711480

Heck, R. H., & Reid, T. (2020). School leadership and school organization: Investigating their effects on school improvement in reading and math. Zeitschrift Für Erziehungswissenschaft, 23(5), 925–954. https://doi.org/10.1007/s11618-020-00969-3

Sleegers, P. J. C., Thoonen, E. E. J., Oort, F. J., & Peetsma, T. T. D. (2014). Changing classroom practices: The role of school-wide capacity for sustainable improvement. Journal of Educational Administration, 52(5), 617–652. https://doi.org/10.1108/JEA-11-2013-0126

 

Furthermore, we added new paper describing the school as unit of action:

Fend, H. Schule als pädagogische Handlungseinheit im Kontext. In: Schulgestaltung: Aktuelle Befunde und Perspektiven der Schulqualitäts- und Schulentwicklungsforschung. Grundlagen der Qualität von Schule 2. Steffens, U., Maag Merki, K., Fend, H., Eds., 1^st ed.; Waxmann: Münster, 2017; pp. 85–102.

Reviewer 3 Report

This is an interesting, well-organized, and very well-articulated piece of work, taking into account the authors' knowledge and analysis of the specialized literature within the area of study, both for reference works and current studies. However, it seems as if they also forgot to develop the construct “technological deficit” as regards “school effectiveness research and school improvement” throughout the paper. This conditions the writing, as authors mainly justify it (the technological deficit concerning school effectiveness research and school improvement) almost exclusively using the following statement: “the claim of evidence-based school improvement processes ultimately fails due to the structural technology deficit of the school” (p. 12). Therefore, it is highly recommended not to lose the focus (main objective) of the paper –see abstract, first line– and to develop the manuscript content from here. Finally, other comments or recommendations to be taken into account are included in the attached file.

Comments for author File: Comments.pdf

Author Response

Thank you for the helpful comment. We explained in more detail in the first section of the manuscript what we mean by technological deficit and why this is crucial from a theoretical and methodological perspective for linking school effectiveness and school improvement.

Page 1, line 32- page 2, line 82

“The recent linking models consider the multi-level structure of organised teaching and learning. However, our further considerations problematise that those linking models are obviously not able to take into account the complexity of organisational and interactional processes in a methodologically appropriate way. Following a relevant educational research tradition, teaching can be conceptualised as a complex interaction process constantly generating unforeseeable and unpredictable events that are potentially able to effectively change the directions and aims of classroom communication. In particular Dreeben’s (1970) initial research on how classroom teaching has to get ‘working’ based on weak ‘technologies’ highlighted the complex ‘nature’ of concrete educational interaction. E.g., a didactical or instructional 'technique' processed by the same teacher may work in one class at one time while failing in another class and/or at another time. Thus, the application of such teaching or instructional techniques should not be conceptualised as an independent variable since the same causes that would explain why it works would also explain why it does not. One common methodological way out of these processual causality issues is to construct a person-related causal chain linked by social background or classroom composition variables embedded in a control-group-design: if students do react differently on the same teaching technology applied by the same teacher, then the discriminatory effect can be assumed to be caused by the students. However statistical likelihood based on person variables overlooks the complexity of the process. This may be one reason for the low effect sizes commonly measured within school improvement and school effectiveness research.

Following Dreeben’s approach, Luhmann & Schorr (1988) theorised classroom education itself as a non-linear, acausal and context-dependant interaction process characterised by a structural and in this perspective insurmountable ‘technology deficit’. This deficit makes the realisation of expected or desired learning effects through classroom interaction quite unlikely. Luhmann and Schorr further argue that the education system compensates for the ‘technology deficit’ by building up organisation-based replacement technologies such as ability grouping, rating, tracking and sorting students. It is the organisational dimension that provides output-related expectability, as well as accountability, by using its ability to make decisions.

Rethinking the ‘technological deficit’ by using Weick’s (1976) work on schools as loosely coupled systems forces the idea of a de facto given re-technologisation of education through organisation to be rejected. According to Weick, the decision-based process of organising is primarily grounded in a loose coupling of actions, interpretations and persons rather than in any decision-making technology. Weick’s considerations imply that schools endemically produce uncertainties about whether or not they did, do and will do the ‘right’ things; whether or not the decisions that have been made were goal-attaining – and if they were not, how and what should be decided next. Modern organisation theory claims a more ‘realistic’ and epistemologically reflected view on organisations as complex social systems. School improvement concepts and strategies apparently depend on a realistic or at least multi-perspective picture of the system they intend to change. Weick’s approach may help to avoid unrealistic ‘technological’ notions of what a school organisation is and how it works.

Bringing the interaction-level back in, school system processes appear to be determined by a twofold technological deficit: on the interaction and the organisation level and, as a 'causal' consequence, on the level of 'linking' both as well. How could a loosely coupled organisation system provide causality for a loosely coupled interaction system operating in the classroom? This is the question we think is crucial for any school improvement/school effectiveness linking approach. Regarding methodological implications, we need to consider both process dimensions as dependent variables varying contingently over time and space.”

 

In addition, we added or made more explicit references to the technological deficit throughout the manuscript in order to make the argument clearer and not to lose the focus.

 

Answers to the comments of review 3 directly in the manuscript

Page 1, line 25

R: Support this statement by adding relevant references. Please apply this to the remaining parts of the manuscript where necessary.

A: Thank you for pointing this out. We added the relevant references at this position and carefully checked the manuscript for other places where a reference to sources is useful and necessary. References were added where necessary.

 

Page 2, line 56 R: Avoid this type of expression.

A: We have removed the expression.

 

Page 2, line 59, R: Taking into account the publication dates of these works, my question is: Does a "technological deficit" still exist today according to the (latest) scientific literature in terms of "classroom action" and "school organisational decision-making in schools"? Or, should it (technological deficit) simply be approached on the basis of other elements such as "model improvement" as it the focus of analysis of the current paper?

A: As we have hopefully made clear through the additions in section 1, the technology deficit is immanent and cannot be fixed by more empirical research based on "classical" causal assumptions or by elaborate methods such as multilevel analysis or structural equation modelling.

 

Page 2, line 81, R: Check the citation style. Please apply this to other cases in the manuscript where necessary.

A: In the current revision, the citation style was adapted to the journal's specifications and thoroughly reviewed once again.

 

Page 4, line 174

A: We removed the typo.

 

Page 6, line 235, R: Perhaps authors should avoid attaching images as tables as red lines appear for certain words, which is common to (word processing) grammar-checking programmes.

A: Thank you for the advice. In our manuscript we did not use images for the table. We assume that the images were created when the manuscript was uploaded to the review platform.  We used the original tables again and made sure that they are preserved when uploading.

 

Page 10, line 398

A: We removed the typo from “Summary” to “summary”.

 

Page 10, line 410, R: So far, little has been said about "technological deficit", which may cause the reader to miss (a large) part of the rationale of the study.

A: see above our answer to your overall comment.

We explained in more detail in the first section of the manuscript what we mean by technological deficit and why this is crucial from a theoretical and methodological perspective for linking school effectiveness and school improvement.

 

 

Page 11, line 436, R: Year of publication.

A: We added the year of the publication.

 

Page 12, R: “structural technology” This concept should be further explained. What does it imply?

A:  see above our answer to your overall comment.

We explained in more detail in the first section of the manuscript what we mean by technological deficit and why this is crucial from a theoretical and methodological perspective for linking school effectiveness and school improvement.

 

R: Page 13, line 541 R: Any suggestion?

A: In section 5 we added a passage to give guidance on how a link between school effectiveness and school improvement can be further developed taking into account the technological deficit

Page 13, line 537-545

“Hence, the question is still whether or not the linking models are based on a ‘realistic’ picture of what occurs in everyday schooling. One apparent problem is that these models are confronted with at least two different process dimensions characterised through incommensurable processual ‘natures’: classroom interaction and organisational decision making. Both dimensions do not operationally ‘link’ in a linear or ‘technical’ manner in reality, nor does each dimension feature processual components that provide causal linkages between process and product. One suggestion our discussion directly implies is the necessity to establish a more advanced and reflexive theory base in a first step and to redesign the processual complexity of education methodologically.”

Reviewer 4 Report

I'd like to thank the author for the manuscript, but I have to say that from my point of view the manuscript should be rejected. 

The topic is interesting and the author adequately describes the different positions and models to link those two. But that is not new and the models part of the body of knowledge. The manuscript fails to enlarge this knowledge and/or makes no a proposal for a new model. 

In addition the manuscript lacks a concise style of writing that makes the article difficult to read.

Author Response

Thank you very much for the feedback. Unfortunately, the feedback does not give us any concrete indications for improving the content of our manuscript. Therefore, we based our revision on the advice given in reviews 2 and 3.

We have marked the changes in the manuscript.

Round 2

Reviewer 4 Report

After revisiting the manuscript I tend to side with my colleague reviewer 2 who wrote that the article provides an exhaustive review of the subject. I also se the use of such a thorough review for the forthcoming research.

The added paragraphs helped to see the argumentation of the manuscript and improved the readability.

The conclusions are now more clear. The implications could be more bold for my test to fuel further discussion of the possible link between school effectiveness and school improvement.

Overall I see the improvement.