What It Really Takes: The Costs and Commitments Behind a Successful Coaching Model for Afterschool STEM Educators

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear author, first of all, I would like to congratulate you on the work you have presented. However, I would like to make some comments on how to improve certain parts of it.
Firstly, formally speaking, your manuscript does not have a methodology.
But in fact, you use three methods:
Conceptual/narrative literature review (Section 3)
Illustrative case analysis (ACRES) (Section 4)
Analytical framework design (CBCA) (Section 5)
The problem is that you never declare or justify them.
You need an explicit section, even if it is brief, establishing that your work is a:
Conceptual and critical review with an illustrative case study
This gives the article epistemological legitimacy.
On the other hand, ACRES is your greatest strength... and your greatest risk.
Right now:
ACRES appears as evidence
But also as a product of your organisation
And also as an example that validates your own framework
This creates a serious risk of authorship bias if you do not explicitly address it.
You need to add three things:
(a) Justification of the case
You need to explain why ACRES is methodologically appropriate:
ACRES was selected because it is a long-running, multi-site, nationally scaled OST coaching model with longitudinal data and diverse funding streams, making it well-suited to illustrate systemic investment dynamics.
(b) Limitations
You must acknowledge limitations:
Because ACRES is a relatively well-resourced and nationally visible initiative, its investment profile may differ from smaller or less formal OST programmes.

(c) Reflective positioning
You must recognise your role:
The author is affiliated with the organisation that developed ACRES. This positionality provides access to rich programmatic data but also requires explicit attention to analytic reflexivity.
This does not weaken you: it protects you.
Finally, the CBCA is powerful, but right now it seems like an “invented” model without process.
You need to state clearly:
•    It is not a psychometric scale.
•    It is not a quantitative instrument.
•    It is a heuristic evaluative framework based on systems theory, infrastructure, and relational learning.
You should add something like:
The CBCA is proposed as a heuristic evaluative framework rather than a standardised measurement instrument. Its purpose is to support structured professional and funder judgement about system-level capacity, not to generate commensurate numerical scores.

 

Author Response

Comment 1: You need an explicit section, even if it is brief, establishing that your work is a:
Conceptual and critical review with an illustrative case study

Response 1: I added a brief section on methodological approach

2. Methodological Approach

I approach this commentary as a theory synthesis–based conceptual study combined with analytic framework construction and an illustrative case analysis. Following Jaakkola’s (2020) characterization of theory synthesis, I integrate scholarship from cost-effectiveness research and equity-centered OST STEM PL to reconceptualize how “cost” and “return” are understood in OST PL systems. The overarching purpose is to engage with these strands of thought to explore how dominant cost frameworks shape decision-making and advance an alternative analytic perspective that program developers and evaluators may find better aligned with equity-centered, capacity-building work.

This synthesis proceeds in three steps. After attending to the national context in which this inquiry is situated, I develop a critical integration of research on cost and cost-effectiveness in education, with particular attention to how these approaches operate in OST STEM contexts. This review surfaces underlying assumptions about efficiency, measurement, and short-term outcomes that structure existing evaluations. Drawing on this synthesis, I then construct a conceptual analytic framework by specifying key domains that may guide alternative analyses (Jabareen, 2009). The resulting framework translates the theoretical integration into a structured approach for examining how monetary and non-monetary investments relate to longer-term system capacities. Third, I include an illustrative case application to demonstrate how the framework can be used in practice.

Comment 2: 

(a) Justification of the case
You need to explain why ACRES is methodologically appropriate:
ACRES was selected because it is a long-running, multi-site, nationally scaled OST coaching model with longitudinal data and diverse funding streams, making it well-suited to illustrate systemic investment dynamics.
(b) Limitations
You must acknowledge limitations:
Because ACRES is a relatively well-resourced and nationally visible initiative, its investment profile may differ from smaller or less formal OST programmes.

(c) Reflective positioning
You must recognise your role:
The author is affiliated with the organisation that developed ACRES. This positionality provides access to rich programmatic data but also requires explicit attention to analytic reflexivity.
This does not weaken you: it protects you.

Response 2: I added text at the top of section 5 clarifying my selection of ACRES and my relationship to the program:

I selected ACRES as an illustrative case for interrogating the cost of OST STEM PL because it represents a long-running, multi-site, nationally scaled OST coaching model that has evolved across more than a decade of implementation. Its development has been supported by multiple funding streams and documented through longitudinal participation and infrastructure data, making it particularly well-suited for examining how investments accumulate, interact, and translate into system-level capacity longitudinally. While ACRES offers analytic advantages for tracing these dynamics, it is also a relatively well-resourced and nationally visible initiative. As outlined below, its investment profile and organizational supports differ from those of smaller, less formal, or locally bounded OST programs. Accordingly, ACRES is presented here as an analytically strategic case rather than as a representative model of all OST PL efforts. 

My affiliation with the organization that developed ACRES, combined with my role as the project’s lead researcher, provides direct access to detailed financial, participation, and programmatic records. While this insider perspective enables a rich, granular view of funding flows, staffing patterns, and longitudinal program operations, it also requires careful analytic reflexivity to avoid overemphasizing the program’s strengths or interpreting outcomes through a personal lens. I endeavor to anchor all claims used to illustrate the point of cost effectiveness calculations in documented evidence, and I use ACRES as illustrative of systemic investment dynamics in OST STEM PL rather than as a judgment of program quality or comparative effectiveness.

Comment 3: Finally, the CBCA is powerful, but right now it seems like an “invented” model without process.
You need to state clearly:
•    It is not a psychometric scale.
•    It is not a quantitative instrument.
•    It is a heuristic evaluative framework based on systems theory, infrastructure, and relational learning.

Response 3: I added a section to address this

6.5. Boundaries of Use and Limitations

The CBCA is intended as a heuristic evaluative framework rather than a standardized measurement instrument. Its goal is to support structured reflection and reasoning about system-level capacity and investment patterns, rather than to produce precise scores. In this way, it is not suited for evaluating or comparing programs. Assigning stages of development relies on the evaluator’s judgment, which introduces subjectivity. However, this subjectivity is by design; it is an opportunity for evaluators to engage deeply with the evidence, make assumptions explicit, and surface nuances that numeric scores often obscure.

In practice, users can approach subjectivity by documenting the rationale for each rating, triangulating across multiple sources of evidence, and engaging multiple evaluators when possible. For example, when assessing the “Relationships” domain, an evaluator might draw on meeting notes, coach logs, and participant surveys to judge whether networks are active beyond the life of a single cohort. Rather than choosing a single “correct” rating, evaluators can annotate why a stage was assigned, note areas of uncertainty, and consider alternative interpretations. This process encourages discussion and reflection between program leaders, funders, and other relevant interest holders, allowing patterns to emerge and systemic strengths and gaps to be made explicit.

The CBCA is best understood as a tool for reasoned judgment rather than precise measurement. It can inform program planning, within-program funding allocations, or research by highlighting where investments support durable capacities, revealing which areas may require further attention, and suggesting where short-term outcomes may underestimate long-term value. While it can be paired with conventional per-participant or cost-effectiveness analyses to provide additional context, its primary contribution lies in capturing dimensions of relational, human, and infrastructural investment that numeric frameworks cannot easily quantify. Future research should explore how CBCA can complement traditional cost-effectiveness approaches, how evaluators apply the framework in varied settings, and how insights from CBCA can reliably and equitably inform funding, policy, and program planning decisions.

Reviewer 2 Report

Comments and Suggestions for Authors The article addresses an urgent issue: the lack of an appropriate cost assessment framework for professional learning in out-of-school time (OST) STEM education. The proposed Capacity-Based Cost Assessment (CBCA) framework represents a valuable contribution, helping shift the focus from "cost-per-participant" to "systemic investment." The paper is logically structured, progressing from context setting → critique of existing frameworks → proposal of a new framework → application guidance → comparison with existing tools. However, the article has several limitations that require improvement:

1. The criteria for selecting ACRES as a case study should be clarified beyond the fact that it is "operated by the authors' organization." It is recommended to include one or two brief examples from other OST programs to enhance generalizability and reduce selection bias.
2. In Table 1, the dimension "Sustainability" is listed separately, but its relationship to the other four investment dimensions is not sufficiently explained. Recommendation: (a) clarify that sustainability is an outcome resulting from the four investment dimensions, or (b) integrate sustainability into the existing dimensions to avoid conceptual overlap.
3. The article should acknowledge practical challenges in applying CBCA, for instance: subjectivity in assigning "stages of development," difficulties in quantifying "relational investments," or barriers in persuading funders to accept this qualitative framework over traditional quantitative metrics.
4. The paper should more clearly differentiate the target users of CBCA, specifically, whether it is best suited for (a) researchers evaluating systemic impact, (b) funders assessing sustainability, or (c) program managers engaged in planning, since each group has distinct needs and varying capacities for implementation.
5. The ranking of "Stages of Development" (grant-bound → field-facing) relies entirely on the evaluator's subjective judgment, without clear quantitative guidelines or explicit criteria to reduce variability across assessors. Dimensions such as "Relational" and "Knowledge" are particularly difficult to measure objectively, raising the risk of "narrative bias," wherein organizations may overestimate their own sustainability capacity. Therefore, a more detailed rubric or calibration method (e.g., inter-rater reliability testing) should be developed for CBCA before broad-scale adoption.

Author Response

Comments 1: The criteria for selecting ACRES as a case study should be clarified beyond the fact that it is "operated by the authors' organization." It is recommended to include one or two brief examples from other OST programs to enhance generalizability and reduce selection bias.

Response 1: I added text at the top of section 1 to clarify the selection of ACRES. Because I do not have the detailed financial and reach information of other programs, I did not include additional examples. 

I selected ACRES as an illustrative case for interrogating the cost of OST STEM PL because it represents a long-running, multi-site, nationally scaled OST coaching model that has evolved across more than a decade of implementation. Its development has been supported by multiple funding streams and documented through longitudinal participation and infrastructure data, making it particularly well-suited for examining how investments accumulate, interact, and translate into system-level capacity longitudinally. While ACRES offers analytic advantages for tracing these dynamics, it is also a relatively well-resourced and nationally visible initiative. As outlined below, its investment profile and organizational supports differ from those of smaller, less formal, or locally bounded OST programs. Accordingly, ACRES is presented here as an analytically strategic case rather than as a representative model of all OST PL efforts. 

My affiliation with the organization that developed ACRES, combined with my role as the project’s lead researcher, provides direct access to detailed financial, participation, and programmatic records. While this insider perspective enables a rich, granular view of funding flows, staffing patterns, and longitudinal program operations, it also requires careful analytic reflexivity to avoid overemphasizing the program’s strengths or interpreting outcomes through a personal lens. I endeavor to anchor all claims used to illustrate the point of cost effectiveness calculations in documented evidence, and I use ACRES as illustrative of systemic investment dynamics in OST STEM PL rather than as a judgment of program quality or comparative effectiveness.

Comment 2: In Table 1, the dimension "Sustainability" is listed separately, but its relationship to the other four investment dimensions is not sufficiently explained. Recommendation: (a) clarify that sustainability is an outcome resulting from the four investment dimensions, or (b) integrate sustainability into the existing dimensions to avoid conceptual overlap.

Response 2: I added a paragraph in 6.1 to explain the inclusion of sustainability as a category:
Taken together, infrastructure, human capacity, relationships, and knowledge investments collectively support the sustainability of the ACRES model. Including sustainability as an additional investment category highlights the deliberate, long-term orientation of OST PL initiatives, drawing attention to how short-term expenditures and efforts compound over time to produce enduring system-level effects. It prompts consideration not just of immediate outputs or participation but also how investments in infrastructure, human capacity, relationships, and knowledge interact to maintain program viability, adaptability, and impact beyond discrete funding cycles.

Comments 3-5:

The article should acknowledge practical challenges in applying CBCA, for instance: subjectivity in assigning "stages of development," difficulties in quantifying "relational investments," or barriers in persuading funders to accept this qualitative framework over traditional quantitative metrics.

The paper should more clearly differentiate the target users of CBCA, specifically, whether it is best suited for (a) researchers evaluating systemic impact, (b) funders assessing sustainability, or (c) program managers engaged in planning, since each group has distinct needs and varying capacities for implementation.

The ranking of "Stages of Development" (grant-bound → field-facing) relies entirely on the evaluator's subjective judgment, without clear quantitative guidelines or explicit criteria to reduce variability across assessors. Dimensions such as "Relational" and "Knowledge" are particularly difficult to measure objectively, raising the risk of "narrative bias," wherein organizations may overestimate their own sustainability capacity. Therefore, a more detailed rubric or calibration method (e.g., inter-rater reliability testing) should be developed for CBCA before broad-scale adoption.

Response to 3-5: 

I added section 6.5

6.5. Boundaries of Use and Limitations

The CBCA is intended as a heuristic evaluative framework rather than a standardized measurement instrument. Its goal is to support structured reflection and reasoning about system-level capacity and investment patterns, rather than to produce precise scores. In this way, it is not suited for evaluating or comparing programs. Assigning stages of development relies on the evaluator’s judgment, which introduces subjectivity. However, this subjectivity is by design; it is an opportunity for evaluators to engage deeply with the evidence, make assumptions explicit, and surface nuances that numeric scores often obscure.

In practice, users can approach subjectivity by documenting the rationale for each rating, triangulating across multiple sources of evidence, and engaging multiple evaluators when possible. For example, when assessing the “Relationships” domain, an evaluator might draw on meeting notes, coach logs, and participant surveys to judge whether networks are active beyond the life of a single cohort. Rather than choosing a single “correct” rating, evaluators can annotate why a stage was assigned, note areas of uncertainty, and consider alternative interpretations. This process encourages discussion and reflection between program leaders, funders, and other relevant interest holders, allowing patterns to emerge and systemic strengths and gaps to be made explicit.

The CBCA is best understood as a tool for reasoned judgment rather than precise measurement. It can inform program planning, within-program funding allocations, or research by highlighting where investments support durable capacities, revealing which areas may require further attention, and suggesting where short-term outcomes may underestimate long-term value. While it can be paired with conventional per-participant or cost-effectiveness analyses to provide additional context, its primary contribution lies in capturing dimensions of relational, human, and infrastructural investment that numeric frameworks cannot easily quantify. Future research should explore how CBCA can complement traditional cost-effectiveness approaches, how evaluators apply the framework in varied settings, and how insights from CBCA can reliably and equitably inform funding, policy, and program planning decisions.