Data Collection and Monitoring in an Educational RCT of a Postsecondary Access Program: Assessing Internal and External Validity

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Congratulations for the work done. It is a solid and well developed proposal, in which all the design factors are very well controlled. I believe that it makes a useful contribution to the design of educational policies and that it also provides a very clear procedural framework for monitoring in educational centers.

Author Response

Congratulations for the work done. It is a solid and well developed proposal, in which all the design factors are very well controlled. I believe that it makes a useful contribution to the design of educational policies and that it also provides a very clear procedural framework for monitoring in educational centers.

We thank the reviewer for taking time to read our work and appreciate the supportive comments on the contribution of the manuscript.

Reviewer 2 Report

Comments and Suggestions for Authors Claims about Texas's college enrollment indicators lack substantive data or visual support. The explanation of monitoring's role in educational policy becomes redundant, requiring narrative tightening.The historical context of Advise TX and the two-stage randomization process is meticulously detailed, offering clear insights into the study's experimental design.   The school categorization process lacks transparency. The discussion about 250 schools applying needs more contextual data and statewide perception analysis.   The use of adviser-student contact tracker data to analyze dosage effects represents a strong methodological highlight, adding granularity to the findings. The connection between adviser turnover and treatment fidelity is superficially explored.   The reduced treatment contrast explanation lacks quantified data. A structured comparison of results across Years 1, 2, and 3 would provide clearer compliance and context evolution.   References are missing.   Author should include figure/table appropriately   data collection method should be discussed in detail  

Author Response

We thank the reviewer for taking time to read our work and for offering comments in the spirit of improving our manuscript.

Claims about Texas's college enrollment indicators lack substantive data or visual support.

We have substantiated this claim with a citation (Texas Higher Education Coordinating Board, 2020) in the introduction demonstrating that postsecondary enrollments in the Fall after high school completion are a critical success indicator for high schools across the state.

The explanation of monitoring's role in educational policy becomes redundant, requiring narrative tightening.

We have tightened the narrative by deleting and rewriting several components of the paper that appeared redundant in the introduction and conclusion as well as throughout the text.

The historical context of Advise TX and the two-stage randomization process is meticulously detailed, offering clear insights into the study's experimental design.  

We thank the reviewer for this kind comment.

The school categorization process lacks transparency. The discussion about 250 schools applying needs more contextual data and statewide perception analysis.  

We have added a paragraph in the Background on the Advise TX Program and the RCT Evaluation section that describes the sampling frame in more detail and explains that these 250 schools were all disadvantaged relative to the state averages on low-income enrollment, college preparation, and college enrollment. This paragraph improves transparency and provides better context of the schools included in the evaluation relative to the state.

The use of adviser-student contact tracker data to analyze dosage effects represents a strong methodological highlight, adding granularity to the findings.

Thank you, we agree that this was a critical component in the data collection and analysis.

The connection between adviser turnover and treatment fidelity is superficially explored.  

We thank the reviewer for pointing out our cursory consideration of this issues. We have added a new paragraph to the Treatment Fidelity section that explains how advisers were designed to serve schools for a year such that year to year turnover is not a fidelity limitation. However, an adviser leaving within a school year would be problematic from a fidelity standpoint. We describe that we used administrative data from the program to track this issue and conclude that it was not a concern for the evaluation.

 The reduced treatment contrast explanation lacks quantified data.

We have added the requested quantified data related to both compliance and the control schools adopting substitute college access programs in the Compliance section of the manuscript. Specifically, we explain that the 75% initial compliance rate fell with 17 of the control schools gaining access to treatment and 20 of the treated schools losing access to treatment in the last year of data we observe. We also specify that the average number of other college access programs at control schools increased 23% from the first to third year of the RCT thereby diluting the treatment contrast.

A structured comparison of results across Years 1, 2, and 3 would provide clearer compliance and context evolution.  

We have added Table 1 to the end of the background section on the program and RCT evaluation. Table 1 summarizes the point estimates of the intent to treat effects for the full sample, Hispanic students, and low-income students in each of the three years of the study. We agree with the reviewer that the inclusion of this information is helpful to provide evidence of the attenuating treatment effect estimates for clearer discussion of issues of compliance and changing context over time.

References are missing.  

Upon thorough review, we do not see any references that are missing. All parenthetical citations appear in the reference list and each item in the reference list is cited in the paper. If the reviewer is referring to the two references listed as “Authors,” this is done to maintain anonymity and will be filled in with the appropriate reference at the time of publication.

 Author should include figure/table appropriately   data collection method should be discussed in detail  

As noted above, we have added Table 1 to summarize the intent to treat effect estimates over time.

We could certainly provide additional detail on the multiple data collection methods employed in this study including interview protocols, survey instruments, and examples of adviser-student tracker spreadsheets; however, we believe the inclusion of such detail would distract from the primary points of the manuscript which are to explain how the idea of monitoring can be used to enhance what researchers can learn from running a randomized control trial to assess the impact of a program or policy. We are happy to revisit this decision if the editor directs us to include these supporting documents.

Reviewer 3 Report

Comments and Suggestions for Authors

Data Collection and Monitoring in an Educational RCT of a Postsecondary Access Program: Assessing Internal and External Validity

Aspects to improve:
1.- Improve the wording of the abstract, considering that it should include: Introduction, objective, methods, results and conclusions so that the reader can have a general overview of the study.

2. In the introduction section, researchers should be clear that they should not describe how the study was carried out, but rather describe the importance of the study, the research problem, and may include a review of related literature on the object of study, so I suggest that they should restructure it. Likewise, the wording should be in the third person.

The development of the article does not include the methodology used (design and type of research, population and sample, techniques and instruments, selection of participants, inclusion and exclusion criteria, informed consent, etc.), the results or discussion.

The article does not have an article structure, but on the contrary, it is a description of the implementation.

Author Response

We thank the reviewer for taking time to read our work and for offering comments in the spirit of improving our manuscript.

Aspects to improve:
1.- Improve the wording of the abstract, considering that it should include: Introduction, objective, methods, results and conclusions so that the reader can have a general overview of the study.

We revised the beginning of the abstract to clearly state the objective of the article is to discuss the advantages of educational monitoring within the context of an RCT evaluation. We believe the rest of the abstract provides a good overview of the article.

2. In the introduction section, researchers should be clear that they should not describe how the study was carried out, but rather describe the importance of the study, the research problem, and may include a review of related literature on the object of study, so I suggest that they should restructure it. Likewise, the wording should be in the third person.

The development of the article does not include the methodology used (design and type of research, population and sample, techniques and instruments, selection of participants, inclusion and exclusion criteria, informed consent, etc.), the results or discussion.

The article does not have an article structure, but on the contrary, it is a description of the implementation.

We acknowledge that this manuscript is not structured like a traditional analysis of an RCT. The goal is to explain how monitoring and multiple sources of data collection can be used to enhance our understanding of the findings from an RCT. We provide a citation for the more traditional analysis of this RCT and believe the current manuscript complements that analysis by further focusing on how the longitudinal data monitoring improves the conclusions and implications of that initial study.

We have included two new sentences at the end of the in the introduction to directly address this point in explaining the purpose and structure of the paper and make reference to the more traditional paper that explains the RCT in more detail.

We appreciate the diversity of stylistic preferences in writing, but we decline to accept the reviewer’s suggestion of the stylistic choice of writing in the third person. We believe it is common and appropriate to use first person when describing the research we undertook.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

paper structure can be properly organized to increase the readability. It seems vague. Context setting is important and it is missing.  Still more figures can be added to present the facts and results.

Author Response

paper structure can be properly organized to increase the readability. It seems vague. Context setting is important and it is missing.  Still more figures can be added to present the facts and results.

It is not clear to us what proper organization the reviewer is referring to. As we previously explained and state in the introduction, the paper is not a standard presentation of RCT results, which have been provided elsewhere and referenced in this manuscript. We believe the paper is well organized. It presents an introduction that includes a literature review and states the contribution, background on the program and randomized controlled trial including a synopsis of the results, data collection, and then proceeds to analyze how each of the components of a well implemented RCT (fidelity, compliance, SUTVA, and external validity) are enhanced through monitoring with a subheading in each relevant section explaining the application to the Advise TX case study.

We have provided additional state context in the first paragraph of the Background section noting that the state was rapidly expanding its college advising efforts and that it has a student to high school counselor ratio of 462:1, similar to that of the national average.

In the absence of specific suggestions, the reviewer seems to suggest generically adding figures for the sake of adding figures. We believe the results presented are concise enough to be easily understood by examining Table 1. It is otherwise unclear to us how figures could be used to improve our discussion of how monitoring can be leveraged to improve fidelity, compliance, SUTVA, and external validity.

Reviewer 3 Report

Comments and Suggestions for Authors

Article: Data Collection and Monitoring in an Educational RCT of a Postsecondary Access Program: Assessing Internal and External Validity

Aspects to improve:

1. The abstract should improve its structure Introduction, Objective, Methodology, Results and Conclusions, this will allow readers to have a better understanding of the study.
2. In the development of the introduction it would be important to organize it in a better way where the background of the study is carried out, the literature review of the last five years, the variables of the study are described as the research objective.
3. A methodology section should be incorporated where the design and type of research, population and sample (inclusion and exclusion criteria), techniques and instruments (their validation process and reliability), information collection process (Informed Consent) and type of analysis (qualitative and quantitative) and how the triangulation was carried out are indicated.
4. Explain in detail the treatment implementation process for each of the phases.
5. Indicate the results obtained and hold a discussion.
6. The conclusions must include two subsections: limitations of the study and new lines of research.

The study seems interesting to me but it should be structured as an article.

Author Response

Aspects to improve:

1. The abstract should improve its structure Introduction, Objective, Methodology, Results and Conclusions, this will allow readers to have a better understanding of the study.
   
   

We have added additional detail to the abstract to explain the methods (both quantitative and qualitative) used in the educational monitoring of this RCT. We discuss the results (e.g., that these efforts revealed explanations for the attenuating treatment effect estimate observed over time), but we note that the results of this paper are not impact estimates of the RCT. Instead, they are examples from a case study of how educational monitoring improves the understanding of the findings from an RCT.

 

2. In the development of the introduction it would be important to organize it in a better way where the background of the study is carried out, the literature review of the last five years, the variables of the study are described as the research objective.

We believe enough of the background context is provided in the opening paragraph and that the additional details are best left left for the Background section which immediately follows the introduction. Is the reviewer suggesting we should only be citing literature from the last five years? If so, we disagree. We are not aware of any relevant literature on the use of educational monitoring when conducting impact evaluation studies or RCTs in the last five years, although we are happy to consider any the reviewer would suggest. We believe it is important to cite the relevant literature regardless of when it was published, as we currently do in the introduction. We also disagree that we describe the variables as the research objective. The objective is to explain and provide examples of how monitoring and data collection can inform the implementation, analysis, and conclusions drawn from an RCT.

3. 
   A methodology section should be incorporated where the design and type of research, population and sample (inclusion and exclusion criteria), techniques and instruments (their validation process and reliability), information collection process (Informed Consent) and type of analysis (qualitative and quantitative) and how the triangulation was carried out are indicated.
   
   

Although not all compiled into a single methods section, many of these details are already present in the manuscript in the Background on the RCT and Data collection sections (for example: describing the RCT, the population and sample, the use of administrative records to track students into postsecondary education, etc.). Many more of these details have been previously published elsewhere as we cite in Authors 2013 and Authors 2019, and we believe they are mostly ancillary to our main points about how monitoring can inform components of an RCT. However, we have added details about how the administrative data was anonymous and how we provided informed consent to the qualitative data collection participants. We have also submitted a sample of the survey directed to the advisers in each school to be included in the supplementary materials for the manuscript.

4. Explain in detail the treatment implementation process for each of the phases.

We discuss the implementation of the treatment in the Fidelity of Treatment Implementation section under the Application to the Advise TX Study subheading. We are not one hundred percent sure what the reviewer is referring to as phases of the treatment, but we think this refers to the different years of the longitudinal study. We have added clarifying language to this subsection that the broad treatment (having an CAC adviser assigned) was consistent over time, but the detail of implementation such as how the adviser prioritized his or her work and the number of hours spent on different types of college preparation and application activities could vary over time.

5. 
   Indicate the results obtained and hold a discussion.

The experimental results are presented and discussed in Table 1 and the accompanying paragraph in the Background on the RCT section of the paper. The other sections of the paper on fidelity, compliance, SUTVA, and external validity discuss how the data collection efforts inform those various considerations of an RCT evaluation.

6. 
   The conclusions must include two subsections: limitations of the study and new lines of research.

This is a good idea, and we thank the reviewer for this helpful suggestion. We have added two paragraphs near the end of the conclusion section that discuss limitations and challenges of the monitoring approach we discuss as well as avenues for future research. Specifically, we believe a promising next step is to consider how monitoring could be incorporated into other quasi-experimental analyses.

The study seems interesting to me but it should be structured as an article.

We appreciate the reviewer pushing us to improve the structure of the paper. In general, we have balanced addressing several of the reviewer’s points with the fact that this manuscript is not a traditional study answering a research question with data and analysis. We have done that elsewhere (as noted and cited in the introduction). The purpose of this piece is to explain how educational monitoring can be leveraged to enhance specific components of a randomized controlled trial, namely fidelity, compliance, SUTVA, and external validity.