Using the Collaborative Problem-Solving Model: Findings from an Evaluation of U.S. EPA’s Environmental Justice Academy

Round 1

Reviewer 1 Report

Dear authors,

Thank you very much for this exciting article, which I enjoyed reading. I think the results of your evaluation study are very relevant for research, even though your study focuses very narrowly on a specific CPS model and thus also has only a small target readership. However, I think your reflections may encourage other institutions to either rethink their own programs or adopt the model you presented in some form, which I think is a good thing. The objectives of your study, the methodological design and the data analysis are well documented and as a reader I was always able to follow your explanations well. I have no objections to the mixed-methods approach and the way the data were prepared and analyzed. I also think the discussion of the results is well done. The results have been contextualized in a meaningful way. You have derived clear policy implications and also provided a successful critical reflection in the limitations section. So far, as you note, I am very pleased with the way you have written and arranged the paper. With this in mind, I will recommend the publication of your study without reservation.

Perhaps a small side note: I would advise you to have Chapter 3 (Data Analysis) start on a new page so that the horizontal page formatting can be switched back.

Thanks again for this very well done contribution.

Author Response

Thank you for your review of the proposed manuscript, we appreciate your comments and reflections on the content and merit of this evaluation work.

Author Response File: Author Response.docx

Reviewer 2 Report

1. The lack of explanation for the effectiveness of the results has affected the credibility of the conclusion.

2. The practical guiding significance of the conclusion was not clearly demonstrated.

3. The summary lacks results and conclusions.

Author Response

Thank you for your review of the proposed manuscript, we appreciate your comments and reflections on the content and merit of this evaluation work.

Author Response File: Author Response.docx

Reviewer 3 Report

The authors examine evaluate the EJA and its use of the CPS Model, using a two-phase explanatory sequential design. Although it is an interesting topic, a rewrite is necessary.

1 Introduction

(1) In the P2: The utility of the CPS model (Figure 1), Why choose the “CPS model”? Other reference(s) ?

(2) Need reasons explaining why it is important to explore “Primary evaluation 4 questions include”.

(3) Research objectives should be clearly specified.

(4) The organization of this article is not depicted in introduction

2. Literature review

The definitions of EJA are stated in introduction. However, they are repeated again in literature review. More discussions on previous findings are required to support why you expect to find the relationships you propose (hypotheses).

3. Data Analysis

(1) This study invited all eligible EJA Fellows (N=67) to participate in this mixed methods. Please explain what criteria you used to make a decision that a response is incomplete.     

(2) There is wording about “IMB SPSS”. It should be “IBM SPSS”?

4. Conclusion and discussion

(1) What are the contributions to the literature? There is a need of more theoretical and/or empirical supports to strengthen the contributions and implications of the research results

(2) Are the outcomes consistent with previous studies?

(3) Managerial implications? There is a need of more implications, since different types eligible EJA Fellows may exist.

Author Response

Thank you for your review of the proposed manuscript, we appreciate your comments and reflections on the content and merit of this evaluation work.

Author Response File: Author Response.docx

Reviewer 4 Report

This reviewer thanks the authors of the paper ‘Using the Collaborative Problem-solving Model: Findings from an evaluation of U.S. EPA’s Environmental Justice Academy’ for a thorough and robust study. This reviewer found the paper to be cogent and well-constructed. In particular, the results section is fulsome, valuable and intriguing.

This reviewer has some minor suggestions as follows:

-        In lines 28/29 (Abstract), it might be helpful to say ‘implementation of (what)’ to be clear at this point as a summary.

-        Lines 38 – the term ‘power’ has some uncertain nuances – do the authors mean generating ‘successful outcomes’ – or do they mean ‘providing agency’ or ‘overcoming hegemony’?

-        The last paragraph of ‘Introduction’ is critical – and this reviewer suggests a clear statement of ‘why’ the study was undertaken is important to isolate. Why was it necessary to ‘build capacity’? Some precise aim of the research seems to be missing.

-        Some of the content in paragraph 2.1 defines the organisation and might be better placed in the Introduction.

-        Line 99 – the S.M.A.R.T. acronym and theoretical basis need expanding on.

-        Figure 1 will need to be larger in the published paper.

-        In section 2.2, this reviewer would like to see some scaffolding around why ‘mixed methods’ was chosen. This is discussed later in the paper (Discussion section) – perhaps it just needs to be moved here to justify the methodology. The question is – why was this methodology selected to achieve the study aim (which, as noted earlier, needs clarification)? What is the theoretical basis for the methodology?

-        Line 648: the idea that the study is a ‘small’ sample is better couched in statistical parameters. The sample is ‘statistically significant’ by observation – were any ‘significance’ tests undertaken? At least the sample size for Gaussian methods needs noting (in the context of mixed methods, the sample does not appear to be ‘small’.)

-        Line 686/687 point is well made (and ably justified) – but it needs to link to a clarified ‘aim’ in the Introduction section.

-        The paper is long – but if the MDPI editors are happy with that, the reviewer did not find this an impediment. The paper's wordiness could be reduced to focus on key messages – not a necessary option.

Author Response

Thank you for your review of the proposed manuscript, we appreciate your comments and reflections on the content and merit of this evaluation work.

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

The innovation of the paper was not found, and the guiding role of the paper in practice was also not found.

Author Response

Thank you for your comments and your consideration of this manuscript. We have revised the text to clarify the innovation of the paper and the guiding role of this paper in practice.  The revisions can be referenced from line 60-79 and now reads as follows:

The EPA’s CPS model is the backbone of the CPS Cooperative Agreement Program that since 2003, has awarded over 120 grants to community-based organizations to assist in fostering collaborative problem-solving to address environmental harms [12] with an additional 83 awards anticipated in 2023. Through the application of strategic planning, consensus building, meaningful involvement, resource leveraging, mobilization, and conflict resolution, the application of this model [11] has been reported as being very advantageous to numerous communities [10-13]. While this CPS Program has been beneficial for building capacity to address environmental change and creating solution-driven, self-sustaining, community-based partnerships, the utility of the CPS model (Figure 1) as a standalone framework has not been assessed until now. Establishing an evidence base for a model that is the centerpiece of both EPA’s CPS grant program and EJA is necessary to document benefits through a formal application of measurement that can identify effectiveness with opportunity for replication.    

Accordingly, the present study sought to evaluate the EJA and its use of the CPS Model as the foundation of their EJ curriculum in building capacity to address local environmental change. This work is novel and has filled a gap in the literature through the application of measurement to the Elements of the CPS Model; and this study is innovative in that it is one of the first efforts to systematically evaluate a capacity-building initiative across multiple theory-informed dimensions of community capacity. Through measurement, this study can serve as a model for evaluating similar capacity-building efforts.

The reviewer suggested additional context with respect to the previous theoretical background and empirical research. 

There are no previous studies that apply measurement to the CPS model for 
comparison. The references that are made to the CPS model are detailed in relation to the CPS grant program with reported community outcomes related to strategic planning, consensus building, meaningful involvement, resource leveraging, mobilization, and conflict resolution. These benefits are detailed with references also in lines 60-66.

The reviewer suggested that revisions be made to more clearly state the research design, questions, and methods.

The research questions for this study are stated on lines 83-87 and read as follows: (1) To what extent did participants (further referenced as EJA Fellows) implement the CPS Model; (2) What were the barriers and facilitators experienced by EJA Fellows in the implementation of their community project; (3) In what ways has the Environmental Justice CPS Model strengthened efforts to address community change; and (4) What community changes have resulted from participation in the EJA. 

Reference to the mixed methods research design for this study has been added to the introduction (line 80); and there is great detail for these methods further referenced in the Measurement section (line 166-170), with the utility of the design mentioned again in Strengths & Limitations section (line 655) as well as Research Implications section (line 684).