Using the Collaborative Problem-Solving Model: Findings from an Evaluation of U.S. EPA’s Environmental Justice Academy

Abstract

The U.S. EPA’s Environmental Justice (EJ) Collaborative Problem-Solving (CPS) Model represents a systematic, community-based approach for bringing together various stakeholders to develop solutions to address local EJ and public health concerns. This mixed methods evaluation study, using a two-phase explanatory sequential design that allows for qualitative data to build upon initial quantitative results, reports findings from an evaluation of the U.S. EPA’s EJ Academy (EJA) program that was comprised of a curriculum grounded in the CPS Model and designed to build capacity in addressing EJ concerns. As a part of participation, each EJA Fellow directly applied learned skills to implement an EJ community change project. All Fellows across four cohorts of the EJA (2016–2019; n = 67) were invited to participate in this evaluation and were asked to complete a questionnaire (n = 35) comprised of measures to assess the degree to which CPS Model elements were utilized in the design and implementation of their community projects. Interviews were conducted (n = 25) to further explore the utility of the CPS Model in building capacity to address local environmental challenges. In an assessment of the full CPS Model using all seven elements in a composite measure, more than half of the EJA Fellows (n = 19, 55.9%) had moderate levels of implementation of CPS elements and 17% (n = 6) had high levels of implementation into their projects. Evaluation findings provide insight into the impact and limitations of the EJA with implications for future program planning and distribution of the Academy curriculum to assist communities in addressing EJ concerns and making environmental change.

1. Introduction

Within the context of environmental justice (EJ), building local capacity is a fundamental first step in generating power to overcome environmental inequities. Strategies that encourage collective action, in particular, are essential in the development of local solutions to problems and enacting higher-level structural and policy change [1,2,3,4,5,6,7,8,9]. Communities impacted by environmental injustices tend to experience an inequitable distribution of power (in terms of agency and influence), lack of resources, and lack of control in decision making, warranting the need for skills development opportunities and strategic approaches to mobilize and enable resolution.

In 2015, the U.S. Environmental Protection Agency (EPA) Region 4, developed an EJ Academy (EJA) leadership program to enhance capacity related to environmental community change. The EJA trains community advocates how to identify assets, develop partnerships, work with diverse community stakeholders, and it emphasizes collaborative problem-solving to address environmental concerns. The curriculum is primarily based on EPA’s Environmental Justice Collaborative Problem-Solving Model (CPS Model) which represents a systematic, community-based approach for bringing together various stakeholder groups to develop solutions to address local environmental and/or public health issues and concerns. The CPS Model is comprised of seven elements, issue identification, community capacity building, consensus building, partnership, constructive engagement, management, and evaluation (Figure 1) [10], and has been successfully applied in the development of vision and change agendas “for the community, and by the community” [11]. It is intended to achieve improvements in: (1) the capacity of overburdened communities to think strategically and to work with diverse stakeholders; and (2) environmental conditions that impact the health of residents living in overburdened communities.

The EPA’s CPS Model is the backbone of the CPS Cooperative Agreement Program that, since 2003, has awarded over 120 grants to community-based organizations to assist in fostering collaborative problem solving to address environmental harms [12] with an additional 83 awards anticipated in 2023. Through the application of strategic planning, consensus building, meaningful involvement, resource leveraging, mobilization, and conflict resolution, the application of this model [11] has been reported as being very advantageous to numerous communities [10,11,12,13]. While this CPS Program has been beneficial for building the capacity to address environmental change and creating solution-driven, self-sustaining, community-based partnerships, the utility of the CPS Model (Figure 1) as a standalone framework has not been assessed until now. Establishing an evidence base for a model that is the centerpiece of both EPA’s CPS grant program and EJA is necessary to document benefits through a formal application of measurement that can identify effectiveness with the opportunity for replication.

Accordingly, the present study sought to evaluate the EJA and its use of the CPS Model as the foundation of their EJ curriculum in building the capacity to address local environmental change. This work is novel and has filled a gap in the literature through the application of measurement to the elements of the CPS Model, and this study is innovative in that it is one of the first efforts to systematically evaluate a capacity-building initiative across multiple theory-informed dimensions of community capacity. Through measurement, this study can serve as a model for evaluating similar capacity-building efforts.

For the purposes of this mixed methods evaluation, community change is identified as any type of community/organizational change (i.e., change of practices, policies, and processes), increased education of decision makers for policy change, or structural change (i.e., sidewalks). Primary evaluation questions for this research include: (1) To what extent did participants (further referenced as EJA Fellows) implement the CPS Model? (2) What were the barriers and facilitators experienced by EJA Fellows in the implementation of their community project? (3) In what ways has the Environmental Justice CPS Model strengthened efforts to address community change? (4) What community changes have resulted from participation in the EJA?

2. Materials and Methods

2.1. EJ Academy Program Overview

The EJA predominantly serves the southeastern states of Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, Tennessee, and six American Indian Tribes who reside predominantly in the southern region. The Academy is an in-depth leadership development program comprised of a nine-module curriculum that focuses on fostering positive change and community revitalization through building partnership with community leaders, community members, and their local stakeholders [14]. The purpose of the EJA is to encourage collaborative problem solving with the local government, small businesses, academic institutions, and industry and create a shared vision among these different local stakeholders so that all who are a part of the community can have a voice [14]. An overarching philosophy that was infused throughout the teachings of the Academy was the practice of Appreciative Inquiry [15], which is a management approach that focuses on what is working well, analyzing why it is working well, and incorporating the positive into change methods. The curriculum is primarily based on the EPA’s seven step CPS Model in an effort to cultivate skills so that EJA Fellows may successfully identify their environmental challenges and accomplish their communities’ environmental improvement goals [14]. Additional teaching in the Academy is coupled with each step of the CPS Model [10] (Figure 1) and involves activities related to setting goals that are specific, measurable, achievable, relevant, and time-bound (SMART) [16]; visioning [17]; letter writing to elected officials; Environmental Justice Screening and Mapping Tool (EJSCREEN) [18]; the application of emotional intelligence [19]; the utilization of the World Café model [20]; Interest-Based Negotiation [21]; stakeholder engagement [22]; financial management [23]; and program evaluation [24].

2.2. Study Population and Data Collection Procedures

Between 2015 and 2020, there were four EJA cohorts comprised of 67 Fellows that completed the training program and were eligible for this mixed methods evaluation study. The EJA program is comprised of a nine-module training course in which each module is dedicated to in-depth learning of one of the seven CPS Model elements with an additional module focused on orientation at the beginning and another module focused on individual project presentations at the end. This curriculum was implemented by the EPA Region 4 Office of Environmental Justice staff and attendance was comprised of community advocates and academic activists, as well as traditional and non-traditional college students. The Academy lectures were conducted by both EPA staff and local/external subject-matter experts, in-class exercises and homework assignments were given, and each Fellow was required to identify an environmental challenge in their respective community. EJA Fellows were then asked to apply these learned skills to the design and implementation of local solutions and engage with the community to overcome challenges. Each EJA Fellow received individual project feedback, developed an action plan to guide organizational activities and created a community portfolio to identify resources, stakeholders, and potential funding opportunities.

All eligible EJA Fellows (n = 67) were invited to participate in this mixed methods evaluation study. Initial contact was made via the EJA alumni listserv with emails and phone calls used to follow-up with interested Fellows. Potential EJA Fellows were directed to an online Qualtrics [25] survey in which, upon giving their consent for participation, they were asked to complete a 56-item questionnaire. Following completion of the survey, EJA Fellows received a USD 35 incentive in the form of a gift card or peer-to-peer direct deposit money transfer (e.g., Cash App, Venmo, or Zelle). Data for this research study were collected from October 2019 to February 2020. Only EJA Fellows who completed the survey (n = 34, 50.8%) were then eligible for participation in a 75–90 min virtual interview and those who participated (n = 25, 37.3%) were given an additional incentive of USD 55 in appreciation for their time. All interviews were conducted by the Emory University research team using a cloud-based meeting platform that allowed for virtual conferencing and video/audio recording. Per the Emory University Institutional Review Board guidelines, this research protocol was reviewed and determined exempt, and all EJA Fellows were informed that participation was voluntary.

2.3. Measurement

Quantitative Measures

The survey data were used as a point of reference to expand the discussion during EJA Fellow interviews. The survey data provided a point of measurement and numerical comparison with respect to EJA projects implementing CPS Model elements and achieving goals. Measures to assess the CPS Model elements were both author-created and adapted from MacLeelan-Wright et al. [26], Bopp et al. [27], Kegler et al. [28], and Laverack et al. [29]. These measures are described in

Table 1. CPS Model Elements, Survey and Qualitative measures [31].

CPS Model Elements	Description	Measurement	Items/Scale Reliability
Element 1: issue identification, community vision, and strategic goal setting.	Element 1 involves identifying the problem, envisioning possible solutions and then setting goals to achieve those solutions. Often people come together when they realize that something is wrong, and awareness of the problem is key to formulating strategies to resolve them. The visioning process is a way to think about what the community should look like and helps to establish community goals. Goals then serve as the foundation for formulating specific solutions or desired outcomes for a particular health and/or environmental problem.	Quantitative Measures: Eight survey items (related to understanding history, community values, and prioritizing concerns); five author-developed and three items adapted from the MacLeelan-Wright et al. [26] community capacity scale related to critical reflection. Qualitative Measures: Three interview questions were asked pertaining to the description of the focal EJ concern and how this issue was prioritized.	8 items α = 0.77
Element 2: community capacity building and leadership development.	Element 2 identifies the necessity of skills, information, and resources to achieve goals. The rationale is that when residents develop a greater understanding of environmental concerns, they can understand the options available to address these concerns and subsequently acquire a greater capacity to meaningfully engage in the decision-making process. Leadership development is an important component to capacity building as certain qualities such as strategic thinking, creating vision, management, effective communication, and consensus-building are necessary to achieve positive results.	Quantitative Measures: Thirteen author-created survey items (related to community participation, community-centered capacity building strategies, and leadership opportunities). Qualitative Measures: Two interview questions were asked pertaining to learned EJ Academy activities and strategies.	13 items α = 0.75
Element 3: consensus building and dispute resolution.	Element 3 details the importance of finding effective ways of making group decisions, involving appropriate constituents, and learning how to resolve disagreements. Collaborative problem-solving and consensus-building is a process that encourages agreement among differences, finding common ground among competing interests, meeting the needs and interests of all EJA Fellows, and creating solutions that are mutually beneficial for all. Differences of opinion naturally arise, and the application of consensus building approaches can be helpful in alleviating tensions and adversity.	Quantitative Measures: Eight survey items (related to diverse stakeholder involvement and conflict resolution) comprised of six author-created survey items and two items adapted from the Bopp et al. [27] resources scale and the Kegler et al. [28] skills development scale. Qualitative Measures: Three interview questions were asked related to a description of the project action plan, methods used to encourage diverse opinions and participation, and strategies used to solve conflict.	8 items α = 0.78
Element 4: multi-stakeholder partnership and leveraging resources.	Element 4 stresses the need for partnership to collectively examine problems, develop action plans, and bring together the resources necessary to achieve goals. Partnerships can be described as consisting of a diverse body of individuals or organizations that represent different sectors of society (i.e., community, government, business, industry, and academia) that are needed for mobilizing resources (i.e., social capital, institutional, technical, legal, and financial) to achieve a community’s vision.	Quantitative Measures: Six survey questions (related to the ability of participants to identify resources and develop partnerships) comprised of three author-created survey items and three adapted items from the Laverack et al. [29] community empowerment scale related to problem assessment. Qualitative Measures: Two interview questions were asked related to stakeholder involvement and network/partnership development.	6 items α = 0.91
Element 5: constructive engagement with other stakeholders.	Element 5 specifically identifies relevant non-community stakeholders (i.e., businesses, academia, civic organizations, and government) who can play an important role in collaborative partnerships. Specifically, business stakeholders who are often perceived as being the source of the problem, can benefit from being actively involved with the local community to increase communication and actively address policies and practices; government stakeholders can act as facilitators, provide technical assistance, assist in coordination and communications, provide services or financial resources, enforce laws/regulations, and provide legitimacy to an effort; and academia and civic organizations can provide training, technical assistance, and act as intermediaries for financial resources.	Quantitative Measures: Eight survey questions (describing the relationship with community partners and level of interaction with stakeholders) comprised of two author-created items and six adapted items from the Bopp et al. [27] community capacity scale related to participation and the MacLeelan-Wright et al. [26] community capacity scale related to resources. Qualitative Measures: One interview question was asked to further discuss engagement with non-community stakeholders.	8 items α = 0.81
Element 6: management and implementation.	Element 6 involves the development of sound management and organization that reflects the ability to identify and carry out work plans with clear goals, creation of a timeframe, and the delegation of responsibility to others. This process involves the identification of a leader/decision-maker who can foster consensus around the vision, establish operating procedures, manage action plans, and communicate properly with other stakeholders.	Quantitative Measures: Seven survey questions (related to leadership) comprised of two author-created items and five adapted items from the MacLeelan-Wright et al. [26] community capacity scale related to leadership. Qualitative Measures: Four interview questions were asked to detail the project action plan, resources needed, and future project plans.	7 items α = 0.75
Element 7: evaluation, lessons learned, and replication of best practices.	Element 7 discusses the necessity of determining whether a project is achieving its goals, identifying what is working or not working, and reviewing lessons learned in order to build on strengths and correct problems for future implementation. This element stresses that a focus on evaluation helps to clarify the underlying assumptions and relationships of the project, as well as identify opportunities and deficiencies that can be adjusted.	Quantitative Measures: Six survey questions (related to utilization of resources, and the degree to which the project has impacted the community) comprised of two items adapted from the Bopp et al. [27] community capacity scale related to resources and four items from the Laverack et al. [29] community empowerment scale related to problem assessment. Qualitative Measures: Four interview questions were asked to identify project goals, long-term vision, lessons learned, and project successes/challenges.	6 items α = 0.86

, which lists the number of items for each element, Cronbach’s alpha, possible range, sample items, and source as appropriate. All items used a Likert five-point scale response of 1 (not at all), 2 (very little), 3 (somewhat), 4 (quite a bit), and 5 (a great deal). The author-created measures were informed using content from the EPA’s Environmental Justice CPS Model Program Guide [30], specifically information detailed in the “Element at a glance” and “techniques to enhance and achieve” sections; curriculum content from the EJA Facilitator’s Guide; and classroom/homework activities from the EJA Fellow’s Guide. In addition to measures related to the implementation of the CPS Model elements, the survey also included demographic information about the EJA Fellows and their projects.

Qualitative Measures

For this mixed methods analysis, a two-phase explanatory sequential design was used that allows for qualitative data to build upon initial quantitative results [31]. The quantitative survey measures provided a point of reference in the application of the CPS model, and the qualitative interview data collected for this study allowed for an understanding of context and further explanation with respect to how and why results and goals were/were not achieved. Briefly, the discussion guide covered details about individual projects, the application of the CPS Model elements, utilization of capacity building strategies, community impact and partnership, as well as success and challenges in making community change. An overview is provided in Table 1 and specific questions are detailed further in Supplemental Table S1. Additionally, two interview questions were posed to identify community changes: “What were you trying to accomplish with this project and what was your desired outcome?” and “Tell me what you view as the greatest accomplishment of your EJA project?”.

3. Data Analysis

Descriptive data from the EJA evaluation survey were analyzed using IBM SPSS [32] Statistics 27 analytic software. CPS Model elements were measured on a 5 pt Likert scale and collective measures used to assess the individual elements were averaged to create an element total score. This total score was then divided by the number of items represented across each element. For example, Element 1 is comprised of eight, 5 pt Likert scale questions, the total score for this element for each participant was the sum of the eight items divided by eight. Similarly, all elements were scored 1–5 for ease of interpretation and this allowed for simple comparisons across elements. Scores were then categorized into three levels: (1) low—ranging from 1.0 to 2.4; (2) moderate—ranging from 2.5 to 3.4; and (3) high—ranging from 3.5 to 5.0. Given the variation in number of items used to assess individual CPS Model elements, this categorization created a more standardized interpretation in the implementation of each element. To obtain a comprehensive assessment of the CPS Model, a total score was averaged across all of the individual items across the seven elements, and this score was subsequently divided by 56 as this was the total number of items in the full CPS Model. Similar to the individual element scores, the full CPS Model was also categorized into low, moderate, and high levels of implementation.

Interviews were transcribed verbatim, participant identifiers were removed, and transcripts were assigned a reference identification code prior to analysis. A code book was developed based on EJA curriculum modules that were grounded in the CPS Model (deductive codes) and salient themes from the interview transcripts (inductive codes). Each transcript was analyzed by a team of two analysts, who were previously trained in qualitative analytic methods. In order to improve reliability and ensure adequate intercoder agreement, each analyst open coded 25% of the assigned transcripts to compare coding patterns and find consensus [33]. After individual review and thematic analysis, the analysts discussed their findings and collaboratively created a central codebook and coded independently to identify key themes. All coding was compared for consensus by the analysis team and discrepancies were resolved through discussion [34]. The coded themes (nodes) were entered into MaxQDA [35] qualitative data management software and an audit trail was created for thematic content analysis using summary grids and tables for specific nodes to identify themes and patterns based on evaluation questions. Primary themes and representative quotes were compiled based on the frequency of mention and consistency across individual interviews.

To acquire a better understanding of EJA Fellow project similarities and differences, the qualitative findings were compiled with survey data to differentiate implementation at high (score of 3.5+), moderate (score 2.5–3.4), and low (score 1–2.4) levels for each element. Summaries across levels of implementation were then detailed and representative quotes were earmarked for theme representation. Specific themes were organized by element of the CPS Model to illuminate implementation of each element: For example, themes related to Element 1 related to the identification and prioritization of community issues, understanding of community history, the visioning process, and alignment of projects with community values; and themes related to Element 2 focused on leadership development, strategic thinking, capacity building activities implemented, and community engagement efforts. To assess barriers and facilitators, themes were compiled that reflected challenges and limitations to project implementation, positive aspects that helped with the project, benefits of developing an action plan, and benefits of participation in the EJA. Additional themes were generated in response to specific questions related to how community efforts were strengthened to address EJ concerns, and detailed project accomplishments were used to identify implemented environmental community changes.

4. Results

4.1. Description of EJ Academy Participants

All EJA Fellows (n = 67) were contacted for participation in this evaluation, of which 51% completed the survey (

Table 2. Participant and project description.

Survey Participants, n = 34	n	%
Professional affiliation
Academic activist	5	14.7
Community EJ advocates	16	47.1
College student activist	13	38.2
Location of EJ project
Alabama	4	11.8
Florida	2	5.9
Georgia	17	50.0
Maryland	1	2.9
Mississippi	3	8.8
South Carolina	2	5.9
Tennessee	4	11.8
Virginia	1	2.9
Interview Participants, n = 25	n	%
Project Focus
Pollution and zoning	6	24.0
Green infrastructure/development	6	24.0
Food security and sustainable farming	6	24.0
Local redevelopment, workforce development, and environmental stewardship	5	20.0
Other: historical contextual research	2	8.0
Project Outcomes
Collaboration, partnership, and networking	9	36.0
Community education and skills development	4	16.0
Grant writing and new funding	2	8.0
Organizational infrastructure (e.g., 501c3 status)	2	8.0
Structural change (e.g., community revitalization)	7	28.0
Policy change	1	4.0

). EJA Fellows were comprised of a mixture of academic activists (n = 5, affiliated with a college or university), other professional/community activist (n = 16), or student activists (n = 13; affiliated with a college or university at the time of project implementation). Only those who completed the questionnaire were then eligible for individual interviews, of which 74% agreed to participate. As shown in Table 2, the majority of EJA projects were conducted in the southeastern region of the United States (n = 32, 94%) with an environmental focus of pollution, green infrastructure, or food security (n = 18, 72%). Slightly more than one-third of projects (n = 9, 36%) resulted in partnership development, less than one-third (n = 7, 28%) resulted in a community structural change, and only one project succeeded in obtaining policy change. The primary community change themes of partnership development, skills development, and community environmental change were further explored and detailed in the qualitative findings (see Community Change, Section 4.5).

4.2. The Collaborative Problem-Solving Model Implementation

In an assessment of the full CPS Model using all seven elements in a composite measure, more than half of EJA Fellows had moderate (n = 19, 55.9%) implementation and 17% (n = 6) had high levels of implementation (

Table 3. Descriptive statistics for CPS Model full implementation.

CPS Model Elements	# of Items	EJ Academy Fellow Element Implementation (n = 34, %)
		M, SD	Low (None/Very Little) Range = 1.0–2.4	Moderate (Somewhat) Range = 2.5–3.4	High (Quite a Bit/a Great Deal) Range = 3.5–5.0
Composite measure seven elements	56	2.6, 0.5	n = 9, 26.5%	n = 19, 55.9%	n = 6, 17.4%
Element 1: Issue identification, community vision, and goal setting	8	3.3, 0.8	n = 4, 11.8%	n = 11, 32.3	n = 19, 55.9%
Element 2: Capacity building and leadership development	13	1.9, 0.6	n = 8, 23.5%	n = 21, 61.8%	n = 5, 14.7%
Element 3: Consensus building and dispute resolution	8	3.3, 0.8	n = 3, 8.8%	n = 14, 41.2%	n = 17, 50%
Element 4: Multi-stakeholder partnership and leveraging resources	6	3.0, 0.9	n = 9, 26.5%	n = 12, 35.3%	n = 13, 38.2%
Element 5: Constructive engagement with other stakeholders	8	2.9, 0.8	n = 7, 20.6%	n = 18, 52.9%	n = 9, 26.5%
Element 6: Management and implementation	7	3.8, 0.7	n = 2, 5.9%	n = 7, 20.6%	n = 25, 73.5%
Element 7: Evaluation, lessons learned, and replication of best practices	6	3.2, 0.7	n = 6, 17.6%	n = 17, 50%	n = 11, 32.3%

). In highlighting variation in the utility of the CPS Model, each element is detailed below, and examples are drawn across EJA Fellow experiences to provide examples of how elements were incorporated into project design thinking, community/stakeholder engagement, and overall project success (additional implementation details of the CPS Model are provided in Supplemental Table S2).

Element 1: Issue identification

The first element of the CPS Model focuses on the identification of the problem, solutions, and development of goals. The overall implementation score was 3.3 (SD = 0.8), which corresponds to being somewhat implemented. In interviews, EJA Fellows identified local pollution concerns related to industry pollution, zoning, legacy contamination, soil testing, and trash; the need for green infrastructure, sustainability planning and urban landscapes such as tree planting, stormwater retention, solar panels, and park revitalization to mitigate community concerns; the use of community gardens to address food insecurity and encourage healthy eating; and the need for local infrastructure change such as sustainability planning with urban landscapes, workforce development, and block revitalization.

EJA Fellows then described how they prioritized their specific EJ issues. Those with lower levels of Element 1 implementation (n = 4, 11.8%) described self-selecting their priority concerns, engaging only with college student communities, and/or minimally working with existing organizations. EJA Fellows with moderate levels of implementation (n = 11, 32.3%) described their work being prioritized through community engagement and discussions or under the existing structure of an organization with an already established project. A high level of implementation (n = 19, 55.9%) was exemplified through more direct community/resident engagement practices and/or the utilization of engagement frameworks that focused on the identification of issues and collaborative partners. One example from a Fellow with high implementation of this element described their use of the Roberts’s Rules of Order to identify community priorities, “we just had different meetings where we went over various things that was going on in our community that we saw as a challenge, like Superfund sites or Brownfields… I think what my organization is doing is pulling together the other organizations in our community that are doing the work, but they don’t have federal funding, or they don’t have a mechanism to get the momentum behind their projects to get it going the way they want it to. … [for example, our organization] worked with [another organization], I built a relationship with them in the farming arena and together [we did] chemical monitoring testing of the water for dissolved solids and [identified] all of these different particles and problems in the water” (EJA Community EJ Advocate).

Element 2: Community capacity building and leadership development

The second element of the CPS Model focuses on the identification of skills and resources as well as methods used to address environmental concerns. The overall implementation score was 1.9 (SD = 0.6), which corresponds to none or very little implementation.

Through interviews, EJA Fellows further identified the specific approaches that were implemented across varying projects. Those on the lower scale of capacity building and leadership development (n = 8, 23.5%) displayed limited direct community engagement with respect to outreach and some did not implement their project as intended; accordingly, in these cases there was minimal application of skills and identification of resources. EJA Fellows with moderate implementation (n = 21, 61.8%) utilized specific community-engaged approaches for building capacity such as outreach that fostered collective decision making, mass media engagement, community surveying, and attendance at and hosting of community meetings/forums. Those who demonstrated high levels of capacity building (n = 5, 14.7%) into their community projects implemented multiple approaches such as partnership development with state agencies, media advocacy, community skill development and citizen science, door-to-door engagement, as well as oral and video storytelling. High implementation is exemplified with comments from one Fellow who details, “working with publicly elected officials to try and change some of the things that are not necessarily law, [but] are written policies that can guide some of the things that are occurring that may adversely impact the community”. This Fellow continues to detail that in her work she is “looking at things we can do in terms of advancing the work [related] to decisions that will ultimately culminate into policies that will benefit the community and keep the community safe from inappropriate siting of facilities or land misuse” (EJA Community EJ Advocate).

Element 3: Consensus building and dispute resolution

The third element of the CPS Model focuses on the importance of collaboration and finding agreement in the best interest of the group. The overall implementation score was 3.3 (SD = 0.8), which corresponds with being somewhat implemented. Interviews were able to further elucidate challenges as the type of conflict often paralleled the degree to which consensus and resolution could be achieved.

EJA Fellows who reported low levels of dispute resolution (n = 3, 8.8%), detailed experiencing conflicts that were unable to be resolved because they were much broader and outside of the scope of the specific project. For EJA Fellows who reported moderate levels of consensus building (n = 14, 41.2%), the type of conflicts detailed appeared to be more localized (e.g., lack of unity or lack of partnership) and within the ability to find resolution and improve communication. Among those who reported high levels (n = 17, 50%) of consensus building, leadership conflict was described most often and the processes for resolution highlighted efforts that were beneficial for the project and community at large. One Fellow details their experience in dealing with conflict, she explains that there was “too much divisiveness in the community … It makes you not want to come back, because nothing is really taking place or happening because there’s just too much divisiveness … [we] established a process for community engagement and participation, ‘we call it a machine’, you come and we will support you and help you and get you in front of the people that you need to be in front of” (EJA Academic Activist).

Element 4: Multi-stakeholder partnership and leveraging resources

The fourth element focuses on ways to partner with others to address concerns, funding opportunities and support, and any other relationship-building opportunities in the community. The overall implementation score was 3.0 (SD = 0.9), which corresponds with being somewhat implemented. In interviews, many were able to further detail the degree to which they were able to engage in partnership development to implement respective projects.

EJA Fellows who reported low levels of partnership opportunities (n = 9, 26.5%) were often a function of their inability to connect with the community or the lack of time invested to develop engagement efforts. Those who reported moderate levels of partnership (n = 12, 35.3%) described their work as being grounded in community engagement and interaction and partnering with local people, groups, and organizations. Among those who reported high levels (n = 13, 38.2%) of implementation of Element 4, EJA Fellows positioned their work as being focused on community partnership development with an emphasis on bringing in larger entities, foundations, and others. From one Fellow’s perspective, community groups in the city of their project “have spent years establishing their voice to government partners and to nonprofit partners”. The Fellow further details their partnership with a state organizing entity that provided fiscal sponsorship, clout, and additional networking opportunities with “50 different organizations in [the state]”. This partnership along with the development of a [local] community oversight coalition, has allowed them to make “a Herculean lift [and instead of] government define their destiny, the community is gaining power in defining their destiny” (EJA Community EJ Advocate).

Element 5: Constructive engagement with other stakeholders

The fifth element identifies other important stakeholders who are critical for project and community partnership. The overall implementation score was 2.9 (SD = 0.8), which corresponds with being somewhat implemented.

EJA Fellows with low implementation (n = 7, 20.6%) utilized partnerships less often and were limited to their own resources and connections to the community to achieve their desired goals. Those who reported moderate levels of implementation (n = 18, 52.9%) described making headway with efforts to engage with stakeholders who were external to the community for present and future partnerships. Those who reported high levels of implementation (n = 9, 26.5%) detailed success in making partnerships that were beneficial for expanding the reach of the organizational goals as well as identifying opportunities for funding. One Fellow with high implementation speaks to the power of diverse partnership and the strategy that they used in “looking to a larger nonprofit [a national funding entity] that has the capacity and ability to manage [large] funds”. The Fellow further emphasizes the importance of this approach as “large funders might not look at a small, black-led organization in Atlanta, and feel confident in their ability to manage a multi-million-dollar grant”; so, they strategically partnered with another nonprofit “so they can pass those dollars thorough [to us]” (EJA Student Advocate).

Element 6: Management and implementation

The sixth element gives attention to the specific details about the Academy project with respect to planning, implementation, specific resources needed, and/or skills used in executing the project. The overall implementation score was 3.8 (SD = 0.7), which corresponds with quite a bit to a great deal of implementation.

Many EJA Fellows had challenges executing action plans, such that some plans were not fully developed and other plans were not implemented at all (n = 2, 5.9%). Those who reported moderate implementation (n = 7, 20.6%) of Element 6 described goals more clearly, detailed plans for engaging with existing organizations or programs, and discussed how action plans were, at minimum, partially executed. Those who reported high levels of implementation of Element 6 (n = 25, 73.5%) included the execution of action plans, established timelines, and, for some, the identification of actions for future program sustainability. One Fellow explicitly stated that their action plan was fulfilled, and that by engaging in this process they have successfully identified “the resources needed in order to continue building the organization, finding [additional] volunteers who can bring different resources to the table, write grants to support the organizational structure … engage with youth … engage with older residents … and establish timelines on each of these actions to ensure that our goals are met and strides made” (EJA Academic Activist).

Element 7: Evaluation, lessons learned, and replication of best practices

The final element discusses the necessity of determining whether goals are being met and identifying course corrections for future programming. The overall implementation score was 3.2 (SD = 0.7), which corresponds with being somewhat implemented.

Those who were categorized with low implementation (n = 6, 17.6%) were largely unable to achieve their desired project success but were able to detail what success would look like. EJA Fellows who reported moderate levels (n = 17, 50%) of Element 7, described their projects as having limited success yet focused on the benefits of their work for the community, future goals, and/or the need for the continuation of their current work. EJA Fellows who reported high implementation of project evaluation efforts (n = 11, 32.3%), demonstrated direct application of learned skills from the Academy, benefits of their partnerships, and/or detailed engagement efforts with the community (e.g., use of surveys, door-to-door activities, opportunities for community feedback, etc.). One Academy Fellow with high implementation proudly spoke of their achievements with respect to the impacts of their community project, “the connections and the skills built, the relationships created, the empowerment of residents … [and] measures the small successes [by] doing work with [the community], testing them, having evaluation components [as a part of] educational events and demonstrations … and measuring the success by the outcome” (EJA Community EJ Advocate).

4.3. Barriers and Facilitators of Project Implementation

Based on a qualitative analysis of the interviews, there were six key barriers and four key facilitators of project implementation that were identified among EJA Fellows (

Table 4. Barriers and Facilitators of Project Implementation.

Barriers to Implementation		Facilitators of Implementation
Lack of communication	A lack of ongoing communication with community stakeholders created challenges in advancing the project and reflected a need for better organizational infrastructure.	Strategic engagements	Those who had pre-existing, good working relationships with organizations, programs, and/or communities were beneficial for advancing project-related work.
Lack of funding	A lack of external financial resources and reliance on personal finances to advance the project was burdensome.	Detailed structure of curriculum	The EJA provided structure, tools, and skills that were beneficial, supportive, and provided a sense of empowerment needed for engaging in the EJ work related to project implementation.
Lack of community engagement	The lack of human/social capital and community involvement made many Fellows feel like “a team of one”.	Credibility of affiliation with the EJA	The teachings of the EJA have had a lasting impact beyond the mere implementation of specific community projects, and training created an opportunity to leverage the affiliation with the EJA for future work.
Lack of community connection	Developing partnerships, building trust, and establishing genuine community relationships were time-consuming and slowed project progress.
Lack of unity	Divisiveness and the lack of unity within the community or among partners of an organization was an impediment to the implementation of project activities.	Opportunities for networking	The EJA allowed for networking opportunities that were beneficial to achieving project goals and further developed skills necessary for future collaborations.
Difficulties in balancing competing priorities	Given the demand and time required with project implementation, many experienced challenges with finding the proper balance while weighing the importance of other life, school, work, and community priorities.

).

4.4. Ways in Which the CPS Model Has Strengthened Community Efforts to Address Community Change

Qualitative themes identified the utilization of the CPS Model as a framework for project development and implementation that provided “structure with how to get the goal accomplished”. EJA Fellows detailed that the Model specifically strengthened community efforts by creating greater understanding around networking and partnership, community stakeholder engagement, and the provision of leadership/professional development around “skills, knowledge and capacity-building”, as well as organizational capacity building.

The development of sound networking and community engagement skills with a focus on building strong connections with others and expanding networks were key takeaways. As explained by one Fellow, “learning how to network with the dynamic of being the only Black person in a predominantly white room… was an invaluable skill”. Another Fellow detailed being able to obtain a greater knowledge with “how to better engage with people as people”. To further this point, another Fellow identified that they now have a “feeling like they have built a stronger connection with community, because I realize that my role is to be a bridge … I can relate as a resident, I can relate as a survivor, I can relate as the person that works at the nonprofit, I can relate that their issues are also my issues”.

For many, professional/leadership development was related to enhancing personal capacity. In particular, learning and practicing emotional and social intelligence were invaluable lessons for communication and strengthened the ability to be successful in making community changes. As explained by one Fellow, “learning how to have important conversations with people that always don’t see eye-to-eye with you, learning how to redirect your anger and not always becoming agitated but changing that conversation or the tone of the conversation to be more productive”, was a skill that was most beneficial for her. To further this point, another expressed that, “having a high social IQ and being able to pick up on body language helped [her] to ensure that [she was] genuinely doing the work on behalf of what is best for the community members”. The benefits of capacity building were also experienced by some at the organizational level. Another Fellow specifically detailed their application of the organizational capacity assessment in helping to identify “where the gaps were”. She further explained having “the director of the board complete one [and] the president of the nonprofit completes it, to see what kind of gaps and opportunities existed with the organization”.

Additional skills that were gleaned from the CPS Model entail understanding the role of history in issue identification. This importance is shared by one EJA Fellows’ emphasis of “being able to understand the history behind different relationships between neighborhoods and government officials and being able to recognize that and acknowledge it and help in the healing process”. Other skills that were garnered relate to enhanced abilities around conflict resolution, appreciative inquiry, and an enhanced orientation to community engagement. As detailed by another, “ appreciative inquiry was a really different framework that I’d never worked from before and I really like … it brings people together in a positive mindset and helps set a different tone for community engagement that I hadn’t really seen in the past, and I’ve been using it quite a bit … let’s go in with more positive and see both what is already good and what you need to be supporting and having that in the plan as well as things that need to be fixed”. Additional emphasis is given to community engagement as another Fellow discussed the value of having this new skill set that can “inform, support, and guide [her] work”; while another further elucidates how these skills have allowed her to “build a stronger connection, because [she is now] a part of the bridge. In every entity, [she] can relate as a resident, as a survivor, as the person that works at the nonprofit and can relate as their issues are also my issues”.

4.5. Community Changes

With respect to community change, three qualitative themes were broadly observed related to direct participation in the EJA: (1) partnership and collaboration development to support future community change were described by all participants; (2) some described skill development for community education and leadership opportunities for community advocacy; and (3) a select few were able to create community environmental and structural change directly resulting from project implementation.

Partnership and Collaboration

Among all EJA Fellows, creating relationships, partnerships, and engaging with the community were essential components of their work. However, for many who did not have a pre-existing network, building trust was a necessary foundational step. For one Fellow, relationship building is not only about the work but speaks to the sincerity of the work as well, “I take great care in how I talk about the community and to the community people, I feel like I’ve made so many friends and these aren’t, it’s not a transactionary friendship, these are very genuine loving relationships”. Another described the relationship-building process as humbling, “I’ve been able to humble myself to work with people that I traditionally probably would not work with and people that others have probably had challenges working with … [and] I think my greatest accomplishment is being able to get people involved and interested in coming up with solutions”. As expressed by another, the creation of community buy-in is fundamental for the sustainability of any project, “life is not always about choosing your team or the people you work with, it is about getting us to work together in achieving whatever our small wins are to get to the big win. So, I’ve just matured to be able to say that I’ve been more inclusive in my communication and plans and programs moving forward realizing that you know, if people are not really engaged and involved then they really have not bought into it”.

Skills Development

Some EJA Fellows were primarily focused on developing and leveraging the skills within their community to create foundations for future work. One detailed her efforts as “a vibration out… just seeing those teachings leave the class and flow into those homes was magical for me to see—experience”. Creating community leadership opportunities for others was equally important for many as well. In this regard, one Fellow highlights his work around initiating “a resident of the month program, where residents are high-fived and fist-bumped and held up on social media for their contributions to talk to the community, some may lead clean-up, some maybe lead educational activities, [because] we want to honor the wisdom holders of community”. Another describes her experiences with transforming community perceptions with respect to the value of their participation and partnership, she further states how her work is helping others “take ownership over environmentalism in a space where black people have not historically been leaders, [or at least] acknowledged leaders”. This point was stressed again by a Fellow who feels that she “can use the privilege that [she’s] been given to help others” and is on “a good path to being able to do that in different ways and in different spaces that would help equip the community to better self-advocate”.

Community Environmental and Structural Change

Based on the qualitative findings, a limited number of EJA Fellows (n = 7) were able to execute their projects in a way that structurally impacted the community. Some projects were related to establishing community gardens and energy efficient homes, while others were related to visually upgrading a degraded community with neighborhood beautification efforts such as weekly community trash pick-up, park revitalization, tree canopy restoration, and the installation of stop signs to mitigate safety concerns. While EJA Fellows were able to make changes, many did not speak of their work within the context of success. One viewed her work as “being able to point to some small achievement that we now have to encourage people to know that we can do more”. Another spoke of his work as “trying to break that dependence [on asking for help or expecting the government to help] … I’m trying to create community power; it’s working but it’s been a slow success”. In making strides with respect to a thriving community garden, a Fellow spoke of their work as a triumph in being “an organization providing opportunity to make a difference in the community”.

5. Discussion

Using the CPS model as a framework for curriculum development, this evaluation of the EJ Academy provides insight as to how model elements have been incorporated into project development and implementation, how community efforts have been strengthened to address EJ issues, and what types of community change have resulted. Evaluation findings suggest that participation in the Academy enhanced efforts to address community concerns by creating greater understanding around the importance of networking, community stakeholder engagement, and providing leadership development around capacity building and assessment. Additionally, participation in the Academy afforded Fellows with a sense of pride and prestige that they were able to leverage in partnership development. Overall, EJA Fellows had moderate levels of implementation of all elements of the CPS Model into their projects. Many also exhibited higher levels of issue identification (Element 1), consensus building (Element 3), and management (Element 6), with the lowest level of implementation being observed with capacity building (Element 2). As a result of participation in the EJA, Fellows were able to build infrastructure for collaboration and partnership; many garnered skills that were directly shared with their respective communities; and a select few were able to create community environmental or structural change.

5.1. EJ Academy Model and CPS Implementation

Findings from this evaluation demonstrate broad variation in the application of CPS Model elements, diverse strategies for community and partner engagement, and varying degrees of community advocacy and structural change. EJA Fellows were comprised of a mixture of academic activists as well as community and college student activists. These differences, coupled with the duration of time that has elapsed since the training Academy, may account for the great variation in the ease with which EJA Fellows were able to execute CPS Model elements. At the time of this evaluation, for some Academy attendees their project was implemented four years ago and for others it was as recent as one year ago. It is possible that EJA Fellows with a lengthier time since attendance may have implemented projects, learned from mistakes, and have revised strategies for continued work. Further, many of the CPS Model elements require a community positioning that is based on an extant grounding with trusting relationships and wherewithal to access supportive structures and financial resources for project implementation and sustainability. Consequently, it might be that EJA Fellows that were engaged in their projects for a shorter period of time have yet to experience the potential impacts of their work and in the future could likely have greater successes.

Accordingly, this variation can lend greater understanding to the lack of success experienced by some and the greater overall success experienced by others. In particular, results indicate that Element 6 (management and implementation) was easiest to implement, and Element 2 (capacity building and leadership development) was most challenging. Element 6 focused on one of the foundational steps of the CPS Model and the EJA more broadly. All EJA Fellows were required to identify a specific environmental concern within their respective neighborhoods and develop S.M.A.R.T. goals related to the execution of their project. Additionally, the EJA curriculum entailed group discussion and homework assignments related to financial management, project management, action plan development, the development of a stakeholder partnership/resource guide, a project readiness checklist, and succession planning. Thus, EJA Fellows were given specific guidance and were fully prepared to navigate, manage, implement, and pivot their respective projects. To the contrary, Element 2 was based less on the implementation of plans and required a different level of practice, support, and engagement. Capacity building is a process and the skills that were acquired with respect to leadership development required direct action, meaningful engagement, and the garnering of resources and support that necessitate a larger investment of time which can take longer to realize. Further, low implementation of Element 2 may be representative of the diversity of experience among those attending the Academy. EJA Fellows were a combination of community advocates, college students, and academic activists; each had a varying knowledge base about their local context, access to community resources, and different levels of skill with community outreach, advocacy, partnership development, and fundraising. These differences undoubtedly can provide some rationale as to why there was great variation in the degrees to which Element 2 was implemented and lend greater understanding as to the ability of some projects to excel while others to fell short of expectations. Capacity building and leadership development are complex concepts, and it may be that this particular element requires more time to observe differences than the time that is allotted within the scope of the Academy. Over time, it is expected that EJA Fellows will continue to engage with their community constituents, practice learned skills, and seek technical assistance that could likely result in greater project successes.

5.2. Alignment of CPS Model with Other Community-Engaged Approaches and Leadership Trainings

The CPS elements are the core foundation of the EJA program and are designed to train community leaders to think and partner strategically to create an active voice and shared vision in addressing community concerns. This model shares many similarities with other community engaged approaches to address EJ concerns such as community-based participatory research (CBPR) [36], participatory action research (PAR) [37], community-owned and -managed research (COMR) [38]. These practices all prioritize working with marginalized and/or overburdened communities, development of partnerships, and the creation of inclusive decision making in the identification of appropriate strategies for addressing environmental health concerns. Further, many who implement these community-engaged efforts recognize the value of community knowledge [39] in driving community-level action and policy change [9,40,41,42,43,44]. To this point, a systematic review of CBPR studies addressing environmental health disparities [45] found that 70% of interventions led to improved community well-being and 50% resulted in enhanced community capacity, mobilization, and skills development. This review also emphasized that when the impetus for CBPR implementation came from within the community, near-term solutions were developed that led to meaningful action and realized change [45].

Although CBPR recognizes the necessity of civic engagement [41], making policy change requires far more advocacy and activism than any one CBPR effort or community mobilization effort, as true policy change requires great attention to the complexity of relationships and power dynamics [46]. Thus, the practices of COMR [38] take capacity building a step further, focusing on citizen science [47], placing greater emphasis on building trust, shared leadership, organizational capacity, policy change, and prioritizing community-led management in addressing the elimination of disparities [48,49]. In many ways, the CPS Model elements are in strong alignment with COMR practices and mimic these capacity building approaches. Additionally, the CPS Model provides a framework that is less dependent on academic partnership, is centered on community priorities, and provides a structured incremental approach for community-developed solutions to complex environmental concerns.

5.3. Making Strides toward Environmental Justice

“Environmental justice is about local people facing local challenges by working collaboratively with the local government agencies, impacted community groups and the responsible state and/or federal agencies” [50] p.27. Accordingly, the EPA’s Office of Environmental Justice developed the EJA with the impetus of creating a leadership development institute that promotes social and educational empowerment as well as strategic thinking among overburdened communities. The incorporation of the CPS Model into the Academy teachings allows for the cultivation of skills with respect to EJ advocacy and also creates a structure for community revitalization and great potential for positive community change. While limited success was seen across EJA Fellow communities with respect to the implementation of new policy or structural change (32%), most projects were impactful in laying a foundation for capacity building. The EJA was successful in assisting the development of an infrastructure for change; the development of partnerships, community mobilization, and community education; as well as tangible environmental change. From an infrastructure perspective, the EJA provided the groundwork and taught EJ advocates how to address injustices in their community with specific skills such as community mapping with the Environmental Justice Screening and Mapping Tool (EJSCREEN) [18] and Community-Focused Exposure and Risk Screening Tool (C-FERST) [51], application of appreciative inquiry [15], development of SMART goals/action plans/logic models, letter writing to elected officials, emotional intelligence [19], and development of a portfolio. These fundamental teachings have provided a space for informed solution-oriented discussions and essential leadership skills, and for many this is just the beginning step of making impactful and meaningful community changes.

Building capacity is an ongoing process and of great importance to address inequities. The CPS Model is particularly useful for capacity building around EJ issues as it provides a systematic approach to develop solutions through an emphasis on skills development, strategic thinking, and the identification of diverse stakeholders. In utilizing the CPS Model as a framework, the EJA developed a leadership program that has provided tools for advocates to become actively involved in problem solving and decision making and to learn how to identify partners for collaboration to develop solutions to environmental issues. While power dynamics often impact the distribution of resources and policy development [29], the EJA is enhancing community power by affording everyday citizens with capacity building skills to counteract inequalities, take ownership of concerns, and bring about much needed social and structural changes.

This research also has direct implications for improvement related to the implementation of the EJA and broader community-led advocacy training programs of the like. Findings from this study suggest that the variation in implementation of CPS Model elements may be related to the diversity of experience among those attending the Academy and different levels of skill with community outreach, advocacy, partnership, and fundraising. Thus, for future iterations of the Academy, a more concerted effort may be needed for enhancing components of community capacity and leadership, specifically in alignment with CPS Element 2. It is suggested that training sessions focus on one particular type of community stakeholder at a time with the opportunity for multiple participants from a community to engage with Academy learnings. This more focused approach to leadership development can ensure the necessary level of training is provided for project implementation, technical assistance with partnership development, and may allow for greater project success and sustainability.

5.4. Strengths and Limitations

Although information for this evaluation is gathered from a small sample, the application of a mixed methods research design adds to the robustness of the overall study, as the combination of survey and interview data are both compelling and informative for understanding the utility of the EJA. These findings provide context and can give a general understanding to the efforts and challenges of many overburdened and underserved communities, while also providing insight for the ways in which communities can become empowered and take ownership in creating change.

In this evaluation, the utilization of a mixed methods design provides comprehensive evidence for addressing the research questions as survey data alone is not able to address context and setting. General limitations with respect to qualitative methods includes the potential for bias as personal interpretations of the data are necessary; however, it is intended that the combination of these methods offsets the weaknesses of both qualitative and quantitative research individually. Both data collection methods also relate to the potential for social desirability bias in that EJA Fellows may have responded to interview or survey questions in a manner that represented themselves and their projects in a more favorable manner. Further, single informant bias may misrepresent the successes and challenges of a particular community, as the findings of this study are reliant on the perspectives and self-reported data of a single representative for each community project. Additionally, at the time of this evaluation, for some Academy attendees their project was implemented four years ago and for others it was as recent as one year ago. Accordingly, it is possible that EJA Fellows with a lengthier time since attendance may have implemented projects, learned from mistakes, and have revised strategies for continued work. Lastly, many of the individual CPS Model elements were comprised of multiple constructs, making it difficult to identify and measure concepts appropriately. Thus, deconstructing elements may be a more appropriate assessment in the future to identify the nuances of these complex dimensions.

5.5. Research, Policy, and EJ Implications

Environmental inequalities are rooted in complexities of public health, policy, city/regional planning, economics, and issues related to social equity [52]. These inequities extend far beyond individual level challenges; thus, programs that build community capacity and power are radical methods for supporting local-level community, higher-level policy, and structural change [2,5,9]. This explanatory mixed methods evaluation approach has provided a deeper understanding of challenges, successes, partnerships, and efforts exerted in the creation of healthier and environmentally just communities. The CPS Model has been advantageous for many communities as a framework for community grant funding, as a strategic approach to address EJ issues, and for intervention development [30]. However, the utility of this model as a framework for building an EJ curriculum has not been assessed until now. Thus, this study has filled a gap in the literature and is the first of its kind to apply measurement to the elements of the CPS Model as well as the first evaluation to identify the effectiveness of the EJA in building community capacity to address environmental change.

Within the EJ field there is a history of bringing together varying stakeholders to create strategies to improve health outcomes; however, the field largely lacks a focus on measurement and assessment of problem-solving efforts. Program evaluation is an underutilized practice in the EJ field, and this research has real-world implications that can directly impact the future of the EJA. The EJA has currently transitioned from EPA leadership to partner institutions, and the findings of this evaluation study stress the importance of continued evaluation efforts. Thus, with continued implementation of the EJA, evaluation is undoubtedly needed to ensure that attendees of different types (students/other academic advocates and community activists), knowledgebase, and skill-level are reaping the maximum benefit and are provided with the leadership development to drive community environmental change. This evaluation study is the first of its kind for the EJA Leadership Program and is a step in the right direction to ensure that communities have the skills, tools, and support to drive policy, systems, and environmental change.