Measuring Social Dimensions of Sustainability at the Community Level: An Illustrative but Cautionary Tale

Abstract

Many communities are working to enhance the sustainability of their physical, economic, and social systems. While economic and physical systems are routinely measured (e.g., money and energy), psychological and behavioral elements of social systems (norms, attitudes, and individual behavior) are seldom tracked. The objective of this research was to evaluate a potentially scalable approach to measure the impact of sustainability initiatives on these variables in a community engaged in holistic sustainability programming. Online survey data were collected in 2012 (N = 155) and 2016 (N = 137), measuring pro-environmental thought and behavior in two towns in Ohio: Oberlin, a community engaged in holistic efforts to enhance environmental sustainability; and a similar community (Berea) used as a control. Survey links were distributed via recruitment letters mailed to randomly selected community residents from a purchased mailing list. We used two (town) by two (time) between subjects’ ANOVAs to evaluate whether Oberlin saw predicted increases in sustainable thought and behavior from 2012 to 2016, compared to the control community. Despite verifiable participation in and awareness of sustainability programs in Oberlin, our survey results did not provide strong evidence that programs resulted in the desired changes in attitudes, norms, and individual behaviors. Recycling attitudes and LED bulb installation were two exceptions. We conclude that assessing the psychological and behavioral dimensions of sustainability poses particular challenges. We encountered ceiling effects and inadequate statistical power. Possibly, norms and attitudes are not easily influenced even by a holistic community-wide effort.

1. Introduction

As climate change and environmental degradation escalate, there is increasing awareness that local communities and governments play a crucial role in successfully implementing long-term sustainability goals. Indeed, some have argued that “effective and integrated solutions can only be found and efficiently implemented through cities and urban areas” [1]. Grassroots efforts have the opportunity to align sustainability programs with the specific development needs of the community, and can often circumvent political resistance that exists at regional or national levels [2]. Leading the charge on the ground, Local Governments for Sustainability (ICLEI) is a global network of more than 2500 local and regional governments committed to sustainable urban development. ICLEI supports members engaged in climate action planning and sustainable mobility.

Because radical transformation towards increased sustainability at the community level is necessary to address existential problems such as climate change [3], the significant growth of local sustainability initiatives is a hopeful sign. However, the consistent and comprehensive evaluation of the implementation and impact of these initiatives remains a challenge [4,5,6]. The UN’s Sustainable Development Goals (SDGs) have emerged from years of discussion as a holistic framework that brings together social, economic, and environmental dimensions of sustainability into a set of 17 goals and 169 targets [7]. However, evidence from a meta-analysis of over 3000 scientific studies on the SDGs published between 2016 and April 2021 suggests only a limited transformative impact of the SDGs on political decision making thus far [8]. One missing piece of the assessment puzzle may be the lack of clear metrics for evaluating the sociological, psychological, and behavioral changes in the individuals that make up a community [9].

This research evaluated a potentially scalable approach to assessing the impact of sustainability initiatives on sociological, psychological, and behavioral variables in a municipality attempting to create a community-wide culture shift toward sustainability. Our research question was whether a low-cost community survey could effectively document the impact of sustainability programming on individual thought and behavior. As we elaborate on below, this research addresses an important gap in the field, as previous sustainability assessment strategies have not evaluated psychological and behavioral outcomes of sustainability initiatives, or evaluated the use of a survey to measure holistic change.

Below, we provide a brief overview of the developments and limitations of metrics to assess community sustainability; articulate the importance of both fostering and evaluating changes in people’s perception, norms, attitudes, and behavior; and summarize lessons from the Oberlin Project, a local effort to promote “Full Spectrum Sustainability” [9]. We then describe the methodology we developed to measure the psychological and behavioral transformations that were expected to occur. We present a candid evaluation of the effectiveness of this approach, identifying strengths and weaknesses. Finally, we offer recommendations to guide communities wishing to conduct similar evaluations of their sustainability programs.

1.1. Assessing Sustainability Efforts

Without accountability and assessment mechanisms, it is impossible for community leaders and their constituents to know whether they are achieving their local sustainability goals. Failure to assess also precludes the opportunity to gather information to make iterative improvements in programming. Sustainability goals are often developed through broad public participation and embodied in the comprehensive plans of cities and towns. Community sustainability indicators should be designed in parallel to measure progress towards these community goals.

Significant progress has been made in developing sustainability indicators. For example, first published in 1993, Sustainable Seattle has emerged as a well-used tool for measuring progress towards achieving sustainable development goals at the community level. It was designed to measure the impacts of past actions and to serve as an adaptive-management tool for ongoing decision making in government, NGOs, and businesses [10]. Sustainable Seattle’s indicators have been used by more than 60 municipalities, cities, and other organizations in the US and abroad. It is cited by the United Nations and other international forums, and by scholars from around the world. Sustainable Seattle incorporates indicators that measure social equity and indigenous rights along with economic development, environmental quality, and climate change response (a breakthrough at the time). However, only two of the indicators can be considered psychological or behavioral in nature: Neighborliness and Perceived Quality of Life, as part of their Health and Community indices (both of which are included in the survey described here).

The UN’s Sustainable Development Goals (SDGs), introduced in 2015, provide another widely used framework for sustainability metrics, with roughly 200 subgoals that communities worldwide are using to categorize and evaluate their sustainability efforts [7]. However, like Sustainable Seattle, the SDGs place little if any direct emphasis on psychological and behavioral outcomes.

Building on the work of Sustainable Seattle and the SDGs, ICLEI is in the early stages of developing a set of EcoLogistics Indicators that incorporate environmental sustainability, social equity, and economic sustainability. However, the current list of indicators being developed under this initiative also does not include any psychological or behavioral measures [11]. A number of researchers have suggested the importance of developing indicators that incorporate values, individual-level variables, and cultural elements into sustainability assessment [9,12,13]. However, from the early implementation of sustainability indicators in urban communities to current initiatives under development, the assessment of psychological and behavioral measures has not yet become standard practice.

This is unfortunate for at least two reasons. First, psychological perceptions and social norms are ephemeral and difficult to capture unless they are measured in real-time before, during, and immediately after programmatic interventions. Research demonstrates that our memory of past subjective states and behavior is fallible and heavily influenced by the present [14]. If one wishes to evaluate the effectiveness of a program designed to change thoughts, values, and individual behaviors, it is essential to collect baseline data before sustainability initiatives have begun as well as after. This requires foresight and planning, and cannot be reconstructed after the fact.

Second, thoughts, values, and individual behaviors are actually quite important for the success of local sustainability efforts [15,16]. While some sustainability transformations can occur via top-down decision making (e.g., a local government or utility can choose renewable energy sources), top-down decision makers must be responsive to constituents to maintain their leadership positions. Further, many sustainability programs and technologies will only be successful if members of the community embrace and engage with them [17,18,19,20]. For example, attempts to electrify residential cooking and heating will fail if homeowners are psychologically attached to perceptions of gas as a comforting and familiar cooking and heating source. These psychological reactions close homeowners’ ability to consider heat pumps and induction stoves as desirable alternatives [21]. But attitudes and norms are not merely side effects of change or stumbling blocks to overcome. They are also potentially powerful leverage points to speed up the transition to the adoption of more sustainable practices within communities. As just one example of the important role individual-level psychological variables can play, a large body of evidence demonstrates the impactful role that social norms play in establishing and maintaining new patterns of behavior [22,23,24].

While it may not yet be common practice in community-level sustainability assessments, social scientists have reliable ways to measure sociological and psychological variables that impact behavior: values [25,26], quality of life [27,28,29], pro-environmental attitudes [30], social norms [31,32], pro-environmental behavior [33], and policy support [34]. There are many examples of researchers successfully including these individual-level variables in their assessments of particular programs [35,36]. For example, a local, community-wide food-waste intervention in Upper Arlington, Ohio, was evaluated using both curbside waste audits (measures of the physical system) and an online survey assessing (self-reported) household food waste, campaign awareness (e.g., in the past 30 days, do you recall seeing or hearing about…), attitudes (e.g., throwing away food is bad for the environment), knowledge, and perceptions [35]. Participants who received the campaign reported attitudes significantly more supportive of avoiding food waste and reported wasting less food, indicating that the campaign was effective. These results also agreed with the waste audit, providing valuable information about the reliability of self-report in this context.

In our study, we attempted to evaluate a community-wide effort (described in detail below) whose goals explicitly included changing cultural and psychological perceptions of the citizens living in the community along with changing physical and economic conditions to enhance sustainability. In measuring household- and individual-level behavior that would otherwise go completely unmeasured, we attempted to expand the scope and impact of the existing methods of assessing community sustainability. Our survey effort is (as far as we know) unique in several ways: we measured psychosocial variables related to sustainability; we attempted to evaluate change at the community level; and our evaluation took place in the context of a comprehensive, community-wide sustainability initiative. Finally, unlike the sustainability assessments we reviewed, our research included a control community, which allows researchers to evaluate whether the changes that occur are community-specific rather than regional- or national-level trends.

1.2. The Oberlin Project and The Small-Town Survey

Many have argued that addressing the existential challenge of climate change demands a systemic approach to fostering change that deeply considers relationships and interdependencies among physical, economic, and social systems [37]. Change in biophysical systems (for example, achieving greater energy efficiency) demands concurrent change in other systems (for example, financial and psychological systems, [21]). Thoughtful sustainability initiatives and programming must consider, measure, and evaluate the potential for synergy in motivations to act and strategies for building momentum based on concurrent changes in social, economic, and environmental systems [38]. This was an explicit goal of the Oberlin Project, a comprehensive time-limited initiative to catalyze a shift towards greater sustainability in a small, rural Ohio community.

This research focuses on lessons that can be learned from the Oberlin Project. In 2012, the city of Oberlin and Oberlin College launched the Oberlin Project as an effort to “improve the resilience, prosperity, and sustainability of our community” (https://oberlinproject.org, accessed on 22 March 2024) at the level of the whole system. As described in internal working documents, the Oberlin Project sought culture change at the institutional and individual level, to “make the goal of sustainability a concrete reality and the norm for decisions and behavior at all levels”.

This initiative resulted in a number of tangible changes to the physical and economic systems in Oberlin. For example, single-stream recycling was introduced, new residential energy-efficiency services were created, community gardens expanded, the Farmer’s Market grew, the city and College dramatically reduced their coal consumption, and City Hall created its first Climate Action Plan. A more comprehensive list of the programs implemented during the time of the Oberlin Project can be found in

Table 1. Initiatives implemented in Oberlin, 2012–2016.

Category	Program Name	Description	Accomplishments
Cross Cutting	City Climate Action Plan 0	Plan to reduce community-wide greenhouse gas emissions below zero, while balancing the environmental, social, and economic interests of the Oberlin community.	Plan formally adopted in 2011 and revised in 2013. Set systematic goals of reducing greenhouse gas emissions by 50% in 2015, 75% by 2030, and below 100% by 2050. Successfully met goal of reducing emissions by 50% in 2015.
	Farm to Feast Agrotourism	Local food touring event where participants explored the farms of the greater Oberlin area.	Two agrotourism events expanded knowledge around the local food economy. Both took participants to local farms and ended with a dinner made by local chefs featuring local produce and meat.
	Hotel at Oberlin	LEED Platinum-certified hotel and conference center designed to be an economic and cultural driver of downtown Oberlin.	Opened in 2016 as a 70-room property. The Hotel at Oberlin is the first hotel in the US to incorporate solar, geothermal, and radiant heating and cooling and one of only five hotels in the US to qualify for the rigorous LEED Platinum certification from the U.S. Green Building Council. The Hotel at Oberlin was central to The Oberlin Project and had substantial community participation in its design.
	Renewable Energy Credit (REC) Debates	Debates over whether to use money from trading renewable energy credits to reduce electricity bills or to create a community fund for sustainability projects	Public discussions occurred in multiple fora. Citizen-ballot initiative to create a community sustainability fund was passed in 2017.
	SEED Ventures	Pop-up marketplace and incubator that allowed local entrepreneurs to test the market and design their business to optimize impact to people, profit, and planet.	Worked with local entrepreneurs in the early stages of development. Provided business coaching, student-driven marketing and graphic-design support, and free retail space in downtown Oberlin to test the marketplace. [One-year project.]
Energy	Ecolympics	Three-week-long event series and scavenger-hunt competition with the goals of fostering conversations about sustainability, highlighting sustainability work in progress, and encouraging participants to engage in sustainability measures on a personal, local, regional, and global scale. Resource-use-reduction competition aimed at reducing both the college’s and the city’s public schools’ water and energy consumption as much as possible within a three-week period.	Encouraged students to practice conservation behaviors that became habitual. Previous Ecolympics have seen residence halls and co-ops collectively save 10,070 kWh of electricity— the average amount of electricity an American home uses in an entire year—and save 54,537 gallons of water, which is equivalent to about 1500 10 min showers. In 2016, the public schools in Oberlin showed averaged reduction rates of up to 32% in electricity and up to 10% in water.
	Efficiency Smart program 0	Program to increase electric efficiency through rebates and custom services.	Assisted residential, commercial, and industrial customers reduce energy consumption by providing information, resources, and incentives. Rebates for energy-efficient appliances increased from 28 in 2012 to 228 rebates by 2016 (700% increase) for a reduction of nearly 6000 megawatts of electricity.
	Environmental Dashboard signs in community	Twelve electronic real-time display dashboards that provided feedback on resource consumption in public spaces in Oberlin.	Created demonstrable impact on social norms of sustainability and conservation behavior among people who frequented sign locations.
	POWER energy efficiency service 0	Non-profit organization aimed to increase the energy efficiency of homes, small businesses, and non-profit organizations. Staff helped homeowners at all stages of the process. Grants decreased the cost of home weatherization.	Reduced emissions and energy costs. Achieved 90% annual increase in residential energy audits and home weatherization. Between 2012 and 2016, POWER weatherized 33 homes.
	Significant expansion of green electricity	Switched from predominately coal to landfill gas, wind, solar, and hydro.	In three years, the city halved greenhouse gas emissions by engaging in more sustainable forms of energy use. The city’s electric portfolio in 2015 was 24% hydro power, 3% solar energy, 3% wind energy, 55% landfill gas, and 15% coal, nuclear power, and natural gas.
	Solar Co-op	Residents of the small Ohio city of Oberlin formed the Oberlin People’s Energy Co-operative, the first solar co-op in Lorain County.	Co-op provided the addition of solar energy to 25 home rooftops.
Food	Legion Fields Community Garden	Large-scale community garden established, offering individual plots to community members and using community areas to grow vegetables for Oberlin Community Services, a local organization that provided assistance to low-income residents.	In 2015, 28 raised beds were planted by 21 gardeners. Additionally, 50 college students and 25 community residents provided over 300 volunteer hours to install fencing, mulch pathways and construct new raised beds as well as plant 400 lineal feet of starter plants in the Open Co-operative Garden.
	Expansion of Oberlin Farmer’s Market (OFM)	Wider selection of locally grown foods at farmer’s market. Acceptance of Food Stamps.	Starting in 2012, from mid-May to mid-October, every Saturday 30 vendors sold fresh, locally grown produce, fruits, meats, eggs, honey, and a variety of breads.
	Produce Perks	Fruit and vegetable Coupon program to make produce more affordable for families. Customers spent USD 10 of food stamps and the Oberlin Farmer’s Market matched it with an extra USD 10 specifically earmarked for produce.	OFM consumers spent approximately USD 1300 total in EBT in 2013. OFM received funding for an additional USD 1300 in tokens to provide a 100% increase in purchasing power for all EBT holders for transactions all summer of 2014 and 2015. The program was discontinued after that for lack of funding.
Transportation	Complete Streets Policy	An approach to design a diverse, multimodal transportation system so all users could stay safe on the road—including pedestrians, bicyclists, motorists, and transit riders of all ages and abilities.	A total of 5284 linear feet of on-street bike lanes, 11,530 feet of shared lanes (added sharrows), 8 bike racks, and 1545 linear feet of new sidewalks were added throughout the community. The Complete Streets Policy was formally adopted by the city in 2015.
Waste	Increase in Recycling Infrastructure	Expansion of recyclable items to include paper and cardboard. Public refuse andrecycling stations added to downtown areas and parks.	The city of Oberlin demonstrated a 30.47% recycling increase in the first-year implementation of the new recycling program and a 123% increase over the study period.
	Single-Stream Recycling	Recycling bins updated with larger single-stream recycling bins. Allowed Oberlin residents to recycle items through a free, subscription-based, curbside recycling program.	Simplified process for consumers. Allowed consumers to fit more items into recycling bins week to week. Single-stream recycling reduced the burden on consumers by allowing them to place materials into one bin without sorting. Recycling increased from 822 tons to 1010 tons from 2012 to 2016.

⁰ denotes initiative started before 2012 but updated/modified between 2012 and 2016.

. The items represented in this table do not portray an exhaustive list of all programs implemented in the city of Oberlin, but provide insight into the intent and breadth of the Oberlin Project.

At the start of the Oberlin Project, our research team at Oberlin College was asked to help evaluate its impact on the social fabric of the community: did the Oberlin Project change how citizens thought, felt, and acted in relation to the city’s sustainability goals? Were people aware of the new programs? We developed a “Small Town Survey”, to be deployed early and again several years into the initiative, as a tool to answer these questions.

1.3. Research Approach, Objectives, and Challenges

To evaluate the overall impact of the Oberlin Project, we used an online survey in a two (town) × two (time) quasi-experiment. We chose this approach for several reasons. A before–after design controls for where the community started out on each variable, and allows us to establish whether change has actually occurred. The inclusion of a control community was essential to evaluate whether any observed changes were specific to Oberlin or occurred broadly across the region. The online survey was a highly efficient, low-cost method of data collection. Participants were randomly selected from purchased mailing lists to minimize sample bias (alternative sampling methods we considered were snow-ball sampling through the mailing lists of organizations and soliciting participants in public locations, but we decided these strategies would almost certainly result in sample bias). Our approach was designed to be a low-cost (<USD 2000) partial turn-key resource. If this approach proved to be successful and useful, we hoped that communities that lack research expertise could adapt our methods to evaluate their own communities.

Early in the planning phase (after the goals of the Oberlin Project had been articulated but before structural changes had occurred), the research team identified a comparable nearby control community: the town of Berea, OH. Rather than attempt to assess each Oberlin Project initiative separately, the study was designed to evaluate the impact of the range of initiatives pursued on a range of psychological outcomes. This was more resource-efficient and helped minimize “survey fatigue” among residents, but also allowed for measuring spillover effects, synergistic impacts, and broader patterns. The survey (described fully below) assessed individual psychological characteristics (e.g., norms, attitudes, identity, and efficacy), household-level behavior (e.g., resource consumption, conservation, food procurement, and waste management), and community satisfaction (e.g., belonging and trust). In measuring household- and individual-level behavior that would otherwise go completely unmeasured, we hoped to evaluate whether the Oberlin Project did, in fact, help make “sustainability the default” in citizens’ thoughts and behaviors. The practical goal of the research was to provide useful and actionable information to inform decision making in Oberlin and in other communities. We also hoped to evaluate our methodology as a potentially scalable approach to capturing a more complete record of social transformation resulting from sustainability efforts; could a low-cost community survey be used to produce information that would help achieve these other goals? This latter goal is the focus of this paper.

To maximize the chances of successfully evaluating change in response to sustainability programming, we adopted the following strategies in developing our survey:

Use previously validated survey items. To maximize the likelihood that our survey items effectively measured the constructs we were trying to capture, we used previously validated survey items wherever possible. However, many of the effects we hoped to document were not easily captured by existing measures, so some new items were developed to assess the particulars of sustainability initiatives initiated in Oberlin. This combination of validated measures with measures targeted towards community-specific impacts is likely necessary in any replication.

Evaluate data for consistency with theory and research on pro-environmental behavior. We used correlations among survey items to evaluate whether these variables related to each other as would be predicted by well-established theories of pro-environmental concern and behavior (e.g., the Theory of Planned Behavior [39,40], or the Value Belief Norm model [41]). Our goal was not to test the theories themselves. Indeed, we did not systematically measure every construct in the Theory of Planned Behavior or the Value Belief Norm model, as this would have made our survey too long. Rather, we examined predicted associations between variables to provide a test of the internal consistency of our survey items, which in turn provides evidence that they measure what they are intended to measure.

Evaluate data for consistency with established demographic differences. The survey results should reflect well-established demographic differences in the measured constructs. A particular effort was made to oversample communities of color and low-income communities, as these sub-populations make up a relatively small percentage of the general population; they are also less likely to respond to surveys [42]. We hypothesized that well-documented patterns found in other research works (e.g., gender differences; see Section 3) would emerge in our data. If these patterns failed to emerge, it would suggest that our methodology was inadequate for capturing differences between subpopulations, and/or that our items did not effectively represent their intended constructs.

Assess a priori predictions based on documented sustainability initiatives. As described in the Methods Section, we used verifiable, publicly available information and interviews with city officials to catalog sustainability initiatives that occurred (summarized in Table 1) and to develop a priori hypotheses about how each measure would change over time in each community (see

Table 2. Norms, attitudes, and beliefs, 2012–2016.

Measure	Prediction		Oberlin Initiative(s)	Observed Change		Matched Hypothesis		Cohen’s d
	O	B		O	B	O	B	O	B
Energy
I feel this community supports efforts to reduce household energy consumption + cost.	↑	—	Efficiency Smart 0 POWER 0	0.004	0.277	0	1	0.005	0.320
There are programs and resources here to help me reduce my household energy consumption + cost.	↑	—	Efficiency Smart 0 POWER 0	0.098	0.266	0	1	0.114	0.297
There is someone in my community who can help reduce my household energy consumption + cost.	↑	—	Efficiency Smart 0 POWER 0	0.168	0.221	0	1	0.188	0.260
I believe that as a community we can work together to reduce our total energy consumption.	↑	—	Efficiency Smart 0 POWER 0 Environmental Dashboard	0.052	0.002	0	1	0.070	0.002
I consciously make decisions to minimize my electricity use.	↑	—	Environmental Dashboard	−0.150	−0.980	0	1	−0.220	−0.158
I consciously make decisions to minimize other people’s electricity use in my household.	↑	—	Environmental Dashboard	−0.124	−0.326 *	0	0	−0.141	−0.365
I have relatively little ability to influence electricity consumption in my household.	↓	—	Efficiency Smart 0 POWER 0 Environmental Dashboard	0.216	−0.110	0	1	0.214	0.116
* How often do you think residents of your town try to conserve energy?	↑	—	Efficiency Smart 0 POWER 0 Environmental Dashboard	0.160	−0.036	0	1	0.025	−0.052
How important is it to renovate to reduce energy use and waste? (Individual homes).	↑	—	Efficiency Smart 0 POWER 0	0.530	−0.037	0	1	0.168	−0.040
I often think about electricity consumption when I turn a light or appliance on or off.	↑	—	Efficiency Smart 0 POWER 0 Environmental Dashboard	−0.132	−0.024	0	1	−0.167	−0.029
How motivated? Conserve energy	↑	—	Efficiency Smart 0 POWER 0 Environmental Dashboard	0.147	0.061	0	1	0.203	0.070
Water
This community supports efforts to preserve water quality in creeks/rivers/lakes.	↑	—	Environmental Dashboard	−0.006	0.228	0	1	−0.008	0.313
I believe that as a community we can work together to improve water quality in creeks/rivers/lakes.	↑	—	Environmental Dashboard	0.069	0.073	0	1	0.103	0.104
I often think about water consumption when I am taking a shower, flushing the toilet, etc.	↑	—	Environmental Dashboard	−0.039	0.138	0	1	−0.047	0.174
I consciously make decisions to minimize the amount of water I use.	↑	—	Environmental Dashboard	−0.041	0.066	0	1	−0.052	0.088
I consciously make decisions to minimize the amount of water other people in my household use.	↑	—	Environmental Dashboard	0.273	0.386 ***	0	0	0.029	0.414
I have relatively little ability to influence local water quality in creeks, rivers, and lakes.	↓	—	Environmental Dashboard	−0.232	−0.229	0	1	−0.248	−0.233
I have relatively little ability to influence water consumption in my household.	↓	—	Environmental Dashboard	−0.007	−0.264	0	1	−0.006	−0.254
My water use has an important impact on the environment.	↑	—	Environmental Dashboard	0.033	−0.010	0	1	0.040	−0.013
How motivated? Reduce water use	↑	—	Environmental Dashboard	0.174	−0.009	0	1	0.198	−0.010
How motivated? Improve water quality	↑	—	Environmental Dashboard	0.244	0.279	0	1	0.247	0.287
Food
I can easily find information on farmers’ markets and where to find locally grown foods here.	↑	—	Oberlin Farmer’s Market Produce Perks	0.116	0.052	0	1	0.136	0.050
I can easily purchase fresh fruits and vegetables here.	↑	—	Oberlin Farmer’s Market Produce Perks	0.093	−0.493 ***	0	0	0.114	−0.508
How motivated? Purchase local foods	↑	—	Oberlin Farmer’s Market Produce Perks	0.586 ***	0.306	1	1	0.602	0.298
Transportation
My community supports efforts to bicycle and walk. ɨ	↑	↑	Complete Streets	0.101	−0.138	0	0	0.149	−0.183
I believe that as a community we can work together to reduce automobile use in town. ɨ	↑	↑	Complete Streets	−0.102	−0.291 *	0	0	−0.118	−0.349
It is easy for me to bike or walk to places I would like to go to in my town. ɨ	↑	↑	Complete Streets	−0.266 *	−0.220	0	0	−0.267	−0.212
How motivated? Bike or walk for transportation ɨ	↑	↑	Complete Streets	−0.070	−0.054	0	0	−0.059	−0.045
Waste
How often do you think residents of your town recycle? ɨ	↑	↑	Single-Stream Recycling Recycling Infrastructure Zero-Waste Plan 0	0.073	0.726 ***	0	1	0.110	0.989
There are resources here to help me recycle. ɨ	↑	↑	Single-Stream Recycling Recycling Infrastructure Zero-Waste Plan 0	0.257 ***	0.414 ***	1	1	0.463	0.554
It is easy for me to be consistent about recycling here. ɨ	↑	↑	Single-Stream Recycling Recycling Infrastructure Zero-Waste Plan 0	0.352 ***	0.625 ***	1	1	0.514	0.738
How motivated? Recycle ɨ	↑	↑	Single Stream Recycling Recycling Infrastructure Zero Waste Plan 0	0.310 ***	0.430 ***	1	1	0.460	0.388
How motivated? Compost ɨ	↑	↑	Recycling Infrastructure Zero Waste Plan 0	0.242	−0.048	0	0	0.172	−0.032
Other
As a community, we can work together to improve the local economy regardless of what happens elsewhere.	↑	—	Oberlin Project SEED Ventures	−0.104	−0.020	0	1	−0.125	0.025
Despite our differences, we can commit ourselves to common community goals.	↑	—	Oberlin Project	−0.099	−0.108	0	1	−0.117	0.156
If people in the community plan something, I would feel that WE are doing it, not THEY.	↑	—	Oberlin Project	−0.142	−0.076	0	1	−0.153	0.087
How satisfied with this here? Local government	↑	—	Oberlin Project	−0.370 *	−0.184	1	1	−0.345	0.186
How motivated? Invest money locally	↑	—	SEED Ventures	0.787 ***	0.717 ***	1	0	0.787	0.669

^ɨ denotes measure relating to intervention occurring in Berea. ⁰ denotes initiative started before 2012 but updated/modified between 2012 and 2016. ↑ = Agreement/frequency/satisfaction would increase. ↓ = Agreement/frequency/satisfaction would decrease. — = No change in agreement/satisfaction. O = Oberlin, B = Berea. * = sig at p < 0.05, *** = sig at p < 0.001.

and

Table 3. Changes in behavior, 2012–2016.

Measure	Prediction		Oberlin Initiative(s)	Observed Change		Matched Hypothesis		Cohen’s d
	O	B		O	B	O	B	O	B
Cross Cutting
Participation last year? Attended a public meeting or council meeting	↑	—	REC Debates	−0.024	−0.029	0	1	−0.021	−0.037
Participation last year? Worked w/other community members to solve a community problem	↑	—	Oberlin Project	−0.187	0.107	0	1	−0.145	0.132
Energy
Did you do this to your home in the last 5 years? Installed energy-efficient appliances	↑	—	Efficiency Smart 0 POWER 0	0.086	0.058	0	1	0.181	0.134
Did you do this to your home in the last 5 years? Other home improvements to reduce energy consumption	↑	—	Efficiency Smart 0 POWER 0	0.031	−0.084	0	1	0.064	−0.174
What percentage of the light bulbs in your home are energy-efficient (compact fluorescent or LED)?	↑	—	Efficiency Smar t 0 POWER 0	0.707 ***	0.322	1	1	0.647	0.261
Do you have and use a programmable thermostat?	↑	—	Efficiency Smart 0 POWER 0	0.057	−0.074	0	1	0.115	−0.152
Do you use the programmable thermostat to adjust the temperature of your home when you are away/asleep?	↑	—	Efficiency Smart 0 POWER 0	0.005	−0.144	0	1	0.011	−0.320
Energy behavior index	↑	—		<0.001	−0.153 *	0	0	0	−0.401
Water
Have you or someone else installed a low-flow toilet in your home?	↑	—	Environmental Dashboard POWER 0	−0.062	−0.135	0	1	−0.100	−0.212
Have you or someone else installed a low-flow showerhead in your home?	↑	—	Environmental Dashboard POWER 0	0.150	−0.180	0	1	0.239	−0.294
Do you have a rain barrel for collecting water run-off from your gutters?	↑	—	Environmental Dashboard	−0.082	0.054	0	1	−0.241	0.151
When taking a shower, about how long do you usually take in minutes?	↓	—	Environmental Dashboard	0.107	−0.059	0	1	0.13	−0.071
Did you do this to your home in the last 5 years? Installed energy-efficient windows	↑	—	POWER 0	−0.290	−0.149	0	1	−0.059	−0.299
Did you do this to your home in the last 5 years? Installed new insulation to reduce heating costs	↑	—	POWER 0	0.109	−0.179 *	0	0	0.221	−0.387
Water behavior index	↑	—		0.042	−0.012	0	1	0.072	−0.022
Food
How many of the meals that you eat at home use unprocessed or minimally processed food? ɨ	↑	↑	Oberlin Farmer’s Market Produce Perks Expansion of community garden	−0.0100	0.086	0	0	−0.011	0.099
Waste
How often do you recycle? ɨ	↑	↑	Single-Stream Recycling Recycling Infrastructure Zero-Waste Plan 0	0.116	0.760 ***	0	1	0.165	0.561
Waste behavior index	↑	—		0.033	0.422	0	0	0.052	0.418

^ɨ denotes measure relating to intervention occurring in Berea. ⁰ denotes initiative started before 2012 but updated/modified between 2012 and 2016. ↑ = Agreement/frequency/number of ‘yeses’ would increase. ↓ = Agreement/frequency/’number of ‘yeses’ would decrease. — = No change in agreement/satisfaction. O = Oberlin, B = Berea. * = sig at p < 0.05, *** = sig at p < 0.001.

).

As an example, Oberlin launched a concerted energy-efficiency effort, while Berea did not. We therefore hypothesized that energy-efficiency norms and behaviors would increase in Oberlin and not in Berea. If both towns demonstrated increases in efficiency norms and behavior, that would suggest that the increase reflects larger cultural shifts, rather than the impact of the initiative. If neither demonstrated an increase, this suggests a failure of the initiative (or a statistical-power problem). If Oberlin stayed the same and Berea decreased, this could suggest a larger cultural shift away from efficiency norms and behavior that was mitigated by the initiatives in Oberlin. Although the community of Berea did not engage in the same sort of comprehensive efforts as Oberlin, they did engage in a number of important sustainability initiatives (summarized in

Table 4. Initiatives implemented in Berea, 2012–2016.

Category	Program Name	Description	Accomplishments
Cross Cutting	Sustainability Grant Program	Funds projects related to energy efficiency, waste reduction, and sustainable products and services up to USD 2000.	Allowed for an increase in sustainability initiatives.
Energy	Energy Efforts	Building renovations with a key focus on changing lightbulbs in addition to conducting energy audits.	Reduced energy use in community buildings.
Food	Berea Community Farm	Community classes at the farm.	Community members provided space to learn about growing food locally.
Transportation	Bike Lane Expansion	New bike lanes added to community.	Bicycle lanes promoted alternate forms of transportation.
Waste	Recycling Expansion	Bigger recycling bins for homes.	Allowed consumers to fit more items into recycling bins week to week, rather than throwing them in the trash. Curbside recycling increased from 472 tons in 2012 to 1551 tons in 2016.
	Recycling Expansion	Recycling expanded to include cardboard and paper.	Reduced recycling burden on consumers.
	Recycling Expansion	Soft recycling picked up from people’s homes once a month and taken to The Salvation Army.	Reduced recycling burden on consumers.

). As in Oberlin, where there were initiatives, we predicted that there would be changes in responses to questions that related to those initiatives.

Consider effect sizes in drawing conclusions. In applied research, it is particularly important to distinguish statistical significance (the confidence with which you reject the null hypothesis of no impact) from effect size (the magnitude of the observed impact on a measured variable). We used the common statistical measure Cohen’s d to estimate and compare the effect sizes. Cohen’s d tells you how many standard deviations apart two group means are. Convention in the social sciences defines a Cohen’s d of less than 0.1 as a trivial effect; 0.1–0.3 a small effect; 0.3–0.5 a moderate effect; and greater than 0.5 a large difference effect [43]. It must be acknowledged that there is ambiguity inherent in teasing apart whether a statistically null finding is the result of a measurement inadequacy, a lack of statistical power, or a genuine lack of program impact. If responses to multiple questions measuring the effects of a particular initiative trended consistently in the predicted direction but did not reach statistical significance, this would suggest a small effect size and inadequate power to detect it. If the effect sizes were not consistently in the predicted direction, that would suggest a failure of the initiative (or a failure of the measures to capture the impact of the initiative) rather than a lack of statistical power.

2. Materials and Methods

2.1. Survey Design

The Small-Town Survey’s social metrics were informed by theory and research from the fields of sociology, social psychology, and community-based social marketing [9]. In particular, we made sure to include measures relevant to the Theory of Planned Behavior [39,40] and the Value Belief Norm theory [41]. However, the goal was not to rigorously test well-established theories of pro-environmental behavior, but rather to utilize them in guiding survey construction. Notably, the social metrics developed for this survey differed from traditional socio-economic variables commonly used as sustainability indicators. Our emphasis was on understanding individual- and household-level thought and behavior. The metrics were also targeted to assess changes in response to sustainability initiatives that were planned as part of the Oberlin Project; a goal was to include questions relevant to each major initiative.

The Small-Town Survey was collaboratively developed by a team of academic researchers and community stakeholders from differing backgrounds. Items from the questionnaire were then tested with constituents. The first round of focus groups consisted of individuals who lived and/or worked in Oberlin and who were engaged in the community through their roles as pastors, librarians, farmer’s-market organizers, daycare workers, etc. Our approach to revising questions was similar to the one advocated by Lesic et al. [44]. Initial focus group participants provided feedback and suggested revisions via several modes that included open discussions during debrief sessions, email comments, and open-ended post-survey feedback questions.

Table 5. Survey design: language feedback.

Questions	Issues Identified	Recommendations/Changes:
Do you have and use a programmable thermostat? A: Yes, No, Not sure	Confusion over the inclusion/necessity of the “not sure option”.	Provide context by defining programmable thermostat in question
It is easy for me to use public transportation to go places I would like to go.	Lack of inclusive/comprehensive answer choices: no option to indicate participants would take public transportation if it was available.	Separate multiple-choice question added: If public transportation were available in [town] how likely would you be to use it?
How often do you shower?	Ambiguous response choices. “Every three days or more” option is misleading since it could be read as showering more than three days or as showering more than every three days but less than every other day, as the option preceding says.	Ensure response options are clear and mutually exclusive. Replaced ambiguous answer choice with “every three days or less often”.
Multiple question sets	Inconsistencies with the ranges of 5-option Likert-scale questions. With recycling, one question provided options always/very often/often/sometimes/rarely or never, whereas the preceding one provided options very often/often/sometimes/rarely/never. Inconsistencies also present in energy-use questions: very important/important/somewhat/not that/not at all vs. extremely important/very/somewhat/not that/not at all and food questions: [insert choices].	Ensure Likert-scale congruence for similar questions.
I feel I belong in Oberlin	Grammar/phrasing ambiguity.	Use appropriate and clear language. Reword to “I feel like I belong in Oberlin”.
Recycling questions	Terminology ambiguity/inaccuracy.	Instead of referring to the recycling receptacle as a “recycling bin”, put “recycling cart/dumpster”.
I believe that the items put in recycling bins are actually being recycled.	Question should be taken out, replaced with something more current and associated with the current recycling program being really interested in the handling of yard waste. Current question not tailored to the stage of the project.	Question removed and replaced with “I know the items that can be put in our recycling container for curbside pickup”.
Overall, I think [town] is a good environment to raise children.; Despite our differences, we can commit ourselves to common community goals.; I believe my neighbors would help me in an emergency.; etc.	Several questions following one another using the same Likert-scale response options.	Condensing questions into a single matrix to allow for ease in participant response.
Roughly what percentage of the vegetables you eat do you grow yourself? A: Less than 25%, Between 25 and 50%, between 50 and 75%, over 75%	Lack of inclusive/comprehensive answer choices: no option for participants who to indicate they do not grow any of their food.	0% answer choice added.

presents examples of survey questions from the Time 1 survey deployed in 2012, potential issues identified during beta testing, and alterations made to improve the questions. The updated survey was taken by a subsequent focus group that included representatives from partner organizations including The Oberlin Project, the Environmental Dashboard, the City of Oberlin Recycling and Recycling Department, and The Oberlin Farmers Market, as well as a panel of college students. Once again, feedback was solicited, and a handful of minor adjustments were made to develop the final survey.

The final survey was designed to assess individual-level psychological questions about attitudes, efficacy, and norm perceptions. It included household-level questions on resource use (water, energy, recycling, etc.), individual-level measures designed to assess core social needs and social-justice concerns [45], and questions addressing specific community initiatives.

Time 1 vs. Time 2

The Time 1 survey was deployed in the summer of 2012, before the Oberlin Project began active community engagement; Time 2 was deployed four years later in the summer of 2016, at which point a variety of sustainability-related initiatives had been implemented in Oberlin. The Time 2 survey was largely identical to the T1 survey. A few questions were dropped because they were not relevant to the initiatives that were actually implemented. A few additional questions were added to assess awareness of the sustainability initiatives. At Time 1, the Small-Town Survey contained 125 questions and took an average of 30 min to complete; at Time 2, the survey contained 90 questions and took 35 min to complete (the questions at Time 2 included some open-ended prompts that took more time to complete).

2.2. Participants

2.2.1. The Control Community

To separate Oberlin-specific trends that might have resulted from concerted sustainability initiatives from regional changes in pro-environmental thought and behavior, we collected survey data in both Oberlin and the nearby town of Berea, OH. Berea was selected as a control because of its proximity and general similarity to Oberlin. Both are near Cleveland OH. Both are college towns; Oberlin College is an important employer and significant cultural presence in Oberlin, and Baldwin Wallace College plays a similar role in Berea. According to 2010 US census data (available at data.census.gov, accessed on 22 March 2024, Berea (B) is a little over twice the size of Oberlin (O), but otherwise similar in demographics as follows: population (O) vs. 19,000 (B); female 58% (O) vs. 54% (B); White 77% (O) vs. 82% (B); Black 10% (O) vs. 8% (B); Native American 0.3% (O) vs. 0.3% (B); Asian 4% (O) vs. 2% (B); Hispanic 8% (O) vs. 3% (B).

2.2.2. Recruitment

A key aim in this project was ensuring that both the survey design and administration elements were replicable across communities. We used Qualtrics, an online survey platform, to minimize recruitment and printing costs (though print copies were available upon request). Potential participants were randomly selected from a purchased mailing list (from Valassis Direct Mail Inc.; Newark, CA, USA) of community residents. However, we intentionally oversampled in census tracts that were low-income and/or had higher percentages of people of color. This oversampling was designed to ensure that information from minority community members were represented in sufficient numbers to achieve enough statistical power for demographic comparison [46]. At Time 1, no financial incentives were offered; we did not exceed USD 1500 in data collection expenses. For Time 2 recruitment, a financial incentive of USD 100 was raffled off in each community. Once again, data collection expenses did not exceed USD 1500.

To recruit participants, households received letters via the US postal service with an individualized code to be entered in the survey. This code allowed us to verify that only one response was received per household, and to allow us to send targeted reminders only to those who had not yet completed the survey. The letter invited their participation with the following text: We are writing to invite you to participate in the Ohio Small Town Sustainability Survey, a survey of community residents living in smaller college towns in Ohio like ours. The letter provided participants a short URL to access the survey; they were also provided with the means to request a paper copy of the survey or to decline participation entirely. Time 1 recruitment letters were signed by a researcher involved in the project. At Time 2, letters were co-signed by an Oberlin College faculty member and well-known community members in each community in the hope of boosting response rates. At both times, participants were assured that their responses were completely confidential. The file matching the name and address to the participant ID was stored separately from survey responses, and at no time was identifying information matched to survey data.

Ten days after the initial recruitment mailing, households that did not respond received a reminder postcard in the mail emphasizing the importance of their participation. Two weeks following, additional reminder postcards were mailed out. A subsequent round of reminder postcards was sent out in the two weeks after; postcards to low-income areas were hand-delivered with a handwritten message, We look forward to your support! (followed by a smile emoticon). Potential participants were not contacted again after four prompts. Informed consent was obtained from all subjects involved in the study.

All participants who began the survey were included in the data analysis. At Time 1, the overall response rate was 36%; Oberlin’s response was 38% (n = 81) and Berea’s was 34% (n = 74). At Time 2, the Oberlin response rate was 40% (n = 86). Due to an insufficient response rate from the city of Berea at Time 2, a second set of 300 Berea addresses was randomly drawn from the purchased mailing list and a new round of recruitment letters was mailed. The final response rate for Berea at Time 2 was 11% (n = 51).

Our samples were largely but not entirely representative, in comparison to American Community Survey data from the year of data collection (available at data.census.gov, accessed on 22 March 2024). In 2012, women were over-represented in our Berea sample, compared to census data (X² = 7.78, p < 0.01). In 2016, women were under-represented in our Oberlin sample (X² = 5.10, p < 0.05). However, as our sample was weighted for gender, this is unlikely to significantly impact our results. Samples for both towns and both years were representative of the populations’ percentage of White vs. POC residents.

2.3. Community Initiatives

Environmental organizations and city governments in Berea and Oberlin were consulted to develop a list of major sustainability initiatives that actually took place in both communities between 2012 and 2016. We erred on the side of inclusivity when deciding what to include as an initiative in our analysis. The full list of these initiatives can be found in Table 1 and Table 4. Table 2, Table 3 and

Table 6. Observed changes in 2016, comparing between Oberlin and Berea.

Measure	Prediction	Oberlin Initiative(s)	Oberlin		Berea		ANOVA		Matched Hyp.	Cohen’s d
			M	SD	M	SD	F	Sig.
Cross Cutting
If I wanted to start a business here, I would feel supported.	O > B	SEED Ventures	3.17	0.850	3.40	0.772	3.614	0.059	0	−0.275
I feel that I have resources that I would need to start a business here.	O > B	SEED Ventures	2.70	0.933	2.80	0.960	0.538	0.464	0	−1.04
I feel that this community could sustain more businesses.	O > B	SEED Ventures	3.72	0.830	3.70	0.865	0.030	0.862	0	0.023
Energy
I have a good sense of the different sources of electricity generation used to provide local residents with electrical power.	O > B	Environmental Dashboard	3.49	1.176	3.17	1.091	3.597	0.059	0	0.274
I feel aware of the total electricity use of the whole community.	O > B	Environmental Dashboard	2.84	1.118	2.48	1.054	4.684	0.032	1	0.323
The amount of electricity I use has an important impact on the natural environment.	O > B	Environmental Dashboard	4.12	0.858	3.95	0.952	1.529	0.218	0	0.187
How often do you try to conserve energy?	O > B	Efficiency Smart 0 POWER 0 Environmental Dashboard	4.30	0.727	4.25	0.806	0.206	0.651	0	0.065
Water
I think about where my water is coming from when I turn on the faucet or hose, take a shower, or turn on appliances that use water.	O > B	Environmental Dashboard	2.96	1.109	3.2	1.149	2.074	0.151	0	−0.210
I think about the water lines that run through the city to my home.	O > B	Environmental Dashboard	2.88	1.048	3.02	1.034	0.805	0.371	0	−0.132
I think about where my water is going once I’ve used it.	O > B	Environmental Dashboard	3.01	0.939	3.03	0.988	0.021	0.885	0	−0.020
I think about the environmental resources necessary to treat drinking water and to treat waste water.	O > B	Environmental Dashboard	3.16	0.985	3.32	0.999	1.227	0.269	0	−0.159
I think about water cleanliness and pollution in local creeks, streams, rivers, and bodies of water.	O > B	Environmental Dashboard	3.52	0.932	3.67	0.997	1.052	0.306	0	−0.154
I have a good sense of the natural body of water that provides tap water for Oberlin residents and how water is stored, treated, and delivered to my home.	O > B	Environmental Dashboard	3.62	1.046	2.48	1.365	42.590	<0.001	1	0.953
The amount of water I use has an important impact on the natural environment.	O > B	Environmental Dashboard	4.02	0.801	3.92	0.904	0.626	0.43	0	0.117
I feel aware of the total water use of the whole community.	O > B	Environmental Dashboard	2.73	1.060	2.61	1.178	0.521	0.471	0	0.107
How often do you think residents of your town try to conserve water?	O > B	Environmental Dashboard	3.15	0.696	3.05	0.581	0.993	0.32	0	0.150
How often do you try to conserve water?	O > B	Environmental Dashboard	3.8	0.844	3.81	0.893	0.011	0.916	0	−0.011
How often do you shower?	B > O	Environmental Dashboard	2.45	0.721	2.58	0.652	1.583	0.21	0	0.183
Transportation
How often do you think residents bike to get around? ɨ	—	Complete Streets	3.57	0.729	2.96	0.662	33.762	<0.001	0	0.849
How often do you bike to get around town? ɨ	—	Complete Streets	3.5	1.287	4.06	1.079	9.35	0.003	0	−0.453
How often do you think residents walk to get around?	O > B	Complete Streets	3.71	0.730	3.09	0.731	32.002	<0.001	1	0.833
How often do you walk to get around town?	O > B	Complete Streets	2.59	1.165	3.16	1.023	11.65	<0.001	1	−0.502
Waste
I know the items that can be put in our recycling container for curbside pickup. ɨ	—	Single-Stream Recycling	4.65	0.538	4.69	0.697	0.205	0.651	0	−0.065

^ɨ denotes measure relating to intervention occurring in Berea. ⁰ denotes initiative started before 2012 but updated/modified between 2012 and 2016. O = Oberlin, B = Berea. O > B = Higher level of agreement or frequency in Oberlin than Berea as Berea had no similar initiative take place. B > O = Lower level of frequency in Oberlin than Berea as Berea had no similar initiative take place. — = No difference in awareness as both communities implemented similar initiatives. 1 = matched hypothesis, 0 = did not match hypothesis.

indicate which survey items are relevant to each initiative.

3. Results

3.1. Analysis Strategy and Overview

We considered weighting our samples on gender, race, age, and/or home-owner status. Sample weights for the Oberlin data (in both 2012 and 2016) were within a reasonable range (none below 0.5 or above 2, [47]). However, in our control community, some weights would have been as high as six. Using a small number of people to represent an entire demographic group is unlikely to lead to accurate results. Therefore, we did not weigh for these demographics in either community. The analyses reported below are weighted for gender, as there are a priori reasons to expect gender differences in variables related to pro-environmental behavior, and the weights were within an acceptable range for both towns in both years.

We first summarize descriptive data demonstrating that, on the whole, our items had an acceptable spread and distributions; we follow with correlations demonstrating that our survey items correlate with each other in ways that are consistent with previous environmental psychology research. We next present demographic comparisons that suggest the survey results mostly reflected expected national trends. Finally, we compare how our main dependent variables—psychological and behavioral measures—changed from Time 1 to Time 2 in both communities and evaluate whether the patterns of significance and effect sizes are consistent with our a priori predictions that sustainability initiatives would enhance these measures.

3.2. Descriptive Statistics

We examined the descriptive statistics (mean, SD, minimum and maximum, and distribution) for all survey items to verify that they demonstrated an adequate spread and to identify variables that had skewed distributions. Participants used all available scale points for nearly all questions. There were two exceptions: for the item “I often feel a strong connection to nature” no one used the lowest value (strongly disagree); and for the item “How safe would you feel walking alone at night in the business district?” no one used the lowest value (very unsafe).

More than half of the survey items had skewed distributions. Generally, respondents’ answers were skewed positively (e.g., feeling satisfied or feeling safe), and pro-environmentally (e.g., I consciously make decisions to minimize my electricity use).

Self-reported behaviors showed more variability. Some behaviors were reported at relatively high rates: 48% reported replacing windows, 53% reported installing low-flow toilets, 56% reported installing low-flow shower heads, 70% reported installing high-efficiency appliances, and 63% reported making “other home improvements” to reduce energy use. Other behaviors occurred less frequently: 29% reported composting food scraps, 38% reported installing new insulation to reduce heating costs, 35% reported installing timers or motion sensors on lights or other automatic devices, and 14% reported having rain barrels for collecting water run-off from gutters.

3.3. Relationships between Variables

To confirm that the items measuring psychological and behavioral constructs related to each other in theoretically predicted ways, we ran correlations between subsets of variables. The correlations between the variables in each domain assessed (transport, waste, water, and electricity) can be found in

Table 7. Energy measures correlations.

#	Measure	1	2	3	4	5	6	7	8	9	10	11	12
1	There are programs and resources here to help me reduce my household energy consumption + cost	1	0.814 **	0.436 **	0.171 **	−0.131 *	0.197 **	0.385 **	0.186 *	0.193 **	0.368 **	0.516 **	0.354 **
2	There is someone in my community who can help reduce my household energy consumption + cost		1	0.487 **	0.189 **	−0.148 **	0.241 **	0.322 **	0.189 **	0.16 **	0.409 **	0.589 **	0.378 **
3	I believe that as a community we can work together to reduce our total energy consumption.			1	0.339 **	−0.205 **	0.408 **	0.321 **	0.296 **	0.218 **	0.308 **	0.393 **	0.292**
4	I often think about electricity consumption when I turn a light or appliance on or off.				1	−0.235 **	0.514 **	0.192 **	0.602 **	0.335 **	0.107	0.055	0.104
5	I have relatively little ability to influence electricity consumption in my household.					1	−0.208 **	−0.204 **	−0.342 **	−0.187 **	−0.136	−0.048	−0.048
6	The amount of electricity I use has an important impact on the natural environment.						1	0.197 **	0.469 **	0.324 **	0.176 *	0.147 *	0.063
7	How often do you think residents of your town try to conserve energy?							1	0.358 **	0.105 *	0.133	0.173 *	0.1588 *
8	How often do you try to conserve energy?								1	0.353 **	0.056	0.01	0.104
9	Energy behavior index									1	0.121	0.201 **	0.17 *
10	T2 Aware of Initiative? Rebate Program for energy efficient appliances (Efficiency Smart)										1	0.653 **	0.342 *
11	T2 Aware of Initiative? The enhancement of services to improve the energy efficiency in homes											1	0.372 **
12	T2 Aware of Initiative? Significant expansion of green electricity												1

* Correlation is significant at the 0.05 level (two-tailed). ** Correlation is significant at the 0.01 level (two-tailed).

,

Table 8. Water measures correlations.

#	Measure	1	2	3	4	5	6	7	8	9
1	This community supports efforts to preserve water quality in creeks/rivers/lakes.	1	0.450 **	0.206 **	−0.116 *	−0.098	0.217 **	0.272 **	0.148 *	0.149 **
2	I believe that as a community we can work together to improve water quality in creeks/rivers/lakes.		1	0.286 **	−0.162 **	−0.148 **	0.313 **	0.285 **	0.281 **	0.209 **
3	I often think about water consumption when I am taking a shower, flushing the toilet, etc.			1	−0.029	−0.212 **	0.509 **	0.289 **	0.636 **	0.398 **
4	I have relatively little ability to influence local water quality in creeks, rivers, and lakes.				1	0.354 **	−0.133	−0.1	0.02	−0.036
5	I have relatively little ability to influence water consumption in my household.					1	−0.003	−0.068	−0.098	−0.108 *
6	T2 The amount of water I use has an important impact on the natural environment.						1	0.217 **	0.424 **	0.323 **
7	T2 How often do you think residents of your town try to conserve water?							1	0.344 **	0.138
8	T2 How often do you try to conserve water?								1	0.374 **
9	Water behavior index									1

* Correlation is significant at the 0.05 level (two-tailed). ** Correlation is significant at the 0.01 level (two-tailed).

,

Table 9. Waste measures correlations.

#	Measure	1	2	3	4	5	6	7	8	9	10	11	12
1	There are resources here to help me recycle.	1	0.604 **	0.615 **	0.379 **	0.379 **	−0.112 *	−0.006	0.279 **	0.314 **	0.356 **	0.271 **	0.378 **
2	T2 I know the items that can be put in our recycling container for curbside pickup.		1	0.715 **	0.186 **	0.503 **	−0.153 *	0.026	0.163 *	0.426 **	0.234 **	0.231 **	0.288 **
3	It is easy for me to be consistent about recycling here.			1	0.377 **	0.669 **	−0.169 **	−0.026	0.186 *	0.548 **	0.273 **	0.161 *	0.272 **
4	How often do you think residents of your town recycle?				1	0.314 **	−0.068	−0.141	0.045	0.276 **	0.141	0.038	0.081
5	How often do you recycle?					1	−0.212 **	0.087	0.096	0.793 **	0.346 **	0.252 **	0.377 **
6	What do you do with yard-waste materials?						1	−0.401 **	−0.047	−0.516 **	−0.316 **	−0.206 **	−0.29 **
7	T2 Do you compost (food scraps)?							1	0.011	0.199 **	0.244 **	0.225 **	0.189 **
8	T2 There are resources here to help me compost.								1	0.102	0.101	0.102	0.107
9	Waste behavior index									1	0.348 **	0.225 **	0.384 **
10	T2 Aware of Initiative? Consolidation of recycling into single stream										1	0.501 **	0.619 **
11	T2 Aware of Initiative? Updating recycling bins with bigger ones											1	0.548 **
12	T2 Aware of Initiative? Expansion of recyclable items to include paper and cardboard												1

* Correlation is significant at the 0.05 level (two-tailed). ** Correlation is significant at the 0.01 level (two-tailed).

and

Table 10. Transportation measures correlations.

#	Measure	1	2	3	4	5	6	7	8
1	My community supports efforts to bicycle and walk.	1	0.204 **	0.318 **	−0.059	0.347 **	0.441 **	0.104	0.165 *
2	I believe that as a community we can work together to reduce automobile use in town.		1	0.383 **	0.253 **	0.186 *	0.176 *	0.348 **	0.164 *
3	It is easy for me to bike or walk to places I would like to go to in my town.			1	0.021	0.253 **	0.249 **	0.581 **	0.289 **
4	T2 If public transportation were available here, how likely would you be to use it?				1	0.083	0.112	0.163 *	0.160 *
5	T2 How often do you think residents bike to get around?					1	0.715 **	0.183 *	0.111
6	T2 How often do you think residents walk to get around?						1	0.261 **	0.087
7	Transportation behavior index							1	0.389 **
8	T2 Aware of Initiative? Addition of bike lanes								1

* Correlation is significant at the 0.05 level (two-tailed). ** Correlation is significant at the 0.01 level (two-tailed).

. Nearly all correlations were significant and in the predicted direction. For example, awareness of the energy-efficiency programs was correlated with items designed to measure their success (e.g., “There are programs and resources here to help me reduce my household energy consumption and cost”). A lack of efficacy (e.g., “I have relatively little ability to influence electricity consumption in my household”) correlated negatively with the corresponding measures of self-reported behavior.

There were some exceptions. Questions about water quality did not correlate with questions about water conservation. Among transportation variables, the question about whether participants would use public transportation if it were available did not correlate with transportation questions focused on current behavior. In the domain of waste, questions about composting did not correlate with questions about recycling. In each of these cases, the non-correlating variables arguably represent distinct concepts or behaviors, and thus would not be expected to correlate with each other.

3.4. Evaluation of a Priori Demographic Predictions

One way we sought to validate the survey tool was to assess whether it reflected common demographic differences found in previous research. Once again, our goal was not to provide an exhaustive exploration of demographic differences, but to select a small subset of particular questions that had the clearest a priori predictions for each demographic category.

First, we predicted that low-income individuals (below the median in our sample) would report lower satisfaction on all questions assessing the cost of living and services in both communities. We also predicted that people of color would report a lower sense of belonging and less satisfaction with living in the communities of Oberlin and Berea that are predominantly (~75–80%) White. We hypothesized this because people of color are a numerical minority and likely to experience systematic oppression. We also hypothesized that POC would be less likely than White participants to endorse the item “This community is respectful and accepting of people with different racial/ethnic backgrounds” for similar reasons.

To identify demographic differences in environmental concern and behavior, we reviewed research conducted on US adults within the last 20 years. A robust collection of literature documents gender differences in environmental concern and pro-environmental behavior, with women engaging in both at higher levels than men [48,49,50]. The relationships between race and environmental concern and behavior is complex [51]; there are differences in how pro-environmental concern is expressed but no consistent overall group differences. We made no a priori predictions about race.

Table 11. Demographic differences: race.

Measure	POC		White		df	t	p	Cohen’s d
	Mean	SD	Mean	SD
How satisfied with this here? Retail and shopping opportunities	3.04	1.176	3.78	0.901	373	4.779	<0.001	0.785
How satisfied with this here? Employment opportunities	2.50	1.159	3.23	0.835	369	4.958	<0.001	0.835
How satisfied with this here? Arts, entertainment, and cultural opportunities	4.30	0.859	4.50	0.739	373	1.625	0.105	0.267
How satisfied with this here? Community events and festivals	4.03	0.998	4.49	0.678	373	3.818	<0.001	0.627
How satisfied with this here? Local government	2.93	1.190	3.70	1.006	373	4.578	<0.001	0.752
How satisfied with this here? Public spaces	3.85	0.859	4.36	0.764	373	4.015	<0.001	0.660
How satisfied with this here? Opportunities for safe biking and walking	4.26	0.670	4.54	0.720	373	2.326	0.021	0.382
How satisfied with this here? Park and recreation opportunities	4.06	0.946	4.49	0.774	373	3.296	0.001	0.542
How satisfied with this here? Public transportation	2.23	1.062	2.71	1.070	373	2.736	0.007	0.450
How satisfied with this here? Public schools	3.24	1.153	3.63	1.013	373	2.316	0.021	0.381
I feel I belong here	4.12	0.869	4.3	0.776	373	1.435	0.152	0.236

,

Table 12. Demographic differences: gender.

Measure	Female		Male		df	t	p	Cohen’s d
	Mean	SD	Mean	SD
How motivated? Conserve energy	4.27	0.822	4.16	0.753	378	−1.283	0.200	0.133
How motivated? Bike or walk for transportation	3.52	1.225	3.61	1.228	375	0.716	0.474	−0.074
How motivated? Reduce water use	4.03	0.86	3.78	0.965	378	−2.686	0.008	0.278
How motivated? Improve water quality	4.06	0.974	3.82	0.985	377	−2.407	0.017	0.249
How motivated? Purchase local foods	3.96	0.916	3.53	1.096	378	−4.182	<0.001	0.432
How motivated? Recycle	4.54	0.948	4.47	0.842	378	−0.749	0.455	0.077
How motivated? Compost	3.41	1.411	3.23	1.504	377	−1.189	0.235	0.123
Energy behavior index	0.0149	0.37041	−0.0214	0.40945	377	−0.904	0.367	0.094
Water behavior index	−0.0739	0.55739	0.0586	0.56477	378	2.286	0.023	−0.236

and

Table 13. Demographic differences: income.

Measure	Above Median		Below Median		df	t	p	Cohen’s d
	Mean	SD	Mean	SD
How would you rate the cost of living here?	3.77	0.934	3.47	1.024	304	−2.661	0.008	−0.309
How would you rate the cost of living here now versus two years ago?	3.58	0.881	3.3	0.941	301	−2.669	0.008	−0.311
In terms of cost, how would you rate this city’s electric services	3.98	0.865	3.83	0.986	299	−1.344	0.180	−0.157
In terms of cost, how would you rate this city’s water and sewer services	3.72	1.021	3.55	1.154	299	−1.371	0.171	−0.16
In terms of cost, how would you rate this city’s garbage collection services	4.21	0.857	3.98	0.924	299	−2.209	0.028	−0.259

summarize the results of the survey items relevant to our hypotheses above. As predicted, we found that low-income community members were significantly less satisfied on three of the five items assessing satisfaction with the cost of living; all effects were in the predicted direction, mean Cohen’s d = 0.239. Similarly, people of color expressed significantly lower satisfaction in their community on all but two items (cultural opportunities and belonging); all effects were in the predicted direction, mean Cohen’s d = 0.538.

However, contrary to our a priori hypotheses, significant gender differences only emerged for a handful of survey items. Among these, most effect sizes were in the predicted direction (i.e., women expressing more pro-environmental concern and behavior than men). For example, women were significantly higher in self-reported motivation to reduce water use, motivation to improve water quality, and motivation to purchase local foods. One exception to this gender pattern is that men reported engaging in significantly more energy-related pro-environmental behavior than women. Across all items, the average effect size supports our prediction that women would report more pro-environmental concern and behavior, but this effect was quite small, with a mean Cohen’s d = 0.120.

3.5. Evaluation of a Priori Predictions Related to Community Interventions

The most important test of the assessment tool’s effectiveness is whether it reflects changes that were known or hypothesized a priori to occur. In this section, we have organized our reporting of the dependent variables by the anticipated likelihood that they might be influenced by the sustainability initiatives that occurred: first, simple awareness of the initiatives; then, changes in norms, attitudes, and beliefs; and finally, behavior change (self-reported). We present a series of two (time) × two (town) ANOVAS evaluating whether the predicted changes in each town occurred. We also present the results of between-town analyses for questions collected only at Time 2. We have not included demographic factors in our analyses because the resulting cell sizes would have been too small.

3.5.1. Initiative Awareness

The first important category in evaluating the survey methodology was “Initiative Awareness”, which assessed residents’ level of knowledge at Time 2 about the initiatives implemented in either community. The Oberlin Project included multiple sustainability-related initiatives. The Berea community engaged in a subset of very similar initiatives. Awareness of an initiative that actually occurred in either community sets an important but low bar for an initiative’s success, as it requires merely recognition.

At Time 2, we asked Oberlin and Berea residents about whether any of the initiatives undertaken in Oberlin had occurred in their community (some of these initiatives also occurred in Berea). We conducted two sets of analyses on the responses. First, we compared participants’ awareness of each initiative to the midpoint of the scale (“I am unsure if this initiative happened”). Answers below the midpoint indicated participants were unaware of the initiative, while answers above the midpoint indicated awareness of or participation in the initiative. We hypothesized simply that on average people should be aware (significantly above the midpoint) of initiatives that actually happened in their community, and not aware (significantly below the midpoint) of those that did not happen in their community. In a second set of analyses, we compared Oberlin participants’ awareness to Berea participants’ awareness. We hypothesized that Oberlin residents would report being more aware than Berea residents of initiatives that happened in Oberlin only; we hypothesized no difference between the two communities on initiatives that occurred in both places. A summary of the results appears in

Table 14. Community initiative awareness in 2016.

Initiative	Measure	Prediction	Oberlin 2016		Berea 2016		ANOVA		Matched Hyp.
			M	SD	M	SD	F-value	p
Cross Cutting
City Climate Action Plan 0	Aware of Initiative? Development of a City Climate Action Plan	O > B	3.13	0.782	2.31	0.677	54.267	<0.001	1
REC Debates	Aware of Initiative? Debates over how to spend dollars from renewable energy credits (REC debate)	O > B	3.45	0.833	2.36	0.539	98.428	<0.001	1
Energy
Ecolympics	Aware of Initiative? Competitions to reduce electricity and water usage in public schools	O > B	3.13	0.83	2.43	0.671	36.577	<0.001	1
Efficiency Smart 0	Aware of Initiative? Rebate Program for energy-efficient appliances (Efficiency Smart)	O > B	3.77	0.964	2.72	1.038	50.427	<0.001	1
POWER 0	Aware of Initiative? The enhancement of services to improve energy efficiency in homes	O > B	3.78	0.95	2.71	1.014	53.684	<0.001	1
Green electricity	Aware of Initiative? Significant expansion of green electricity	O > B	3.22	0.788	2.44	0.658	49.984	<0.001	1
Solar Co-op	Aware of Initiative? Development of a co-operative to add solar energy to home rooftops	O > B	3.2	1.006	2.07	0.683	70.354	<0.001	1
Food
Oberlin Farmer’s Market	Aware of Initiative? Wider selection of locally grown foods in farmer’s markets	—	4.29	0.948	2.85	1.248	81.724	<0.001	1
Legion Fields Community Garden	Aware of Initiative? Expansion/creation of community gardening	—	3.69	0.805	3.18	1.025	14.494	<0.001	0
Transportation
Complete Streets	Aware of Initiative? Addition of bike lanes	—	4.1	0.945	3.91	0.94	1.911	0.169	1
Waste
Recycling Infrastructure	Aware of Initiative? Updating recycling bins with bigger ones	—	4.57	0.905	4.21	1.189	5.739	0.018	0
Recycling Infrastructure	Aware of Initiative? Expansion of recyclable items to include paper and cardboard	—	4.83	0.536	4.5	0.995	8.71	0.004	0
Single-stream Recycling	Aware of Initiative? Consolidation of recycling into a single stream	O > B	4.62	0.901	4.33	1.111	3.837	0.052	1

⁰ denotes initiative started before 2012 but updated/modified between 2012 and 2016. O > B = Oberlin more aware of initiative than Berea as Berea had no similar initiative take place. — = No difference in awareness as both communities implemented similar initiatives.

.

In the comparison to the midpoint of the scale, 85% of comparisons matched our hypotheses; across both communities, respondents were aware of initiatives that had actually occurred in their community and not aware of initiatives that had not occurred in their community. The effect sizes were all in the predicted direction (mean Oberlin Cohen’s d = 1.022 and mean Berea Cohen’s d = 0.228 for initiatives that happened in Berea, and −0.160 for those that did not). In general, each sample was appropriately significantly above or below the midpoint, depending on whether or not the initiative happened in their community. There were a few exceptions for particular initiatives. For example, Oberlin participants were only marginally above the midpoint for awareness of the city’s Climate Action Plan and resource-reduction competitions in local schools (which would have impacted only parents/guardians of school children). Berea was not significantly below the midpoint on awareness of an increase in local foods at the farmer’s market (possibly because local foods had indeed increased in the absence of an explicit initiative). Berea participants were not above the midpoint on awareness of the expansion of community gardening (possibly because the programming in Berea involved classes on gardening, not expanding or creating gardens as it did in Oberlin).

In the town comparisons, 77% of items matched our hypotheses, with effect sizes all in the appropriate direction (mean Cohen’s d = 0.842). As predicted, Oberlin residents indicated a higher level of recognition than Berea residents of initiatives that happened only in Oberlin. The items that did not match our a priori predictions were all related to initiatives that happened in both communities: the Berea sample reported being significantly less aware of the initiative than the Oberlin sample (but as reported above, was still above the midpoint, indicating some level of awareness). The fact that Oberlin residents were overall more aware than Berea residents of initiatives that happened in their community may be a result of the coordinated effort in Oberlin represented by the Oberlin Project.

3.5.2. Changes in Norms, Attitudes, and Beliefs

Table 2 summarizes the results from 38 questions that measured norms, attitudes, and beliefs about a variety of issues, and our a priori predictions based on the initiatives that took place in each community. Responses to 21 of the 38 questions (55%) changed in a direction (d > 0.05) that was consistent with our a priori hypotheses regarding how norms, attitudes, and beliefs would change over time in Oberlin. However, these changes were statistically significant for only six of these questions. In total, 37% of the items expected to increase yielded negative effect sizes. Across the suite of items, the average effect size was only 0.084, suggesting that, as a whole, the programs implemented in Oberlin resulted in small and inconsistent changes in norms, attitudes, and beliefs. One domain was an exception: in Oberlin, three of the five questions related to recycling and composting showed significant changes in the predicted direction (average Cohen’s d across all solid waste questions = 0.344).

In the control community of Berea, we predicted that there would be no change in 29 questions because there were no local initiatives related to these questions during the study period. Consistent with our hypotheses, we observed no statistically significant change in 25 out of 29 variables (86%). We predicted changes in nine variables related to initiatives that occurred in Berea and observed significant changes in the predicted direction for four of these variables. It should be noted that a prediction of no change is much easier to support than a prediction of change. Similar to Oberlin, the Berea sample showed significant predicted changes in the domain of recycling and composting: Four of the five solid waste questions increased significantly (average Cohen’s d = 0.527).

Table 15. Effect size averages summarizing changes from Time 1 to Time 2.

Category	Norms, Attitudes, and Beliefs		Self-Reported Behavior
	Oberlin	Berea	Oberlin	Berea
Energy	0.042	0.038	0.170	0.025
Water	0.026	0.087	0.017	−0.199
Food	0.284	−0.053	−0.011	−0.099
Waste	0.344	0.527	0.057	0.072

provides average effect sizes for four major categories: energy, water, food, and waste. This provides a high-level snapshot of trends across individual survey items.

3.5.3. Behavior Changes Measured at T1 and T2

Table 3 summarizes the results for self-reported behavior-change questions. On items that measured self-reported behaviors at both Time 1 and Time 2, only 1 of the 15 comparisons matched our hypotheses for Oberlin: the reported percentage of LED bulbs in the home increased from 2012 to 2016. Across all self-reported behaviors for Oberlin, the mean Cohen’s d = 0.066, indicating a tiny (and perhaps spurious) increase in pro-environmental behavior. Among Berea participants, all but two comparisons matched the hypotheses, primarily in cases where we predicted no change. Only 1 out of 15 questions showed a significant (and hypothesized) increase: the self-reported frequency of recycling increased from 2012 to 2016. Across all items for Berea, the mean Cohen’s d = −0.074, indicating a tiny (and perhaps spurious) decrease in pro-environmental behavior.

3.5.4. Behaviors Measured at T2 only

Among the questions that we asked, only at Time 2, only 4 of 23 (24%) were consistent with our expected (hypothesized) change (see Table 9). As predicted, Oberlin residents reported being more aware of content that appeared on the Environmental Dashboard and reported composting more. Interestingly, Oberlin residents believed other residents of their town walked and biked more in 2016 (compared to 2012) than Berea residents did, but Berea residents actually reported walking and biking more in 2016 (compared to 2012) than Oberlin residents.

4. Discussion

This research evaluated a potentially scalable approach to measuring the impact of a community-wide effort to promote sustainability on social, psychological, and behavioral variables. We sought to document whether specific programs had targeted impacts on attitudes, norms, and behavior; we also sought to evaluate whether the holistic approach to sustainability represented by the Oberlin Project created broad cultural shifts in the community. (We note that the data and discussion in this paper are not indicative of Oberlin’s current condition). Below, we provide a summary of our findings and a candid assessment of our survey methodology as a tool for achieving our goals.

4.1. Evaluation of Survey Items

Our descriptive statistics, correlations, and demographic comparisons suggest that our survey items measured what they were intended to measure and had reasonably good psychometric properties. Participants used all scale points for the vast majority of questions, and the survey items correlated with each other as predicted by well-established theories of pro-environmental behavior. With the exception of gender, our a priori demographic differences emerged, suggesting that our survey questions produced the same differences between demographic groups found in previous research.

It should be noted, however, that responses to many of the survey items were skewed, with a majority of responses clustered at the end of the scale representing the positive views of the communities and pro-environmental beliefs. This was true in both Oberlin and Berea. The positive skew could accurately reflect the broad cultural tendencies of these two college towns in Ohio, or they could be the result of sample bias (both possibilities are discussed further below).

4.2. Support for a Priori Predictions (or Lack Thereof)

Despite the apparent validity of the survey items themselves, support for our a priori predictions was limited. One major exception was the Time 2 questions assessing citizens’ awareness of the initiatives that occurred in their communities: on the whole, the questions assessing awareness of the various sustainability initiatives matched our a priori predictions. Residents of both communities accurately identified initiatives that happened in their towns, and Berea residents correctly indicated they had not heard of initiatives in Berea that had only occurred in Oberlin. These data also provide evidence that the sustainability programs associated with the Oberlin Project achieved a level of success with respect to communication: residents of Oberlin had heard of nearly all of them (the exceptions being the Climate Action Plan and resource-reduction competitions in the school, whose ratings only marginally differed from the midpoint of the scale, “I am unsure that this happened”).

However, despite the basic integrity of the survey instrument and evidence that participants were accurately aware of the sustainability initiatives that occurred in their communities, the changes in norms, attitudes, beliefs, and self-reported behavior that we hypothesized largely did not emerge as significant effects. Two exceptions were positive attitudes towards recycling (predicted to increase in both communities) and the increased use of LED light bulbs in Oberlin (and we note that in these cases our study design does not allow us to establish causality). There are several possible explanations for the null findings, none of which are mutually exclusive. These include the failure of initiatives to create changes in the variables we measured; bias in the sample; ceiling effects; and inadequate statistical power. We discuss each of these potential explanations below.

4.2.1. Lack of Program Impact

The simplest explanation for the null findings is that the sustainability programs that were implemented failed to create changes in attitudes, norms, and behavior. If we assume that the survey was capable of documenting all changes that occurred, one would conclude that recycling programs in both towns and efforts to disseminate LED bulbs in Oberlin were the only ones that were successful.

While it is certainly possible that some programs were not successful, we do not find this a compelling blanket explanation. First, as noted above, there is evidence that citizens were at least aware of the initiatives that happened in their town, which on its own is a limited kind of success. More importantly, evidence available from program administrators clearly documents tangible program success. For example, in Oberlin, the number of homes weatherized during the study increased by 90%, the amount recycled increased by 123%, and the number of people who received rebates for energy-efficiency appliances increased by 700% (see Table 1). Thus, the self-reported behaviors reported in survey responses do not accurately reflect aggregate community behavior as documented by program administrators. In short, the survey methodology did not reliably capture the program successes that we know occurred. This failure can be attributed to multiple factors discussed further below.

4.2.2. Bias in Our Sample

No sample is truly random; there will always be people who do not wish to complete surveys or do not bother to. Similarly, many survey efforts struggle to adequately represent low-income people and people of color (although our samples were representative of race) [52,53]. However, it is possible that our sample was biased beyond these common sample biases. While Time 1 response rates were typical of response rates for contemporaneous mailed surveys at 37% [54], we struggled at Time 2 to recruit an adequate sample from Berea in particular. Thus, the sample we did recruit from Berea at Time 2 was even farther from the ideal of random, and therefore less likely to be representative of the general population. In several cases, pro-environmental behavior went down significantly in Berea, while remaining unchanged in Oberlin. While it is possible that the initiatives in Oberlin counteracted a broader regional trend of decreased pro-environmental behavior, it is also possible that the challenges we experienced in obtaining an adequate Time 2 Berea sample explains the drop. This potential bias cannot explain the lack of predicted positive changes in Oberlin, however.

Another source of bias could have arisen from the name we gave the survey: describing the survey as a “sustainability” survey may have tilted the sample to over-represent those who care about issues related to this term. This could provide one possible explanation for the relatively high levels of pro-environmental attitudes and behavior we observed at both times and in both towns. In further support of this explanation, we note that a fully representative sample of Oberlin at Time 2 would have contained only one participant in Oberlin’s solar co-operative, but four of our participants indicated they had participated in this initiative.

There are reasons to question this sample-bias explanation, however. Oberlin community survey data collected by the research team in 2016 using a completely different method (soliciting respondents in public places) and no mention of sustainability observed similar patterns of high pro-environmental concern (unpublished data). This suggests the high levels of pro-environmental attitudes and behaviors observed in this data reflect actual population levels.

To see whether other researchers also observed high levels of pro-environmental attitudes and behavior during this time period, we reviewed research published between 2012 and 2016 that used community samples. Researchers indeed found similar patterns in their samples. For example, in surveys focused on rural areas in Ohio, Kansas, Iowa, and Oregon, indicators for pro-environmental attitudes and behaviors were above the midpoint of the Likert-type questions [35,55,56]. This pattern was not only found in the United States, but also in studies across the world using participants with various socio-economic backgrounds, including the Netherlands, Romania, and England [57,58,59]. Relatively high levels of pro-environmental attitudes and behaviors are thus not unique to Oberlin or Berea.

Further, many behaviors that were reported at relatively high rates seem to reflect actual population uptake, as these numbers are in line with data from this time period from large studies of penetration rates of resource-saving technologies. For example, a 2007 study of Canadian households found that 64% of households had low-flow showerheads, and 42% had low-flow toilets. We were unable to locate comparable numbers for the United States; however, data on the Energy Star program [60] suggests that up to 76% of the market was made up of Energy Star-qualified units on some appliances in 2012. The relatively high percentages of participants who reported installing energy- and water-efficient appliances at Time 1 are roughly in line with these data.

4.2.3. Ceiling Effects

Whether our sample over-represented people concerned about sustainability, or whether the skewed responses accurately reflect population attitudes and behaviors, the high means we observed could have resulted in ceiling effects that prevented us from detecting changes resulting from sustainability initiatives. In other words, participants may have started out so high on our variables of interest that there was not room to observe an increase in environmental concern and behavior.

This is a plausible explanation for many of the null findings among norms, attitudes, and beliefs. For example, the average Time 1 mean ratings for Oberlin and Berea on the item “I consciously make decisions to minimize my electricity use” were 4.48 and 4.38, respectively, on a five-point scale. For the survey item “My community supports efforts to bicycle and walk” the means at Time 1 for Oberlin and Berea were 4.29 and 4.24. These high ratings left little room for an increase in response to community programming.

This issue also exists for many of the behaviors we asked about. For example, at Time 1, Oberlin residents responded an average of 4.57 on a five-point scale to a question about how often they recycle (with 5 = always). In contrast, Berea’s average at Time 1 was 3.76. Both towns made significant changes to their recycling program, but only Berea saw a significant increase in self-reported recycling behavior. A ceiling effect in Oberlin seems a likely explanation for the null result there.

However, there was still room for attitude and behavior changes for a number of variables. Statements such as “There are programs and resources here to help me reduce my household energy consumption and cost” were designed to evaluate the effectiveness of Oberlin’s energy-efficiency program in shifting citizens’ efficacy. In 2012, the mean was 3.84, leaving room for an increase. In terms of behavior, only 37% of respondents in Oberlin had insulated their homes at Time 1, again leaving plenty of room for an increase. The 2016 data revealed small increases on these variables (Cohen’s d = 0.10 and 0.22, respectively), but the effects were not statistically significant. In sum, the ceiling effects were likely a limiting factor for some, but not all, of the variables we hypothesized would change.

4.2.4. Sample Sizes and Statistical Power

The above examples suggest that another major issue with our survey as an assessment tool was the small sample size and resulting lack of statistical power. For programs that had relatively limited uptake, we were most certainly underpowered. For example, only 25 homes in Oberlin participated in the solar co-operative. At only 1% of the Oberlin population, we would expect perhaps only 1 of these 25 households to appear in our sample. In fact, we had four participants, but that is still not enough people to expect to find a statistically significant increase in the installation of solar energy on our survey. For programs with very targeted or limited uptake, a generalized community survey will be unlikely to adequately capture changes in behavior.

However, we still might expect changes on questions about norms or perceptions of community resources among the general population, given that most participants were aware of these programs. These changes did not emerge, perhaps because norms and attitudes are not as easily influenced as one might wish or as common psychological theories suggest. This raises interesting questions about how well results from controlled experiments, which typically ask about behavioral intentions rather than behavior, generalize to the messy and noisy real world and to real behaviors.

Low statistical power is also a potential explanation for the null findings in other areas. The data in Table 15 suggest that three areas resulted in a general positive shift in Oberlin: waste and food attitudes (Cohen’s d of 0.344 and 0.284, respectively), and energy behaviors (Cohen’s d = 0.17). The energy behavior in particular may represent an underpowered but real change. At Time 2, 20% of our Oberlin sample indicated they had taken advantage of the efficiency program. From a community-penetration perspective, this number is quite high and suggests a very successful program. However, from a statistical point of view, it is still a minority of participants. A retrospective power analysis revealed that with a presumed effect size of 0.17 and our sample sizes, our test had only a 24% chance of finding a significant effect, which is clearly inadequate.

It is worth examining the areas with (limited) success. We note that recycling is a relatively main-stream, low-barrier behavior. It is also a public behavior, as neighbors’ bins can be seen along the street. This may explain why we saw significant shifts in norms and attitudes for recycling in Oberlin and Berea (we discussed the ceiling effect on recycling behavior above). Similarly, installing LED bulbs is the simplest energy-efficiency behavior a household can engage in. While the efficiency program did not track the numbers of bulbs given away, it was the most frequent way the organization engaged with the public. Larger participation rates would make it easier to detect an effect via a survey with limited sample size. Food is another area in which the effect sizes suggest a small, underpowered effect. The farmer’s market had a regular social media presence, which may explain the success in shifting attitudes in this area. Overall, our results on behavior change suggest that the difficulty of the behavior is a crucial variable (one that the TPB and VBN theory do not include).

While there is some evidence of changes in attitudes and behaviors in select domains, there is little evidence for the holistic change in attitudes and reported behaviors that the Oberlin Project hoped to achieve. The average effect sizes were quite small and not always in the predicted direction. However, we reiterate that program evaluators documented success via actual participation in programming. We note that self-reported attitudes and norms do not always move in lockstep with actions. These results add to previous work on spillover effects [61], and suggest that a concerted, cohesive attempt to shift a community’s culture on sustainability does not necessarily lead to broad shifts in attitude and behavior.

5. Conclusions

Our goal was to evaluate a systematic community-wide effort to promote sustainability, and to do so using a model simple and inexpensive enough to be replicated in other communities. Our vision was that a holistic approach to assessment would be more efficient and allow us to document hoped-for spillover effects. While a number of concrete changes in behavior were documented in Oberlin over the 4 years of this assessment by program evaluators, these changes were rarely reflected in individual thoughts, feelings, and behaviors as captured by our survey. Overall, we conclude that while the survey items functioned mostly as intended, our objective of developing a scalable, replicable method of community-sustainability assessment was not achieved. Further, we also wish to acknowledge that the predicted patterns we did find (e.g., recycling attitudes) are correlational; we cannot be confident that the changes that occurred were the result of sustainability programming.

Our analysis points to multiple challenges to employing a survey focused on psychological dimensions of sustainability initiatives. Awareness of new programs proved easy to measure, but changes in attitudes and behaviors were documented only in areas where programs had low barriers and high penetration rates (recycling, LED bulbs) or high visibility (farmer’s market). Many of the attitudes and behaviors we measured started out at such high levels that the ceiling effects made it difficult to measure changes. We found it difficult to recruit an adequately sized sample with the limited resources available. The sample we were able to collect in these two small college towns was also too small to yield enough statistical power to document changes that were small or impacted only a subset of the population. Finally, even though we included a control community, our study design is fundamentally a correlational one, which prohibits us from drawing causal conclusions. This is a necessary condition for field studies, but an important limitation to acknowledge.

Based on these observations, we put forward the following recommendations for those seeking to evaluate social and cultural shifts in sustainability.

• For programs with very targeted or limited uptake, a generalized community survey will be unlikely to adequately capture changes. Targeted assessment methods will be more appropriate. On the other hand, for programs that attempt to reach the whole community (or most of it), a community survey such as ours may be effective (if it is adequately powered);
• Estimating statistical power and appropriate sample sizes is essential for success. Our research team, in the absence of better information, assumed moderate effect sizes (Cohen’s d ~ 0.30). This proved to be overly optimistic. Unless more specific information is available, we suggest that researchers adopt relatively conservative estimates of effect sizes in a field study;
• The ceiling effects in pro-environmental attitudes and behavior may be a serious problem. Researchers should consider proactively addressing this issue by using larger response scales (e.g., a 100-point slider scale instead of a 5-point Likert scale) or using a fully labeled unbalanced response scale (e.g., with finer-grained distinctions on the positive end of the scale [58]);
• Our recruitment method should be completely rethought, particularly in the face of decreasing response rates to mail surveys [62]. Creative sample recruitment through multiple channels (e.g., data collection via iPads in diverse public places) may help boost sample size and address response bias. Similarly, it may be advisable to engage with multiple community partners representing a range of subpopulations in each community to encourage participation.

We conclude that creating community-wide cultural changes and assessing those changes are both difficult tasks. While hindsight bias makes it tempting to feel that these conclusions should have been obvious from the start, in fact they were not. We hope this report provides a cautionary learning opportunity for others interested in assessing community-wide sustainability, and look forward to the development of more effective methods. Future research will need to identify methodologies for capturing ephemeral social and psychological variables relevant to sustainable behavior, as well as to evaluate more systematically how—or if—culture shifts occur when communities engage in widescale sustainability programming.