Enhancing Urban Accessibility: Reliability and Validity Assessment of the Stakeholders’ Walkability/Wheelability Audit in Neighbourhoods Tool

Abstract

As Canada’s population ages and disability prevalence increases, understanding the built environment’s impact on mobility and social participation is essential. This study evaluates the measurement properties of the Stakeholders’ Walkability/Wheelability Audit in Neighbourhoods (SWAN) tool, a user-led instrument designed to assess environmental factors affecting older adults and individuals with disabilities. Using community-based participatory research, we recruited 54 participants from five cities to assess the SWAN tool’s inter-rater reliability, construct validity, and internal consistency. The results indicated a high overall inter-rater reliability of 85.22%, with substantial Cohen’s Kappa coefficients across domains, particularly in the Safety domain (0.73). The construct validity was confirmed through moderate to strong correlations with established measures, notably a correlation of 0.79 between the Street Crossing subdomain and the Sidewalk Index. The internal consistency analysis showed excellent reliability in the Functionality domain (α = 0.95) and a lower consistency value in the Social Environment domain (α = 0.63), suggesting the need for further refinement. These findings provide preliminary evidence of the SWAN tool’s potential for evaluating neighbourhood accessibility. By identifying barriers and facilitators to mobility, the SWAN tool can guide urban planning efforts aimed at creating inclusive environments for aging populations and individuals with disabilities. Future research should focus on larger samples to explore structural validity. Ultimately, the SWAN tool can contribute to improving the quality of life of vulnerable populations and promote more equitable urban policy development.

1. Introduction

Canada’s population is aging, and the prevalence of disability is also increasing. In 2010, 14.1% of Canada’s population was aged 65 or older, a proportion that has notably climbed to 19.0% in 2022. Projections from Statistics Canada indicate a continuation of this trend, reaching an estimated 22.5% by 2030 [1]. Likewise, the prevalence of disabilities among Canadians aged 15 and older has significantly increased from 22% in 2017 to 27% in 2022, with a notable correlation between age and disability prevalence [2]. This dual demographic trend signifies the evolving character of Canada’s population, suggesting the necessity for nuanced policies and better health and social services to address the needs of these populations.

The built environment has a profound impact on the mobility of older adults and people with disabilities, which can affect their health and quality of life [3,4,5,6,7]. For instance, individuals who are deaf or hard of hearing feel less safe when navigating the pedestrian environment as they struggle to hear traffic on the road [8]. Persons living with cognitive disabilities often experience challenges with navigation of their own environment, which is heightened when communities do not integrate sufficient green spaces and landmarks (such as large shops, libraries, community centres, and senior centres) to help reduce the stress of wayfinding [9]. Moreover, people using mobility assistive devices may find it challenging to navigate the built environment, especially during rain and snow because of inadequate drainage or snow removal [5]. This aligns with the International Classification of Functioning, Disability, and Health (ICF), which posits that disability and functioning are outcomes that result from the interplay between health conditions (such as diseases, disorders, and injuries) and contextual factors, including the built environment [10].

Environmental variables that can affect the experience of being mobile in a place fall into two broad categories: macroscale variables, consisting of structural features such as street inter-connectivity and land use mix [11,12]; and microscale variables, or details such as aesthetics, sidewalk design, and maintenance [13]. While much research has concentrated on macroscale variables that define walkability [14], investigating microscale features is also valuable for understanding the mobility experience [15,16,17]. Microscale characteristics of the built environment can often be modified at a lower cost and within a shorter timeframe compared to restructuring macroscale designs [18].

A persistent challenge in this field of study revolves around developing valid and reliable micro-level metrics to assess sidewalk and street design, which can impact non-vehicular travel behaviour [19,20]. Assessment instruments, including audit tools, must demonstrate both reliability and validity for study results to be credible. Reliability refers to whether an assessment instrument yields consistent results each time it is used in the same setting with the same type of subjects, essentially indicating dependable outcomes [12,21]. Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores or other modes of assessment [22]. Thus, an instrument is considered valid when its construction and applicability allow it to accurately measure its target [23].

Various audit tools have been developed and tested to evaluate the microscale qualities of the built environment, particularly at the street level, through on-site visits [24]. The Stakeholders’ Walkability/Wheelability Audit in Neighbourhoods (SWAN) is a microscale user-led audit tool designed to evaluate both objective and subjective aspects of the built environment that affect the lives of older adults and individuals using mobility assistive devices, people who are deaf or hard of hearing, and people living with mild cognitive impairment, including dementia [25]. Various audit tools have been developed and tested to evaluate the microscale qualities of the built environment, particularly at the street level, through on-site visits [24]. The Stakeholders’ Walkability/Wheelability Audit in Neighbourhoods (SWAN) is a microscale user-led audit tool designed to evaluate both objective and subjective aspects of the built environment. Compared to other audit tools, the SWAN tool captures features that affect the mobility and social participation of older adults and individuals using mobility assistive devices [25]. The SWAN tool was later updated to capture other streams of disability, including mild cognitive impairment and sensory (hearing and vision) impairment [26].

Given the critical need for tools to assess the built environment’s impact on the mobility and social participation of older adults and people living with disability, this paper aims to address this gap by investigating the measurement properties of the SWAN tool. In this paper, the results for the inter-rater reliability (IRR), construct validity, and internal consistency of the SWAN tool used by individuals using mobility assistive devices, people who are deaf or hard of hearing, and people living with mild cognitive impairment, including dementia, are reported.

This paper is structured as follows: In Section 2, we describe the materials and methods used in this study, including participant recruitment, data collection tools, and the evaluation processes. Section 3 presents the results, including the IRR, construct validity, and internal consistency of the SWAN tool. In Section 4, we discuss the implications of the findings, limitations, and potential directions for future research. Finally, Section 5 concludes the paper, summarizing the key findings and their relevance for enhancing mobility and social participation in built environments for individuals with disabilities.

2. Materials and Methods

2.1. Research Design

2.1.1. Recruitment

This study was based on community-based participatory research (CBPR), in which participants were considered as co-researchers. The CBPR promotes self-advocacy among participants and facilitates policy changes that are reflective of community needs [27]. Fifty-four participants were recruited in 5 cities including Vancouver, North Vancouver, Burnaby, Richmond, and Surrey. For recruitment, the SWAN team established contact with community centres, relevant organizations, and healthcare networks to identify potential participants. In these communications, they provided a brief overview of the study and its objectives. If an organization expressed interest, research assistants proceeded to distribute flyers and put up posters in these locations. Individuals who were interested then connected with the team through email or phone to assess eligibility criteria. To ensure a streamlined participant onboarding process, we centralised the procedure by appointing a researcher as the Central Administrator (CA). When an interested person contacted the Central Administrator, the CA arranged a suitable date and time for discussing eligibility and participation. The person’s eligibility was determined by answering the following questions:

• Do you have a mobility/hearing/cognitive disability?

  ○

  If so, are you 19 years of age or older?

  ○

  If not, are you 60 years or older?

• Can you independently walk or wheel for four city blocks over a period of up to three to four hours?
• Do you reside in the city of Vancouver, Burnaby, North Vancouver, Richmond, or Surrey?
• Are you capable of providing informed consent?

2.1.2. Locations for Data Collection

Once the eligibility was confirmed, the date and location of data collection were finalised based on the participant’s preference. Locations for collecting data were identified meaningfully through undertaking a stepwise approach. In the initial phase of the study, the research team employed Geographic Information System (GIS) layers to identify problematic intersections and their adjacent areas within the Lower Mainland. These GIS layers included data on transportation hubs (locations with public transportation like SkyTrain stations or bus stops), number of collisions, presence of commercial buildings, and other relevant features. The primary outcome of this step was the creation of a map showing up to ten hotspots (intersections) within each of the five municipalities. In the next stage, an interactive forum was arranged, bringing together individuals with disabilities, professionals, community partners from senior centres, and municipal officials. The purpose of this forum was to gain a comprehensive understanding of the difficulties encountered at the selected intersections and their adjacent areas. After the forum discussions and consultations with stakeholders, the decision was made to focus data collection efforts on four intersections in each city.

On the day of data collection, two research assistants accompanied each participant. The lead research assistant provided support as needed; if the participant preferred, the lead RA would ask questions and assist them to complete the form. If the participant preferred to fill out the form independently, the lead RA was available to provide clarifications on any questions. Meanwhile, the secondary research assistant completed a separate audit form to assess the reliability of the tool. The secondary RA also took photographs, if requested by the participant, and recorded any additional comments the participant provided.

2.2. Data Collection

Data were collected using various tools in this study to assess different aspects of the built environment. First, a demographic form was used to gather participants’ demographic information. After completing the SWAN tool, three complementary tools were utilised to provide a more comprehensive assessment of the environment: the Home and Community Environment (HACE) Instrument, the Sidewalk Index (SI), and Walk Score. In the following sections, details on each of these measurements are provided.

2.2.1. Demographic Form

Demographic data were collected on age, gender identity, marital status, employment status, living situation, education level, and income. Participants were also asked to indicate their housing type (single or multi-unit housing), location of living (city or suburbs), and familiarity with the neighbourhood.

2.2.2. Stakeholders’ Walkability/Wheelability Audit in Neighbourhoods Tool

The SWAN is a microscale user-led audit tool that allows users to assess the facilitators of and barriers to mobility, access, and social participation in their neighbourhoods. The SWAN tool is an adaptation of the SWEAT-R tool that captures the perspective of older adults on neighbourhood accessibility [28]. Moreover, the development of the SWAN tool included a comprehensive literature review and incorporated aspects of other user-led tools, such as the Microscale Audit of Pedestrian Streetscapes (MAPS) [29,30], the Built Environment and Active Transport (B.E.A.T.) Neighbourhood Assessment [31], and Jane’s Walkability Checklist [32]. The tool asks users to assess the presence of specific design features in their environment, such as: “Is there a crosswalk?” or “Is there a pedestrian signal?”.

Previous research using the SWAN tool was primarily conducted with older adults and people using mobility assistive devices. To encompass a broader spectrum of disability experiences, the SWAN tool was updated using a Community-Based Participatory Research (CBPR) approach, allowing for the inclusion of individuals who are deaf or hard of hearing and those with mild cognitive disabilities, such as those in the early stages of dementia, to systematically assess their neighbourhoods. The new version of SWAN tool can be used for people with mobility, hearing, or cognitive disabilities. The CBPR approach involves collaboration between researchers and community members as partners throughout the research process [1]. Individuals with disabilities and other stakeholders provide valuable input in concept validation, tool adaptation, and pilot testing through the establishment of a committee (SWAN Development Committee) comprising individuals with lived and/or professional experience.

Within the SWAN tool, the assessment of walkability and wheelability extends beyond the physical infrastructure of streets and sidewalks. It encompasses a wide range of neighbourhood components that are strategically chosen to enhance the social participations of older adults and individuals living with disability.

The SWAN tool employs a structured framework that consists of 185 items spread across five distinct domains including the following: 1. functionality of (a) street crossings and (b) sidewalks; 2. safety of (a) street features and traffic and (b) personal safety of pedestrians; 3. appearance and maintenance; 4. land use and supportive features; and 5. social aspects. The significance of these domains has been verified in focus groups with people with lived experience [25,28]. To facilitate data collection, the SWAN tool utilises a presence/absence checklist to objectively record the presence or absence of specific environmental elements in a paper format.

Additionally, it employs a 5-point Likert scale for each domain and subdomain to gauge the subjective score for each domain and subdomain. This rating scale ranges from “poor” (scored as “1”) to “excellent” (scored as “5”). This allows one to comprehensively assess the neighbourhood environment from both objective and subjective perspectives.

2.2.3. Home and Community Environment Instrument

The HACE tool assesses various factors within an individual’s home and community environment. The initial HACE prototype covered five domains, including the following: Home Mobility; Community Mobility; Basic Mobility Devices; and Communication Devices, Transportation, and Attitudes. It has adequate internal consistency (α = 0.70–0.85 across subscales) and construct validity through correlations with functional outcomes in rehabilitation populations [33]. Notably, five questions from the Community Mobility domain were integrated into the data collection.

2.2.4. Sidewalk Index

The SI evaluates the performance of sidewalks and public spaces based on the needs of wheelchair users, considering comfort and safety variables. This is an efficient and easily applicable tool to identify the current infrastructural conditions of sidewalks and street crossings, as well as their design characteristics [34]. During data collection, the SI focused on evenness, maintenance, width, and surface quality, with no inquiries about the suitability of pedestrian crossings.

2.2.5. Walk Score

Walk Score measures the walkability of neighbourhoods. Analysing walking routes to nearby amenities, Walk Score points are awarded based on distance, with a decay function for more distant amenities. The scale categorises walkability into four levels based on scores, ranging from Very Walkable to Car-Dependent. A score of 70–89 indicates that most errands can be accomplished on foot, classifying the area as Very Walkable. A score of 50–69 suggests that some errands can be completed on foot, making it Somewhat Walkable. Scores between 25 and 49 mean that most errands require a car, indicating a Car-Dependent area. Finally, scores ranging from 0 to 24 signify that almost all errands necessitate a car, also classifying the area as Car-Dependent.

2.3. Data Analysis

Microsoft Excel software was used to enter and organise all data collected by SWAN and other tools. The process of coding/scoring responses based on the codebook and calculating scores for each domain were performed in the same environment (Microsoft Excel). Further analyses for IRR, tool validity, and internal consistency were performed using R programming language in RStudio (version 2023.9.1.494).

Measurement properties were assessed using the COSMIN taxonomy [35]. The reliability of the SWAN was determined by calculating inter-rater reliability (IRR) using the paired observer method. The IRR compares the objective scores of the older adults or people with disabilities and the secondary research assistant. This is calculated both in terms of the percentage agreement and Cohen’s kappa to compare the results [36]. Cohen’s kappa result was interpreted as follows: values ≤ 0 indicate no agreement, 0.01–0.20 indicate no to slight agreement, 0.21–0.40 indicate fair agreement, 0.41–0.60 indicate moderate agreement, 0.61–0.80 indicate substantial agreement, and 0.81–1.00 indicate almost perfect agreement [37].

• The study assessed construct validity through hypotheses testing, using Pearson correlations to examine relationships between SWAN tool domains and related constructs in the Home and Community Environment (HACE) Instrument, Sidewalk Index (SI), and Walk Score. The following hypotheses were tested and we expected moderate positive correlations between SWAN domains and corresponding questions in complementary tools:
• Sidewalk functionality in SWAN tool versus question on sidewalk evenness in SI;
• Appearance and maintenance in SWAN tool versus question on surface condition in SI;
• Street crossing functionality in SWAN tool versus question on intersection design in SI;
• Pedestrian safety in SWAN tool versus question on sidewalk evenness in HACE;
• Traffic safety in SWAN tool versus question on safety in HACE;
• Social domain in SWAN tool versus question on places to rest in HACE;
• Land use and supportive features in SWAN tool versus Walk Score.

To assess the internal consistency of the SWAN tool, Cronbach’s alpha is calculated. This is a statistical measure widely used in the field of psychometrics and social sciences to assess the internal consistency reliability of a set of items or questions within a questionnaire, test, or scale. It plays a crucial role in determining the extent to which these items measure a common underlying construct or dimension consistently. In essence, Cronbach’s alpha helps researchers and practitioners evaluate the reliability of a measurement instrument. Raw Cronbach’s alpha accounts for differences in item variances and is suitable for the SWAN tool’s heterogeneous domains. As a general guideline for interpreting Cronbach’s alpha, the following scale applies: values of 0.9 or higher are considered excellent; 0.8 to 0.9 are adequate; 0.7 to 0.8 are marginal; 0.6 to 0.7 are questionable; and values below 0.6 are deemed unacceptable [38]. In this analysis, missing data are addressed using pairwise deletion, allowing for the inclusion of all available cases for each item in the calculation of Cronbach’s alpha. This approach maximises the use of available data while being mindful of the potential biases introduced by missingness.

3. Results

The demographic data of the study participants are outlined in

Table 1. Sample demographic data.

Stream	No. of Participants	Age (Mean)	Gender Distribution (%)
People using mobility assistive devices	24	51	Female (42%)
People who are deaf or hard of hearing	10	41	Female (40%)
People living with cognitive impairment	10	70	Female (30%)
Older adults aged 60+	10	65	Female (80%)

. In total, data were collected for the four groups shown below.

3.1. Inter-Rater Reliability

The IRR assessment for the SWAN tool was conducted between the secondary RA (research assistant) and the individuals with disabilities across various domains. The findings, as shown in

Table 2. The inter-rater reliability (IRR) result.

Domain/Subdomain	% Agreement	K
1. Functionality	84.58	0.69
1-1. Crossing Functionality	84.58	0.69
1-1-1. Crossing C Functionality	84.58	0.69
1-1-2. Crossing D Functionality	83.48	0.67
1-2. Sidewalk Functionality	70.04	0.47
2. Safety	90.97	0.73
2-1. Traffic Safety	90.97	0.73
2-2. Pedestrian Safety	79.57	0.59
3. Land Use and Supportive Features	81.38	0.61
4. Appearance and Maintenance	79.79	0.66
5. Social Aspects	80.64	0.64
Total	85.22	0.67

, indicate a high level of agreement, with an overall percentage agreement of 85.22%. The Safety domain exhibits the highest agreement at 90.97%. Moderate agreement is observed in the Sidewalk Functionality and Pedestrian Safety domains, with percentages at 70.04% and 79.57%, respectively. The SWAN tool demonstrates its reliability in assessing aspects such as Land Use and Supportive Features, Appearance and Maintenance, and Social Aspects, with substantial agreement ranging from 79.79% to 81.38%. The Cohen’s Kappa coefficient ranged from 0.47 (Sidewalk Functionality) to 0.73 (Traffic Safety) across the different domains of the SWAN tool, demonstrating substantial agreement.

3.2. Construct Validity, Hypotheses, and Testing

As shown in

Table 3. Construct validity, hypotheses, and testing results.

SWAN Domain	Comparator Measure	Correlation (r) (95% CI)	Interpretation
Sidewalk Functionality	SI (Sidewalk Evenness)	0.44 (0.27, 0.58)	Moderate
Street Crossing Functionality	SI (Intersection Design)	0.79 (0.71, 0.85)	Strong
Appearance and Maintenance	SI (Surface Condition)	0.12 (−0.07, 0.30)	Weak
Pedestrian Safety	HACE (Unevenness)	0.31 (0.04, 0.53)	Moderate
Traffic Safety	HACE (Safety)	0.29 (0.02, 0.52)	Moderate
Social Aspects	HACE (Places to Rest)	0.30 (0.03, 0.53)	Moderate
Land Use and Supportive Features	Walk Score	0.28 (0.01, 0.51)	Moderate

, the Street Crossing subdomain exhibits the highest correlation with questions in the SI tool regarding intersection characteristics (0.79). The Sidewalk subdomain also shows moderate correlations (0.44) with questions in the SI tool regarding sidewalk evenness. Traffic Safety demonstrates a moderate correlation (0.29) with question regarding safe sidewalks in the HACE tool, and Pedestrian Safety is similarly correlated with questions about sidewalks in the HACE tool. The Land Use domain moderately correlates (0.28) with the Walk Score. The Social Aspects domain exhibits a moderate correlation with questions in the HACE tool concerning places to rest. Among all the correlations, the lowest is observed in the Appearance and Maintenance domain, which is weakly correlated (0.12) with questions about surface conditions in the SI tool.

3.3. Internal Consistency

As shown in

Table 4. Internal consistency results.

Domain	Raw Alpha
1. Functionality	0.87
1.1. Street Crossing Functionality	0.87
1.2. Sidewalk Functionality	0.80
2. Safety	0.74
2.1. Traffic Safety	0.81
2.2. Pedestrian Safety	0.54
3. Land Use and Supportive Features	0.69
4. Appearance and Maintenance	0.55
5. Social Aspects	0.45

, the internal consistency of the SWAN tool was assessed for each domain using the raw Cronbach’s alpha, showing varied reliability across the domains. The Functionality domain demonstrated strong internal consistency with a raw alpha of 0.87, indicating a reliable measurement of its underlying construct. The Safety domain also showed acceptable consistency with a raw alpha of 0.74, effectively capturing safety-related factors. The Land Use domain had a raw alpha of 0.69, suggesting moderate reliability. The Appearance domain exhibited lower consistency with a raw alpha of 0.55, indicating potential variability among items. The Social Aspects domain had the lowest reliability, with a raw alpha of 0.45, suggesting that its items may not cohesively measure a single construct and may need refinement.

4. Discussion

As the population continues to age and the prevalence of disability rises, mobility and social participation for these groups remain critical challenges. This study evaluated the reliability and validity of the SWAN tool, which aims to assess the impact of the built environment on the mobility and social participation of older adults and individuals with disabilities.

The SWAN tool shows a high level of IRR, with an overall agreement of 85.22% and moderate to substantial Cohen’s kappa coefficients across different domains, highlighting the tool’s reliability in this sample. Additionally, the study provides evidence of the construct validity through the hypothesis testing of the SWAN tool in terms of the correlations between its domains and well-established measures from complementary tools, such as the HACE, SI, and Walk Score. The significant correlations that are observed, particularly in areas like Street Crossings, Sidewalk Functionality, and Pedestrian Safety, confirm the SWAN tool’s ability to accurately assess neighbourhood accessibility. While the internal consistency of the SWAN tool is generally good, the Social Aspects domain may be improved (e.g., by removing or modifying existing items in that domain).

This study also has limitations that should be addressed in future research. First, while the study included participants from multiple cities in the Lower Mainland of British Columbia, the results may not be fully generalizable to other regions of Canada with different urban landscapes or infrastructure. Additionally, as this is emerging evidence on the SWAN tool’s reliability, we recommend future studies with larger sample sizes that involve a factor analysis to provide psychometrically robust results.

5. Conclusions

This study suggests that the SWAN tool is a valid and reliable measure for assessing the built environment’s impact on the mobility and social participation of older adults and individuals with disabilities. With a strong level of IRR and meaningful correlations with established measures, the SWAN tool can help identify barriers to and facilitators of neighbourhood accessibility. While the Social Aspects domain requires some adjustments to improve internal consistency, the other domains demonstrate good internal consistency.

The inclusion of individuals with mobility, hearing, and cognitive disabilities in this study broadens the scope of the SWAN tool, enabling a more comprehensive assessment of the built environment’s impact. By incorporating diverse disability experiences, this research emphasises the need for an inclusive design that caters to a range of mobility challenges, from physical barriers like uneven sidewalks to sensory limitations such as difficulty hearing traffic signals. The CBPR approach employed in this study further enhances the validity of the tool by ensuring that it reflects the real-world experiences of people with disabilities, thereby promoting the design of more inclusive environments.

This study has several limitations that should be addressed in future research. First, though the participants were recruited from multiple cities in the Lower Mainland of British Columbia, the findings may not be fully generalizable to other regions of Canada with different urban landscapes or infrastructure. Additionally, as this is emerging evidence on the reliability of the SWAN tool, we recommend future studies involve larger sample sizes and incorporate a factor analysis to provide more robust psychometric results.

Another limitation is the relatively small number of participants, underscoring the need for future research with a larger and more diverse sample. Given demographic trends in Canada, such as its aging population and the increasing prevalence of disabilities, the SWAN tool holds potential as a valuable resource for promoting more inclusive and liveable communities. By addressing environmental barriers, the tool could contribute to improving the quality of life of vulnerable populations and support more equitable urban planning and policy making.