A Cluster-Randomised Stepped-Wedge Impact Evaluation of a Pragmatic Implementation Process for Improving the Cultural Responsiveness of Non-Aboriginal Alcohol and Other Drug Treatment Services: A Pilot Study

There is limited evidence regarding implementing organisational improvements in the cultural responsiveness of non-Aboriginal services. Using a pragmatic implementation process to promote organisational change around cultural responsiveness, we aimed to (i) identify its impact on the cultural responsiveness of participating services; (ii) identify areas with the most improvement; and (iii) present a program logic to guide cultural responsiveness. A best-evidence guideline for culturally responsive service delivery in non-Aboriginal Alcohol and other Drug (AoD) treatment services was co-designed. Services were grouped geographically and randomised to start dates using a stepped wedge design, then baseline audits were completed (operationalization of the guideline). After receiving feedback, the services attended guideline implementation workshops and selected three key action areas; they then completed follow-up audits. A two-sample Wilcoxon rank-sum (Mann–Whitney) test was used to analyse differences between baseline and follow-up audits on three key action areas and all other action areas. Improvements occurred across guideline themes, with significant increases between median baseline and follow-up audit scores on three key action areas (median increase = 2.0; Interquartile Range (IQR) = 1.0–3.0) and all other action areas (median increase = 7.5; IQR = 5.0–11.0). All services completing the implementation process had increased audit scores, reflecting improved cultural responsiveness. The implementation process appeared to be feasible for improving culturally responsive practice in AoD services and may be applicable elsewhere.


Introduction
There are substantial inequalities in health status and health care access between Aboriginal and Torres Strait Islander people (hereafter referred to as Aboriginal) and non-Aboriginal people in Australia [1], including disproportionate drug and alcoholrelated morbidity and mortality [2]. Although all health services should be culturally safe, effective and welcoming to Australians from any cultural backgrounds, there is evidence that Aboriginal people receive less benefit from non-Aboriginal health services than non-Aboriginal people [3]. Ensuring that mainstream health services (that is, services that are not specifically developed for Aboriginal people) are responsive to Aboriginal peoples' needs is a key strategy to reduce inequalities in healthcare access and enhance the quality of care provided to Aboriginal people [4][5][6]. Cultural responsiveness is an ongoing process of adapting systems, services and practice to fit with culturally diverse user preferences [7], and providing high-quality care that is culturally appropriate and safe [8]. While the Int. J. Environ. Res. Public Health 2023, 20, 4223 2 of 15 importance of culturally responsive health services is well acknowledged [8], there is a lack of consensus on effective methods to develop health services that are culturally responsive [6].
Cultural responsiveness initiatives have been shown to improve healthcare worker cultural knowledge, awareness and sensitivity [9][10][11][12], improve patient satisfaction with providers [9,12,13] and increase access and frequency of visits by Aboriginal people [14]. However, the quality of many existing studies is low, frequently using observational study designs and interventions that provide one-off staff training, but which tend to be ineffective if not implemented as part of a systematic approach [6,15].
While many non-Aboriginal clinicians are individually committed to practising in a culturally responsive way, improving cultural responsiveness needs to be a whole-ofservice activity that involves multiple strategies across all levels of the workforce and organisational policy, management and practices to be effective [5,6,[16][17][18]. There is limited evidence, particularly in the Australian context, regarding effective systematic methods for implementing organisational-level change to improve cultural responsiveness [6]. Aiming to provide a structured method to implement best-evidence cultural responsiveness practices, the current project developed a pragmatic implementation process for facilitating organisational change in services. The first step of this process involved combining a number of recommended cultural responsiveness strategies [19] into a best-evidence guideline for improving the cultural responsiveness of non-Aboriginal AoD services [20]. The co-designed best-evidence guideline details a wide range of evidence-based strategies including: engaging management [21,22]; enhancing communication and relationships between mainstream and Aboriginal services and communities [3,23,24]; improving staff knowledge of the social and historical determinants of health [25]; and tailoring programs to suit the local community [26].
The core components, or themes, of the guideline were then operationalised into flexible activities that could be tailored to suit each service [27][28][29], and implemented in non-government organisation (NGO) non-Aboriginal AoD services. The implementation fidelity, barriers and facilitators to implementation, and their acceptability and feasibility, are described elsewhere [27,30]. The current study aims to identify the impact of the implementation process on the cultural responsiveness of participating services, as measured by the mean change in audit scores from baseline to three-month follow-up. Secondary aims were to identify the areas of the guideline that were most frequently selected as priority areas for change and most successfully actioned by services during the project. We also aimed to build on the services' insights to develop a program logic to identify how the standardised core components were flexibly applied by services to support future implementation.

Study Design
The project was co-designed and implemented using a community-based participatory research approach [29,31] that facilitated iterative development of the best-evidence guideline and the pragmatic implementation process through collaboration between the project team (RW and JA, who have experience working in NSW AoD services), the researchers (SF, AH, AS), the Network of Alcohol and other Drugs Agencies (NADA; the peak organisation for the NGO AoD sector in NSW), the Primary Health Networks (PHNs) as the project funders and an Aboriginal Advisory Group (which included Aboriginal community members with professional and community connections to NGO AoD treatment services or government treatment services). The project was overseen by the Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project. Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project. n = 17 services invited to participate n = 15 services consented to participate (88%) n = 2 services did not provide consent n = 14 services completed baseline audit (93%) n = 1 service did not complete due to limited staff/time available n = 13 services had staff attend workshop and completed action plan (87%) n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. n = 15 services consented to participate (88%) Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project. n = 17 services invited to participate n = 15 services consented to participate (88%) n = 2 services did not provide consent n = 14 services completed baseline audit (93%) n = 1 service did not complete due to limited staff/time available n = 13 services had staff attend workshop and completed action plan (87%) n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.  n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. n = 1 service did not complete due to limited staff/time available Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.  n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.  n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. n = 1 service did not complete due to staff turnover and non-response Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.  n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. Aboriginal Advisory Group to ensure the priorities and world views of Aboriginal experts were centralised into the guideline and the project implementation. Members of the Group were offered reimbursement for expenses arising from their involvement. Project implementation expenses were covered by the project. The impact of the project on the cultural responsiveness of participating services was evaluated using a cluster-randomised stepped-wedge design with 12 services and six clusters.

Participating Services
Seventeen non-Aboriginal NGO AoD treatment services from six PHN districts in New South Wales (NSW) were identified by the PHNs as being potentially willing to participate, with fifteen providing formal consent to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types, including residential rehabilitation (n = 3), day programmes (n = 2), centre-based counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2). Twelve services completed all project activities (80%; Table 1). No data related to Aboriginal clients or organisations were accessed or used in this phase of the project.  n = 1 service did not complete due to staff turnover and non-response n = 12 services completed 3-month audit (80%) n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium. n = 1 service did not complete due to limited staff/time available

Cultural Responsiveness Project
The project was delivered in these sequential phases: (i) engage stakeholders, develop co-design structures and secure approvals from ethics and participating sites; (ii) co-design the implementation process and best-evidence guideline; (iii) implement the guideline and monitor uptake. Phase 1 and the process evaluation outcomes are described in detail elsewhere [27] and the co-design, implementation and monitoring steps are described below (phases 2 and 3). Aboriginal author RW was involved in all aspects of the project and was provided training in research methods, manuscript development and presenting research findings. Non-Aboriginal members of the research team (JA, AH, SF, AS) have extensive experience of working with Aboriginal communities over multiple projects and have completed training in cultural responsiveness. RW provided cultural mentoring to non-Aboriginal researchers. Findings from the project were presented to participating services and local Aboriginal peak bodies via ongoing discussions about the project and at formal events, such as the Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) Symposium.

Co-Designed Best-Evidence Guideline for Cultural Responsiveness in Non-Aboriginal AoD Services (Phase 2)
A best-evidence guideline that describes key elements of culturally responsive service delivery in non-Aboriginal AoD treatment services was co-designed at the beginning of the project and this process is described fully in the guideline document (20) (See Supplementary File S1 or also published online at https://www.nada.org.au/resources/alcohol-andother-drugs-treatment-guidelines-for-working-with-aboriginal-and-torres-strait-islanderpeople-in-a-non-aboriginal-setting/ (accessed on 1 June 2022)). Briefly, the guideline codesign process was facilitated by an Aboriginal project team member (RW) and overseen by the Aboriginal Project Advisory Group (27). The guideline identifies six themes: (1) Creating a welcoming environment, (2) Service delivery, (3) Engagement with Aboriginal organisations and workers, (4) Voice of the community, (5) Capable staff, and (6) Organisation's responsibilities.

Clustering of Participating Services and Randomisation to a Starting Date
Services were clustered based on PHN district/geographical region (n = 6). Each cluster of services was randomised to an implementation starting date between June and October 2019, with approximately one month between clusters, as shown in Table 2. Cluster randomisation was conducted by a statistician independent of the project using random number generation. Owing to varying numbers of services within regions, and attrition, clusters included different numbers of services; cluster 1 included one service, cluster 5 included three services and the remaining clusters included two services each. The following implementation and monitoring steps were completed with each participating service.

Baseline Audits of Participating Services
Services were advised of their allocated start date and structured baseline audits of current culturally responsive practice, using a standardised audit tool, were completed individually with each participating service. The audit process identified the extent to which services addressed the guideline, rating cultural responsiveness according to 21 actions areas which corresponded with the six guideline themes. Audit tools were developed which framed the 21 action areas as questions in order to collect information from staff at participating services. Audits were conducted by two trained auditors (RW, JA or another trained auditor) in the setting where the service is delivered and took between 90 min to two hours to complete. Auditors were independent of the service being audited and at least one auditor at each audit was Aboriginal.

Audit Feedback to Participating Services
Individualised written feedback from the audit findings was provided to each participating service, listing all guideline action areas with a descriptive assessment for each area reflecting the level of evidence observed during the audit (limited, some, good or excellent) and recommendations for areas where potential improvements could be made.

Guideline Implementation Workshops with Participating Services
Implementation workshops were held with key staff from services (CEOs/managers and direct service delivery staff) to explain the guideline, review the audit feedback, set goals for improvement and develop a detailed action plan tailored to their service (to operationalise action areas from the guideline themes). Workshops were facilitated by JA and RW. Staff identified and prioritised specific activities that they would implement from the 21 action areas and were encouraged to select three key action areas for their service to progress over the next three months. For example, activities that operationalise guideline Theme 1: Creating a welcoming environment, might include processes to ensure that all clients are welcomed respectfully at first contact with the service, providing tea/coffee/water in the waiting room, accommodating children or other family members in the service, or displaying local Aboriginal artwork. These self-designed activities provide flexibility in how individual services operationalised and implemented the core components, enabling the practice change activities to be tailored to the needs and resources of individual services and the communities they serve [1-5].

Follow-Up Audits of Participating Services
Follow-up audits of services were conducted after three months to assess change in culturally responsive practices in the 21 action areas, following the same procedure as for the baseline audits. Where possible, the same service staff attended the follow-up audit. Services were provided with a second individualised feedback report, including discussion of any changes that had occurred.

Measures
To privilege Aboriginal values and views throughout analysis and reporting, we used the guideline themes that were developed by the Aboriginal Advisory Group to assess culturally responsive practices. The study aimed to identify the impact of the project on the cultural responsiveness of services using the following outcomes: • Change in audit score from baseline to follow-up audit on the three key action areas identified by staff at the implementation workshops (possible score 0-9). • Change in audit score from baseline to follow-up audit in all other action areas from the guideline (other than the three key action areas selected by each service (possible score 0-54).

Statistical Analysis
The audit responses provided by staff were recorded into the audit tool. After each audit was completed, ratings of 0-3 were allocated to each of the 21 audit criteria, according to pre-specified rating rules, by one of the researchers conducting the audit (RW). A second researcher (SF) then independently reviewed the audit tool and rated the 21 criteria. The two sets of ratings were compared and any disagreement around ratings were resolved by discussion, until a consensus was reached. A two-sample Wilcoxon rank-sum (Mann-Whitney) test was used to analyse the difference in audit scores between baseline and follow-up audits, on the three key action areas (outcome 1) and all other action areas (outcome 2). All analyses were conducted using Stata 16 [32].
The extent of change across the six guideline themes was identified by summing item ratings within each theme and calculating the rates of change for each theme. The frequency with which each individual action area was selected by service staff (during workshops to operationalise their improvement goals), and whether improvements were subsequently observed in those action areas, were descriptively explored.

Development of a Program Logic
We used a program logic structure developed in previous work [5,7] to build a standardised logic model specifically for improving cultural responsiveness in non-Aboriginal NGO AoD services. The program logic model was developed by reviewing the audit findings and activities chosen by staff during the workshop and linking these to the core components (themes) of the guideline.

Implementation Process
Twelve of the fifteen participating services completed all service-specific project components. Some delays in completing the three-month follow-up audits occurred, with an average time between audits of 18 weeks (range 14-28 weeks) (See Table 2). The longest delays in completing the follow-up audits were for services "J", "D" and "F", with the audits completed at 19, 24 and 28 weeks, respectively. Service "B" was part of cluster 5; however, due to delays in completing the baseline audit, service "B" ultimately completed the project components in line with cluster 6. Further detail on implementation and process outcomes are reported elsewhere [27].

Change in Cultural Responsiveness of Services in Three Key Action Areas
Outcomes are reported for services that completed baseline and follow-up audits (n = 12). Ten of 12 services increased their audit score on their three key action areas at followup. The median follow-up scores were statistically significantly higher than the median baseline scores (median change = 2.0, IQR = 1.0-3.0, z = −2.79, p < 0.005) ( Table 3).

Change in Cultural Responsiveness of Services in All Other Action Areas
All 12 services showed an increase in score on all other action areas (excluding the three key action areas). The median follow-up scores were statistically significantly higher than the median baseline scores (median change = 7.5, IQR = 5.0-11.0, z = −1.97, p < 0.05) ( Table 3).

Action Areas Most Frequently Selected (by Staff) and Most Frequently Improved
Service staff chose a wide variety of action areas from the guideline to prioritise; 16 of the 21 areas were selected at least once. Those most frequently selected as key action areas were: 1B: The physical environment is welcoming to Aboriginal people (n = 6 services); 3Ai: Aboriginal community engagement to develop relationships (n = 4); 3Aiii: Local history and protocols are reflected in practice and/or policy (n = 5); and 4A: Developing connections with Aboriginal organisations and workers (n = 5) (see the guidelines in Supplementary File S1 for further description). The action areas that services most frequently improved on were in Theme 2 (2B: Immediate triage options are available for Aboriginal people (n = 8 services) and 2C: Staff are culturally responsive in therapeutic practice (n = 7)), Theme 3 (3B: Local Aboriginal protocols are reflected in practice and/or policy (n = 7)) and Theme 6 (6Aii: There are Aboriginal-identified positions and Aboriginal publications and networks are used to advertise jobs (n = 7), 6Aiii: Service induction includes materials about working with Aboriginal people and materials are developed/reviewed by a local Aboriginal person (n = 8)).

Development of a Program Logic
To facilitate future implementation of improvements to cultural competence, the researchers (AH, AS, SF) developed a program logic that is directly tied to guidelines, shown in Table 4. This program logic was developed post-implementation to clearly delineate the standardised core components (guideline themes), flexible components (service level activities) approach and likely mechanisms of change, for future iterations of this project, based on previous work by the authors [28,29]. The second column lists the six bestevidence themes/principles that comprise the core components of the guideline. These are standardised across all services, as are the aims/goals/target areas for improvement (first column) and the articulation of why these core components would impact on cultural responsiveness (third column). The fourth column provides examples of specific activities that services can implement, with flexibility to choose practice change activities tailored to the needs and resources of individual services [28,31,[33][34][35]. The remaining columns identify the measures of processes (the extent to which services engaged in the intervention process), outcomes (the extent to which indicators of culturally responsive practice improved) and data sources. Target problem Some Aboriginal Alcohol and other Drug (AoD) clients will access non-Aboriginal AoD services. There is a lack of guidance for non-Aboriginal services around processes involved with culturally competent service delivery. Goals 1.
Use a co-designed best-evidence guideline for culturally responsiveness practice to assist services in improving cultural responsiveness.

2.
Services will be able to achieve changes (improvements) to cultural responsiveness through a range of one-off or ongoing activities that relate to Guideline Themes Referrals and assessments C.
Direct practice A-Providing flexible service delivery options may increase visibility and access to a service. Providing services via outreach can create opportunities for access by people who might otherwise not attend. B-Seeing the same staff member each time can establish trust and rapport. Yarning to someone before requiring them to fill out forms or answer personal questions helps build rapport. Local history and protocols A-When a service is involved in the community it may help to build relationships, trust and awareness of the service. By building relationships within the community, the service may become more known from people talking or 'vouching' for it within community. B-Having knowledge of local history, culture and protocols is critical to understanding the local Aboriginal community. Recognition of local language groups demonstrates respect.   Aboriginal staff support C.
Training provided D.
Organisation wide practices E.
Policies and procedures

Discussion
The current study used a community-based, participatory research approach to develop a best-evidence, service-level practice change process, supported by a program-logic framework previously developed by the authors [28,29]. The pragmatic implementation approach is supported by existing evidence [33][34][35] and means that individual services could implement areas of the guideline that were most relevant to their local context and current level of cultural responsiveness. All participating services increased their overall audit scores and most increased scores on their chosen priority action areas, reflecting an increase in compliance with the guidelines and improved cultural responsiveness. The results are consistent with previous research demonstrating that audits and practice improvement interventions can be effective methods of identifying where improvements are needed, engaging with workers, and improving culturally responsive practices in a variety of health settings [14,[36][37][38]. Our results support the effectiveness of the guideline and implementation process as a meaningful way of identifying and operationalising best-evidence principles of cultural responsiveness and enabling staff to understand and enact components of the guidelines that were relevant to their service. The program-logic model links the best-evidence core components of the guideline to the flexible service level activities, likely mechanisms of change, processes and outcomes, and can be used to guide future work on improving cultural responsiveness in AoD and potentially other health and human services. Our approach demonstrates a process to improve the cultural responsiveness of service delivery, and it is hoped that this approach impacts on inequalities in health status and healthcare access between Aboriginal and non-Aboriginal people.
An important strength of the project is that the co-design and implementation was led by an Aboriginal researcher and AoD worker, with extensive consultation with senior Aboriginal AoD clinicians (via the Aboriginal Advisory Group), funders, researchers, as well as links with workers (via the peak organisation for the NGO AoD sector) [6,21,24]. Furthermore, the guidelines recommend multiple evidence-based strategies across all organisational levels [6,16], such as: tailoring of service delivery to local communities [26]; enhancing relationships with Aboriginal services and communities [3,23,24]; improving staff knowledge and competency [25]; and implementing organisation-wide policies and practices [39]. The improvements in audit scores observed across all themes of the guideline, indicating that these concepts and strategies were clearly operationalised. Improvements were frequently observed in areas related to enhancing relationships with Aboriginal communities (e.g., having local Aboriginal protocols reflected in practice and/or policy), improving staff knowledge and skills (e.g., improved crisis triage options and staff demonstrating cultural responsiveness in direct service delivery) and organisation-wide policies or practices (e.g., including materials about working with Aboriginal people in service induction training). A larger evaluation with a longer timeframe would allow a more detailed exploration of specific components of the audits and guidelines and whether there are critical activities that services can enact to improve cultural responsiveness.
In addition to measuring change in audit scores (reflecting change in cultural responsiveness), future studies should also aim to examine the impact of these changes on service delivery or utilisation outcomes, potentially through using routinely collected administrative data. Previous reviews of cultural responsiveness programs have highlighted the need for valid indicators of change and objective outcome measures [6,12], and routinely collected administrative data represent objective, pragmatic, low-cost and easily tracked outcomes. As services improve their levels of cultural responsiveness, we would hope to also see improvements in service utilisation by Aboriginal people (for example, the number of episodes of care provided to and completed by Aboriginal people). The short time frame of the current evaluation limited our ability to examine these types of outcomes. Not only was the time for services to enact changes limited [27], but three months was likely not sufficient for any changes implemented to impact on service utilisation or client outcomes.
The project used a methodologically strong assessment process involving a standardised audit tool that reflected the best-evidence guidelines and a double-scoring system to enhance inter-rater reliability [40]. The possibility of practice effects should be noted; service staff may have had a more thorough knowledge of the audit criteria after completing the baseline audit, leading to more positive reporting of activities in follow-up audits. The practical implication of this is that some of the improvement in follow-up audit scores may be due to improvements in staff understanding of the audit, rather than the specific cultural competence activities they enacted. This is an issue about the true mechanisms of change: it is likely that the observed changes in cultural responsiveness are a combination of both the activities themselves and greater familiarisation with the audit process and content. Some services had limited capacity for improvement in audit scores for their three priority action areas; three services chose to prioritise an area that already had a full score at baseline, and one service only selected two priority areas. For future implementations, services should be encouraged to choose priority areas that have room for improved practice, providing maximum opportunity for improvements.
Participating services were self-selected, and it is possible that they may have had a pre-existing active interest in and/or resources to dedicate to improving their cultural responsiveness. The significant improvements in audit scores achieved by these services may not occur so quickly in other services. However, the participating services do represent a broad geographic and demographic area of NSW (including both urban and regional locations), as well as a variety of service delivery types. Service frontline and managerial staff rated the project as highly acceptable [27]. A key next step is a longer-term follow-up of participating services to establish whether the improvements in culturally responsive practice can be maintained or extended over time. Importantly, in line with the logic model presented, this will include an examination of administrative data to assess any changes in service utilisation. Then, if indicated, a randomised controlled trial evaluation of the implementation process in a larger sample of services may be warranted to demonstrate the generalisability, and costs and benefits of the process.

Conclusions
The co-designed best-evidence guideline and pragmatic implementation process represents a feasible and acceptable method (27) for implementing service-wide improvements in cultural responsiveness and may be applicable to improving the cultural responsiveness of a wide variety of health and human services. The randomised stepped-wedge evaluation design, double-rated audit scoring, and standardised core intervention increased methodological rigour, while the flexibility with which individual services can operationalise and implement the guidelines allowed tailoring to available resources and needs.
Supplementary Materials: The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/ijerph20054223/s1, Supplementary File S1: Alcohol and Other Drugs Treatment Guidelines for Working with Aboriginal and Torres Strait Islander people in a non-Aboriginal setting.
Author Contributions: The project was overseen by the Network of Alcohol and other Drugs Agencies (NADA), implemented by the project team (R.W. and J.A.), and the evaluation was completed by NDARC (S.F., A.S. and A.H.). NADA, the project team, the Aboriginal Project Advisory Group and the evaluation team all worked closely to design, implement and evaluate the project (R.W., J.A., S.F., A.S. and A.H.), reporting to the coalition of PHNs who funded the project. R.W., J.A. and S.F. acquired the data; A.H. conducted the data analysis; S.F. and A.H. wrote the introduction, methods, results and discussion, with input from all authors. All authors provided substantial contributions to the interpretation of the results, critically revised the manuscript, and read and approved the final manuscript as submitted. The guideline was developed with consultation with the Aboriginal Project Advisory Group, whose intellectual contribution is formally acknowledged in the guideline document. To ensure the knowledge and resources arising from the project remain with Aboriginal people, the project materials (e.g., audit documents) are owned by Aboriginal Corporation Drug and Alcohol Network of NSW (ACDAN) to support future work. All authors have read and agreed to the published version of the manuscript.

Funding:
The Alcohol and Other Drugs Treatment Guidelines for Working with Aboriginal and Torres Strait Islander people in a non-Aboriginal setting project was initiated and funded by a coalition of six PHNs in NSW, Australia. The project arose from the intention by the six PHNs to enhance and standardise culturally responsive practices at non-government AoD Treatment Services. The PHNs identified and contacted potential services within their jurisdictions regarding participation and provided contact details of the services to the project team. Funding was provided to National Drug and Alcohol Research Centre (NDARC) to conduct the project evaluation. The PHNs had no role in the evaluation design, implementation, data collection, analysis, interpretation of the data, or in the writing of this manuscript. Informed Consent Statement: Informed consent was obtained from service CEOs/management for services to participate and from all individuals involved in the study.

Data Availability Statement:
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. The data are not publicly available due to potential for re-identification of participating services.