Evaluation of a Multipart Implicit Bias Educational Program Designed for a Non-Profit Organization

Abstract

Children’s access and opportunities to play sports are influenced by categories of difference, such as gender, race, and socioeconomic status. In order to provide an inclusive community and facilitate the recruitment and retention of diverse youth, athletic organizations should be aware of implicit bias and how this can affect the relationship between volunteers and the children they serve. This paper presents a formative process evaluation of a diversity, equity, and inclusion (DEI) training program for a non-profit athletic organization. Training was implemented in person in a group setting with multiple opportunities for group discussion. Mixed methods were used to monitor (1) the implementation of training, (2) its effects on attitudes throughout training, and (3) impact one month after training. Findings demonstrate that the program increased participants’ ability to identify DEI initiatives and sense of belonging to the organization, but these effects had declined by the end of the first month after training. However, participants’ self-beliefs regarding DEI and overall willingness to engage in DEI activities remained enhanced one month after training. Qualitative data were helpful in providing insight into how training impacted participants and their interactions within and outside of the organization. Through this mixed methods approach, we can conclude that DEI training did in fact have a positive impact on the organization, but further evaluation and training may be necessary to address the decline in some effects seen one month after training.

1. Introduction

DEI (diversity, equity, and inclusion) training has become more commonplace in order to facilitate a diverse, equitable, and inclusive environment in organizations, particularly for members from under-represented or historically marginalized communities. The term “DEI training” typically encompasses any instructional program with a goal of helping people with different backgrounds and attributes work together more effectively [1]. There are many different forms of DEI training, such as allyship training, bystander communication training, and training on hiring practices. Training may consist in self-reflection, background on historical and legal context, workplace scenarios, and role-specific training. The DEI workshop in this study utilized unconscious bias training (UBT), which is a structured intervention that aims to increase awareness of unconscious biases and their effects on others and to provide strategies for mitigating these biases [2]. Key components of UBT include education on the nature and impact of bias, self-reflection and self-awareness exercises, and historical context and skill-building exercises aimed at reducing the influence of bias on decision making [3].

The current literature remains inconclusive on the efficacy of DEI training and whether the effects last in the long term to make a difference in an organization. Studies range from showing effective long-term improvement to short-term change to no effects at all [1,4,5,6,7]. In a literature review focused on evaluating the effectiveness of diversity training in promoting inclusivity and diversity, the authors encountered challenges to deriving clear and definitive conclusions on the efficacy of DEI training. They report that many DEI training courses are not relevant enough to the context of the organization and not rigorous enough to make a difference. Further, the authors note that many studies on DEI training focus too much on individual-level self-reported outcomes and not enough on consequential, behavioral, and systems-level outcomes to shed light on the effects of a DEI program [5]. In this study, behavioral outcomes are measured after the completion of a bias training program relevant to the goals of the organization.

To add to the uncertainty of DEI training effectiveness, some studies have found that DEI training can improve awareness, knowledge, motivation for change, and attitudes towards diversity, while other studies have found that only the knowledge and awareness components last in the long term, while attitude and motivation for change diminish over time [1,4]. One study evaluated the effects of DEI training on healthcare professionals in the Netherlands in the settings of a nursing home, a mental institute, and a hospital. Participants completed four training modules, each of which was four hours long, and the effects of training were monitored using surveys and interviews for three years afterwards [4]. The four modules consisted of a general introduction to diversity, diversity and the professional, diversity and the institution, and working on a plan for action. This study found that the participants’ awareness, knowledge, motivation for change, and attitude all changed and lasted throughout the three years of follow-up [4]. In contrast, a meta-analysis on diversity training evaluation found that although DEI training can strongly enhance people’s knowledge about other groups, it can only affect people’s beliefs and how they behave towards others in the short term [1]. The authors proposed that as time passes, people remember the new knowledge, but over time, since beliefs are strongly linked to our emotions and personal identity, beliefs and behaviors revert to what they were before training [1]. This study also found that DEI training is the most effective when the participants are concurrently involved in other diversity-related initiatives and can actively incorporate what they have learned [1]. These observations were corroborated by another group that examined the effectiveness of cultural competence programs in ethnic minority patient-centered care. Although an increase in cultural knowledge and awareness was noted, there were no significant findings in terms of patient health outcomes [8]. This suggests that although knowledge may significantly increase, actions might not change, at least according to the measurement of patient-reported outcomes as behavioral change of the staff [8].

There have also been mixed results on whether DEI training leads to behavioral change, with some studies finding that participants with a baseline level of a more positive and supportive attitude towards diversity had a successful behavioral change after DEI training, while study participants with a less supportive attitude at baseline had no behavioral change [9]. This suggests that participants’ backgrounds and baseline attitudes may affect the efficacy of DEI training and the results of the studies. For instance, the study mentioned above on diversity training in the Netherlands found that participants from the nursing home started off with less motivation to take action on diversity and had a smaller change in willingness to take action than the participants from the hospital, who went from “plan” to “do”, and those from the mental health practice, who went from “unawareness” to “plan” [4]. This illustrates the importance of measuring the baseline level of DEI awareness with at least a pre-event survey, as is performed in this study. This study attempts to elucidate if training can effectively change participants’ awareness, knowledge, motivation to change, and attitudes, and whether these changes are only short-term or long-lasting.

Specifically in the non-profit sector, studies have shown that diversity training needs to be adapted according to the size of the organization in order to maximize success [10]. Another study on non-profit training found that only 25 percent of diversity trainers include steps for organizational development in their training [11]. A review of research on non-profit organizations also calls for investigation of how diversity can be worked through in organizational practice [12]. Overall, research on DEI training for non-profit organizations is currently limited, and this study attempts to provide research on organizational diversity training created for the nonprofit setting.

Hockey Haven Inc. is a 501(c)(3) non-profit organization established in 2021 with a goal of increasing the representation and diversity of youth hockey for under-resourced children in the Greater New Haven area, Connecticut. Hockey Haven aims to accomplish this by increasing access and creating an inclusive community. Specifically, the organization provides completely free-of-cost opportunities for children from any demographic to participate in a learn-to-play ice hockey program. One of the challenges Hockey Haven faces is the lack of diversity among the staff and board members, who primarily work with an under-resourced and marginalized population. With the goal of creating a more diverse, equitable, and inclusive environment in the organization, Hockey Haven applied for and received funding through Quinnipiac University Perspectives on Equity Advancement: Research and Learning Symposium (PEARLS), to organize a diversity, equity, and inclusion (DEI) training program for all staff, volunteers, and members of the local community [13]. DEI training included an in-person presentation and a group discussion facilitated by Randi McCray MS, who specializes in DEI training and leading group activities on these topics. Randi McCray is the founder of RM consulting and has started multiple DEI initiatives at Yale University while pursuing a PhD. The purpose of this study is to evaluate the effectiveness of DEI training on implicit bias among those who participated in training.

2. Intervention: Implicit Bias Training

Participants of DEI training on implicit bias were recruited primarily by the Hockey Haven Inc. organization and the Frank H. Netter M.D. School of Medicine community. Emails, flyers, and the school’s event calendar were utilized for advertising. A total of 19 study participants attended the DEI training event held on 24 March 2023 at Frank H. Netter M.D. School of Medicine.

Training on implicit bias was split into four key components: (1) introduction to implicit bias, (2) identification of implicit bias, (3) learning skills to manage bias, and (4) applying skills to manage bias (

Table 1. Content of DEI training focused on implicit bias.

Module	Aim	Content
General Introduction	Explore implicit and unconscious bias	Introduction
		Pre-Event Survey
		What is Bias?
		Explicit vs. Implicit Bias Definitions
		Video on Thinking Under the Influence
Identification of Bias	Recognize how bias may impact the lived experiences of ourselves and others	Video Example of Implicit Bias in Action
		Large Group Discussion on Video
		Microaggressions and their Impact on Others
		Mid-Event Survey
Learning Skills	Explore techniques to manage bias in day-to-day interactions	Reflect on our own Identity
		“Identity Lenses” Activity
		Small Group Discussion on our “Identity Lenses”
		Overview of Lens Method for Recognizing Bias
Applying Skills	Explore how to utilize skills to minimize bias	Strategies to Mitigate Bias
		Overview of the Ladder of Inference
		Post-Event Survey

). Participants carried out several exercises individually, in small groups (three–four people), and as a large group to gain different perspectives and explore how bias affects everyone.

The first module of training began with a general group introduction for training participants to introduce themselves to everyone. Before beginning the content of training, a pre-event survey was given out to study participants. The content of training began with participants answering the question “What is bias?” with the group. The theoretical background of bias and differences between explicit and implicit bias were then discussed. A video showing the harmful effects of bias was then presented to the group.

In the second module of training, participants watched a video of how implicit bias can specifically affect interactions when working with children. A group discussion on the video ensued. Following group discussion, microaggressions and how they pair with implicit bias in affecting our interactions with others were further discussed. A mid-event survey was then given out to study participants.

In the third module, training participants were asked to take some time to reflect on their own personal identity. A group activity on creating “identity lenses” was then performed. Participants split into small groups (approximately three–four people) to discuss their identity lenses and how they affect the way they see and interact with the world. The lens method was then presented as a way to recognize bias in our own personal lives.

The final module presented training participants with strategies to mitigate bias in their daily lives, known as the ladder of inference. The ladder of inference was then described in further depth as a way in which humans default to making decisions based on the observations made. The strategies to mitigate bias involve trying to modify and not solely rely on the ladder of inference when making decisions. Finally, a post-event survey was given to training participants.

3. Mixed Methods

All participants of DEI training on 24 March 2023 were invited to participate in a pre-event, mid-event, post-event, and one-month follow-up survey. Participants could withdraw from participating in the surveys at any time.

Data were anonymously collected online and involved a total of four surveys delivered as follows: beginning of training, middle of training, end of training, and one month post-training. At the event, before the interactive session began, the pre-event survey was released by handing out QR codes with links to the survey. At the end of the second training module, a second QR code was distributed with the link to the mid-event survey. Then, following the interactive session by Randi McCray and at the conclusion of the event, a third QR code was distributed with the link to the post-event survey. Finally, on 24 April 2023, a fourth QR code was distributed by email to all training participants with the link to the one-month follow-up survey.

The survey questionnaires consisted of Likert scale and ranking questions, utilized for quantitative analysis, and open-response questions, utilized for qualitative analysis. Survey questions were originally created by Julianne Hall PhD and adapted from a study by Yoder et al. assessing the effects of a similar DEI event [10]. Survey questions are listed in Appendix A.

3.1. Quantitative Analysis

Likert-scale survey questions were on a scale from 1 to 5 (1 = strongly disagree; 5 = strongly agree) with an option for N/A. The analysis of the Likert scale questions was performed using Prism 9 software by GraphPad (Version 9, GraphPad, Boston, MA, U.S.A., 2023). The mean of the Likert scale responses was calculated and analyzed using a Mixed Effects analysis.

3.2. Qualitative Analysis

The analysis of the qualitative portion of the surveys was conducted with an individual review by authors A.E.G. and A.M., followed by a meeting to develop a codebook, which includes definitions for what would constitute a participant’s response being coded for each theme (Appendix B). This ensured that the two authors used consistent measures for deciding how to code the participants’ responses. Biweekly meetings were held between the two authors to review the coding and ensure the trustworthiness and consistency of the qualitative analysis. Once codes for each question and each response were agreed upon by both authors, the codes were tallied and divided by the total number of responses for each respective question. This depicted the number of times each theme was referenced in each survey as well as the percentage of participants that referenced each theme in their responses.

4. Results

4.1. Implicit Bias

This study aimed to compare participants’ knowledge on implicit bias at the beginning of, middle of, end of, and one month after DEI training. Participants were asked to define implicit bias at these different time points. The codes “unconscious beliefs/thoughts” and “unconscious beliefs/thoughts with actions” were consistently the top two themes described by respondents across all four surveys. Both themes showed an increase in the percentage of respondents who referenced them in their definition of implicit bias from the pre-event survey to the post-event survey, although there was a higher increase in “unconscious beliefs/thoughts with actions” (

Table 2. Most referenced themes—“How would you define implicit bias”?

Survey	Unconscious Beliefs/Thoughts	Unconscious Beliefs/Thoughts with Actions	Unconscious Beliefs/Thoughts Based on Social Grouping	Conscious Beliefs/Thoughts
Pre-event	63% (12)	42% (8)	11% (2)	11% (2)
Mid-event	66% (12)	56% (10)	6% (1)	28% (5)
Post-event	66% (12)	56% (10)	6% (1)	28% (5)
Follow-up	70% (7)	40% (4)	30% (3)	40% (4)

). However, the number of respondents using these two themes in their definition remained equal from the mid-event to the post-event surveys. The following participants’ response shows “unconscious beliefs/thoughts” as an initial theme, with “unconscious beliefs/thoughts with actions” being introduced as a theme in the mid-event and post-event surveys.

Pre-event: “Our thoughts and attitudes that are driven by our unconscious mind”.

Mid-event: “The way we think and behave under the influence of our subconscious mind”.

Post-event: “Letting your subconscious mind direct your thoughts and attitudes. Which can lead to actions”.

In the follow-up survey, the percentage of respondents who referenced “unconscious beliefs/thoughts” increased, while the percentage of respondents who referenced “unconscious beliefs/thoughts with actions” decreased (Table 2). Additionally, in the pre-event survey, there was an equal number of respondents who referenced “unconscious beliefs/thoughts based on social grouping” and “conscious beliefs/thoughts” in their definitions. However, there was an increase in the number of respondents who referenced “conscious beliefs” compared with “unconscious beliefs/thoughts based on social grouping” in the mid-event, post-event, and follow-up surveys. The largest increase in the percentage of respondents referencing a certain theme from pre-training to post-training and follow-up levels was in the theme “conscious beliefs/thoughts”, which increased from 11% to 28% throughout training and up to 40% at follow-up.

4.2. Awareness of DEI in Organization

Responses to survey questions measured on a Likert scale showed an increase in the self-reported ability to identify DEI issues and programs in the organization throughout training, which receded over the 1-month period following training (Figure 1). The self-reported sense of belonging to the organization followed a similar pattern, increasing throughout training but then dropping back to pre-event levels 1 month after training. In addition, participants’ self-reported ability to engage with DEI in the organization did not have a significant change from pre-event levels.

However, in the follow-up survey, participants were asked how, if at all, DEI training had impacted their interactions at Hockey Haven. An example of some participants’ responses to this question are shown below:

“The DEI training has made me more thoughtful and reflective on the kids’ lives outside of Hockey Haven and more about how either their parents or teachers treat them and how that affects their behavior. I think it has made me more thoughtful about what else could be going on in the children’s lives and just being more reflective on that front”.

“After the training I have more diligently looked for ways in which I may be biased, and I also think it help to reaffirm the important role that we have in shaping the experiences of these young kids and what a privilege and responsibility that is”.

“It has enriched my interactions with those in the organization, in that I find myself able to appreciate differences more consciously, and in so doing, reflect and then react with greater understanding”.

4.3. Organization Values

In the pre-event, post-event, and follow-up surveys, Hockey Haven volunteers who participated in DEI training were asked to list what they believed was the organization’s most important value. In these three surveys, “access/opportunity” and “inclusion” were the most referenced themes. However, the number of respondents who referenced “access/opportunity” decreased from the pre-event to the post-event surveys, and the number of respondents who references “inclusion” increased (

Table 3. Most referenced themes—”For Hockey Haven staff/volunteers, what do you think is one of our most important values”?

Survey	Access/ Opportunity	Inclusion	Eliminating Barriers	Youth Engagement
Pre-event	37% (7)	26% (5)	16% (3)	11% (2)
Mid-event
Post-event	22% (4)	44% (8)	11% (2)	11% (2)
Follow-up	30% (3)	10% (1)	0% (0)	20% (2)

). For example, one participant’s responses in the pre-event and post-event surveys are shown below:

Pre-event: “Increasing access and providing opportunities for kids to learn a sport where there is limited diversity”.

Post-event: “Inclusivity”.

This participant’s response theme changed from “access/opportunity” to “inclusion”. However, in the follow-up survey, “access/opportunity” remained in the top two most referenced themes, but inclusion was replaced by “youth engagement” (Table 3). The follow-up survey saw a trend in responses similar to pre-event responses.

4.4. One-Month Post-Training Analysis

Questions assessing training participants’ self-reported DEI beliefs were included in the one-month follow-up survey. The majority (80%) of respondents reported that their awareness, knowledge, attitude, and willingness to take action in DEI all increased after training (Figure 2). No participants reported a decrease in any of their beliefs regarding DEI.

In addition, training participants were asked in all four surveys how likely they were to participate in DEI efforts, in any capacity, over the next year. Throughout training and upon follow-up, the percentage of participants who said they would have participated in DEI efforts frequently or every time increased (Figure 3).

Additional questions regarding training participants’ use of their skills and reflections on DEI were included in the one-month follow-up survey. A minority (20%) of respondents reported that they had not had an opportunity to use their skills and had not reflected on their attitudes towards DEI. However, 70% agreed that they had the opportunity to use their skills, and 80% of respondents agreed that they had reflected more about their views on DEI (Figure 4).

In the follow-up survey, respondents were asked about any instances since training in which they had the opportunity to use the skills or knowledge they had learned in training. Below are examples of some respondents’ answers:

“I think in a general sense I have been more patient with behavioral issues that arise at Hockey Haven. Giving myself a space to not react to bad behavior immediately, but instead reflect on why it might be happening, has caused me to divert bad behaviors instead of reprimanding or scolding the kids”.

“Yes, when trying to resolve conflict among players on the ice, I have really been careful with what I say and the approach I take to resolving the conflict”.

“I remember specifically before the training that when I was talking to a kid who was not following directions or was quiet or in a bad mood, I did not consider the range of possibilities as to why that might be. After the training I have been able to have a vastly more open mind when approaching kids in these situations and instead have focused on what I can do in that moment to help them enjoy the session as much as possible”.

5. Discussion

The objective of the present study was to evaluate the effectiveness of a DEI training program for a non-profit organization focused on increasing representation and diversity in youth sport. Through self-reported analysis of awareness and motivation of DEI in the organization, our study found that the positive effects of DEI training in these areas declined over time. This remains consistent with the prior literature that has shown the decline in DEI training effects over time [1,2,6]. However, even though awareness and motivation towards DEI initiatives specific to the organization declined, the majority of respondents agreed that their general beliefs towards DEI had all remained positively affected one month after training. In addition, most respondents, at the one-month follow-up, felt that they had opportunities to utilize their skills and reflect on what they had learned in training. It seemed that the key takeaway from training was the aspect focused on implicit bias.

This brings into question whether implicit bias training was the most effective topic for the specific audience of a non-profit organization focused on increasing diversity in sport. Though the key takeaway of training was implicit bias, the overall values and principles of diversity promotion seemed to have an overall positive effect on participants. In addition, the decline back to pre-training levels in the ability to identify DEI initiatives and the sense of belonging to the organization could be explained by the relatively high baseline that training participants already possessed, consistent with what was seen in another study [4]. It is safe to assume that employees of a non-profit focused on diversity and representation in sport maintain a high level of awareness towards DEI even prior to formal training. The potential for bias in this group of subjects that all come from the same organization is a possible explanation for this finding.

The qualitative portion of the results also shows that participants of this training had basic, pre-existing knowledge about implicit bias, as demonstrated by most participants defining implicit bias as “unconscious beliefs/thoughts”. However, the increase in participants referencing “unconscious beliefs/thoughts with actions” throughout training shows that training did have a positive effect on informing participants that implicit bias can affect actions. Additionally, training emphasized the need to recognize and challenge implicit biases. This may explain why more respondents mentioned “conscious beliefs/thoughts” in the mid-event and post-event surveys compared with pre-event. The follow-up survey showed that participants’ knowledge about implicit bias dropped back to pre-training levels; however, the theme “conscious beliefs/thoughts” was still referenced at a much higher level than in the pre-event survey. This could suggest that the training’s focus on becoming aware of implicit bias lasted beyond training.

Training also seemed to influence participants’ definitions of cultural competency. While many of the participants came into training having a basic understanding that awareness and knowledge of cultures go hand in hand with cultural competency, the number of respondents listing this as a theme slightly decreased after training began, while the number of respondents listing “appreciation and acceptance” increased. This finding suggests that participants may have realized that simply being aware and knowledgeable about cultures may not mean one is culturally competent, but rather it is this knowledge alongside appreciating and accepting cultural differences that may constitute this term. The follow-up survey data suggest that this knowledge lasted beyond training in some participants, as there was still a slight increase from baseline in the percentage of respondents that referenced “appreciation and acceptance” and a slight decrease in those referencing “cultural/background awareness/understanding” as a theme one month post-training.

Interestingly, as a consequence of the program, a decrease was observed in Hockey Haven participants who listed “access/opportunity” as the organization’s most important value and an increase in those choosing “inclusion” as the most important value. As an organization that focuses on increasing representation in hockey and providing learn-to-play programs at no cost to the families that participate, it is no surprise that these two themes were referenced as the top two most important values. However, it is possible that throughout training and the group discussion integrated into training, Hockey Haven volunteers were reminded that the core mission of the organization is to increase representation in hockey. Increasing representation in the sport ultimately comes from the inclusion of everybody, regardless of demographic or socioeconomic background. This move to expand representation requires Hockey Haven staff to be actively cultivating an inclusive environment with youth participants, whereas increasing access and opportunity through free programs provides the platform to make that happen. The importance of implementing debriefs and conversations about DEI and the organization’s mission is illustrated by the reported Hockey Haven value “inclusion” dropping down below pre-event-survey levels at follow-up.

Given the above, to maintain and stay consistent with DEI efforts, employees, at follow-up, suggested that routine debriefing after Hockey Haven sessions and continuous conversations surrounding DEI can help to mitigate the decline seen at the one-month follow-up. It is important to note that the low response rate in the follow-up survey makes it more difficult to draw a definitive conclusion as to whether the lasting effects or lack thereof held true for the group as a whole.

6. Conclusions

A formative evaluation of a DEI training program for a non-profit organization demonstrated that such a program can increase participants’ ability to identify DEI initiatives and the sense of belonging to the organization. Yet, these specific effects of training declined by the end of the first month after the training program. However, one month after training, participants reported that their beliefs towards DEI had increased since training. Qualitative data show that training positively influenced participants’ ability to identify that implicit bias can affect actions; however, the recognition of implicit bias influencing actions in their definitions returned to pre-training levels at the one-month follow-up. A higher percentage of participants were able to identify conscious beliefs as a component of implicit bias at the one-month follow-up, suggesting that training left lasting effects on participants’ recognition of the importance of becoming aware of implicit bias. Additionally, the training’s effects on participants’ definitions of cultural competency persisted at the one-month follow-up, demonstrating that training was able to emphasize the appreciation of cultures as a component of cultural competency. In terms of identifying the non-profit organization’s most important value, the changes associated with training returned to pre-training levels.

The responses from training participants suggest that implicit bias was the biggest takeaway from DEI training. The qualitative portion of the survey shows that becoming aware of implicit bias was the portion of training that left a lasting impact, as did the appreciation of different cultures. The responses suggest that multiple DEI training sessions and follow-up discussions focusing on different aspects of DEI, such as cultural competency, gender diversity, or religious observance, could prove useful for the organization. In addition, follow-up sessions could also address the decline in the ability to identify DEI initiatives, sense of belonging, recognition of implicit bias having the potential to influence actions, and identification of the organization’s important values seen one month after training.

There are several limitations to this study. One limitation is the small sample size of this study, ranging from n = 19 to n = 10 for the pre-event and one-month follow-up surveys, respectively. With a small sample size, it is not possible to extrapolate quantitative results or make strong conclusions. The quantitative results provide an insight into how participants were impacted by DEI training, which supplements the qualitative aspect of this study.

Another limitation of this study is volunteer bias in the selection of participants. All participants are members of a non-profit organization focused on bringing equity to the sport of hockey, and they voluntarily chose to participate in DEI training. This could have led to participants having a similar attitude towards the importance of DEI before training, thus impacting the results. Studies have shown that it is more likely that organizations with servant leaders have more empathy and more servant attitudes, which may influenced the effects of training [14,15]. However, teams with servant leaders have been shown to flourish and build better task-focused communities, so this limitation is not inherently negative [14].

These results can help guide non-profit organizations in creating impactful training curriculums that help to address the importance of different aspects within DEI. However, we recommend further research on this subject to gain more insight into the effects of DEI training at an organizational level and if follow-up training proves to be effective. Additionally, future research needs to explore the role that gender may play in the effectiveness of DEI training. As a non-profit organization, there is a need to foster and nurture DEI initiatives to better help the community we serve. The utilization of the outcomes of DEI training can help to implement and improve diversity and policies within an organization.