Culture of Interculturality, Diversity, Equity, and Inclusion (IDEI) Assessment: Lessons from a Social Justice-Based Intercultural Learning Certificate Program for Preservice Teachers

Abstract

Despite the pillars, rubrics, and standards provided by national education organizations and accrediting bodies, many educator preparation programs (EPPs) struggle to prepare teacher candidates to engage effectively with all children across differences in an intercultural context. The ability to engage across differences is especially important for teacher candidates as America’s public schools are more diverse than ever. To increase teacher candidates’ knowledge skills and dispositions, we propose a theory-based program focused on interculturality, diversity, equity, and inclusion (IDEI) that aligns with professional accreditation standards and weaves in effective assessment practices. By intentionally embedding assessment activities in program development, we hope to create a culture of IDEI assessment that not only meets accreditor standards but also results in program improvements and learners’ development.

1. Introduction

America’s public schools are more diverse than ever before. As such, educator preparation programs (EPPs) must intentionally and systematically prepare preservice educators to engage with all children, especially those children, families, and communities who are marginalized and disenfranchised. EPPs struggle with educating preservice teachers about interculturality, diversity, equity, and inclusion (IDEI), and they often falter in assessing learning, development, mindset, and engagement with these concepts. According to Ryan et al. [1], “these concerns around teaching diversity provide a significant challenge for teacher education. Issues of student diversity, social justice, and educational equity have featured prominently in teacher education research for more than five decades” (p. 1). Therefore, educating preservice teachers to engage across differences in an intercultural context, as well as with diversity, equity, and inclusion (DEI), is paramount, necessary, and an ethical and moral imperative for EPPs to truly contribute to social change.

EPPs in professional schools do not exist in a bubble, as national education organizations and societies as well as education accrediting bodies provide pillars, rubrics, and standards to engage with IDEI across the curriculum. These organizations provide a pathway to approach the vexing problem of assessment and continuous quality improvement that EPPs can build on to create a culture of assessment for equity work with preservice teachers. In this era of accountability and heightened scrutinization of teacher effectiveness, it is important to note that “identifying common approaches to assessment has rarely been met with unbridled enthusiasm and agreement but has instead been met with skepticism, debate, and frustration” [2]. Creating a culture of assessment that is holistic and takes into consideration the EPP’s mission, vision, learning outcomes, institutional climate, curriculum, instruction methods, the school district’s mission and community demographics, student, faculty, and staff demographics, and the sociopolitical context of the surrounding county and state is an important reframing for centering assessment in teacher education.

The American Association of Colleges and Universities (AAC&U) defines intercultural knowledge and competence as a “set of cognitive [knowledge], affective [attitude] and behavioral skills and characteristics [skills] that support effective and appropriate interaction in a variety of cultural contexts” [3]. Likewise, the Council for the Accreditation of Educator Preparation (CAEP), the accrediting body that oversees the institution that houses the program that is the center of this reflection paper, offers guidance on engaging DEI in standard one (see Table 1) of its initial-level programs [4].

In addition, centering IDEI in programs, policies, curriculum, and instruction is the role of effective programmatic assessment. Not only must EPPs enact a culture of engaging with IDEI, but they must also have a culture of assessment to measure preservice teachers’ growth and development in these areas. Cochran-Smith [5] suggests that teacher education programs should consider multiple factors in their assessment efforts as “transforming teacher education into an enterprise that is grounded in research, revolves around continuous assessment of learning, and makes decisions driven by evidence is nothing short of a culture shift in our field” (p. 189).

As such, this reflection paper offers a set of tools in the form of a framework for institutions to use if they are interested in creating similar programs to bolster a culture of IDEI assessment within EPPs. Importantly, this reflection paper aims to provide guidance on the important role of assessment within an equity ethos in the design of programs that support preservice teachers’ development as efficacious border crossers. Hence, the development of the Intercultural Learning Certificate Program (ILCP), which is a co-curricular IDEI initiative that is designed around a culture of assessment within a private, selective liberal arts university’s preservice teacher preparation program is the focus of this reflection paper. The lessons learned from the development of this program inform the framework for the culture of IDEI assessment to support EPPs presented in this work. The central purpose of this programmatic reflection paper is to provide a framework for faculty, staff, and administrators at EPPs, campus-wide leaders, other teacher education foundations and policy institutes, accreditation bodies, and other concerned constituencies who are looking for mechanisms, systems, and processes for creating educational programming that centers on IDEI within a culture of assessment from the onset of program implementation. According to Rowan et al. [6], “teachers consistently report working with diverse learners as one of the most challenging aspects of their work, and teacher educators should therefore be able to answer questions about the ways we respond to this context” (p. 115). Therefore, educational learning programs that complement existing coursework, like the one that is the focus of this paper, which centers on assessing IDEI efforts, are essential promising practices for the future of preparing preservice educators.

This reflection paper brings together our life experiences, research, teaching, and the practitioner expertise that we have accumulated around issues related to interculturalism, diversity, equity, and inclusion. We value preparing preservice teachers from a justice-oriented, inclusive, and anti-bias perspective. Epistemically, the important roles of critical self-reflection and critical consciousness are central to the work we carry out in teacher education. As such, this reflection paper sits at the intersection of our social identities and our quest for EPPs to create a culture where IDEI is systematized, concretized, and woven into the fabric of policies, practices, and curricula. The following positionality statements, in order of authorship, reflect our engagement with this work.

(1)

Preparing preservice teachers to interrogate problematic frames of reference, privilege, bias, discrimination, and oppression is central to my teaching, research, service, and consulting. As the lead author of this study, my positionality in the academy is at the intersection of my immigrant, Black, and Christian faith identities. As a tenured, Black woman and teacher educator at a historically White institution, I am cognizant of the complexities of my identities in carrying out the work of dismantling oppression in a politically heightened environment. In addition, I have over twenty years of social justice work with marginalized and disenfranchised communities, and I bring these experiences to the modules, activities, and discussions in the ILCP. More specifically, I approach the important work of preparing the next generation of preservice teachers from a liberatory and critically conscious perspective, using Black feminist thought, to create opportunities for deep critical self-reflection. Thus, my identities are interconnected with the anti-racist, inclusive, and culturally responsive practices I use to support preservice teachers’ examinations of their epistemological, ontological, and axiological frames to engage in equity-minded teaching.

(2)

As one of the authors of this study, I acknowledge my positionality and privilege as a mid-career academic and a Black American woman with a doctoral degree. I have served dual roles in higher education as a teacher educator and an assessment administrator. As a teacher educator, my goal is to help develop pre-service teachers’ ability to develop and implement culturally responsive formative and summative classroom assessment practices. While striving for objectivity, I recognize that my experiences, beliefs, and privileges may impact my teaching and the research process. As the quantitative methodologist on this research team, I understand that these aspects of my identity may shape my perspectives and influence how I perceive, interpret, and analyze data. In assessing programs aimed at supporting diversity, equity, and inclusion, my goals are to consider individual growth, habits of inquiry, and curiosity about power, inequality, and change via quantitative data. I commit to considering the cultural assumptions that may go into the creation of assessments and acknowledging that data that appear to be neutral can be intensely political. By openly acknowledging my positionality, I aim to enhance the credibility and trustworthiness of this study.

(3)

I acknowledge my standpoint in this research as an educated, able-bodied, White, transgender, and queer American. I also come from a working-class and Christian upbringing. My identities, combined with my professional experiences as a public high school special education teacher and an employee in a college queer resource center, have undoubtedly influenced my perceptions of the Intercultural Learning Certificate Program (ILCP) and my understanding of interculturality, diversity, equity, and inclusion as both dispositions and areas of knowledge. I sought involvement with the administration of the ILCP as a graduate intern due to the teacher–student interactions I observed as a teacher and my passion for social justice education. Just as my positionality guided my work within the ILCP, I believe my positionality has affected my views on how the ILCP is administered and assessed, which is the focus of this paper.

2. Description of the Intercultural Learning Certificate Program

The extant research literature is clear that IDEI education should be an important component of preparing teachers [7]. Despite the clarion call for such efforts, the preparation of preservice teachers to engage with DEI remains a vexing problem for many EPPs [8]. Milner [8] states:

the task of preparing teachers for the diversity they (will) face in P-12 schools is shaped and grounded in a range of complex realities in U.S. society and in education. Whether through traditional or alternative teacher education programs, preparing teachers for diversity, equity, and social justice are perhaps the most challenging and daunting tasks facing the field.

(p. 119)

This paper focuses on the ILCP, a social justice grounded program within a School of Education at a selective, mid-sized private university in the South.

2.1. Institutional and Community Context

The institution’s curriculum is grounded in the liberal arts and sciences and its student population is predominantly White, with 80 percent of undergraduates identifying as White. The School of Education is less diverse by both gender identity and race/ethnicity than the institution’s overall student population, with 90 percent of its students identifying as White and nearly 95 percent identifying as female. The School of Education includes both licensure and non-licensure programs. Licensure programs are offered in early childhood education, elementary education, middle grades education, and in secondary licensure areas, such as English, history, and mathematics.

The community surrounding the institution and the school system into which the EPP places most of its student teachers is far more racially and ethnically diverse than the EPP’s population of student teaching candidates. According to data from the National Center for Education Statistics [9], 35 percent of K-12 students in the district in which the majority of the EPP’s student teachers are placed identify as non-White. Over 20 percent come from homes where languages other than English are spoken, and nearly 22 percent of K-12 students are from families with incomes below the poverty line. Most of the district is suburban, but some schools in the district are classified as rural. The district is supported by several community agencies that focus on education as a key pillar of their operations.

The ILCP is a graduation requirement and, foundationally, frames intercultural competence as a necessary requirement for equity-minded educators. In the ILCP, interculturalism is the first unit of analysis in a journey toward dismantling systems of oppression that infiltrate the teaching and learning process, the curriculum, habits of mind, policies, and practices for preparing preservice teachers. The ILCP is designed to supplement teacher education students’ coursework and to help them make sense of their own experiences.

Structurally, preservice teacher candidates in the School of Education seeking licensure are enrolled in the ILCP in their sophomore year and must complete the program by the fall of their senior year before they begin student teaching. Students are required to complete specifically designed modules and attend debriefs on these modules, participate in IDEI training or workshops on or off-campus, reflect on courses and field experiences, and complete a final capstone poster. Students’ experiences on the program are measured via the Intercultural Development Inventory^® (IDI) in a pretest and posttest format, as well as through qualitative analysis of their reflections from the start to the completion of the program.

2.2. Centering Accreditation Standards in IDEI Assessment

The Council for Higher Education Accreditation’s (CHEA) provides approval to accrediting organizations and represents over 6000 institutions; in 2022, its first requirement related to diversity, equity, and inclusion took effect [10]. Institutions and programs are more likely to embrace DEI efforts as a priority if they are explicitly addressed in accreditation standards. Medical schools and teacher education programs have long been subject to DEI standards to comply with their professional accreditors. Creating a sustainable and meaningful culture of assessment requires EPPs to align state, national, and accreditation standards with program learning goals. Accredited EPPs align their curricula with professional standards set by the CAEP or the Association for Advancing Quality in Educator Preparation (AAQEP) (see Table 1). The accreditation process necessitates self-assessment within programs and encourages innovation in educator preparation. Both CAEP and AAQEP require EPPs to demonstrate a commitment to diversity in their curricula and during their clinical experiences.

The standards require EPPs to intentionally embed knowledge of IDEI in their programs and to assess the impact on candidates’ knowledge, skills, and dispositions, and their ability to work with families and communities in enhancing learning for all PK-12 students.

Commissioned by the Council of Chief State School Officers (CCSSO), the Interstate Teacher Assessment Support Consortium (InTASC) offers a systematic approach to promote and improve teacher effectiveness and growth. The ten InTASC standards encompass a common core of teaching knowledge, dispositions, and skills [11]. CAEP’s adoption of the InTASC Standards and its revised standards handbook moves diversity from a cross-cutting theme and makes it an overarching feature of educator preparation as it requires EPPs to demonstrate the integration of diversity throughout their programs, policies, admission criteria, curricula, and clinical experiences [4]. The EPP under study is accredited by CAEP; thus, its program is aligned with CAEP, InTASC, and required state standards.

For the ILCP, the program that is the focus of this reflection paper, marrying those standards with concrete programmatic learning goals aligned with established theories [12,13,14,15] and the mission of the EPP allowed us to develop an assessment process that garnered buy-in from faculty and shows students that they are being provided with a coherent and intentional learning experience. Paramount to this alignment is the belief by stakeholders, both university teacher educators and PK-12 teachers and administrators, that the standards and competencies are those that the profession needs and values. The intentional development of ILCP learning goals and the alignment with not only professional standards but also theory-based programming allows us to determine whether the program is having the desired impact on students’ knowledge of and dispositions related to IDEI. As a result of this intentionality, we can systematically assess the program’s performance against its goals as well as against relevant standards. Overall, prioritizing accreditation standards and aligning them with IDEI goals is a necessary step in creating a culture of assessment.

Table 1. Professional Standards and Competencies Related to Knowledge of Interculturality, Diversity, Equity, and Inclusion.

Standards	Sample Sub-Standards Related to the ILCP
CAEP Standards (R1.2 and R1.3 also apply) [4]	R1.1. The Learner and Learning (e.g., candidates examine and understand their personal biases, respect learners as individuals, make learners feel valued, promote respect among learners).	R1.4 Professional Responsibility (e.g., deepening understanding of own frames of reference and potential bias, seeing the role as one of advocacy for learners and accountability for learner success, embracing the challenges of continuous improvement and change)
AAQEP Standards [16]	Standard 1: Candidate/Completer Performance 1c. Culturally responsive practice, including intersectionality of race, ethnicity, class, gender identity, and expression, sexual identity, and the impact of language acquisition and literacy development on learning.	Standard 2: Completer Professional Competence and Growth 2b. Engage in culturally responsive educational practices with diverse learners and do so in diverse cultural and socioeconomic community contexts.
NCPTS Standards	Standard 2. Teachers establish a respectful environment for a diverse population of students.	2b. Teachers embrace diversity in the school community and in the world.
(2A, 2C, and 2D, also apply) [17]		2e. Teachers work collaboratively with the families and significant adults in the lives of their students.
InTASC Standards	Standard 2: Learning Differences	Standard 3: Learning Environments
	The teacher uses understanding of individual	The teacher works
(1, 5, 9, and	differences and diverse cultures and	with others to create environments
10 also	communities to ensure inclusive learning	that support individual and
apply) [11]	environments that enable each learner to	collaborative learning and that
	meet high standards.	encourage positive social
		interaction, active engagement in
		learning, and self-motivation.

Table 1. Professional Standards and Competencies Related to Knowledge of Interculturality, Diversity, Equity, and Inclusion.

Standards	Sample Sub-Standards Related to the ILCP
CAEP Standards (R1.2 and R1.3 also apply) [4]	R1.1. The Learner and Learning (e.g., candidates examine and understand their personal biases, respect learners as individuals, make learners feel valued, promote respect among learners).	R1.4 Professional Responsibility (e.g., deepening understanding of own frames of reference and potential bias, seeing the role as one of advocacy for learners and accountability for learner success, embracing the challenges of continuous improvement and change)
AAQEP Standards [16]	Standard 1: Candidate/Completer Performance 1c. Culturally responsive practice, including intersectionality of race, ethnicity, class, gender identity, and expression, sexual identity, and the impact of language acquisition and literacy development on learning.	Standard 2: Completer Professional Competence and Growth 2b. Engage in culturally responsive educational practices with diverse learners and do so in diverse cultural and socioeconomic community contexts.
NCPTS Standards	Standard 2. Teachers establish a respectful environment for a diverse population of students.	2b. Teachers embrace diversity in the school community and in the world.
(2A, 2C, and 2D, also apply) [17]		2e. Teachers work collaboratively with the families and significant adults in the lives of their students.
InTASC Standards	Standard 2: Learning Differences	Standard 3: Learning Environments
	The teacher uses understanding of individual	The teacher works
(1, 5, 9, and	differences and diverse cultures and	with others to create environments
10 also	communities to ensure inclusive learning	that support individual and
apply) [11]	environments that enable each learner to	collaborative learning and that
	meet high standards.	encourage positive social
		interaction, active engagement in
		learning, and self-motivation.

3. Review of the Literature: Assessing Diversity, Equity, and Inclusion Programs in EPPs

Pragmatically, assessing the knowledge, attitudes, skills, and dispositions relating to DEI among students who are enrolled in an EPP is challenging [8]. While one can reliably measure the performance of a student and, thus, infer their acquired knowledge and skills, assessing their attitudes and dispositions is more complicated. Learning dispositions and attitudes are highly situational in nature, which makes these aspects difficult to reliably assess [18]. In addition, there is frequently a lack of consistency among researchers and administrators on how to define teacher beliefs about cultural diversity [19].

Measuring the potential growth or the lack thereof in students within the areas of DEI is further complicated if the EPP lacks a culture of assessment. A culture of assessment is best characterized as “the overarching institutional ethos that is both an artifact of the way in which assessment is conducted and, simultaneously, a factor influencing and augmenting assessment practice” [20]. Without a culture of assessment, measuring the progress of students within EPPs along the functional areas of DEI is at risk of becoming an afterthought rather than a foundational characteristic of the EPP. These factors have led to many EPPs using multiple forms of assessment to measure the knowledge, attitudes, skills, and dispositions of their students [21].

Below is a summary of three common assessment methods, i.e., surveys, standardized test instruments, and written reflections, used to measure diversity, equity, and inclusion programs within EPPs. These three methods are highlighted due to their relevance to the ILCP.

3.1. Surveys

Surveys are perhaps the most common method for assessing diversity, equity, and inclusion programs in teacher education. Many EPPs are assessed by utilizing a pre- and post- survey design [22,23], whereas others administer a survey once and compare two or more student populations, usually with the differential between groups being a diversity-based educational experience [24,25]. In most cases, the diversity-based educational experience accompanying either survey method is a diversity course [22,23].

3.2. Standardized Test Instruments

In contrast to relying upon self-reported confidence, some EPPs assess candidates using standardized test instruments. Inventories such as the Racial/Ethical Sensitivity Test or Defining Issues Test are administered to students at the start and conclusion of their EPP journey [26]. When administered in this manner, standardized test instruments, which are often rigorously evaluated for reliability and validity, can chart patterns of progress or the lack thereof in groups of students.

3.3. Written Reflections

Perhaps the second most common method for assessing diversity, equity, and inclusion programs in EPPs is to analyze the written reflections of students. This can be accomplished by having students write a single reflective essay, multiple reflective essays, and/or engage in repetitive reflective journaling [23,26,27] during an EPP experience, such as a course or fieldwork. Depending on how the assessor constructs the prompts for the written reflections, the submissions from students can provide a qualitative wealth of insight into their knowledge of diversity, equity, and inclusion topics, as well as their attitudes and dispositions.

In summary, the intersection of how this research cross-pollinates with current professional standards in preservice teacher education is paramount in creating a culture of assessment.

4. Creating a Culture of Assessment: A Four-Step Approach to Program Implementation

According to Farkas [28], few institutions have developed a true culture of assessment, and many faculties view assessment as a top-down requirement set by administrators or accreditors. The true purpose of assessment is improving student learning and helping the faculty to embrace and internalize the purpose that is essential to creating a culture of assessment. Considering the importance of curating a culture of assessment for IDEI programs, we recommend a four-step approach for EPPs interested in creating a comprehensive co-curricular program that aligns with the curriculum and relevant professional standards. The assessment of IDEI should be intentional and embedded into the fabric of an EPP’s mission and vision. Consequently, EPPs that have not cultivated a culture of assessment are in danger of a lack of synchronization, flawed conclusions, and may cause harm to students engaging with IDEI. The four-step approach described below is recommended for designing and implementing co-curricular programs, like the ILCP, that intentionally embed relevant program theory and assessment.

4.1. Step 1. Articulating the Goals and Theories That Guide the ILCP

Building programs that prepare equity-minded educators must first begin with an EPP having a mission dedicated to diversity, equity, and inclusion. Guided by the mission of the EPP, the learning goals should be supported by professional standards and theory and research in the genealogy of social justice education. In some cases, programs that support teacher learning in relation to the issues of diversity, equity, and inclusion are enacted without learning goals that mirror state or professional organizations’ standards as well as requirements for accreditation, theories of social justice education, and the EPP’s mission. Goals for the ILCP were selected based on examining CAEP and InTASC standards, state professional teaching standards, and the mission of the EPP. Therefore, in the planning of the ILCP, measurable goals were created to center awareness, knowledge, skills, and dispositions that are often the outcomes expected not only in research on social justice education for teachers but also in state and professional accreditation standards. Hence, creating measurable goals from the onset of the program is a key tool in creating a culture of IDEI that also centers assessment.

To further develop learning goals for programs like the ILCP, designers of such programs must consider the paradigmatic lens that informs the goals and overall experiences of the program. According to Tackas [29], it can be difficult to see beyond our own world views and biases. In the development of the ILCP, careful attention was given not only to the extant research in the field of IDEI presented above but also to theories [12,13,14,15] that guide preservice teachers’ development in interrogating their assumptions, worldviews, identities, problematic frames of reference, biases, and cultural and familial backgrounds. Theories that supported preservice teachers’ learning about marginalization, disenfranchisement, bias, oppression, and educational inequities were the foundation of all learning outcomes, policies, processes, and practices of the ILCP.

Thus, a focus on theories [12,13,14,15] that inform IDEI and overall social justice work is an essential step. Theory guides practice and shapes how one views the world. Learning goals should be based on theories, and preservice teachers enrolled in the ILCP are specifically taught that “whenever we ask “how” or “why” about anything, we are engaged in theorizing: theory can be conceptualized as the learned cultural maps we follow to navigate and make sense of our lives and new things we encounter” [30]. As such, ILCP program participants engage in a continuous critique via theory about how they enact their positionality to understand, engage, support, and create equitable teaching and learning environments for a diverse group of students. In the design of the ILCP and its goals, theories such as culturally relevant teaching [14], social justice education [13,15], and antiracist education [12] are the foundation of the learning goals, modules, reflection prompts, capstone project, and pedagogy used in the ILCP.

Essentially, culturally relevant teaching [14] highlights the important role of culture in the learning process and identifies student learning as a key component of success and social change. Social justice education [13,15] is another theoretical lens used to help students critically engage with issues of oppression and justice related to contemporary and historical inequities in education as well as the persistent and widening opportunity gap. In addition, Gillborn [12] provides a necessary historical and contemporary analysis of race and racism in schooling, focusing on policies and practices that have concretized systemic oppression. Antiracism as a paradigmatic lens involves preservice teachers examining their racial background and how race and racism are manifested in the socio–political context of schooling, the equity gap, distribution of resources, and student success.

Using antiracism as a theoretical frame for program design and implementation enables a culture of IDEI assessment that de-marginalizes racial equity and centers it as crucial to student success.

Careful consideration should be given to theories in the construction of learning goals for co-curricular programs and how these theories are actualized in the curriculum, instruction, policies, practices, and culture of the EPP. Overall, these theories form the foundation for all work in the ILCP and are amplified in coursework as students matriculate at the EPP. In a culture of IDEI assessment, these theories should be part of the lexicon of program design, development, and operation and should help to form the epistemological, ontological, and axiological pillars of learning for preservice candidates.

In addition, in the creation of learning goals, consideration should be given to professional, state, and accreditation standards, as well as the theories that influence learning, by focusing on two essential questions for garnering faculty, staff, and student buy-in to create a culture of assessment. Based on what we have learned in developing the ILCP, the first question to consider when creating learning goals for a co-curricular program is “Are the learning goals of the co-curricular program reflective of the mission and vision of the EPP?” Wilson et al. [31] examined the mission statements of 80 higher education institutions and concluded that “an institution’s mission statement may provide insight into whether or not it truly values diversity” (p. 126). Learning goals that engage IDEI must be aligned with the EPP’s mission for transformative learning. Because mission and vision statements reflect the intent and overall philosophy of the kind of preservice teacher an EPP wants to attract and prepare, co-curricular programs like the ILCP that are designed with the mission and vision at the center will be able to utilize the full resources of the EPP (e.g., its human, social, cultural, and aspirational capital) to galvanize and support students who are often resistant to required diversity, equity, and inclusion programs.

The second question is “Where in the EPP’s curriculum will the learning goals be further supported?” To develop an EPP-wide culture of equity-mindedness, an examination of courses is necessary so that the co-curricular program is not seen as an add-on but as a natural part of the curriculum, instruction, and experiences that each teacher candidate will receive as they matriculate. Gorski [32] supports the importance of a curriculum-wide approach to social justice programming by stating “others have urged teacher education programs to be more intentional about infusing multicultural and social justice education throughout students’ coursework so that what students learn in their multicultural teacher education courses is supported rather than undermined in other courses” (p. 144). Below, we provide two learning goals from the ILCP and refer to Figure 1 to view how these program learning goals align with the professional standards.

Examples of ILCP Learning Goals

• Increase awareness of diverse social identities (self and others) and how identities inform values, beliefs, behaviors, and interactions with others.
• Apply necessary attitudes, knowledge, and skills to interact across differences effectively and appropriately.

4.2. Step 2. Collaborate with Assessment Professionals to Create an IDEI Framework of Assessment Integrated with Accreditation Standards

The second step in creating a program like the ILCP requires consideration of the EPP’s current culture of assessment. Consistent with how data are often required by accreditors at various levels (e.g., institution-wide, program-level, general education, etc.), the EPP’s academic program assessment process consists of three tiers. Tier I consists of data collected at the EPP level, meaning that all teacher education programs regardless of major/discipline assess the same learning outcomes and use the same measures. Tier II consists of program-specific data tied to programmatic learning outcomes. Tier III consists of data collected from program graduates and their employers related to satisfaction, preparation, and impact on P-12 learners. The tiers ensure a systematic review of the EPP’s mission, program goals, and learning outcomes. The EPP’s academic programs have also adopted several high-quality measures, both direct and indirect, that are used EPP-wide to gather accurate information about candidate achievement and to assess candidates’ success in achieving program outcomes.

To further assess programs like the ILCP, EPPs must consider the intentionality of their assessment methods. The literature suggests that some co-curricular program offerings are not assessed intentionally across EPPs and in co-curricular assessment at the institution level.

According to Waryas [33], co-curricular programming and its assessment should be “intentional and purposeful, rather than structured as an afterthought” (p. 72). Given the ILCP’s grounding in program theory, its developers understood the need for assessment in highlighting not only the program’s effectiveness but also in situating IDEI within and outside of the EPP’s traditional classrooms. ILCP staff understood the need to shape assessment activity to meet the needs of the EPP’s setting and mission and its accreditation requirements. With that in mind, the program developers thought deeply about who needed to be involved and in what ways. They focused on the school’s assessment staff as an important thought partner in identifying both curricular and co-curricular changes.

To that end, the EPP’s assessment staff were invited to program meetings and even orientation sessions with students. This important partnership helped to ensure that the program developed specific and measurable goals aligned with relevant standards, as well as a practical and sustainable process for collecting, evaluating, and analyzing data on students’ growth and development. This collaboration between the ILCP and assessment staff can also help to penetrate departmental or institutional silos [34]. The research suggests some programs may be developed without intentional alignment with program learning outcomes or professional standards and without collaboration from assessment professionals [33]. Collaborating with assessment or institutional research professionals for the assessment of co-curricular programming is beneficial because they are often able to support data collection, analysis, and storage of student artifacts, as well as translate co-curricular outcomes into accreditation discussions [33]. Figure 1 illustrates the alignment of the program’s learning goals with the accreditation standards provided in Table 1.

The program identified a valid and reliable measure, the IDI, to assess the students’ intercultural competence at the start and end of the program. The assessment staff were also able to identify complementary measures currently in use by the EPP that could supplement and/or confirm the findings of the IDI. These discussions revealed the importance of sharing and using evidence of student learning and development more broadly so that evidence collected within the classroom could be triangulated to evidence collected outside the classroom. Aligning co-curricular learning outcomes with academic program learning outcomes and professional standards/competencies embeds them in the activities of the program and “allows stakeholders the opportunity to understand more easily the relationships between a program’s ‘moving parts’ and the importance of co-curriculars” [33]. Hence, we suggest that collaboration between IDEI co-curricular program staff and assessment and/or accreditation professionals can enhance engagement with assessment data across the EPP for the evaluation of students’ growth and program effectiveness.

Relevant Questions:

• What if my School of Education does not have a dedicated assessment professional?
• How do you build a professional relationship with assessment experts who may be working beyond capacity?
• What strategies can be utilized to align IDEI goals with academic program learning goals and accreditation standards?

4.3. Step 3. Administering a Standardized Measure to Complement Locally Developed Measures

CAEP and many other accreditors require evidence of the use of normed instruments to assess students’ learning and development. The ILCP adopted the IDI, a standardized instrument that has been evaluated psychometrically for reliability and validity. The IDI is administered as a pre- and post-assessment, and ILCP program staff can use the results both formatively and summatively. Its pre- and post-administration serves as a value-added measure, whereby staff can attempt to measure growth over time from the start of the program to its culmination. The pre-test results allow the program to establish a firm benchmark against which to measure growth, as most students enter the program in their sophomore year before they have engaged substantively with coursework in education. Pre-test orientations also allow the program to conduct group and individual debriefing sessions. During individual debriefing sessions, program staff assist students with developing customized intercultural development plans that identify targeted activities that can help students increase their intercultural competence. In addition to the course modules, program staff develop a list of different learning opportunities (e.g., workshops, speaker series) and recommend strategies that can support students’ intercultural development.

Post-test results are used to present evidence-based stories of student learning and development. Evidence-based storytelling [35] is a promising assessment practice that includes describing the assessment process and includes a narrative detailing how a program knows its students are learning by outlining assumptions and making explicit connections between program components and students’ learning and development. Post-test results provide an opportunity for students and ILCP staff to reflect on the development of their IDEI knowledge, skills, and dispositions. Students use their results to develop their capstone projects and to discuss whether and how they have changed and grown. ILCP staff use the results to provide evidence that the modules and programming influence students’ understanding of their own social identities and how society and systems impact education. In addition to the normed IDI, the program has a multi-faceted assessment process that includes multiple measures. These measures, which are a mix of both locally developed and tiered data, include written reflections, the capstone project, standardized observation instruments, and end-of-program evaluations.

Therefore, not only does the ILCP administer a standardized instrument that supports the requirements of the accreditation body, but other locally created measures such as the written reflections, the capstone project, and tiered data are also used, which allows an expanded view of how students engage and experience the program. These locally created measures are discussed in step four of creating a culture of assessment that places an emphasis on program improvement and reevaluation.

Relevant Questions

• How do you ensure that your standardized measure is aligned with program goals/theory?
• Considering the culture of your EPP, how and when do you administer the standardized measure?
• How do you prioritize and secure funding for a standardized instrument over time?

4.4. Step 4. Assessing Program Effectiveness and Using Results for Program Improvement and Reevaluation

According to Boyce [36] “evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity” (p. 33). Hence, the fourth step in building a culture of assessment around co-curricular programs that focus on IDEI is to engage in the iterative process of program improvement and reevaluation. Currently, the ILCP staff are immersed in this step based on the data gathered from the most recent graduating cohorts. A culture of assessment must include mechanisms that allow for a continuous critique of the processes, policies, practices, tools, experiences, and overall outcomes desired. For the ILCP, those mechanisms of reevaluation and program improvement include the written reflections over the duration of the program, the information collected from the students’ final capstone project, and our role as participant observers in the presentation of these projects to the campus community, as well as the feedback from the tiered data about students’ experiences on the program collected via surveys administered by the assessment team in the dean’s office of the EPP. For EPPs seeking ways to engage in reevaluation and improvement, we suggest utilizing the aforementioned data sources as examples in this final step of creating a culture of assessment for programs focused on IDEI. As program reevaluation and improvement is a cyclical process, the four avenues below are currently in process, and the information presented is what we have learned thus far.

4.4.1. Written Reflections

As noted in the research literature, written reflections [23,26,27] are also used to assess IDEI programs and practices in EPPs. According to Blanchard et al. [37]:

Purposeful and meaningful learning opportunities related to diversity, inclusion, equity, and social justice can help preservice teachers better understand perspectives different from their own, support children with a wide range of strengths and abilities, and celebrate diversity within their classrooms.

(p. 347)

The written reflections in the ILCP are purposeful and meaningful in pushing students to decentralize monocultural ways of knowing through more intercultural paradigmatic lenses. Hence, to assess growth over the four semesters in the program, students use the same prompt for each reflection, which is a set of questions based on the goals of the program (step 1), as well as research on IDEI development to guide them in writing their reflections. On average, each student in the ILCP program submits 15 written reflections over four semesters. A key factor in the program reevaluation and improvement process in the ILCP is the mapping of change or growth in reflections from the onset of the program to its completion. The ILCP team is currently engaged in this process, and this is informing and helps us critique the content we use in the program, identify what content needs to be included, and unearth areas for further mentoring and coaching as well as opportunities for the faculty at our EPP to adjust the curriculum to support IDEI work. We suggest including written reflections in IDEI programmatic offerings in a longitudinal manner as this will help the EPP evaluate the students’ engagement incrementally and adjust content in a sustainable manner.

4.4.2. Examining the Capstone Experience

Central to the program improvement and reevaluation process adopted by the ILCP is the analysis of the capstone project completed by students and presented in the form of a poster presentation to the campus community. The capstone project is a deeply reflective exercise that showcases not only what students have learned but also allows them the opportunity to engage interculturally and formally with the campus community to specifically discuss the knowledge, skills, and dispositions garnered from their time on the program. Examining capstone projects provides an overarching view of what students have gained from the program, expressed in their own voices and within their unique social positioning. Analogously, capstone projects help to concretize the learning goals from the students’ perspectives and, according to Goldstein and Fernald [38], capstone projects “should integrate and cap previous academic learning… [and] prompt self-examination leading to both personal and professional growth” (p. 28). IDEI programs that include a capstone project or course are an important tool for measuring learning outcomes for future program improvement.

4.4.3. Examining Tiered Data

In addition to the capstone project and the written reflections, the EPP surveys graduating students about their experiences during their course of study. We suggest using information captured from the tiered data needed for accreditation to further inform program reevaluation and improvement. Tiered data represent an important pathway for assessment personnel into IDEI programming, since they often hold a panoramic view of the tiered data used and needed for accreditation.

4.4.4. Revising the End-of-Program Evaluation Tool

Examining the capstone projects, written reflections, and tiered data is a key step in revising the end-of-program evaluation survey that the ILCP team continues to refine. This end-of-program evaluation survey measures dispositional descriptors such as ethical practices, professional demeanor, communication with students and their families, intercultural competence, responding to individual needs of students, and engaging with students and families from diverse backgrounds. The program improvement and reevaluation process can be difficult and fraught with barriers, such as student fatigue in completing the requirements of the ILCP while they are on their program of study. As we moved from the pilot to an intermediate stage of the ILCP, the pandemic was a salient factor in how we reimagined program reevaluation and improvement. Currently, the ILCP has moved to a more intermediate and sustainable stage, and our aim is to administer an ILCP-specific survey to gather the necessary data for triangulation with other data sources. Therefore, program improvement and reevaluation are not the last steps in the culture of assessment; they are merely cyclical and iterative processes for continuous improvement.

4.4.5. Relevant Questions:

• Based on the evidence that you have, did student learning outcomes improve?
• How do you prioritize data mining with limited personnel?
• How do you determine which data sources are most meaningful?

5. Implications

Implications of the proposed framework and reflections for EPP administrators and the faculty, professional schools and programs, and the field of higher education are discussed below.

First, understanding how the ILCP is driven by a culture of IDEI assessment is highly relevant for EPP practitioners. As accreditation agencies and research [6] continue to demonstrate an urgent need for meaningful diversity education within EPPs, teacher education programs will need to be able to demonstrate the effectiveness of their preparation programs. Adopting a culture of IDEI assessment allows administrators and faculty of such programs to more accurately determine whether their diversity education initiatives are meeting accreditation standards and fulfilling their EPP’s mission. In addition, having IDEI assessment ingrained into an EPP’s design can create an important foundational baseline for assessing student teaching experiences and serve as an evaluation tool for graduating candidates. As students approach their student teaching semester, administrators and faculty who assess student teaching experiences would benefit from having multiple points of data that indicate each student’s progress with IDEI competencies.

Second, professional schools and programs would also benefit from the proposed framework of building a culture of IDEI assessment. Like EPPs, professional schools and programs around the country are struggling with how to effectively teach and assess IDEI education within their functional area [39,40,41]. For example, Johnson-Mallard et al.’s [40] research demonstrated that the following:

Diversity among the nursing populations is critical, to reflect the society nurses serve. In 2017, only 19.5% of all registered nurses in the United States identified as racial/ethnic minorities. Medical schools continue to struggle with increasing and retaining minority students and faculty.

(p. 298)

Though the context among the different types of professional schools and programs can vary widely, implementing a culture of IDEI assessment and utilizing elements of the proposed framework would allow administrators and faculty to prepare equity-minded students to work directly with diverse communities and individuals. The framework used to develop the ILCP can be easily adapted to suit the context, accreditation standards, and missions of other types of professional schools and programs. How IDEI is connected to the program’s curriculum may need to be re-strategized. However, the research-based methods of teaching IDEI competencies and assessing those competencies, as outlined in the framework above, are useful and complementary for program design.

Third, the field of higher education, including programs and majors where the faculty and administrators may not see an obvious connection to IDEI education and competencies, stand to gain valuable insight from understanding how to implement a culture of IDEI assessment. In its earliest stages of development, the ILCP was originally intended to be a co-curricular program in which any student of any major or program could participate. As the planning progressed, the ILCP found a natural first home within the institution’s School of Education. However, the rationale behind its initial design remains valid. Universities are increasingly concerned with preparing students to engage with others across differences [42,43], and employers desire college graduates who can display a greater mastery of IDEI competencies than their peers [44,45]. Infusing curriculum design and co-curricular initiatives with a culture of IDEI assessment is necessary for higher education institutions to keep pace with demands in an increasingly diverse world.

6. Recommendations

Based on the lessons we have learned implementing the ILCP, we offer the four-step framework outlined earlier as an active tool for creating initiatives that address current IDEI issues within EPPs, embody a culture of assessment, align with accreditation standards, and can be a catalyst for satisfying diversity education requirements. In addition to the provided framework, we also make the following recommendations for EPP practitioners who may want to implement a culture of IDEI assessment:

-

Spend a significant amount of time considering how your goals for assessing IDEI outcomes within your EPP align with accreditation standards and relevant theories on teaching IDEI skills and knowledge;

-

Ensure that your team of collaborators comprises one or more experts on assessment in higher education. Implementing a culture of IDEI assessment requires practitioners to blend IDEI and assessment expertise, which can be accomplished through strategic partnerships;

-

Consider the possibilities and limitations of your program’s budget and the capacity of the staff involved. If possible, expand your budget and staff through interdepartmental partnerships;

-

Carefully select your assessment methods and when they should be implemented during a student’s journey through the EPP. Be sure to consider how cultural bias may be embedded within assessment methods, especially when assessing student learning and growth. We highly recommend having several forms of researched- based assessments spread throughout the timeline of the EPP;

-

Routinely reconvene your team of collaborators to review the data collected and to make iterative adjustments to both the EPP initiative and how you conduct assessments. Creating a culture of IDEI assessment requires a cyclical process of continuing to critically question both the effectiveness of the IDEI education and the assessment of that education;

-

Share the data, your team’s analysis, and intended changes with both students and faculty staff who are a part of the EPP. Effective IDEI education and assessment requires commitment from all parties involved;

-

Continually engage with EPP faculty so that the goals of the IDEI initiative and the student learning that occurs can be incorporated into the EPP coursework and curriculum.

7. Conclusions

It is important to recognize that the implementation of the ILCP was heavily impacted by the COVID-19 pandemic. Due to this global health crisis, administrators were forced to rapidly alter the implementation of the program to primarily rely on virtual components. Despite the radical changes to its implementation, assessment has always remained a focal point of the ILCP. This pivot highlights the need for assessment methods to be so intertwined with the program that it is impossible to disregard assessment when forced to make difficult programmatic decisions during stressful moments.

Effectively teaching and accurately assessing IDEI competencies is a new frontier for many EPPs. The IDEI assessment framework outlined in this paper and the lessons learned from implementing the ILCP will assist EPP administrators with designing successful programs. Embracing a culture of IDEI assessment encourages EPP practitioners to create multiple layers of assessment in order to continually improve the effectiveness of diversity education and, ultimately, student outcomes.