Instrument to Evaluate Intercultural Competence in Pedagogy Students

: Confronting the current educational reality involves developing intercultural competence during and after training to produce an impact on educational practices that promote positive classroom interactions with culturally and linguistically diverse students. In this context, this article presents the results of the construction and validation of the Intercultural Competence for Pedagogy Students (ICPE) instrument in a Chilean university. Taking into account the focus of interest, intercultural education in initial teacher training, the validation sample of the survey included 521 students in initial teacher training. This research aimed to establish the goodness of fit of the scale by analyzing the reliability and validity of the construct through confirmatory factor analysis. The scale presented adequate psychometric properties, contributing to the measurement of intercultural competence in initial teacher training to support attention to students in their cultural diversity.


Introduction
Globalization has intensified the communicative interaction between cultures, giving rise to the need in higher education institutions to develop the relevant skills to deal with it in students.One is intercultural competence (IC), conceptualized as a set of skills, attitudes, and behaviors necessary for positive interactions between culturally diverse speakers [1].There are several relevant theoretical models of IQ.One model traces the evolution of speakers from an ethnocentric plane of communication (exclusive use of one's own cultural frameworks to interpret other cultures) to an ethno-relative one (acceptance and adaptation of cultural diversity in a society) [2].Another model emphasizes self-awareness as a fundamental component in interaction with other cultures [3].A third model posits that IC is established as a mechanism of attitudes, knowledge, and skills to interact appropriately in intercultural situations [4].
The Pyramid Model of Intercultural Competence [5] considers the convergence of the dimensions presented in other theoretical models, such as attitude, knowledge, and ability.The latter model is shown as processual and dynamic, where the different stages of competition are communicated.The essential characteristic of this model is given by the pyramidal representation, meaning that the higher the degree of acquisition of the components of IC, the higher the degree of intercultural competence obtained.The Pyramid Model of Intercultural Competence, at its base, is composed of required attitudes, such as respect (valuing other cultures), openness (learning about other cultures without judging them), curiosity, and discovery (tolerance towards ambiguity and uncertainty).The second rung of the pyramid is shared by knowledge and understanding in conjunction with skills.Knowledge is composed of elements: cultural self-awareness, in-depth understanding and knowledge of cultures, culture-specific information, and sociolinguistic awareness.The skills component integrates them: listening, observing, interpreting, analyzing, relating, and evaluating.
The third level of the pyramid corresponds to the desired internal outcome component.Its elements are adaptability (to communication behaviors as well as to new cultural environments), flexibility (using appropriate behavioral styles with a new cultural context), ethnorelative vision (understanding of culture from the specificity of each cultural context), and empathy (ability to identify with a cultural other).The desired external outcome component, which involves behavior and communication effectively and appropriately, is at the top of the pyramid.
In the international context, culture is considered in the European Framework of Reference for Language Learning, Teaching, and Assessment [6] as an essential element for learning a foreign or second language, with IQ being a theoretical construct used for these purposes.In intercultural education, international organizations define interculturality as the equal interaction of diverse cultures through dialogue and mutual respect [7].From a critical perspective, interculturality is not accessible in education, but, rather, a colonizing, modern, white, and Western gaze is overcome.Intercultural education, then, would seek to approach cultural knowledge denied and subordinated by modernity/coloniality [8].
Following the previous line, Chilean public policies preferentially emphasize interculturality in education, focusing on indigenous peoples without recognizing other cultures in curricular and pedagogical spaces [9].This has put even more pressure on multicultural coexistence in Chile, which has been deepened by the sustained increase in the arrival of the Latin American migrant population, estimated at 1,736,691 million at the end of 2022 [10].This fact impacts the education system, generating new challenges for schools and teaching [11].
The migratory phenomenon has shown that opening borders affects cultural contact and communication, bringing with it effects on culture, manifested in social interactions where individuals or groups relate [12].This interaction is also affected by linguistic diversity, and there is a need to deal with the different languages that interact in school classrooms [13].This interaction in schools has increased in Chile, especially with the arrival of a population of Haitian origin that communicates in Creole as their mother tongue and not in Spanish, making it urgent to pay attention to the processes of linguistic integration in schools as a result of this new scenario [14].
It is also observed that cultural diversity predominates in the current era's globalized historical, social, and cultural context.This is an age where communication technologies have globalized, benefiting the teaching of various languages [15].There is, therefore, a paradigm shifts from a monolithic society to a society of cultural pluralism, which leads to the need to promote intercultural competence in teacher training.In this way, educational interactions with the immigrant population are favored [16].However, interculturality is also part of the pedagogical challenges of the 21st century, so it is necessary to develop skills in teacher training that help the coexistence of relationships [17].Consequently, it provokes the urgent requirement of pedagogical attention to culturally and linguistically diverse students in teacher training, which is relevant for integrating interculturality in a transversal way in the training curriculum in pedagogical careers [18].
However, in higher education, intercultural competence has only been studied in recent years, giving importance to its development.Numerous communication problems result from multicultural coexistence, where cultural and linguistic differences converge or different interpretations of the communicative behavior between interlocutors [19].In this subject, studies on teaching second languages predominate in initial teacher education and teaching foreign languages [20][21][22][23][24][25].There are also IC studies in professional health training [26,27].As for the instruments that measure intercultural competence, the literature mentions, for example, a scale to measure intercultural empathy [28] or the measurement of adaptation to various cultural contexts [29].In the Chilean case, based on the Pyramid Model of Intercultural Competence, a scale of cultural intelligence is translated, operationalized, and constructed to be used in pedagogy students from three Chilean regional universities [30].
Consequently, studies related to the teaching of a foreign language in contact with another culture predominate in the literature on intercultural competence in initial teacher education, showing few studies in the field of mother tongue teaching.In addition, there is a need for more instruments that measure the dimensions established by different theoretical models on intercultural competence that are valid and reliable [31], presenting a methodological gap regarding the matter.Thus, there is a need for instruments that measure this construct, guaranteeing the identification of the impact of teacher training.
Therefore, through this communication, the research team takes on the challenge of having valid instruments to measure the construct of intercultural competence.It is understood that, in the Chilean context, with the arrival of a migrant population, mainly from Latin America, teachers must possess intercultural competence to interact with culturally and linguistically diverse students.For this reason, this research seeks to construct and validate a scale of Intercultural Competence for Student Teachers (from now on, CIEP) in the context of a Chilean university.From this perspective, the research team wonders if it is possible to measure the perception of intercultural competence through this new Likert scale in pedagogy students, understanding that future teachers mediate the interaction between the different cultures present in the classroom through language.

Materials and Methods
This study uses the quantitative research method as a way to construct and validate an intercultural competence scale with a non-experimental and cross-sectional design in order to search, in a short period, for traits, factors, characteristics, and other important properties that affect the intercultural competence to be studied, given that the data are collected at a single point in time [32] using a questionnaire.

Participants
The study was carried out using a non-probabilistic convenience sample of university students (n = 521) from two pedagogical careers at a Chilean university (Pedagogy in English and Pedagogy in Basic Education), belonging to four branches of a private university with regional headquarters.In the capital of Santiago de Chile, it was applied in two offices: the San Miguel Campus (n 1 = 69) and the Providencia Campus (n 2 = 34).In Southern Chile, they were applied at the Talca (n 3 = 297) and Temuco (n 4 = 121) sites.The distribution of the sample according to sex corresponded to 450 women and 71 men.On average, the students evaluated were in semester 5.04 (SD = 2.4), with an average age of 22.2 years (SD = 3.37), with a minimum age of 18 years and a maximum of 40 years.

Instrument
The ICPE instrument was designed to contribute to the knowledge and measurement of intercultural competence in pedagogy students.This goal was established based on the Pyramid Model of Intercultural Competence [5].Four of the five components proposed by the model were considered scale components.The fifth was discarded because it was a component that pointed to the performance of the competition and, therefore, required direct observation for its evaluation.The four components selected were as follows: Attitude, made up of 15 items and associated with three conceptual categories (respect, curiosity, and openness); Knowledge and Comprehension, composed of 15 items and associated with three conceptual categories (cultural self-awareness, deep cultural knowledge, and sociolinguistic awareness); Integrated Skills with 9 items and associated with the conceptual category (knowledge processing); and, finally, Desired Internal Results, made up of 15 items and associated with four conceptual categories (integrated with adaptability, ethnorelative perspective, empathy, and flexibility).

Procedure
The research was carried out in five phases.The first phase (a) corresponds to the revision of the existing theoretical models and instruments for collecting information on IQ, an issue incorporated in other publications.The subsequent phases are phases (b), the construction of sentences based on the dimensions of the selected and operationalized theoretical model, (c) validation of content by expert judgment, and (d) analysis of the validity of the theoretical model of the instrument.
In the stage of content validation by expert judgment, a Likert scale [33] was designed with 54 items in the format of a polyatomic response with three categories: Disagree, Agree, and Strongly Agree.The literature considers these categories sufficient and relevant [34].The scale was subjected to content validation by four highly experienced experts who evaluated the suitability of the sentences based on their extensive experience and knowledge in the area.To this end, the inter-judge agreement calculated with Lawshe's strategy was used, which proposes the evaluation of the relevance of each item through two categories: "essential" and "non-essential".Although the criteria analyzed by the judges were wording, content, congruence, and relevance, only the latter was used to calculate the CVR.Subsequently, qualitative data (comments) and quantitative data (QOL) were triangulated for the definitive retention of items [35].
The questionnaire was administered to students in their classrooms, with prior authorization from the institution's academic units, and it was approved by the university's institutional ethics committee.Informed consent was obtained from all participants, informing them that their response was voluntary and respecting their rights to privacy.

Data Analysis
The confirmatory analysis of the data provided by the application of the scale sought to statistically establish the unidimensionality of the components of the construct to be measured through the factorial loads of each sentence with the WLSMV (Weighted Least Squares Mean and Variance-Adjusted) method, for ordinal rating scales [36] and, in turn, to investigate the relationship between the general construct and the components defined by the CFA to analyze the suitability of the model The comparative fit index (CFI), the Tucker-Lewis index (TLI), the normalized Chi-square index (χ2/d.f.), the goodness-of-fit index through the mean square error of approximation (RMSEA), the convergent validity of the components of the model through the extracted variance (ρvc), and, finally, the reliability through the Jöreskog rho index.The statistical processing of the data was carried out using the R project software, version 4.0.0., with Yve Rouessell's Iavaan package.

Instrument
The content analysis considered the level of agreement among the panelists through the CVR index-Lawshe's new strategy for calculating CVR [37,38].It also considered the expert opinions' coding, considering each item's relevance criterion [39].Finally, the results obtained from the CVR and the judges' comments were triangulated to eliminate the items that did not present convergence.
Once the initial items were defined, a confirmatory factor analysis (from now on, CFA) was developed to establish the validity of the scale construct.In the confirmatory analysis, the reliability of the internal consistency is checked by the Jöreskog coefficient [40], which integrates the errors into its calculation.The use of TFA is justified by the need to evaluate the fit of the data to a specific factor structure previously hypothesized [36,41].For this purpose, the WLSMV (Weighted Least Squares) estimator was used.Adjusted mean and variance for categorical data fit non-normal distributions, such as that presented by the study data.
The factorial loads are observed to evaluate the proposed theoretical model, which must be significant according to the Wald test; the adjustment statistics of the comparative Bentler-Bonett index (CFI) and the Tucker-Lewis index (TLI) must be greater than 0.9, a criterion used by [42,43] and the mean square error of approximation (RMSEA) (acceptable if it is less than 0.08 and good if it is less than 0.05), the Joreskog index for the reliability of each verified construct (more significant than 0.7), and the extracted variance index (greater than 0.5) [44].Using the extracted variance's rho allows us to determine if the indicators measure the associated construct.The acceptance criterion for the index of variance extracted from a construct is defined as 0.5 as the minimum value, meaning that the construct shares more than half of its variance with its indicators [45].

Content Validation
Regarding the content validation of the items of the study in question, it is possible to affirm that the Content Validity Ratio (CVR) of the Lawshe-Adjusted Model index 125 out of 131, which exceeded the value of 0.71.On the other hand, the value of the CVI calculated for the complete instrument was 0.77, which is considered acceptable, for which the critical value of the CVI is 0.7.The result of this statistical analysis resulted in the elimination of six items.After Lawshe's analysis, according to the analysis of the judge's comments, the suggestions and observations of the content of each item [46] were coded.This qualitative analysis revealed some items as redundant or complementary, adjusting the instrument by eliminating 71 items.The information obtained from the data triangulation finally allowed us to maintain 54 items that made up the instrument applied to the CFA sample.

Results of the Instrument Applied
Below are the characteristics of the instrument applied and its components to be reviewed through the CFA (Table 1).After the CFA, the attitude dimension, initially composed of 15 items, was reduced to 8 items after the factor contribution analysis.The items eliminated included the following: item 2, "It is legitimate to ignore people of different cultures, races, and nationalities than mine", item 38, "I appreciate that people of different cultures, races, and nationalities than mine have different rights than I do", and item 41 "I like to always be right when I am in a communicative situation with people from another culture".
The reduced items show an adequate fit according to the reported statistics (χ2 = 127.8,d.f.= 43, χ2/d.f.= 2.9; p-value = 0.0; TLI = 0.988, CFI = 0.981, extracted rho variance = 0.51, RMSEA = 0.062, rho_Jöreskog = 0.89) (Table 2).explains part of the variation in the concept of intercultural competence with the measure of error.Finally, convergent validity greater than 0.80 is considered very good, while RMSEA is acceptable when it varies between 0.05 and 0.08 [45].As shown in Table 7, in order to analyze the complete model, the reliability of the constructs was calculated with two indices: the coefficient of determination of individual reliability (CFI) and the composite reliability (Jöreskog's rho).As the table indicates, the individual reliability of each of the items is significant since the value is close to 1. Regarding composite reliability, the value of Jöreskog's rho index (0.94) indicates that each set of items explains part of the variation in the concept of intercultural competence with the measure of error.Finally, convergent validity greater than 0.80 is considered very good, while RMSEA is acceptable when it varies between 0.05 and 0.08 [45].Finally, Figure 1 shows a graphical representation of the proposed model for intercultural competence where the factor loads of IC are expressed as a theoretical construct in the components of the initial theoretical model.In this sense, IQ is strongly explained by the components of Knowledge and Internal Results, followed by the attitudinal component and the Skills component, respectively.This seems to coincide with studies in the area where it is found that the greatest challenge for institutions that train professionals is to develop skills associated with IC, as opposed to learning some notions related to the understanding of IC, which is the first thing that students in vocational training reach [47].

Discussion
This study sought to communicate the process of construction and validation of a measurement instrument for intercultural competence in university teacher training Finally, Figure 1 shows a graphical representation of the proposed model for intercultural competence where the factor loads of IC are expressed as a theoretical construct in the components of the initial theoretical model.In this sense, IQ is strongly explained by the components of Knowledge and Internal Results, followed by the attitudinal component and the Skills component, respectively.This seems to coincide with studies in the area where it is found that the greatest challenge for institutions that train professionals is to develop skills associated with IC, as opposed to learning some notions related to the understanding of IC, which is the first thing that students in vocational training reach [47].

Discussion
This study sought to communicate the process of construction and validation of a measurement instrument for intercultural competence in university teacher training based on the Pyramid Model of Intercultural Competence [5] for the design and construction of the items, unlike other assessment instruments that feature only some dimensions of theoretical models [28][29][30].The instrument is a Likert scale that allows four dimensions of the chosen theoretical model to be measured, given that studies have pointed out that it is essential to integrate interculturality in higher education transversally [18].Therefore, this instrument contributes to the purpose of the evaluation.
The instrument proved valid and reliable for the purposes for which it was created.It should be noted that the validation of the instrument's content is used through the inter-judge agreement, calculated with Lawshe's proposal, ending with the triangulation of this analysis with the categorization of the qualitative comments made by the judges on the instrument.According to Romero et al. [38], such decisions made it possible to gather a broader spectrum of information beyond pertinence and relevance to interpret the experts' answers later and understand their frame of reference, theoretically and representationally.
In this sense, the values obtained from the CVI above 0.70 are consistent with studies that applied these indices in content validation, such as that of Zamora de Ortiz et al. [48].Confirmatory factor analysis was used for the instrument's construct validity, where the four theoretical dimensions of the model [5] included in this study were evaluated: Attitude, Knowledge and Understanding, Skills, and Desired Internal Outcomes.
The results show that when applying the CFA, the instrument fits the model hypothesized in the first instance [49].The model validated in this research allows us to affirm the composition of the construct in four dimensions present in the Intercultural Competence for Pedagogy Students (ICPS) model.This fact corroborates the initial underlying structure ratified by analyzing the factor loads of the analyzed dimensions (between 0.84 and 0.94), with items that present a high factor load.The literature shows that applying the CFA to validate the construct with which the instrument was constructed is recurrently used in education [50][51][52].
This goodness of fit makes it possible to define an instrument with a consistent factorial structure and items that adequately measure intercultural competence in pedagogy students.Thus, the final instrument consisted of 32 items, eliminating all items representing the theoretical concept of openness.This fact evidences the appropriate fit, given that this dimension presented a higher factorial score of 0.3.It can be interpreted that the items related to this theoretical concept would not discriminate, probably due to the precarious teaching in university education about the intercultural pedagogical actions received by the members of the sample or as a result of assimilationist and monocultural models in the university curriculum, such as the Chilean national school curriculum [53].
The goodness of fit between the CFA-derived and hypothetical models may vary depending on the sample size [48].The consulted literature indicates that a sample of 500 people is considered very good [54].Therefore, the adequate goodness of fit presented by the instrument after applying the TFA can be interpreted based on the size of the study sample, which is 521 students.In other words, the sample size considered more than 500 people, which influenced the adequate goodness of fit of the instrument.
However, even though the sample is considered very good, the researchers note limitations in the methodological aspects.Regarding reliability, Jöreskog's rho could have been complemented with the Test-Retest strategy, which this study did not consider as a methodological procedure due to the scope of the research objective.However, it is hoped to respond to these challenges in future studies by applying the instrument to students from other careers that could be included in a second stage.
Even with the above, the results of the construction and validation of the instrument allow us to affirm that the instrument is adequate to measure intercultural competence in students of initial teacher training in higher education.Its contribution lies in constructing and validating an instrument that contributes explicitly to teacher training, considering the diversity in the Chilean school classroom that teachers in training must face.In addition, it is observed that most instruments that assess intercultural competence are based on something other than the totality of the theoretical dimensions of the theoretical models on the subject.On the contrary, they include one or two theoretical dimensions for their understanding.
On the other hand, as a projection of the study, the instrument allows for exploration of the development of intercultural competence in initial teacher training.For example, the theoretical dimensions of the instrument can be worked on didactically in a transversal way in the students' professional practices.In that learning space, like internships, preservice teachers find themselves in classrooms with culturally and linguistically diverse students.Therefore, using this instrument makes it possible to project teacher training in intercultural competence.

Conclusions
In the first place, the objective set in the research is considered to have been achieved because it validates the theoretical decision to incorporate four dimensions associated with the proposed Pyramid Model of Intercultural Competence and the subsequent construction of the instrument.This makes it possible for the components to explain the construct of intercultural competence, given that all dimensions have excellent goodness of fit, with significant reliability and perfect convergence.
It is also essential to have been able to design and validate a model to measure intercultural competence with these characteristics and with a sample considered to be very good.The instrument has the value of offering educational teacher training to measure intercultural competence through diagnostic strategies.At the same time, students can be pedagogically trained through methodological devices associated with the validated dimensions to serve culturally and linguistically diverse students in the Chilean context.
Consequently, the results of this research open up a series of future investigations that allow us to work with the validated dimensions to access genuine reflections that deconstruct beliefs and judgments and enable professional competencies that strengthen professional development and interculturality in the classroom, in teachers in training and also in active practice.

Figure 1
Figure1shows a graphical representation of the proposed model for intercultural competence.

Figure 1
Figure1shows a graphical representation of the proposed model for intercultural competence.

Figure 1 .
Figure 1.Proposed measurement model for IC.

Figure 1 .
Figure 1.Proposed measurement model for IC.

Table 1 .
Theoretical dimensions, conceptual categories, and items retained by content analysis.

Table 7 .
Summary of the proposed model.

Table 7 .
Summary of the proposed model.