A Preliminary Content Validity Analysis of the Receptive and Expressive Language Pre-Referral Protocol for Bilingual Learners (RELPP-BL)

Abstract

The number of bilingual-bicultural students in the US continues to grow exponentially. With this growth, educators have an increased need for ensuring that all bilingual-bicultural students have equal language learning opportunities. It is, therefore, crucial that bilingual educators have access to valid tools that can serve as guides for determining if a speech and language referral is needed. The Receptive and Expressive Language Pre-Referral Protocol for Bilingual Learners [RELPP-BL] was developed as a data gathering tool to assist educators in the decision-making process. The purpose of this study was to explore the content validity of the RELPP-BL. The preliminary results demonstrate that the RELPP-BL is a viable, valid tool for use in the pre-referral process; it is not intended as an evaluation measure.

1. Introduction

The US is becoming more culturally and linguistically diverse (PEW Research Center 2015, 2019), yet our educational practices appear to be inequitable in serving linguistically diverse populations. According to the PEW Research Center (2013):

Spanish is, by far, the most spoken non-English language in the U.S. The next most spoken non-English languages are Chinese (with 2.8 million speakers), Hindi, Urdu or other Indic languages (2.2 million), French or French Creole (2.1 million), and Tagalog (1.7 million).

(2nd paragraph)

Within the Latino population, the majority are bilingual (PEW Research Center 2015) and native-born (PEW Research Center 2019). The number of young Latinos under the age of 18 in the country increased by 22% between 2006 and 2016, and they represent 25% of the current school population (PEW Research Center 2018).

Given the diverse representation of US students, a great concern is that a disproportionate number of racial-, ethnic-, and language-minoritized learners continue to be overidentified as having a disability, which results in their placement in special education programs (Yates et al. 1998; Artiles 2003; Ortíz et al. 2011, 2020). Two compounding issues that impact the referral process are: (1) the paucity of research on bilingual learners referred for special needs; and (2) the limited research-based tools that are culturally and linguistically sensitive for assisting educators in bilingual learners’ proper identification.

Foremost, limited research specifically examines the number of English learners (ELs) or emergent bilinguals (EBs) who are misdiagnosed for speech and language services; trends for over- and under-referral for bilingual learners have been noted (Bedore and Peña 2008; Kraemer and Fabiano-Smith 2017; Ortíz et al. 2020). Specifically, nearly 77% (n = 44) of emergent bilingual (EB) children (i.e., English learners) referred for special education services including speech and language were misdiagnosed (Ortíz et al. 2011, 2020). As Ortíz et al. summarized, “Inaccurate identification and/or classification has serious consequences for EBs, including lowering teachers’ expectations for student performance, regardless of their actual academic performance (Kangas 2020) and limiting their long-term social, academic, and economic opportunities” (Kangas 2020, p. 246).

Secondly, in reference to the need for culturally and linguistically sensitive approaches, Bedore and Peña (2008) stated that educators need information that helps them distinguish between typical language development and language disorders. In recognizing the need for speech-language pathologists (SLPs) to make evidenced-based decisions, they suggested the following:

Development of appropriate assessment tools and practices for identification of LI [First language] in bilingual children should use comparison to appropriate normative groups (e.g., bilingual children), development of appropriate targets for the language (e.g., markers of LI), and consideration of the way that two languages might interact or influence each other. Attention to these three points for test development is an initial start in improving assessment practices for bilingual children.

(Bedore and Peña 2008, p. 20)

We concur with these recommendations and suggest that the paucity of appropriate tools and information affects not only SLPs, but also bilingual classroom educators. Typically, educators are the ones who initiate referrals because of the concerns they observe daily (Klingner and Harry 2006; Prezas and Jo 2017). Thus, their role is critical in the referral process. Therefore, we propose that there is a need for the development of tools for bilingual educators to assist them in their pre-referral decision-making when they are concerned about a bilingual learner’s language development.

We must recognize that, in the case of bicultural-bilingual learners, determining atypical from typical language development is a complex process for educators and SLPs. Bilingual learners with typical language development demonstrate age-appropriate language skills, whereas bilingual learners with differential developmental language profiles or developmental delays may be diagnosed as having a language disorder. Regardless of whether the bilingual learner has acquired the second language simultaneously or sequentially, typical and atypical bilingual language development share commonalities (Flores et al. 2022a; Bedore and Peña 2008; Prezas and Jo 2017). For example, in the case of sequential bilinguals, there may be a silent period in which the language learner is unable to express their thoughts in the second language (Krashen 1981, 1982). Similarly, learners experiencing language delays can present with limited capacity to express themselves. Herein lies the quandary, in that bilingual learners in the silent period could erroneously be identified as having a language delay, when in fact they are engaged in the acquisition of the second language. Another commonality between bilingual learners and individuals who present with a language disorder is the use of vocabulary and word order (Ortíz and Maldonado-Colón 1986; Roseberry-McKibbin 2008; Willig and Greenberg 1986). Often, there is an assumption that students having difficulty in expressing themselves or understanding words is reflective of a language disorder. Prezas and Jo (2017) conceded that not understanding bilingual language development could result in the improper identification and diagnosis of bilingual learners who are in the process of acquiring a second language.

1.1. Rationale

While there is research (e.g., Ortíz and Maldonado-Colón 1986; Roseberry-McKibbin 2008; Willig and Greenberg 1986) that discusses language similarities and differences between typical and atypical bilingual language development, there appears to be a disconnect between research and implementation-based practices (Flores et al. 2022a). As aforementioned, these findings have not led to the development of a variety or plethora of culturally and linguistically sensitive tools for SLPs or for bilingual educators. This may be due to assumptions that research findings will be read and applied into daily practice. Further, it is estimated that it takes seventeen years for research to be implemented into practice (Douglas and Burshnic 2019).

It is also important to consider sociocultural differences in designing tools/instruments that include bilingual learners’ language acquisition history within the home and school (Restrepo 1998; Roseberry-McKibbin et al. 2005). Early schooling experiences may also impact children’s vocabulary acquisition. As Bedore and Peña (2008) surmised, “Children’s specific sociocultural experiences additionally influence the kinds of words they learn” (p. 5).

It is also essential that bilingual educators realize that bilingual learners who are diagnosed with a language disorder will struggle in both languages. Given the critical role that bilingual educators play in supporting bilingual learners’ language development, they must demonstrate specialized knowledge and skills. These include: (1) an understanding of typical language versus atypical language development; and (2) the knowledge and skills to identify the need for further referral and evaluation (Prezas and Jo 2017). Furthermore, it is imperative that bilingual educators have tools to make data-driven decisions, such as during the pre-referral and referral stages when learners are being considered for special services. The pre-referral stage serves as an initial step in gathering information about the learner; for example, when the educator has concerns regarding a bilingual learner’s language skills. Further, the data gathered during the pre-referral stage serves as evidence that can then be used during the referral process, such as during the Response to Intervention (RTI) or Multi-Tiered System of Support (MTSS) meeting. In the US, RTI and MTSS are intervention processes that provide learners with equitable educational opportunities in the least restrictive environment, usually in the general education classroom. During RTI or MTSS, educators and specialists meet to discuss the learning potential of the student and identify possible goals to address the referring educator’s concerns. Rather than only relying on standardized measures, which may not accurately reflect bilingual learners’ language abilities, we suggest that a pre-referral tool specifically designed for bilingual learners allows for an equitable decision-making process (Castilla-Earls et al. 2020).

Thus, to guide bilingual educators during the pre-referral stage in making decisions and reduce the likelihood of inappropriate referrals, the Receptive and Expressive Language Pre-Referral Protocol for Bilingual Learners (RELPP-BL) was developed (Flores et al. 2022b). One of the drawbacks of RTI and MTSS is that there is often a delay in diagnosis because of the lack of evidence; thus, a pre-referral tool such as the RELPP-BL would ensure a timely referral process. The purpose of this paper is to provide an overview of the RELPP-BL as a pre-referral tool specifically for bilingual learners, explain how it was developed, and examine its validity as a pre-referral tool.

1.2. Review of Literature

Prior to developing the RELPP-BL, Flores et al. (2022b) noted that there was a dearth of educator tools and resources that could be used in referring bilingual learners when there are language concerns. We observed that the American Speech-Language Hearing Association (ASHA) and Texas Speech-Language Hearing Association (TSHA), while providing eligibility guidelines and informational materials for SLPs, offer no specific referral guidance for educators. The best practice prior to making a referral is to gather information about the learners’ language skills. According to VanDerHeyden et al. (2003), the pre-referral process is often subjective and based on the educator’s perceptions. Therefore, there is a need for a pre-referral tool in which objective data is collected in order to facilitate a referral for services.

For this review, we conducted a search for tools that could be employed in the pre-referral of bilingual learners in which language concerns were present. We limited our search to instruments/tools that included gathering information from educators and/or parents. We also looked for any comprehensive review on available tools, specifically for use with bilingual learners. As a result of our search, we found only instruments that are used primarily by SLPs. These instruments will be presented in the subsequent paragraphs and will be examined regarding their utility for bilingual classroom educators.

In their comprehensive review, Pratt et al. (2020) examined available questionnaires used to screen for language disorders in bilingual learners. Their review included a variety of tools: Ages and Stages Questionnaire- Spanish ([Spanish ASQ], Squires et al. 1999); Desarrollo de Habilidades Comunicativas Palabras y Enunciados ([INV-II], Jackson-Maldonado et al. 2003); Alberta Language Development Questionnaire ([ALDeQ], Paradis et al. 2010); Parent Report of Speech and Language Problems ([PRSLP], Restrepo 1998); and the Bilingual Language Assessment Battery ([BLAB], Pua et al. 2017). The Spanish ASQ, INV-II, ALDeQ, and PRSLP engage parents as reporters of children’s information, whereas the BLAB utilizes both parents and educators. Overall, Pratt et al. (2020) concluded that these instruments lacked the sensitivity required to identify bilingual learners with language disorders. If an instrument cannot clearly measure the constructs or identify a language disorder, then the validity is in question as it does not serve the intended purpose and, thus, impacts accurate assessment.

One instrument we found in the literature that is widely used among SLPs for assessing Spanish-English bilingual learners is the Bilingual English-Spanish Assessment ([BESA], Peña et al. 2014, 2018). The BESA assists in determining phonological and language impairments and serves to guide the speech-language pathologist (SLP) in assessment decision-making. The BESA incorporates the use of two questionnaires, the Bilingual Input-Output Survey (BIOS) and Inventory to Assess Language Knowledge (ITALK) to gather information from parents and educators regarding English and Spanish language exposure and usage.

The BIOS seeks input from parents and educators about the language exposure of the bilingual learner (Peña et al. 2014, 2018). Demographic data is gathered, such as the child’s grade level, date of birth, and age. The SLP also documents whether the interviewee is a parent or educator. BIOS allows for questions or prompts to be asked in either English or Spanish. Detailed information about the child’s language exposure and use at home and school is gathered.

The ITALK is a parent and educator questionnaire, with the interview being conducted in English or Spanish by the SLP (Peña et al. 2014, 2018). Parents and educators are asked about the use of the bilingual child’s language proficiency for each of the five language domains: language comprehension, grammar, phonology, sentence production, and vocabulary (Pratt et al. 2020). In Pratt et al.’s (2020) study, they observed the importance of gathering data from multiple sources, such as parents and educators, that can be used as part of the SLP’s assessment and decision-making process. In addition to the BESA, sources of evidence can include questionnaires, bilingual language and speech samples, learning potential evaluation (dynamic assessment and RTI), and standardized testing (Castilla-Earls et al. 2020).

In sum, there are limitations in the instruments that we reviewed. As a result of Pratt et al.’s (2020) extensive review, it was surmised that some instruments are questionable for SLP’s use and, hence, would not assist the bilingual classroom educator in the pre-referral or referral process. Moreover, while the BESA—including the BIOS and ITALK questionnaires—are excellent tools for SLPs, these are not geared towards guiding bilingual educators in their referral decision-making. Lastly, the BESA is limited to children ages 4–6, and when we consider sequential bilinguals, the acquisition of the second language may occur at a later age. Another concern is the limited availability of instruments that can be used for the specific purpose of referring bilingual learners who the educator observes as potentially having a language disorder. This brief review supports the need for a valid instrument that can be used by bilingual educators in the pre-referral process.

2. Materials and Methods

In this preliminary study, we explored the content validity of the RELPP-BL (Flores et al. 2022b). Generally, there are different types of validity that are considered in assessing a measure: content, construct, and criterion. Messick (1998) proposed that, in all assessments of measures, we must consider consequential validity. Some researchers have argued for a unitary theory of validity, yet Messick (1998) suggested that all these types of validity are “complementary forms of evidence to be integrated into an overall judgment of construct validity” (p. 37).

Of import, some researchers consider content validity to include both face and logical validity (Rubio et al. 2003); however, the approaches to assess face versus content validity are distinct. As opposed to face validity, in which individuals simply review a measure, content validity requires the use of a specific protocol and process to evaluate the validity of the measure (Rubio et al. 2003). Content validity refers to the extent to which the sample items in a measure are representative of the entire theoretical construct that is being assessed (Rubio et al. 2003; Polit et al. 2007; Zamanzadeh et al. 2015). Construct validity assists in determining if the hypothesized construct(s) are observed in the scale as a result of a factor analysis (Thompson and Daniel 1996). Criterion validity demonstrates the relationship between a measure and a criterion (Salkind 2006); for example, how well the RELPP-BL predicts a bilingual educator’s referral decision. Consequential validity extends the conversation, in that a study of the RELPP-BL would examine the social consequences of bilingual referral decisions. Given that the intent was to do a preliminary validity analysis, our inquiry was based on the following research questions:

• What is the content validity of the RELPP-BL subscales?
• What is the overall scale content validity of the RELPP-BL?

In the subsequent section, we will provide a description of the tool, its development, and the process for analyzing the content validity of both the RELPP-BL subscales and the overall scale.

2.1. Measure

The RELPP-BL was developed to assist bilingual educators in determining the need for a referral for bilingual students (Flores et al. 2022b). The primary developers are four SLPs who have worked at six different school districts across a multicultural/multilingual city in south central Texas; in addition, a bilingual education researcher also assisted in the development of the tool. According to the US Census Bureau (2019b), 64.2% of the population in this city identify as Hispanic or Latino, 24.7% as White, 7% as Black or African American, and 2.8% as Asian. Additionally, 42.8% of the population 5 years or older report speaking a language other than English at home. It is also estimated that 17.8% of this population live in poverty.

As previously noted, the RELPP-BL is specifically designed as a guide that is completed by a referring bilingual educator who has concerns about a bilingual learner’s language skills. While the RELPP-BL does not provide scores or an index of risk, its intent is to be used as a complementary tool to gather objective data. The data can assist the educator and the referral committee in making informed decisions as to whether a bilingual learner should be referred for further speech and language assessment. Importantly, the RELPP-BL should not take the place of the district/campus referral process.

2.1.1. RELPP-BL Development

In developing the protocol, Flores et al. (2022b) followed these steps: (1) content domain determination, (2) item generation, and (3) protocol formation (Almanasreh et al. 2019; see also Zamanzadeh et al. 2015). In identifying the theoretical constructs, Flores et al. (2022b) considered ASHA’s (2020b) recommendations and followed their definition of language disorders:

A language disorder is impaired comprehension and/or use of spoken, written and/or other symbol systems. The disorder may involve (1) the form of language (phonology, morphology, syntax), (2) the content of language (semantics), and/or (3) the function of language in communication (pragmatics) in any combination.

(https://www.asha.org/policy/rp1993-00208/ (accessed on 15 June 2020))

Additionally, as recommended by ASHA (2020a), Flores et al. (2022b) incorporated an ethnographic interview and considered the following multi-cultural/linguistic characteristics in the development of the protocol to ensure that the data collected are reflective and relevant to the bilingual learner’s cultural and linguistic background. For example:

• Languages: 22% of Americans speak a language other than English (US Census Bureau 2019a) and 22% of US students are bilingual (PEW Research Center 2015).
• Dialects spoken: There are varieties of dialects within certain languages (e.g., Spanish-Caribbean, Mexican, Spanish/Castilian).
• Gender roles: Some ethnic groups have different roles for boys and girls.
• Generational Status: Country of birth and how long the student has been in the country.
• Exposure: Education/play groups which provide language opportunities outside the home.

Further, to ground the protocol within the research literature, Flores et al. (2022b) used Prezas and Jo’s (2017) recommendations for determining the characteristics of English language learners (ELs) who present with a language concern (see

Table 1. Some Initial Considerations for Suspected Language Disorder.

1. Whether parent(s)/caregiver(s) have concerns 2. Languages spoken by the student/dialects 3. Language use (both at school and at home) 4. Proficiency in both languages 5. Whether concern is present in both languages 6. Age of child 7. Age of US public school exposure/enrollment 8. School program (e.g., Dual-Language, English immersion).

Prezas and Jo (2017, p. 10).

). Importantly, these initial considerations do not imply that an EL or bilingual learner presents with a language disorder.

As a result of the review of literature and Flores et al.’s (2022b) professional experience, the following six constructs were identified and developed into subscales: ethnographic interview and parent/caregiver language background, student profile, language exposure, language use, student educational background, and language checklist.

2.1.2. RELPP-BL Process

In generating the items for the RELPP-BL, Flores et al. (2022b) considered the aforementioned research recommendations. Flores et al. also examined various school district referral packets and guidelines. Prior et al.’s (2011) pre-validation process for screening instruments was also followed: (1) identify various items, (2) cluster items based on the identified theoretical construct, and (3) review items for inclusion. Duplicate and non-relevant items were removed. A final review of the wording was completed to ensure consistency and relevancy.

2.1.3. RELPP-BL Description

The RELPP-BL is a comprehensive pre-referral tool that a bilingual educator can use to document detailed information about a bilingual learner’s language skills. As aforementioned, the tool consists of six subscales based on the constructs identified in the literature (Castilla et al. 2009; Castilla-Earls et al. 2020; Pratt et al. 2020). It is vital to use converging evidence when gathering information about bilingual learners so that appropriate and objective pre-referral decisions can be made (Castilla-Earls et al. 2020). This converging evidence will also assist the SLP in making diagnostic decisions if a referral is made (Chabon et al. 2010). In addition, language data gathering recommendations are provided (See Flores et al. 2022b for link to RELPP-BL). Each of these subscales are described in the subsequent sections.

Ethnographic Interview and Parent/Caregiver Language Background

The ethnographic interview includes ten open-ended questions that collect demographic and background information about the bilingual learner and their communicative performance, needs, and concerns from the parent/caregiver. As is recommended in the literature (ASHA 2020a; Paradis et al. 2010; Pratt et al. 2020), gathering this type of data provides information about the bilingual learner’s first language (L1) and/or second language (L2) development within the home. Sample questions in this subscale include:

Tell me about how your child communicates his/her needs?

What happens if your child cannot communicate with others?

The parent/caregiver language background includes a table that aims to gather information about parent/caregiver language use in the community as well as with children and adults in the home. Paradis et al. (2010) stated that parent reports are useful in determining bilingual learners’ emerging language abilities. The educator uses the category prompts to gather information from the parents and/or caregivers and then completes the table.

Table 2. Parent and Caregiver Profile.

Parent/Caregiver Language Profile
Relationship to child	Dominant language	Country of birth	Languages used in community	Languages used with other adults in home	Languages used with children in home	% of time spent with student per week

illustrates the sample items in the Parent and Caregiver Profile subscale:

Parent/Caregiver Language Profile Sample Prompts:

What is parent/caregiver dominant language?

What language is used in the home?

Student Language Profile

The student language profile includes eight open-ended questions that collect the student’s language history and development. Collecting a student language profile is a common practice in the speech and language field (ASHA 2015; Restrepo 1998) and is a practice that educators should also use. This information can help identify the acquisition of languages over time and help understand the language abilities of the student. Sample prompts include:

What are the languages that the student has been exposed to?

At what age was the student exposed to each and for how long was the exposure?

Language Exposure and Language Use

These two rating subscales aid in the collection of the language experiences of the student in different settings and scenarios. For each subscale, the educator rates (e.g., not at all, some of the time, most of the time, and all of the time) the amount of exposure or use for each language. Research indicates that language exposure is associated with language comprehension, and language use is associated with the opportunity to interact using the language. Language exposure and language use can have an impact on language performance (Peña et al. 2021; Prezas and Jo 2017).

Table 3. Language Exposure.

What is the Student’s Recent Language Exposure in the Following Scenarios?
Check the amount of time for each language in a typical week									Note any special circumstances
Language	English				Other: ________
	Not at all	Some of the time	Most of the time	All of the time	Not at all	Some of the time	Most of the time	All of the time

presents a Language Exposure subscale sample item.

Table 4. Language Use.

What is the Student’s Recent Language Use in the Following Scenarios?
Check the amount of time for each language in a typical week									Note any special circumstances
Language	English				Other: ________
	Not at all	Some of the time	Most of the time	All of the time	Not at all	Some of the time	Most of the time	All of the time

illustrates a sample item for the Language Use subscale.

Educational Background

According to many researchers (Bedore and Peña 2008; Restrepo 1998; Roseberry-McKibbin et al. 2005), when a bilingual learner presents with a language concern, it is important to gather educational history. The educational background subscale includes a checklist about the educational bilingual program or English as second language model and open-ended questions about the classroom’s language(s) of instruction as well as the language used with the bilingual learner. It is important to gather information about program models because subtractive models such as English as a Second Language (ESL) or transitional bilingual education can lead to language loss, whereas additive models such as dual language allow children to become bilingual and biliterate (Huang et al. 2022). Sample prompts include:

Language used by the teacher to communicate with student

Languages used by student to respond/communicate with teacher

Table 5. Bilingual Program and English as a Second Language Model and Type.

Program Model & Type
ESL1	Immersion	Pull-out
Transitional BE2	Early-exit	Late-exit
Dual Language3	One-way	Two-way
Other:	_____________

illustrates Educational Background sample items:

Language Checklist

Lastly, the language checklist subscale provides an opportunity for bilingual educators to identify language milestones exhibited by the bilingual student in either language. Gathering a student’s language milestones assists in gauging their language abilities (ASHA 2015; Prath 2016) and may reveal language loss (Paradis et al. 2010). This subscale is composed of a checklist of receptive and expressive developmental milestones listed by grade level.

Table 6. Language Checklist.

Grade Level	PK/K
Receptive Language (listening)	• Follow 1–2 step directions
	• Listen to and understand stories
	• Follow simple conversation

presents a sample item from the Language Checklist subscale:

Language Data Gathering Tools

The language data gathering tools section provides recommendations for obtaining language samples using an inventory and/or play-based approach. Using appropriate approaches and tools allow the bilingual educator to observe the student in a natural/authentic context (Kelly-Vance and Ryalls 2008). 

Table 7. Language Data Gathering Tools.

Language Data Gathering Tools:
a. Language Sample Inventory
A language sample can provide insightful information regarding the child’s language usage and understanding
Tip: It is helpful to ask parents/guardians for areas of interest (favorite toys, favorite books, favorite games, favorite show, etc.)
Tip: Engage the child in conversation through child-led play
Tip: Record the interaction
Tip: Obtain information over various situations (e.g., classroom, playground, lunch, art, music, etc.)

presents Language Data Gathering Tools sample recommendations.

3. Results

3.1. Content Validity

The first six aforementioned subscales were evaluated for content validity; the language data gathering tools section was excluded as the intention is for it to serve as a resource for educators. Initially in examining the content validity of a scale, it is important to ensure that the content domain is being measured (Polit et al. 2007). In the subsequent section, we will describe the development of the content validity tool and the process for obtaining content validity for the RELPP-BL.

3.2. Development of the Content Validity Tool

In Qualtrics, a content validity tool was created, as recommended by Rubio et al. (2003), to assess the relevancy and comprehensiveness of each of the six subscales. To develop the response items to evaluate the content validity of each of the subscales, we used an iterative process. Likert and open-ended items were developed, reviewed, and revised to ensure that accurate data would be collected. To rate the relevancy and comprehensiveness of each of the protocol subscales, the following Likert scale was used: 1—Not at all, 2—Little, 3—Much, or 4—Very. The relevancy of the subscales was evaluated in terms of how relevant the scales were for the purpose of collecting data about bilingual learners’ language skills. The comprehensiveness of the subscales was evaluated on the basis of how the scales addressed concerns an educator may have about a bilingual learner’s language skills. Additionally, the content validity included open-ended prompts that asked for suggestions on additional question items that needed to be added or deleted and to provide any other additional feedback to make this protocol a useful resource for bilingual educators.

3.3. Content Validity Process

As compared to face validity, establishing content validity is considered a rigorous process. Rubio and colleagues (2003) stated that using a panel of experts “provides constructive feedback about the quality of the newly developed measure and objective criteria with which to evaluate each item” (p. 95). We conducted the content validity with bilingual classroom educators.

Content Validity Study

The first author disseminated the content validity tool to a panel of experts that included bilingual educators enrolled in a bilingual assessment graduate class taught by the third author at a university in south central Texas. Since this protocol was specifically developed for bilingual educators, they were invited to participate in the content validity study. A total of eleven (n = 11) participants responded to the survey, but only eight (n = 8) met the criteria to participate in the study. The other three respondents were eliminated because they were not bilingual educators. The number of experts does meet guidelines that indicate a minimum of two experts can be used; however, it is preferable to have at least five to eight experts in order to reduce chance agreement. The number of experts also determines the acceptable Content Validity Index ([CVI], Rubio et al. 2003; Polit et al. 2007; Yusoff 2019).

According to various researchers (Guion 1977; Lawshe 1975; Rubio et al. 2003), in selecting experts, they should have experience in the content being assessed as needed to engage in the content validity process. Since the RELPP-BL is to be used by educators and there is no specific language disorder domain knowledge needed, eight bilingual educators were selected to be experts. They were experienced bilingual educators and in their final semester of their graduate work in bicultural-bilingual studies. These bilingual educators had the knowledge, experience, and familiarity with the constructs presented in the RELPP-BL.

The first author asked the panel of experts to examine and rate each of the RELPP-BL subscales’ (1) relevancy and (2) comprehensiveness by completing the content validity tool. Additionally, the experts were asked to provide feedback and suggestions for improving the RELPP-BL. They were also informed that their feedback would be kept confidential. Extra credit was awarded at the end of the semester to those who completed the content validity tool.

4. Analyses

Of note, we do not consider the individual items within each of the six subscales as representative of the previously identified constructs. Rather, in each subscale the items collectively assist in operationalizing the construct. Further, some of the subscales consisted of only a table or a checklist and not individual items per se. Given that we intended to establish the content validity of each of the six subscales, in the first phase, we only analyzed the subscale content validity index rather than the individual item content validity.

To obtain the subscale content validity index (Ss-CVI), Polit and Beck (2006), Polit et al. (2007), and Zamanzadeh et al. (2015) suggested using the ratings of either 3 or 4 as universal agreement, divided by the number of experts. Several researchers recommended obtaining a CVI of 0.80 in order for the scale to be considered valid (Rubio et al. 2003; Polit et al. 2007). Thus, we used these guidelines for determining the validity of the subscale and scale. We primarily employed Polit et al.’s (2007) and Zamanzadeh et al.’s (2015) procedures in order to obtain the subscale and scale CVI for relevancy and comprehensiveness, which were pertinent in answering our research questions. These two articles provided a comprehensive overview of these procedures and for conducting the kappa statistic.

As per the aforementioned researchers’ recommendations, we used universal agreement to determine the Ss-CVI. Thereby, a rating of 3 or 4 suggested the universal agreement that the subscale was valid in terms of relevancy and comprehensiveness. Thus, if all eight experts rated the scale with either a 3 or 4, then this would imply that eight out of eight agreed that the subscale was valid, and the CVI would equal one (CVI = 1). Given that there were eight experts, the minimal acceptable CVI = 0.83 (Polit and Beck 2006; Polit et al. 2007; Yusoff 2019). We transferred the expert ratings for each of the subscales onto tables (see

Table 8. Expert Ratings on Relevancy.

Expert
Subscale	1	2	3	4	5	6	7	8	S-CVI	Fleiss k
Ethnographic Interview and Parent/Caregiver Language Profile	4	4	3	3	3	4	4	4	8/8 = 1	1
Student Language Profile	4	4	3	3	3	4	4	4	8/8 = 1	1
Language Exposure	4	4	3	3	3	4	4	4	8/8 = 1	1
Language Use	4	4	3	3	3	4	4	3	8/8 = 1	1
Student Educational Background	3	4	3	3	3	4	4	2	7/8 = 0.87	0.869
Language Checklist	3	4	3	3	3	4	4	3	8/8 = 1	1
Total SCALE									0.98	0.978

and

Table 9. Expert Ratings on Comprehensiveness.

Expert
Subscale	1	2	3	4	5	6	7	8	S-CVI	Fleiss k
Ethnographic Interview and Parent/Caregiver Language Profile	4	4	3	3	3	4	4	4	8/8 = 1	1
Student Language Profile	4	4	3	3	3	4	4	4	8/8 = 1	1
Language Exposure	4	4	3	3	3	4	4	4	8/8 = 1	1
Language Use	4	4	3	3	3	4	4	4	8/8 = 1	1
Student Educational Background	3	4	3	3	3	4	4	2	7/8 = 0.87	0.869
Language Checklist	3	3	3	3	3	4	4	3	8/8 = 1	1
Total SCALE									0.98	0.978

). We determined the Scale CVI (S-CVI) using a liberal method in which we calculated the average of the Ss-CVIs. In addition to determining the subscale and scale CVI, we used the Cohen’s kappa statistic as modified by Fleiss for multiple raters (Fleiss 1971; Polit et al. 2007) to determine chance agreement and to demonstrate that the content validity index was not due to chance.

$$k=\frac{\mathrm{ProportionAgreement}-\mathrm{ProportionChance\; agreement}\left(.5^{\mathrm{n}}\right)}{1-\mathrm{ProportionChance\; agreement}}$$

Following this analysis, the feedback from the bilingual educators was examined to see if there were any common recommendations for refining the instrument. The researchers reviewed the comments obtained to determine if modifications were necessary.

5. Findings

In this section, we will present the findings in which we explored the content validity of the RELPP-BL with bilingual educators.

5.1. Content Validity

To examine the RELPP-BL’s content validity, the first author analyzed the results and constructed a table to report the findings. The findings were discussed among the three researchers to ensure that there was agreement.

As reported in Table 8 and Table 9, the Ss-CVI resulted in a CVI equal to and greater than 0.87 for both relevancy and comprehensiveness, thus meeting the threshold criteria of 0.83, as previously determined. In addition, the overall Scale-CVI for relevancy (0.98) and comprehensiveness (0.98) both exceed the threshold criteria. In using Fleiss/Cohen’s kappa meta-analysis to correct for chance agreement, we were essentially attempting to determine if the agreement among the experts was a result of chance. For example:

Subscale

$$k=\frac{0.87-0.004}{1-0.004}$$

k = 0.866/0.996 = 0.869 (0.87)

Scale

$$k=\frac{0.98-0.004}{1-0.004}$$

k = 0.976/0.996 = 0.978 (0.98)

Applying the Cohen’s kappa statistic indicates that the subscales k ranged from 0.89 to 1 agreement and scale agreement (k = 0.98), which are considered excellent and not a result of chance.

5.2. Expert Feedback

Following these content validity index analyses, we examined the bilingual educators’ feedback and recommendations. We received a few recommendations; the suggestions included changes or modifications to the following subscales. The bilingual educators recommended deleting the following questions from the Ethnographic Interview subscale:

“Does the student communicate her feelings?”

They also suggested adding the following question to the Student Profile subscale open-ended prompts:

“Does the student request help from peers when needed?”

Additionally, changes were proposed for the Language Exposure and Language Use subscales, in which the educator is asked to identify the language in various scenarios, such as within the home or with grandparents. The bilingual educators suggested rephrasing the following open-ended scenario prompts: ‘Other, Special Circumstance, and TV/radio’ to ‘reading and access to media’. One bilingual educator indicated: “I think that ‘other activity’ and ‘special circumstances’ are sort of asking the same question.”

The overall feedback obtained from the bilingual educator experts was positive, and they suggested that this protocol could be a useful tool for bilingual educators in determining the referral of bilingual learners due to language concerns. One stated that the protocol “helps, it does help because it kind of narrows down and it makes it more focus[ed].” Another bilingual educator expressed that the protocol was useful “because it gave [her] kind of ideas and like things to look out for.” As a group, the bilingual educators agreed on the relevancy and comprehensiveness of the protocol. Their recommendations were used to improve the items in the respective subscales. All participants expressed that they would be extremely likely or somewhat likely to use this protocol.

6. Discussion and Conclusions

The over- and under-referral of bilingual learners is of great concern as it impacts educational opportunities and decreases students’ academic performance. In an attempt to help educators to gain access to valid tools that can guide them in their decision-making process, we developed the RELPP-BL. Our goal is that this tool can help educators identify the students who need referral for further evaluation. In addition, we wanted to ensure that this tool demonstrated content validity.

In establishing the content validity of the RELPP-BL, we followed the protocol and process as recommended in the literature and used in the fields of education, social work, psychology, and medicine (e.g., Almanasreh et al. 2019; Polit and Beck 2006; Polit et al. 2007; Meurer et al. 2002; Rubio et al. 2003; Zamanzadeh et al. 2015). The experts selected during the content validity phase were bilingual educators.

We propose that our preliminary findings suggest that the RELPP-BL demonstrates content validity. As such, the instrument can serve as a guide for bilingual educators during the pre-referral process. Importantly, the RELPP-BL is not limited to only English-Spanish bilinguals, as the data gathered is needed regardless of the bilingual learners’ first or second language. We propose that the six subscales create a comprehensive tool that allows for the collection and reporting of the converging data needed to determine the language skills of bilingual learners. The six subscales of the RELPP-BL measure critical information recommended by the literature and aid in the collection of language performance skills of bilingual learners (Castilla et al. 2009; Flores et al. 2022a). While the RELPP-BL and the BESA questionnaires (BIOS and ITALK) are similar in that these collect data on a child’s language performance, the RELPP-BL is unique because it is created as a pre-referral tool for bilingual educators to collect and report information about students’ language skills. The RELPP-BL also provides educators with data about a bilingual learner for further evaluations if needed. Additionally, this study contributes to the availability of viable pre-referral tools that educators can use to assist them in their decision-making process.

This study is an initial step in establishing the content validity of the RELPP-BL; as Messick (1998) recommended, this evidence, along with other complementary validity exploration (e.g., criterion, consequential), can be used to evaluate overall construct validity. As a preliminary study, there are limitations that we must acknowledge, such as the fact that the bilingual education experts were recruited from a graduate class and were given extra credit, which may have influenced their ratings.

Recommendations for Further Studies

Future comprehensive studies could include a representative group of bilingual educators from across the country. It is also important to test the RELPP-BL with other language combinations other than Spanish-English and other bilingual populations in other contexts. Additionally, SLPs who work with bilingual learners could also be asked to engage in evaluating the relevance and comprehensiveness of the RELPP-BL. Other studies could continue to examine the criterion validity of the instrument once it is used by a larger sample group. For example, how accurately does the RELPP-BL assist in making appropriate decisions based on referral guidelines? Consequential validity could also be explored; that is, does the RELPP-BL result in over- or under-referral of bilingual learners? We also think that pre-referral tools could explore other speech-language concerns, such as articulation, fluency, pragmatics, etc. These studies could add to the literature and extend our understanding of the speech and language concerns that bilingual learners present. In sum, to ensure that bilingual learners receive equitable educational experiences and to reduce the likelihood of over- or under-referral, it is critical that pre-referral tools such the RELPP-BL be further developed, validated, and implemented.