Decoding Narrative Statements in Child Protective Services Hotline Calls: A Methodological Approach

Abstract

There is clear evidence that non-safety-related concerns abound in child protection hotline calls. In the United States, over half of Child Protective Services (CPS) calls are screened out because they do not meet criteria for a child welfare investigation. While reporter bias is one factor theorized to contribute to this level of screened out calls, the field has neither used methods that account for culturally specific socialization processes involved in bias nor analyzed hotline calls to determine if these biases were present. This paper describes cultural domain analysis (CDA) as an innovative method to inform the measurement and assessment of bias in reporters’ narratives about children and families during calls to a CPS hotline. We describe CDA, which involves a rapid interviewing technique (freelisting), a participatory method for coding (pile sorting) and how the resultant findings can be used to inform the development of a measurement framework (codebook and scale), which may be tested using recorded hotline calls. Together, these methods provide a useable framework that can help surface common and shared ways bias is conceptualized and defined in the context of CPS hotline calls. This proposed approach provides a socially valid and reliable way for measurement to make generalizable inferences across a jurisdiction. When applied in practice, data collected and analyzed from the proposed measurement framework can inform jurisdictional CPS hotline policy, practice, and training.

1. Introduction

Child protection system hotlines experience high call volumes consisting of numerous non-safety-related inquiries (Nadon et al. 2023). In the most recent federal fiscal reporting year, 4,399,000 maltreatment calls were made, over half of the reports (52.5%) were screened out and did not meet the criteria for a child welfare investigation (U.S. Department of Health & Human Services et al. 2026). Significant heterogeneity across states (e.g., Day et al. 2022; Piersiak et al. 2023) and reporter groups exists (Edwards 2019; Gandarilla Ocampo et al. 2024). In the most recent reporting year, there were substantial differences in screen-in decisions by state, for example Mississippi had a screen-out percentage of 25% whereas Vermont screened out 83% of its reports (U.S. Department of Health & Human Services et al. 2026). Nationally, the total number of child protection hotline reports (including calls that do not result in an investigation); disaggregated data by reporter type, is not tracked. However, within reports that are accepted for investigation, these data show relatively low substantiation percentages for some reporters compared to others: anonymous callers (69%), educators (69%), family and friends (68%), medical professionals (64%), and legal and law enforcement professionals (53%) (Children’s Bureau et al. 2024). The low accuracy and diverging reporting decisions and practices raise critical questions about the institutional reporting guidance given to reporters as well as the mental models and cognitive lenses that reporters use to interpret the world (Genter 2001). These cognitive frameworks can be biased or flawed in ways that lead to inaccurate assessments of specific groups of families based on stereotypes or prejudice.

Research has not thoroughly explored the role of reporter bias at the initial touchpoint of the Child Protective Services (CPS) hotline, leaving a significant gap in understanding regarding how bias may shape reporting behavior. Measuring such bias is methodologically challenging given that coded and loaded terms (i.e., words with strong emotional connotations or hidden meanings), are deeply contextual, dependent on history, geography, and culture (Caparoso and Collins 2015; Holmes and Wilson 2022; Massey 1995). Given strong sociocultural underpinnings of language, standard quantitative methods are difficult to apply in child welfare settings, demonstrating the need for measures that can effectively account for the context-specific nature of bias (Faulkner et al. 2024).

These concerns and questions were brought to our team by a jurisdictional partner who sought to understand the breadth and severity of bias in reporters’ descriptions of families during calls made to CPS in their jurisdiction. To respond to this request, our team developed a four-stage, mixed methods study that situates linguistic bias (e.g., pejorative, coded, essentializing, and loaded terms) as shared, cultural artifacts (Borgatti 1994; Brewer 2000; Romney et al. 1986), specific to geography, history, and culture. This form of Cultural Domain Analysis (CDA) involves collecting and analyzing data to understand how a group of people think about concepts that are negotiated through social and psychological processes (Borgatti 1994). Hence, the present study introduces CDA as an innovative method to inform the measurement and assessment of bias in reporters’ narratives about children and families during calls to CPS hotlines. By adapting techniques from cultural anthropology and integrating them with concepts from social psychology and sociolinguistics, this research seeks to “decode” narrative statements by establishing a socially valid measurement framework for assessing bias in hotline reporters’ descriptions of children and families.

2. Background

The decision to report a family to CPS is influenced by a variety of factors, both structural (statute, training, institutional norms) and individual (knowledge, facts, experience, personal beliefs, bias or malintent) (Douglas and Fell 2020; M. Harris 2021). We were asked by a child welfare jurisdiction to explore one of these factors, bias. The jurisdiction’s core research questions were: (1) Does bias exist in reporters’ descriptions of families in their CPS hotline calls; and (2) what is the extent and severity of bias in reporters’ narratives about families in calls made to CPS? We were tasked with designing a methodology to assess and characterize the level and extent of bias in the narratives used by reporters when describing concerning situations to CPS. Our jurisdictional partner identified the need to select a rigorous methodological approach that centered cultural insiders’ understanding of bias to ensure social validity. In addition, they sought methodological transparency, reliability, and replicability.

2.1. Theoretical Framework

This study presumes that language and culture are inextricably linked, whereby language acts as a central vehicle for transmitting and shaping cultural beliefs and encoding specific social experiences (Holmes and Wilson 2022). Within this cultural context, schema theory posits that people interpret, organize, and assign meaning to thoughts and new experiences, in part through socialization processes which are mediated by language (Vygotsky 1978; Brewer 2000), resulting in cognitive schema, or mental structures, that are built upon earlier experiences. However, meaning is not exclusively expressed through vocabulary. Prosodic features such as pitch and volume, function as evaluative qualifiers that transform factually neutral language into markers of bias (Scherer 1986). While cognitive schemas support individuals in processing information quickly, these mental frameworks can result in rigid thinking, such as stereotypes or assumptions, particularly when schemas intersect with systemic social hierarchies of privilege and power (L. Harris 2021).

Indeed, a relational view of place suggests that cognitive schema, particularly mental frameworks related to the social world, like bias, are influenced by local histories, migration patterns, institutional norms, and state policies (Massey 1995; Soja 1989). Consequently, groups of people develop shared cognitive frameworks of thought and meaning, that are specific to shared relational geographies and (re)negotiated overtime. Through this lens, the vocabulary of bias (i.e., pejorative, loaded, essentializing, and coded language), used by reporters to the hotline, is not merely a collection of universal stereotypes, but a “localized linguistic repertoire” deeply embedded in local cultural and geographic contexts (Agha 2005; Johnstone 2011).

Together, we posit that the exploration of explicit and implicit bias in language and communication necessitates a method that captures place-based knowledge and socialization practices negotiated through cultural practices embedded in local geographies and histories (Dwyer and Bressey 2008; Jackson 2008). Therefore, actionable and context-relevant strategies for child welfare jurisdictions seeking to recognize, characterize, and respond to ethno-racial and poverty-based bias in reporters’ descriptions of families must be grounded in contemporary place-based understandings of the nature of bias and how it manifests in everyday discourse.

2.2. Research on Bias in Child Welfare

Over the last 30 years, scholars have used a range of methodological approaches to explore the role of bias in disproportionality and disparity across the child welfare continuum, focusing largely on system outcomes and experiences of African American and Indigenous families (Fluke et al. 2011; Harris and Hackett 2008; Hill 2006; Roberts 1997, 2002). This body of work has generated diverse findings about racial and economic disparities, spanning from rich, deeply personal experiences of families of color impacted by the system (Miller et al. 2013) to statistical inferences generalizable across distinct settings and populations (Drake et al. 2009; Jonson-Reid et al. 2009) and that permit causal inferences to be made about drivers of racial disparities in child welfare practice (Baron et al. 2023). While each methodological approach brings distinct strengths for identifying and conceptualizing bias, a substantial portion of the dialog and scholarship has focused on differential treatment and disparities during the intake process and post-system entry rather than potential bias amongst hotline reporters. Expectedly, findings and perspectives published pertaining to the existence of bias amongst hotline reporters are divergent (Barth et al. 2026; Merritt et al. 2025). To our knowledge, quantitative research aiming to explore bias in system outcomes has yet to utilize a direct psychometric measure of bias in their models, relying instead on statistical methods that infer consequences of bias rather than directly observing it through discrete linguistic features or decision-making patterns of reporters or jurisdictional staff.

2.2.1. Qualitative Research

Qualitative research is a powerful option to develop rich contextual detail about how bias appears and operates and for gaining insights from individuals most proximate to and impacted by it. Qualitative studies on bias in child welfare primarily gather information via in-depth interviews and focus groups. Through focus groups, scholars have examined visibility bias (increased public system contact and surveillance adds to the likelihood of being reported to CPS) and cultural sensitivity (awareness, appreciation, and respect of values, norms, and beliefs characteristic of other cultural and racial-ethnic groups that is not one’s own) among reporters and case workers (Miller et al. 2013) and to better understand factors contributing to racial disproportionality (Dettlaff and Rycraft 2008; Kokaliari et al. 2019). Whereas Fong’s (2017) in-depth interview techniques elicited information about impacted parents’ experiences with poverty-based bias (form of class-based prejudice or discrimination based upon socioeconomic status). Further, researchers have used phenomenological approaches to record the lived experiences of African American mothers impacted by bias and CPS reporting (Merritt et al. 2025). The exploration of bias has occurred via the use of vignettes which endeavored to isolate race as a variable in decision-making processes (Ibanez et al. 2006). Qualitative research is vital because bias is context-dependent, experiential, and deeply personal; however, these findings are not intended to be extended to the broader population which can limit our understanding of the breadth and prevalence of bias.

2.2.2. Quantitative Research

Quantitative research aiming to explore the role of bias in child welfare disparities has generally relied on outcome-based statistical inferences. These approaches, such as decomposition methods (e.g., Maloney et al. 2017), benchmarking (e.g., Drake et al. 2023), proportional comparative analysis (e.g., Drake et al. 2017), and causal inference techniques (e.g., Baron et al. 2023; Font et al. 2012), infer the presence or absence of bias, with varying levels of sophistication, by evaluating whether disparities persist after adjusting for factors known to covary with outcomes. Collectively, these methods share a common reliance on indirectly estimating bias by identifying differences between groups and/or data sources. While such approaches cannot provide direct observational evidence of bias in decision-making or thought processes, they are relevant for generating broad systems-level inferences about the nature of differential outcomes attributable to race and income at scale.

Findings from this body of research have produced mixed results about the extent to which racial and economic bias contributes to differential systems outcomes for families of color and those experiencing significant socioeconomic disadvantage, in part, given variability in methodological rigor. For studies utilizing statistical techniques that can largely rule out observable and unobservable confounds (e.g., Baron et al. 2023; Font et al. 2012), findings broadly suggest that while child and family race account for disparities in systems outcomes, different mechanisms (e.g., risk versus case-worker bias) may account for outcome disparities. While limited conclusions can be drawn about a single universal cause, it is reasonable to posit that mechanisms accounting for differences are context-dependent, revealing complex interactions among caseworker bias, socioeconomic risk, jurisdictional environments, and historical context.

Together qualitative and quantitative approaches for exploring bias in the child welfare system have generated findings appropriate to understand (1) deep, personal experiences related to differential treatment based on family race and socioeconomic contexts and (2) broad systems-level inferences about the nature of differential outcomes attributable to race and risk. Yet, work to date is limited in significant ways. First, quantitative research has yet to use a direct psychometric measure of racial or economic bias in research attempting to account for the effects of prejudicial treatment in decision making on systems outcomes, resulting in imprecise understanding of these complex cognitive and behavioral processes. Conversely, while qualitative research has directly explored personal experiences and perceptions about differential treatment based on race and income, these methods cannot estimate the breadth or severity of bias broadly across settings and populations.

2.3. The Approach

Building off previous efforts to develop and validate a psychometric measure of bias specific to the child welfare context, (Faulkner et al. 2024) the framework offered in this paper aimed to create a direct and quantifiable measure to examine the breadth and depth of bias in CPS hotline calls. This paper introduces Cultural Domain Analysis (CDA) as a methodology to bridge the gap between qualitative and quantitative methods for exploring bias. We propose CDA’s use as the foundation for a measurement framework that assesses bias in reporters’ descriptions of concerning situations during hotline calls. This paper outlines the processes and strategies for the development of a measurement framework that has the capacity to produce findings about bias that are generalizable and rooted in shared cultural experiences.

2.4. Cultural Domain Analysis: Background and Application

2.4.1. Cultural Domain Analysis

Cultural Domain Analysis (CDA) is a set of methods for collecting and analyzing data to understand how a group of people think about concepts that are negotiated through social and psychological processes (Borgatti 1994). Rather than imposing the researcher’s own understanding and definitions of a particular concept, attribution of meaning is elicited from a group of informed stakeholders. CDA uses both quantitative and qualitative data to identify terms in a cultural domain and uncover salient dimensions of meaning (Haile and Woldu 2024). These are perceptions and are, in theory and principle, shared (Borgatti 1994). Terms in a cultural domain should have an internal structure, be related or similar to one another, and the assigned attribute(s) should fit all items in the domain. For example, when thinking of the concept/domain sadness, other similar items in this domain may include feeling blue, feeling down, or being melancholy.

2.4.2. Freelisting

Freelisting is a rapid interviewing technique derived from cultural anthropology used to establish a common set of words and phrases that are shared among a group of people and used to describe and define a phenomenon specific to their shared experience (Borgatti 1994, 1998). Participants are given a prompt and asked to generate as many words and phrases as possible that are associated with that topic. Interviews are conducted with a purposeful sample of participants who have experiences or background congruent with the concept or research questions under investigation. While there are no specific guidelines for the number of participants required to reach saturation, generally a minimum of 20 to 30 participants is needed to achieve research saturation, though this threshold varies significantly and depends upon features of the research question (Borgatti 1994).

2.4.3. Pile Sorting

Pile sorting is a participatory method for sorting, organizing, and coding qualitative data generated from freelisting based on items’ similarities, differences, and their overall meaning. To facilitate pile sorting, a subsample of participants who completed freelisting are asked to engage in this activity. The goal of pile sorting is to have participants organize words and phrases from freelisting into groups based on their meaning and context (e.g., how phrases are similar, different, and/or related in context). Then, in individual interviews, participants are asked to group and label categories. Once complete, researchers ask participants a series of questions about their work to further explicate the meaning and relevance of these domains and the individual phrases that comprise them (Borgatti 1998).

2.5. Use of CDA in Understanding Bias

Cultural domain analysis has previously been used as a framework for conceptualizing and categorizing bias. Specifically, CDA has been the basis for understanding racial stratification (Caparoso and Collins 2015), perceived deservingness and discrimination in refugee care (Ziegler and Bozorgmehr 2024), gender and body type bias (Monocello and Dressler 2020), and systemic bias in education leadership (Hines 2023). Within child welfare, components of CDA (freelisting and pile sorting) have been used to understand how workers understood and responded to domestic violence (Collins and Dressler 2008) and the way in which they defined and made sense of child welfare agency culture (Spielfogel et al. 2016). In furtherance of the application of the CDA framework in child welfare, we propose this new use case, applying CDA to understand and measure the breadth and depth of bias in CPS hotline calls.

3. Materials and Methods

Reporters’ decisions to report a family are influenced by a complex interplay of state and federal statute, institutional norms and practices, and individual beliefs and attitudes, including bias (Sedlak et al. 2022). To date, research has primarily used quantitative methods to examine disproportionality in investigation and placement outcomes, inferring bias through statistical approximations (e.g., decomposition methods), rather than directly measuring bias in stakeholder mental models or cognitive schemas. Conversely, qualitative research has tended to explore personal experiences of differential treatment but cannot estimate the prevalence or severity of bias across broad settings and populations.

As requested by our jurisdictional partner, we describe a measurement framework using CDA to capture the breadth and depth of bias in reporter’s narratives during hotline calls. By bridging the gap between qualitative depth and quantitative rigor, this framework provides a foundation for producing findings about bias that are both generalizable and rooted in the shared cultural experiences of those at the system’s front door. A standardized measurement framework for assessing the breadth and depth of bias in reporters’ characterizations of families could support jurisdictions in developing context-relevant narrative change strategies, inform training for callers and reporters to help lessen the likelihood of bias and to understand how reporter bias might contribute to dimensions of hotline worker call fatigue and overall well-being.

3.1. Application of Cultural Domain Analysis for an Observational Measurement Framework

This section describes a proposed step-by-step approach for the use of CDA to inform a measurement framework that identifies pejorative, loaded, essentializing, and coded language in reporters’ descriptions of families during CPS hotline calls. Given that non-verbal elements of speech underpin subtle emotional cues and communication, including implicit and coded forms of bias (i.e., emotional paralinguistic cues) (e.g., Guyer et al. 2021), the framework also accounts for prosodic features of speech (e.g., volume, tone) to ensure a holistic assessment of bias. Together, the design integrates methodologies from cultural anthropology (cultural domain analysis), sociolinguistics (prosodic and narrative analysis), and psychometrics (observational coding and validity testing). And proposes the use of audio recorded CPS hotline data for validity testing. The mixed methods research design was largely informed by the study aims and hypotheses presented by our public child welfare agency partners. The intent was to produce generalizable findings from audio-recorded hotline data about constructs that are deeply embedded in cultural and social processes (Sue et al. 2007).

3.2. A Phased Approach

The research project is organized into four distinct phases (see Figure 1) Phase I: Planning and Preparation; Phase II: Cultural Domain Analysis; Phase III: Measurement: Codebook and Scale Development, and Phase IV: Validity Testing. Phase I includes convening key research and jurisdictional stakeholders to ensure project feasibility and establishing an advisory committee of people with professional and lived expertise. Phase II involves executing CDA to generate a consensus-driven taxonomy of categories, terms, and examples that reflect the spectrum of biased language, including pejorative, loaded, essentializing, and coded descriptors, used to characterize families’ social identities and circumstances. Phase III involves translating findings from CDA into a formal codebook that systematically defines, operationalizes, and qualifies target language, including prosodic elements of speech (e.g., intonation, volume) that indicate complex linguistic features of speech like condescension and judgment, which are conceptually related to bias. Phase III also encompasses the development of a bounded ordinal response scale informed by the codebook, allowing the measurement of the presence, frequency and intensity of biased narratives, including paralinguistic markers (e.g., intonation, volume). Phase IV involves feasibility testing of the codebook and scale with a representative sample of real-world hotline data to ensure its practical utility.

3.2.1. Phase I: Planning and Preparation

The proposed study requires experts from the social sciences and child welfare practice. The social scientists need technical expertise and knowledge in cultural domain analysis, prosodic and narrative analysis, social psychology and intergroup relations, CPS hotline reporting, and psychometrics/measurement. Using that knowledge, they will provide the needed expertise to execute and manage the project. The core project team will also include CPS leadership, veteran CPS hotline staff, and jurisdictional administrators, across different phases of the project, as needed, to ensure that the research design is grounded in realities of frontline practice. An advisory committee (AC) of people with lived and professional expertise, such as systems impacted families, hotline staff, and cross-sector leaders from callers and mandatory reporting groups (e.g., education, law enforcement), will enhance the ecological validity of the research design and findings, ensuring they reflect the complexities of the child welfare frontline. Concurrently, the AC ensures the social validity and content validity of the research materials (i.e., codebook and scale), confirming that the tools are grounded in the lived and professional realities of the system stakeholders (Lätsch et al. 2021; Zumbo and Padilla 2020). The AC will advise on all phases of the research, including refining recruitment materials and providing member checking of qualitative findings and study materials derived from CDA (e.g., codebook, scale). The AC will also co-develop dissemination strategies to ensure findings reach academic, policymaker, and practitioner audiences.

3.2.2. Part II: Cultural Domain Analysis

The study will use a stratified, purposeful sampling design to include maximum variation in terms and expressions related to implicit and explicit bias across the jurisdiction (Robinson 2014). Key stakeholder groups will include impacted families, hotline staff, case workers, and various reporter groups. This design maximizes ecological validity by ensuring that the resulting codebook and scale are grounded in the authentic child welfare hotline environment (Lätsch et al. 2021). And it increases social validity by ensuring the research constructs are co-defined by, and remain relevant to, the stakeholders most impacted by these systemic narratives (Wolf 1978; Zumbo and Padilla 2020). Insights into the types of terms and expressions that convey bias about families require the involvement of families impacted by child welfare systems, given that people who directly experience discrimination are uniquely positioned to describe, understand, and, most importantly, transform inequitable conditions (Freire 1970).

Additional sampling strata will be informed by jurisdictional partners’ hypotheses about how terms and expressions of implicit and explicit bias vary across the jurisdiction, which may include geospatial and demographic stratification, targeting specific zip codes and neighborhoods characterized by diverse socioeconomic statuses and ethno-racial compositions. The AC can inform the research team about important organizations, institutions, settings, or community groups that align with these priorities to inform the recruitment strategy for initial phases of the project. A diverse range of professional and personal perspectives will increase the likelihood that the terms and expressions derived from the CDA are reflective of linguistic realities across the jurisdiction.

Given the centrality of reporters to this study, the study will sample all primary CPS hotline reporter groups listed in state statute (e.g., educators, health and medical professionals, law enforcement), given that institutional and professional affiliations contribute to variability in narratives used to describe concerning situations. For example, the term “hostile” may suggest implicit or subtle bias in law enforcement settings whereas “non-compliance” may be a term more commonly used in medical or health settings. Related, the sampling of each group should be proportional to the volume of calls each group generates. For example, if the 45% of all calls to the hotline derive from educators, then the sampling strategy should intend a similar proportion, capturing the breadth and depth of variability.

Freelisting: Interview Protocol and Activity

The freelisting sessions will be short in duration, lasting approximately 20 to 30 min. Individual interview sessions, virtually or in-person, will increase participant feelings of safety and privacy. The protocol will proactively address the potential for uncomfortable or difficult emotions that participants may experience when eliciting pejorative, loaded, essentializing, and coded terms about families and children. To ensure procedural fidelity, participants will engage in a practice round of freelisting on a topic that is communicated with subtle, coded, and explicit terms but not as charged as social bias (e.g., depression, death). This exercise will increase familiarity with the process before participating in the primary activity.

To begin the activity the interview will provide a prompt like this “Please list the words, phrases, or euphemisms you believe a reporter might use to describe a family or child that would signal, either explicitly or implicitly, a biased perspective. As you generate this list, consider how a reporter’s preconceived notions might shift depending on a family’s race, socioeconomic status, or neighborhood, which may increase or decrease the likelihood that they are reported to the hotline.” Interviewers will audio record sessions and take field notes to increase accuracy of data collected. Once the participant stops eliciting terms, the interviewer will summarize content by re-listing all terms generated, with the intent of providing the participants with another opportunity to recall or generate additional terms.

After completing freelisting interviews, researchers will need to clean and analyze the data. The aim of cleaning data is to combine and consolidate listed terms so that each word or phrase holds unique meaning. This process involves several iterative rounds of cleaning, starting with identifying root stems (e.g., nice ≈ nicest, nicer), synonyms (e.g., good ≈ nice), and similar concepts (e.g., kind and nice) (Keddem et al. 2021). For example, the words “satisfied” and “pleased” may be collapsed into a singular item “satisfied” as these two terms are generally considered to be interchangeable. While time intensive, the near synonym matching process benefits from multiple rounds of matching and feedback from topical experts. Documenting decision making about which word matches were made is recommended.

Once the original list of words is consolidated to a set of items with each holding distinct meaning, words and phrases are organized and ranked by frequency (i.e., the number of times the same word or phrase is listed by different participants) and saliency (i.e., the order in which words are listed). Borgatti (1998) suggests that items of central importance are generally listed earlier in the interview (e.g., listed first, second, third), revealing important information about cognitive structures underpinning domains. Then using graphing techniques, unique words and phrases are plotted by their frequency count and rank order, (i.e., saliency factor), from highest to lowest, to determine where the counts start to drop off, commonly known as a scree plot. Smith and Borgatti (1997) propose a formula to calculate saliency, which equals the sum of the item’s percentile ranks divided by the total number of lists. Using this metric, the research team examines the plot for the “elbow”, demarcating a big jump between word frequencies and ranks (i.e., salience factor) or looks for instances where they start to stabilize. Figure 2 displays an example of a scree plot, with stabilization occurring initially at frequency counts of 5, aligning with the top ~20 words and a second area of stabilization frequency counts of 4, corresponding to the top ~40 words. Final decisions about the retention of terms from freelisting phase will be informed and guided by the AC to maximize context relevance, with a maximum number of terms ranging between 40 to 50 to minimize cognitive fatigue.

Pile Sorting: Interview Protocol and Activity

Once the most frequent and salient terms are identified, the research team will transfer words and phrases onto note cards in preparation for pile sorting interviews. Pile sorting interviews can be conducted virtually or in person. Online platforms with virtual whiteboards and digital cards (i.e., sticky notes) provide several advantages, including data preservation for codebook auditing and asynchronous member checking once pile sorting is complete.

Pile sorting interviews will occur individually and involve a minimum of 30 to 40 participants, mirroring the sampling strata used in the freelisting phase, which is generally sufficient for reaching consensus (e.g., Borgatti 1998; Romney et al. 1986). The total duration for each session is anticipated to be 60 to 90 min, with 30 to 45 min allocated to sorting and categorizing terms and the latter 30 to 45 min allotted to interviews. Participants will be directed to organize the digital cards into conceptually distinct clusters based on their perceived meaning within the specific context of a CPS hotline narrative. For each resulting cluster, participants will provide a descriptive title that captures underlying “higher-order” constructs. Participants will be permitted to duplicate terms, allowing a single descriptor to be in multiple categories if meaning shifts across different social or institutional contexts.

Following the sorting task, the research team will conduct semi-structured interviews to elicit participants’ underlying decision-making processes. Probing questions will aim to identify the boundaries among crucial information such as implicit bias, explicit bias and terms that carry dual meanings with a focus on exploring how tone, intonation, and volume contribute to meaning. Some of these questions will include the following:

• What was your strategy for sorting these terms?
• What makes this pile different from others?
• How would you infer that this group of terms or phrases is indicative of implicit bias: what other information in the call would you consider?
• When you hear a reporter use a word like ‘uncooperative,’ how does their tone or volume help you decide if they are reporting a factual safety concern or expressing a personal bias against the parent?

3.2.3. Part III: Measurement Framework

Codebook

The research team will use findings from freelisting and pile sorting to inform the development of a formal codebook, which will be organized into a hierarchical taxonomy of higher- (broad and general) and lower (narrow and specific)-order constructs. The codebook will serve three primary functions: (1) cataloging biased terminology into discrete categories; (2) providing operational definitions for each code; and (3) establishing qualifications to provide clear decision rules that allow raters to distinguish between (a) objective safety concerns, (b) biased descriptors, or (c) situations where both may be present. The codebook will undergo a formal member-checking process to increase social validity and procedural rigor. This process will involve structured reviews with AC members (e.g., jurisdictional and hotline staff, lived experts) to verify that the coding categories are authentic to the child welfare context. This iterative feedback loop remains grounded in the lived realities of those impacted by the system while maintaining the technical precision required for large-scale applications.

In collaboration with an expert in sociolinguistics, the codebook will define paralinguistic markers, such as tone, pitch, and volume, that function as evaluative qualifiers of implicit and explicit bias (Guyer et al. 2021; Maass et al. 1989; Scherer 1986). Research suggests that prosodic cues transform factually neutral language into markers of bias, signaling skepticism or exasperation. For example, research indicates that ingroup-outgroup distinctions are communicated through specific shifts in voice like a sudden, sharp increase in loudness on specific words and that negative judgments are generally associated with a narrowing of pitch range, which sounds “flat” or, conversely, a sharp upward inflection that signals incredulity or condescension (Cheng et al. 2016). As well, findings indicate that drawing out a word (e.g., “thoooose”) may suggest a speaker is “distancing” themselves to categorize or separate themselves from others (Arafat and Hamamra 2021). In summary, the codebook will provide raters with specific examples and qualification criteria to ensure high inter-rater reliability (IRR) when distinguishing between legitimate safety information and linguistically encoded bias.

Scale Development

The research team will synthesize findings regarding the breadth, depth, and intensity of bias to construct a standardized observational measure. This instrument will utilize a bounded ordinal scale with four defined response categories (e.g., 0–3), ensuring that the intervals between scores represent a meaningful progression in the severity or explicitness of biased language. Unlike a simple frequency count, an ordinal structure allows the measure to capture the evaluative weight of biased narratives, distinguishing between subtle, “coded” microaggressions and overt, essentializing descriptors.

The scale will be operationalized through two primary dimensions: diagnostic (the degree to which language aligns with implicit or explicit bias constructs) and confidence in assessment (a rater-reported metric of certainty regarding the presence of bias versus a legitimate safety concern). To ensure the tool remains clinically useful for jurisdictional partners, a secondary Safety Assessment Scale should be integrated. This concurrent measure allows raters to document observable safety indicators alongside biased descriptors, providing a critical data point for the “transformation” goal; identifying calls where a high volume of biased language may be obscuring or conflating the actual types and overall level of risk to the child.

3.2.4. Part IV: Validity Testing

To evaluate the instrument’s ecological validity, the research team should apply the consensus-driven codebook to a stratified sample of archived hotline calls. Working in coordination with jurisdictional partners, researchers should extract a minimum of 50 audio-recorded calls. Given that jurisdictional data systems often restrict filtering to metadata, such as call date, time, and call type, the team should employ a limited stratification strategy. Calls will be weighted proportionately based on call volume frequency across a 24 h cycle; for example, if 40% of reports are received between 12:00 PM and 4:00 PM, 40% of the sample should reflect that window. This ensures the tool is tested against the variability of the authentic hotline environment, accounting for the different reporter profiles and pressures present at various times of day for hotline staff.

During this phase, a multidisciplinary rater pool, comprising researchers in social sciences (e.g., social psychology) and child welfare, alongside practitioners with direct experience in casework and hotline operations, will independently assess archived calls. This cross-section of expertise ensures that the framework is evaluated through both theoretical and clinical lenses. Raters will conduct five iterative rounds of assessment, with each round consisting of 10 calls, to test the codebook’s applicability to the authentic hotline environment. This process will continue until the rater pool has achieved full scale saturation, ensuring that the team has successfully identified and calibrated examples for each of the four defined response categories (0–3).

Throughout this process, raters will maintain detailed field notes to document their decision-making logic, specifically regarding the differentiation between scoring levels (e.g., distinguishing a score of 1 from a 2). Following each round, the team will meet to discuss discrepancies and document the rationale for final coding decisions. These deliberations are critical for refining the qualifications section of the codebook, making sure that the decision rules for semantic and prosodic markers are clear and socially valid. This iterative loop ensures the measurement framework is robust, transparent, and grounded in the operational realities of the child protection hotline and intake process. This four-step approach provides guidance for the use of CDA to inform a measurement framework that identifies and categorizes bias in CPS hotline calls.

4. Discussion

4.1. Implications for Practice

This paper proposed and described a framework for measuring bias in CPS hotline calls. The proposed methodological framework moves the field forward in a variety of ways. First, this paper proposes that CDA be used to measure bias in hotline calls. Second, it outlines specific steps to iteratively test the framework. Third, when this framework is applied in practice, it could systematically document the presence and severity of bias in hotline calls, which will offer avenues to further educate hotline reporters and stakeholder groups about what constitutes reasonable suspicion of child maltreatment. Finally, the data may provide agencies with a springboard for discussions around the necessity to reduce non-safety-related call volume and consider new ways to increase the quality and relevance of reports made to CPS. In other words, there are meaningful lessons to be gleaned from using this early-stage framework to understand the prevalence and severity of bias in reporters’ mental models. There are real world child welfare practice challenges that this exploration can help inform, namely: call volume, call fatigue, narrative framing, caseworker and caller training, and better understanding and identifying bias in hotline calls.

4.1.1. Call Volume, Call Fatigue and Related Practice Changes

High call volume drains and diverts vital resources and bandwidth away from more urgent child safety concerns. As agencies navigate how to best manage the hotline and uncover the right levers for process improvements, they can further explore the question: where should we best focus our time and resources—on reporters or hotline workers? Is it easier to change the beliefs and attitudes of a large and broad group of hotline reporters? Or is it more feasible to focus on re-training hotline workers to filter out superfluous information and/or potential bias? Under the current conditions, call takers for the hotline face the prospect of experiencing call fatigue as well as compassion and auditory fatigue. Call fatigue occurs when professionals are overwhelmed by frequent alerts, non-critical alarms, and calls such that they can become desensitized or delayed in their responses to actual safety risks and events. The oversaturation of non-safety-related calls or alerts can contribute to a sense of overwhelm, reduced urgency, lack of focus, exhaustion, and burnout (Albanowski et al. 2023; Issitt 2024; Smith et al. 2019). Finding ways to correctly size the call volume and the type of calls received by the hotline may increase worker job satisfaction and their ability to be thoroughly responsive to child safety-related calls. Further, this paper provides an opportunity for discussions around the reduction in non-safety-related call volume via reexamining ways to increase the quality and relevance of CPS calls.

4.1.2. Informing Reporter Training and Narrative Framing

For system-impacted families, non-safety-related CPS reports are not harm neutral. The CDA framework can help agencies and reporting groups better understand and navigate reporter sentiments, mental models, narratives, and behaviors. This knowledge can inform ongoing efforts to better educate and train reporters about what reasonable suspicion is, and what constitutes a grounded and objective child maltreatment report. Further, this study can assist with broader maltreatment reporting reform efforts by elucidating the role of bias in reporter narratives. With information gleaned from this study, child welfare agency leadership, where applicable, can develop targeted, context-specific narrative change strategies to counter-act and dismantle harmful rhetoric about the fitness and capacity of marginalized families to support the wellbeing of their children. More specifically, this paper provides the opportunity to specifically address the realities of real or perceived bias in hotline calls, and train call takers to look past it to get to the factual underpinnings of hotline calls, ideally reducing call volume and further prioritizing child safety-related calls.

4.2. Policy

Over 50% of hotline calls are screened out and for various reasons do not meet the criteria for investigation. Given this, a thorough policy analysis is needed to examine whether child maltreatment reporting laws and practices, as they stand, are accomplishing their stated goal(s). There is a disconnect or misalignment between the concerns that some reporters believe should be reported to CPS and what needs to be reported from a statutory standpoint. Reporters’ widely variable understanding and adherence to child protection reporting laws and criteria is a function of the ways the laws were designed and implemented, training quality and requirements. Policy changes may be needed to create higher accuracy, safety-specific reporting, and to create uniformity in training criteria, quality, and frequency. Utilizing the proposed CDA approach can provide a process to gather key ideas and insights to inform proposed policy modifications.

4.3. Future Research

Conceptualizing and measuring bias in CPS hotline calls is an essential task for both researchers and practitioners. This framework serves as one step in a broader series of necessary investments. To accomplish this, start by arriving at a proper conceptualization of bias that is reflective of a wide variety of key stakeholders’ perspectives. CDA is a mechanism to help achieve consensus on ways to best measure and categorize bias. Given the complexity of measuring bias, future research is needed to formally test the validity and reliability of the CDA measurement framework. Engaging in applied research by using all elements of cultural domain analysis (e.g., freelisting and pile sorting) to rate hotline call data would provide additional insights and help build knowledge and further advance the fields’ ability to decipher biased narrative statements in hotline calls. While several possible benefits of this methodological approach were named, some jurisdictions or agencies may face practical challenges with implementation and execution. This level of codesign requires significant trust, buy-in, capacity, and resource allocation (e.g., time, personnel, and expertise). To help shoulder the burden and make the process more feasible and navigable, a university and/or philanthropic partnership would be beneficial.

5. Conclusions

This paper provided a rigorous and innovative methodological approach to cataloging and characterizing bias in the narratives used by reporters when making reports to the CPS hotline. It detailed the processes and strategies for the development of a measurement framework which has the capacity to produce findings about bias that are generalizable and rooted in shared cultural experiences. Beyond its research utility, this paper discussed practical applications that can help child welfare agencies develop targeted, context-specific narrative change strategies to curb potentially harmful, non-safety-related commentary about families. Findings can be leveraged to educate callers regarding the intention, spirit and purpose of child maltreatment reporting and provide guidance on ways to report and frame child safety concerns. Ultimately, this work may help limit families’ unnecessary contact with CPS, unburden hotlines that are inundated with calls that are not safety related, enabling child welfare systems to reallocate time and attention to instances where legitimate safety concerns are present.