“This Kind of Technology Can… Treat Students Like Threats”: Black Youth Experiences, Reflections, and Articulations of Digital Discipline Under the New Jim Code

Abstract

Believed by many to be the “silver bullet” that will bring an end to educational inequality, AI technologies continue to proliferate within schools and classrooms, promising to bolster academic achievement, spark student engagement, and ensure campus safety while lessening the burden of overworked and systemically underpaid teachers. Despite this hype, a growing body of critical research is revealing that many of the AI technologies used in schools are rife with algorithmic biases that exacerbate, rather than remediate, educational inequity for historically marginalized students. We extend the work of scholars who have called attention to the rise of tech-mediated racism and the New Jim Code to consider how the proliferation of AI technologies into K-12 schools has worked to hide, speed up, and automate educational inequities for Black students, giving rise to a techno-educational carceral apparatus. To do so, we analyze youth interviews, youth-generated video blogs, and weekly journal reflections of 46 Black students that participated in a critical technology summer course.

1. Introduction

Since the high-profile release of ChatGPT in 2022, efforts to leverage the power of artificial intelligence (AI) to transform teaching and learning have hit a fever pitch. Believed by many to be the “silver bullet” that will bring an end to educational inequality, (Fortis, 2024), AI technologies continue to proliferate across schools and within classrooms, promising to bolster academic achievement, spark student engagement, and ensure campus safety while lessening the burden of overworked and systemically underpaid teachers. A growing body of critical and educational research, however, is revealing that many of AI technologies used in schools are rife with algorithmic biases that exacerbate, rather than remediate, educational inequity for Black and Brown youth.

For instance, generative AI chatbots meant to depict historical figures have been found to disseminate racially hostile and historically inaccurate information about Black history, often favoring white supremacist, pro-segregationist, and pro-eugenics stances on major historical events (Brockell, 2023). AI image generators meant to make lessons more culturally relevant work to disrobe and hyper-sexualize Girls of Color (Tiku et al., 2023; Heikkila, 2022) and generate images of People of Color as Nazis and slave owners (Grant, 2024). At the same time, AI-powered search results bring antiblack and misogynistic results to the top of the information pile, positioning these outputs as the most relevant and the most accurate in an era of increasing polarization, misinformation, and antiblack racism (Noble, 2018a). Teachers are also increasingly employing AI grading systems that score Black students more harshly than non-Black students, and more harshly than human reviewers would, while AI writing supports work to erase, correct, and penalize African American Vernacular English (Smith, 2025; Feathers, 2019). Collectively, these algorithmic harms, whether through racially offensive content or artificially reduced grades, work to widen longstanding ‘gaps’ in Black students’ academic achievement and feelings of belongingness (T. Tanksley, in press).

Notably, algorithmic mishaps that occur within learning technologies are only the tip of the iceberg. Research is beginning to show burgeoning connections between the proliferation of AI in school and the increased surveillance, dehumanization, and punishment of Black students both within and beyond the classroom. For instance, early warning systems identify Black students as future dropouts and disciplinary problems at disproportionately high rates (Feathers, 2023), while school safety software flags Black English (Chung, 2019) and searches for Black history as explicit, dangerous, or in violation of school disciplinary policies (Feathers & Mehrotra, 2023). Anti-cheating technologies also use facial detection systems that are unable to ‘see’ Black faces, resulting in failed courses, locked exams, and heightened disciplinary action (Clark, 2021). Importantly, since many of these school-based AI systems are directly connected to law enforcement, algorithmic biases that synonymize Black language, identity, and personhood with criminality can expose Black students to carceral pathways and campus-based violence at increasingly high rates (Farmer, 2014). These algorithmic biases and carceral entanglements become especially concerning when we consider that Black youth are six times more likely to die from everyday interactions with law enforcement than non-Black youth (see Heyward & Costa, 2020). Though many have deemed concerns about AI-mediated harms against Black students as ‘unfounded’ or ‘alarmist,’ the material harms of encoded antiblackness in school-based AI technologies was recently made manifest: in October 2025, Taki Allen, a 16-year-old Black student, was arrested by Baltimore police after his school’s AI-powered security system mistook his empty Doritos bag for a gun (Stanley, 2025).

As these and other examples show, the rapid, uncritical integration of AI into schools has created a complex matrix of algorithmic racism and sociotechnical domination that not only mediates Black students’ access to high quality teaching and learning, but also shapes their experiences with school discipline, pushout, and carceral contact. We thus echo scholars who have called attention to the rise of tech-mediated racism, coded bias and the ‘New Jim Code’ (Benjamin, 2019; Noble, 2018b; Buolamwini & Gebru, 2018), extending this work to educational landscapes. In particular, we interrogate how AI technologies hide, speed up, and automate educational inequities for Black students, giving rise to what we explicitly name a ‘techno-educational carceral apparatus.’ As schools become increasingly inundated with AI technologies, more scholarship is needed to caution against the mass adoption and normalization of AI as a purveyor of educational equity. Instead, researchers must examine how these tools alter, update, and automate the school discipline landscape in ways that widen, rather than remediate, existing concerns about physical safety, academic achievement, socioemotional wellness, and feelings of belonging for Black students. To address this, we pose the following research questions:

(1)

What effects do AI technologies have on the educational experiences of Black youth, particularly as it relates to discipline, surveillance, and school pushout?

(2)

How do Black students make sense of the growing connections between AI, education, and the carceral apparatus?

Our findings are structured by four main themes regarding AI-powered technologies in public schools: (1) AI technologies restrict Black students’ access to high quality learning experiences; (2) AI technologies restrict Black students’ access to culturally relevant and racially just content; (3) AI technologies produce racialized harms across socioemotional and academic domains; and (4) AI technologies expand the scope and scale of the school-prison apparatus. Importantly, the first three Findings directly address our first research question, and the fourth finding primarily provides insight for our second research question.

2. Literature Review

We situate our article within scholarly conversations spanning interdisciplinary bodies of scholarly, journalistic and grey literature on AI and educational technologies (EdTech). In particular, we examine AI technologies beyond generative AI chatbots, including school safety technologies, web filtering and blocking systems, and content moderation software to name a few. We couple these analyses with conversations happening at the national level that stress the need to “keep humans in the loop” in order to mitigate unintended harms to marginalized groups. This section provides a general overview of the arguments central to these sets of literature, and it is by no means exhaustive of the entirety of the fields of study for each.

2.1. Weapons Detection Systems and Facial Recognition Technologies

Though AI has proliferated across nearly every facet of teaching and learning, from automated grading and lesson planning to chatbot tutors and writing support, its most ubiquitous adoption continues to be within the realm of school discipline and campus safety. Weapon detection systems, facial recognition software, content moderation systems, and anti-cheating technologies are just a few of the AI-powered tools designed to support behavioral management and early threat detection in public schools primarily serving low-income Students of Color (Okoh, 2023). While the adoption of these AI technologies into schools are meant to ensure all students are safe, protected, and behaving in alignment with the schools’ disciplinary code (Feathers, 2020), research suggests that the most salient threat to Black students’ safety and well-being is often the AI technologies themselves (Herold & Harris, 2019). Specifically, weapon detection systems are riddled with antiblack biases that misidentify Black hairstyles, such as locs and braids, and personal hair-care items, such as combs or brushes, as deadly weapons (Young, 2025). Likewise, facial-recognition technologies utilized to track attendance, determine ‘on taskedness,’ and ‘verify’ the identity of campus visitors are fundamentally incapable of accurately identifying people with melanated skin, and regularly misidentify Black students as suspected criminals at exceedingly high rates. In one study within New York public schools, researchers found that campus-based facial-recognition software misidentified Black boys at a rate four times higher than white boys, and Black girls at a rate sixteen times higher than white girls (Feathers, 2020; McCormack & Coyle, 2020). Altogether, these algorithmic misfires lead to more frequent bodily searches, pat downs, and physical assault by school resource officers and/or local law enforcement, which is an increasingly common side effect of navigating zero-tolerance policies in schools with high surveillance infrastructures (Crenshaw et al., 2015; Love, 2019; Johnson & Jabbari, 2022).

2.2. Content Moderation Software and Web Filtering Tools

While facial recognition technologies and weapons detection systems are meant to stop threats that emerge at the ‘borders’ of school campuses, AI-powered content moderation systems are adopted to provide more comprehensive and expansive threat detection within and beyond school boundaries. These systems, which are used by a majority of K-12 schools in the United States, proactively scan students’ school-issued laptops for signs of self-harm, aggression, explicit content, and plans of violence. However, despite having unbridled 24-h access to student data, including website cookies, email content, biometric data, location services, and device usage statistics, these AI-powered systems are rarely successful at proactively detecting or deterring threats to student safety (Laird & Dwyer, 2023; Dwyer & Laird, 2024). Researchers have also found content moderation tools to be riddled with racial, gendered, and cultural biases that disproportionately harm students from historically marginalized groups (Laird & Dwyer, 2023). For instance, some of the most widely used systems have been known to identify queer students and conversations about sexual identity as ‘dangerous,’ accidentally outing students to teachers, families, and in some cases, the police. These tools have also been known to flag AAVE and other languages spoken by Black students as dangerous, resulting in Black students being contacted by law enforcement both on school grounds and in the privacy of their homes (see Herold & Harris, 2019).

In addition to flagging “suspicious” or “concerning” content created by students, AI-powered content moderation systems also block students’ access to websites and digital content deemed to be dangerous, explicit, or inappropriate. However, because of antiblack encoded racism, these content moderation systems inadvertently block students from accessing content about Black life, history, and culture (Feathers & Mehrotra, 2023). Consequently, Black students are often unable to research topics of interest related to their own identity while connected to the school’s Wi-Fi or while using their school-owned devices at home. Since systemic racism and socially engineered poverty make it more likely for low-income Youth of Color to need a school-owned device, they are concomitantly the groups to be most directly impacted by racially disparate content filtering systems. Ultimately, these tech mediated ‘stop-and-frisks’ place more students on the discipline track, putting them in direct contact with the carceral apparatus at higher rates and in different forms. In doing so, these tools quietly expand the scope and scale of school-carceral relations, automating school push out and the New Jim Code (Benjamin, 2019).

2.3. The Algorithmic Fail Safe: Keeping Humans in the Loop

In response to growing concerns about algorithmic harms, schools are encouraging teachers to act as the ‘human in the loop;’ that is, the person who can audit AI outputs and ensure that AI decision-making systems work in ways that ensure equity, safety, and protection for all. However, according to research, teachers are not only more likely to assume Black students are using AI to cheat than their non-Black peers, but they are more likely to punish them for using AI compared to other students (Laird & Dwyer, 2023; Dwyer & Laird, 2024; Madden et al., 2024). These punitive disparities persist even when Black students are using AI in ways that align with their schools’ student conduct policies, such as to augment the curriculum, implement accommodations, or make instructional content more accessible. Further research shows that regardless of whether the AI-powered tool was designed to support learning or to manage student behavior, teachers report a steady decrease in uses of these tools for teaching and learning and an overall increase in uses for monitoring and disciplining student behavior (Laird & Dwyer, 2023). Without efforts to remediate antiblackness from the technological and the sociopolitical landscape of schools, the adoption of AI-powered technologies will continue to produce racialized disparities in school discipline, pushout, and carceral contact, even when educators and administrators are positioned as ‘humans in the loop.’

Importantly, the close links between schools and prisons, mediated by AI and emerging technologies, have been found to have devastating consequences that extend beyond disciplinary pathways to include both socioemotional and academic harms. In their examination of “high surveillance schools,” that is, schools where both digital (e.g., content moderation, weapons detection systems, etc.) and analog (e.g., drug sniffing dogs, school resource officers, etc.) surveillance technologies are in high use, Johnson and Jabbari (2022) found that “schools ranking highest in surveillance infrastructure suspend more students than schools that rank among the lowest in their surveillance capability, even when controlling for school social disorder and student misbehavior” (p. 1). They simultaneously observed spillover effects at the school level, noting that high surveillance schools produce heighted feelings of stress, chronic worry, deflated feelings of belonging, as well as reductions in mathematics performances, and lower college enrollment rates for suspended and non-suspended students alike (Johnson & Jabbari, 2022). In the end, this study showcases how anti-black approaches to AI-powered technologies can have far reaching and multifaceted harms, posing a salient threat to the physical, socioemotional, and academic wellbeing of the entire school community.

2.4. Toward a More Critical and Expansive Study of AI in Education

We offer these macro- and micro-level analyses as a point of departure from ahistorical, race-evasive (Annamma et al., 2017), and apolitical discussions of AI as a purveyor of educational equity in general, and for historically marginalized students in particular. In this article, we articulate a need for more holistic analyses that move beyond a singular focus on ChatGPT and chatbot tutors to concurrently consider the wide range of AI systems that create, reproduce, or exacerbate inequitable schooling and social conditions for Black youth. To do so, we adopt a more critical notion of equity than the one currently used in mainstream characterizations of AI in education; one that moves beyond grades and test scores to simultaneously consider how these tools mediate Black students’ ability to experience inclusion, protection, safety, nurturance, and acceptance in schools (Love, 2016). We also need analyses that consider how antiblackness continually manifests within socio-technical infrastructures (e.g., code, algorithms, data) of AI technologies, creating new algorithmic barriers to equity, access, wellness, and safety for Black students that often go unnoticed in broad scale, race-evasive analyses. In doing so, we move from a purely technological perspective of AI in education towards a socio-technical approach: a perspective that understands that technologies are imbued with logics of antiblackness, racial capitalism, and misogynoir, and as such, operate in ways that hide, speed up, and automate the inequitable schooling conditions they were purportedly designed to remediate (Benjamin, 2019). In the following section, we describe how we leverage critical race technology theory (CRTT) in education (T. Tanksley, 2019, 2025a) to conduct a historically anchored, algorithmically conscious, and systems-focused analysis of AI in schools, and do so in collaboration with, rather than on behalf of, Black youth.

3. Theoretical Framework

Designed in response to calls for more intersectional, algorithmically conscious and ecological examinations of digital technologies in educational settings (Vakil, 2018; Nichols & Garcia, 2022; Selwyn, 2010), CRTT works to expose the inter-centricity of antiblackness as the default setting and organizing logic of schools and digital technologies (T. Tanksley, 2019; T. C. Tanksley, 2024). This framework disrupts majoritarian narratives that characterize digital systems as post-racial, apolitical, and inherently democratic. Instead CRTT acknowledges the matrices of domination encoded within twenty-first century technologies (Noble, 2012, 2014, 2018a), and in doing so, we shift discourses toward more critical engagements with AI by taking serious how digital technologies are a site of power and control over Black life (Noble, 2018b). Ultimately, CRTT exposes the racialized layers of subordination embedded within digital technologies that have historically restricted Black youth’s access to, representation in, and agency over digital systems that influence their educational, sociopolitical, and technological experiences (T. Tanksley, 2023, 2025a). The following tenets form anatomical underpinnings of CRTT in education.

3.1. The Intercentricity of Algorithmic Racism

CRTT in education acknowledges that racism is permanent and deeply ingrained within the fabric of American society, both on and offline, and should therefore be centralized in discussions of educational equity for Students of Color (Bell, 1995; Pérez Huber & Solorzano, 2015; Ladson-Billings, 1998). Foundationally, CRTT acknowledges that the ‘institutional disease of white supremacy” (Solorzano, 1997) has been digitally upgraded and algorithmically codified, and that its invisible presence within school-based technologies, learning platforms, and AI systems must be regularly interrogated (Crooks, 2024). In our present study, this tenet illuminates antiblackness as not only the animating force of analog algorithms (e.g., racially biased discipline practices that criminalize Black hairstyles, clothing, and language use), but also of digital algorithms embedded within school-based technologies (e.g., content moderation software that block and penalize searches for Black history) that make Black students hyper-vulnerable to educational pushout.

3.2. Challenging Dominant Ideology and Techno-Solutionism

CRTT in education encourages scholars to interrogate dominant narratives of race, education, and technology, and challenge oversimplified constructions of digital technologies as post-racial, ungendered, and politically neutral. In our present study, CRTT disrupts popular characterizations of AI and EdTech as inherently progressive and capable of advancing equity on behalf of marginalized students. By expanding definitions of equity to include a focus on safety, belonging, consequential learning and wellness, this study exposes the central, yet largely invisible, role AI systems play in automating school discipline, dehumanization, and carceral contact for Black youth.

3.3. Commitment to Socio-Technical Justice

In its struggle toward sociotechnical justice, CRTT aims to abolish algorithmic racism, as well as to eliminate all other forms of sociotechnical oppression along axes of class, gender, sexuality, and more. We posit that a social justice agenda in education must expose and eradicate algorithmic oppression that currently pervades digital technologies and sustains educational inequities both on and offline. In our present study, CRTT demands a thorough justice-focused interrogation of the sociotechnical systems that automate the school-prison apparatus, including a concomitant commitment to dismantling these systems.

3.4. Centrality of Experiential Knowledge

CRTT in education fundamentally recognizes that lived experiences of racially minoritized students are legitimate and critical to understanding current conditions of educational inequity and sociotechnical oppression. In turn, we position the voices, experiences, and sociotechnical insights of Black youth as invaluable to the production of knowledge on AI, educational (in)equity, and school-prison apparatus landscapes.

3.5. Interdisciplinary Perspective

CRTT also actively integrates race and racism within an interdisciplinary context by drawing upon scholarship from ethnic studies, feminist theories, media studies, digital humanities, and critical science and technology studies, affording it the possibility to expansively intervene theoretically and empirically across disciplinary boundaries. With roots in Black feminist thought, and Black feminist technology studies (BFTS) in particular, CRTT is attuned to the carceral geographies and technologies of surveillance that have been erected throughout history to constrict, constrain, corral, and cage Black bodies both within and beyond the Imperial Core. Consequently, CRTT can be used to unearth and interrogate the underlying currents of surveillance (Browne, 2015; Noble, 2018b), captivity (Wun, 2016a), discipline (Annamma, 2017; Love, 2016; Morris, 2016), spirit murder (West, 1992; Williams, 1991), and Black death that exist within broader macro-structural antiblack schooling apparatuses.

Through this socio-technical and historically grounded lens, we come to understand heightened school disciplining for Black students as not solely a result of AI’s enmeshment with schooling, but rather a contemporaneous manifestation of antiblack surveillance practices developed during enslavement, refined during Reconstruction and Jim Crow, and algorithmically updated in the digital era. Notably, Black feminist technologists, education scholars, and historians alike have remarked that increased surveillance, punitive responses, and criminality have been central to the state’s technological approach towards Black people, dating back to the creation of slave brandings and slave patrols, the use of lantern laws and the Book of Negros, the creation of Big Brother and COINTELPRO, and the refashioned modes and development of the panopticon (Browne, 2015; Benjamin, 2019; Ross, 2020). These early efforts to identify, track, quantify, and control Black bodies in motion (Tillman, 2025) helped create the racially disparate datasets upon which contemporary AI technologies are trained, inevitably producing digital systems wherein antiblackness exists as neither a bug nor a glitch, but as a foundational design feature (Broussard, 2023). Historic practices of datafied surveillance and quantified control are, therefore, part and parcel of the AI technologies that currently undergird public schools, where the ubiquitous presence of biometric detection systems, content moderation systems, predictive analytics, and computer vision systems not only allow for but actively normalize constant observation, control, and punishment of Black youth both within and beyond the classroom.

4. Methods

This study centers the voices, experiences, and sociotechnical insights of forty-six Black high school students that participated in a college bridge summer course on race, education, and technology in Southern California. The data for this paper is a smaller subset from a larger, multi-year study on how Black students develop critical algorithmic literacies that enable them to read, resist, and reimagine antiblack technologies within and beyond school. Our data for this study includes three subsets of data from the larger study: semi-structured interviews, weekly digital journal reflections, and student video blogs (vlogs; see

Table 1. Data Source Descriptions and Examples.

Data Source	Description	Example Prompts
Student-Generated Video Blogs (vlogs)	- Completed at the end of each class session - Lasted about 7–10 min each - Students gathered into groups of 4–5	- “What stood out to you about today’s lesson?” - “What experiences, if any, have you had with the AI powered educational technologies we discussed today?”
Weekly Student Journal Reflections	- Students would select 1 of 2 assigned news articles and write a 5–7 sentence response	Sample articles: - Inside America’s School Internet Censorship Machine (Feathers & Mehrotra, 2023) - AI is Supercharging the School to Prison Pipeline (Okoh, 2023)
1:1 Semi-Structured Interviews with Youth	- Conducted over zoom at the end of the 5-week program for 45–60 min	- “What AI technologies, if any, does your school allow you to use?” - “Tell me about technologies your teachers, administrators, or school resource officers use.” - “What effects, if any, do anti-cheating technologies have on your educational experiences?”

below). Our decision to include multiple sources of data enables us to triangulate our findings for purposes of validity and reliability (Rodwell, 1998), but more importantly, to ensure that this work thoroughly honors and accurately showcases Black students’ voices, insights, and perspectives (Bhattacharya, 2017). We also selected data sources that would allow us to demonstrate youth perspectives that arise in various settings (e.g., individual, small group, one-on-one), modalities (e.g., written work, live class discussions, recorded videos), and at different points in the course (e.g., daily, weekly, once per program).

The first author conducted semi-structured interviews at the conclusion of the 5-week summer program, which were done synchronously via Zoom. Each interview lasted between 45–60 min, and participants were asked to share their thoughts on the course, on AI and EdTech, and the types of everyday experiences they have with AI at school. For the purpose of this article, we analyzed student answers to questions related to AI and school discipline (e.g., “What AI technologies, if any, does your school allow you to use?” “Tell me about technologies your teachers, administrators, or school resource officers use.” “What effects, if any, do anti-cheating technologies have on your educational experiences?”; etc.).

In addition to analyzing student interviews at the close of the program, we analyzed their reflective journal entries, which were assigned weekly on Fridays and submitted to our online portal by Sunday evening. Each week, students were given a choice between two different articles related to the week’s discussion topic (e.g., EdTech and surveillance, social media algorithms and antiblack racism, AI-generated content and digital blackface, chatbots and mental health, etc.) and were asked to write a short journal reflection on the article of their choosing. This was not a graded assignment, and students were simply asked to respond casually to what they read, either making connections between the article and their lived experience, or between the article and class discussions/readings more broadly. Students were encouraged to interrogate, further research, and ‘fact check’ articles against their own experiential knowledge. The decision to make this an ungraded assignment, remove any requirements for length or ‘writing style’ (e.g., formal vs informal), and explicitly name that students are not expected to agree but to analyze and make connections was an attempt to ensure their responses were candid. It was also another way to help them build skills in critical thinking, systemic critique, and counter-storytelling, encouraging them to practice speaking truth to power.

Finally, student-generated video vlogs were completed at the end of each class session and lasted about 7–10 min each. Students gathered into self-selected small groups and used an iPad and a mini microphone to record their thoughts on the day’s lesson. Students were instructed to treat these sessions as video blogs (‘vlogs’) similar to social media podcasts, TikTok lives, and Twitch streams where teens and young adults discuss trending topics for either a live or asynchronous audience. The activity, which encouraged the students to speak to an imaginary audience of other teens and young adults, allowed them to engage in practices of communal knowledge construction (Collins, 1986) that was already familiar to them. At the end of each class, the first author uploaded the recordings and had them transcribed, while also summarizing and documenting the main themes of the interaction, and entering the files in a comprehensive video data log for record keeping.

Data Analysis

As authors, we were interested in understanding whether and how AI technologies impacted students’ disciplinary experiences. We adopted an expansive view of equity-related impacts of school discipline, including mental health, peer-to-peer and student-teacher relationships, sense of safety and bodily autonomy, and academic performance, to name a few – impacts that are documented extensively across school discipline scholarship. We thus began our analysis by focusing on student interviews, as we were most interested in analyzing students’ thoughts after completing 5 weeks of discussion and reflection during the summer program. Specifically, we began an inductive open-coding process (Charmaz, 2006; Corbin & Strauss, 1990) for all forty-six interview transcripts for themes related to students’ on-the-ground experience with AI-powered technologies in their schooling context. These interviews were key to understanding the types of AI-related encounters that were most frequently and most saliently impacting students’ learning experiences. Once we established preliminary codes from the interview dataset, we conducted a review of reflective journal entries and student vlogs, looking specifically for instances where students discussed educational technologies they used or encountered in school, including AI writing assistants (e.g., Grammarly), learning technologies (e.g., ChatGPT), anti-cheating technologies (e.g., TurnItIn), school safety technologies (e.g., GoGuardian), and the like. Our iterative review yielded forty-six reflection papers and fifteen student vlogs that met the criteria. As we completed this coding process, moving across all three datasets, our codes were refined, collapsed, deleted, and expanded until we left with four major themes (Saldaña, 2013), which we showcase in our Findings.

5. Findings

Our findings are structured by four prominent themes related to AI, educational (in)equity, and disciplinary practices anchored in the experiential wisdom, cultural insights, and sociotechnical analyses of Black students: (1) AI-powered technologies restrict and restrain Black students’ access to high quality learning experiences; (2) AI-powered technologies restrict and restrain Black students’ access to culturally affirming and racially just learning experiences; and (3) AI-powered technologies cause racialized harms across socioemotional and academic domains; and (4) AI-powered technologies exacerbate the scope and scale of the school-prison apparatus. Notably, these findings are structured via a narration that intentionally centers Black student and youth articulations in written (journal entries), digitized (vlogs), and verbal (interviews) forms.

5.1. AI-Powered Technologies Restrict and Restrain Black Students’ Access to High Quality Learning Experiences

Black students in our study navigated a wide range of AI-powered technologies meant to block inappropriate content, ensure students stay on task during the school day, and detect concerning or inappropriate behavior in real time. While many of the study participants were sympathetic to the presence and use of AI by school staff and understood why educators would want to employ such tools, they found the technologies to be overly aggressive and noticeably harmful to both the teaching and learning process. For example, in discussing her schools’ content moderation software, Malia explained:

We have a lot of sites that we can’t access. You can’t go onto YouTube anymore, which I found problematic because that’s a big one for me, a way that I learn, especially if I’m not understanding lessons in class. I would go to YouTube in search of videos, but they had a lot of sites restricted. We can’t really use them all together, [even] for specific research things. One of my friends was trying to research the ties between ballerinas and prostitution in history for a research project, and she wasn’t able to access any sites that she needed for her research.

Tatiyana echoes these sentiments, noting that her school’s content moderation policy is “probably too aggressive.” She recalls a time when the system interfered with her college going efforts. She states, “[there’s] some websites that we can’t get on. I was trying to apply to a scholarship the other day, and it wouldn’t let me access it since the website was blocked. It was a scholarship website, so I don’t know why it was blocked.” When asked if she could simply wait until after school to access the scholarship website, Tatiyana noted that because she does not own a personal laptop, she must use her school-owned device at home as well. Since the content moderation systems exist both on the school Wi-Fi and are integrated on the devices themselves, the restrictions extend beyond the campus setting and include the students’ homes. Omar encountered something similar, noting, “there have been things where I might want to look something up for a class and haven’t been able to get the information, or, even when I’m at home, because the blocking software is still up at home. So, if I was trying to look something up [at home] or there’s no teacher near me [in class], I can’t get help.”

For Eva, the additional stress of having to navigate overly aggressive content moderation systems at home and at school is so taxing she often gives up on trying to access learning content or complete her assignments. She reflects, “When I would look up something, it would say ‘application blocked,’ and then to bypass it, an administrator would have to insert a password. I’m not even sure who knows the password, because I don’t even ask.” She then explains that the only viable workaround would be “getting my own device,” but because her family doesn’t have the means, she is “probably just going to wait ‘til college to get one.”

Importantly, the effects of overly aggressive content moderation systems do not merely extend to students, but also to the educators’ instructional content as well. Melody notes that because her schools’ content moderation system “will literally block anything and everything,” it also keeps educators from sharing informational resources and interactive learning content in their lessons. She shares, “it’s also irritating for the teachers too, when they’re trying to show us stuff but it’s blocked.” Angelo had a similar experience, noting, “when you’re not allowed to get on the website that your teacher needs you to get on, I would say that’s a big time [frustration]. I feel like sometimes they have the wrong things banned. There’s this one extracurricular I’m in for computer science, and it’s a coding class and the website that we use is banned on [our school-owned] device.” Eva states, “I remember my teacher put a video link in one of our assignments. Obviously, our teacher is not going to put anything inappropriate in our assignment. And then we couldn’t watch the video because it was restricted.” The heightened surveillant monitoring is thus extended to teachers’ (in)ability to conduct certain pedagogical and curricular learning activities, further constraining what Black students can engage in educationally at school.

In her small group vlog with Deondre and King, Natalia shared a similar critique about overly aggressive content moderation systems, reflecting:

What’s also kind of crazy is even before you search things, like, just typing it into the search bar, and before you even press enter… it’ll just shut down your entire page. I just thought that was pretty crazy. You’re monitoring the activity on my screen without me even having to click ‘Enter’ or anything.

These forms of hyper-surveillance were also concerning for Malia, whose school uses Hapara, an AI-powered classroom management tool that provides real time tracking, assessment, and remote control of students’ school-owned devices. In her final interview she reflects,

Our school is very big on Hapara. It’s a tool that teachers use, basically to look at what students are doing on their screens and they can use it throughout different classes as well. So you can be out of their class and they’re still looking at what you’re doing on their screens… it’s like you’re constantly being surveyed about what you’re doing online.

Malia was shocked to learn that her teachers watch students throughout the day, even when they are in other classes. She admits, “I just actually learned that today. One of my English teachers was talking about how she was looking on her [Hapara] screen, and she saw that somebody was off task, but then she remembered that they were in a different period. I was like, ‘I didn’t know that it didn’t stop once we left your class.’”

Unsurprisingly, this AI-powered surveillance culture had chilling effects on students’ academic performance. Malia admits, “it does affect me. I’m always thinking, what if she’s looking at what I’m doing or what I’m writing? I think it disrupts the flow of my work or my writing.” Malia was not alone in this feeling. King’s teacher will use Hapara to remotely lock his students’ pages when they are in other classes or even at home, directly impeding their ability to take notes, complete assignments, or fully participate in their other courses while outside of his classroom. In the small group vlog, King tells his groupmates, “I also have one teacher that just uses [Hapara] way too much. I think his is on a schedule, where it’ll lock out all your tabs at this time, and this time, and this time. And we might not even have him for that time period, but it still locks us out [of our computers].” Deondre agreed with King, and shared a similar story in the small group vlog:

Sometimes, with the blocking and closing of tabs, I’ll have past tabs, like articles and stuff open from classes that I’ll need the next period, but they’ll use [Hapara] and just delete all my tabs, so I have to figure out how to get them all back for the next period. And it’s annoying to have to do that over and over each time.

It appears that for the students in this study, AI technologies meant to support their focus and attention, protect academic integrity, and to ensure high quality learning actually operated in ways that impede and constrain those efforts.

5.2. AI-Powered Technologies Restrict and Restrain Black Students’ Access to Culturally Affirming and Racially Just Learning Experiences

In addition to restricting students’ access to educational content, resources, and opportunities in general, AI-backed content moderation systems also blocked students’ access to digital content related to Black culture, language, history, and activism. After reading an article about how content moderation systems work on an algorithmic level, Taj reflects on the racial disparities in the type of content that is restricted or flagged as inappropriate at his own public school. He writes:

In my school, we use filtering technologies similar to those discussed in the Wired article “Inside America’s School Internet Censorship Machine.” These technologies are intended to enhance safety by blocking access to harmful content, but in my experience, they often act as a restriction to certain information rather than a protective measure. For example, when I was working on a project about the civil rights movement, I found that many resources on figures like Malcolm X and the Black Panther Party were blocked. Also while trying to access information about LGBTQ+ history for another assignment, I encountered similar restrictions… This lack of social movements and historical content prevents students from accessing critical knowledge and engaging with relevant social issues. The system is filtering out the reality and realness of the world. This is reminiscent of the banking method in education where students are fed base-level information without challenging them or sparking social activism.

Tatiyana also read this article and found similar connections between the school district in the article and her own school. She writes:

… I realized that [the school district in the article] blocks a lot of things pertaining to race, gender, historical context, health/mental health, etc. This blocks their students from learning about things that might be important to them. In my school we have similar blocking systems that prevent us from looking at certain things. One time I was doing a research paper on the “Central Park 5” but when I tried to open an article it was blocked.

In his reflection, Angelo discusses the negative role that algorithmic racism plays in Black students’ access to and experiences with AI tools meant to support their learning, saying, “These systems have biases in place, holding stereotypes in the keywords used, and denying cultures of their interest in their heritage.” He goes on to explain his own experiences with biased AI technologies in school, writing:

Throughout my entire education [in Southern California], I have solely if not entirely used school provided technology, and I can connect my school’s online safety tools to similar protocols spoken in “Inside America’s Schools” article. These safety walls have been nothing but useless in my opinion over the years… Their solutions to solve [safety] issues aren’t genuine. Back in third grade my entire class got in trouble because one student searched [a racist slur] and airdropped a photoshopped photo to the entire class. My teachers and school staff solution to the problem was to give a short lecture to the entire class, and to reset the iPads, and giving them right back to the students without any updates or modifications to the censor system.

Jeremiah had a similar critique of schools’ use of AI-powered technologies to ensure “safety,” highlighting the uneven burden placed upon Students of Color that are also low income:

Schools have [content moderation systems] in place to make sure kids are “safe” but in reality I think it’s because of the information that they don’t want kids to find out about. Often when I was in my AP U.S. History class almost all of the links were blocked to do research on school computers. This also makes things harder for the kids that can’t afford personal computers to access all their work at school while other kids can.

Knox argues that algorithmic biases in school-based technologies result in Black students being “exploited into receiving base level information that doesn’t challenge them or spark social activism.” He believes “it’s not fair that [Black] students should have to be filtered out the reality of the world.” These incisive reflections, paired with Angelo’s and Jeremiah’s, suggest an insidious automation of marginalized epistemologies that render particular Black knowledge and historical events as inaccessible and out of reach within the hyper-surveillant structures of content control and management. This is exemplified further by Bella, who wrote:

My school-provided laptop uses filtering technologies similar to those mentioned in the article, and I find it to be more of a restriction than “safety.” When I try to access articles for assignments or YouTube videos, I am sometimes blocked even though the content is not “obscene” or “harmful to minors” as the “Children’s Internet Protection Act (CIPA)” aims to block… it is more of a learning restriction…Technology “safety” restrictions are suppressing [Black] students’ access to knowledge and therefore hindering their education.

To exemplify this point about encoded forms of antiblackness and other forms of identitarian biases, Melody adds:

A lot of school districts have begun to block information regarding identities, especially those part of the LGBTQIA+ community. One time I had to do a project and make a collage on the different types of media that influenced my life. At that time, I was using a school computer and was blocked from looking up PICTURES of artists or apps… doing this assignment on a school computer made it hard because everything that shaped who I am was blocked.

In their small group video blogs, Malia, Erica, and Iyana begin to analyze the larger implications of these restrictions, noting how AI-backed safety technologies contribute to antiblack racism and hyper-surveillance on a larger scale. As Malia theorizes,

… with the blocking of websites that show our history, I think that plays into what you [Erica] were talking about: how they structure schooling to teach you just one version of our history, like, enslavement. And if we were to go and search things that were radical or more accurate to our history, then it’s threatening to them. So, then it’s blocked on our websites, and we can’t view it. Then that plays into society as a whole feeling that black people are less than because then it’s even embedded into us, like the way that we think about ourselves.

Iyana makes a similar connection, noting the subtle, but dehumanizing message that uncritical and inequitable approaches to AI have on Black youth specifically. She asserts:

… the content restricting things when you get flagged for something, that happens at my school. There’s two schools on my campus. There’s a [charter] school which is predominantly white, and there’s my [public] school which is predominantly Black. But when you’re on my school’s WIFI, you can’t open up certain tabs and a lot of the stuff that we have is restricted. They actually monitor what we’re looking up and what we’re doing. But as soon as I would switch over from my internet to [the charter school] Wi-Fi, I would have no restrictions. I could look up whatever I wanted to look up, and it would be completely fine. So, it just shows the distrust they have in Black students compared to any other students.

Iyana’s reflection forces us to consider how surveillance and digital discipline operates differentially, positioning Black students as incapable of ethically or responsibly utilizing web-based programs to access pertinent learning content without adult-centric hyper-monitoring or institutionally restrictive mechanisms in place. It simultaneously highlights how the use, design, and deployment of digital technologies are differentially employed based on racism, classism, and antiblackness, with Students of Color from low-income backgrounds enduring heightened surveillance, limited learning opportunities and harsher disciplinary responses as a result.

5.3. AI-Powered Technologies Cause Racialized Harms Across Socioemotional, Academic, and Disciplinary Domains

While the AI-backed content moderation technologies worked in ways that restricted Black students’ access to intellectually engaging and culturally affirming learning content, their teachers’ increasing reliance upon anti-cheating technologies like AI detectors created a host of socioemotional, academic, and disciplinary issues that threatened to widen gaps in educational equity for hyper-vulnerable students. As Malia explains, “they don’t want you using [AI] for writing, so they’ll use [an AI detector] or some other method, like judging your word choice or sentence structure.” Despite being promoted as objective and effective, the anti-cheating technologies were believed by the students in this study to be inherently faulty at identifying AI-generated writing. In fact, nearly all the participants had been falsely flagged at least once, with many of the students reporting multiple false accusations that negatively impacted their grades, their relationship with their teacher, or their feeling of respect and belonging in the class. Eva recalls a time her Advanced Placement (AP) language teacher flagged her for using AI to cheat on a poetry assignment. She recalls:

I used to get so mad, because I would write my work, and she would say, “Oh, well, the AI detector said that because you use the word “delve” too much and that it’s AI.” So she would give me zeros on my work… Because I use “too many words that AI uses.” … it’s like, I know words like delve. I swear I’m not cheating.

Ashton also recalls a time he was accused of using AI for a major course assignment, stating:

I got mad, because I had to do a visual arts class, so I had to do a comparative study on different artworks. I took so long to compare the artworks. And then [my teacher was] like, “make sure you use your citations because if you didn’t write it all yourself [you’ll fail]. And she was like, “you can’t copy from AI.” I was like, “this isn’t the AI. I know my writing sounds good but it wasn’t AI”… but she literally put that on my feedback for my paper.

Angelo had a similar experience, noting:

I go to a predominantly Black school, and last week, I wrote an essay. Sometimes, I like to add a couple things. So instead of saying “after,” I’ll say something like, “post,” you know? Be a little fancy. So for my homecoming essay, one of my teachers came up, and she was like, “did you use AI?” And I was like, “No, I just, I genuinely just write like that sometimes.”

During a whole group discussion about being falsely flagged for cheating with AI, students were asked to explain what they thought teachers meant when they would say their work “sounded too much like AI.” The youth used words like “intelligent,” “smart,” and “scholarly” to describe the surface level assumptions, and “not Black” and “whitewashed” to describe the underlying racial connotations of these statements. When asked how they know this was a racially coded assumption or micro-aggression, the students describe disparities they have witnessed between students that are more regularly flagged for cheating and the students that are not as experiential evidence. For instance, Eva recalls:

I’m not gonna lie, there’s this group of three white Hispanic girls in my class, and they all have perfect grades. None of their stuff ever got flagged. And they’re actually some of the smartest people I know… They use the same words I use… our papers look the same most of the time, and all their grades are really good. But they never got flagged. I think [the teacher] probably just never checked it.

As Eva notes, flaws present within AI technologies can be compounded by biased beliefs and practices maintained by educators, particularly if those educators let racialized assumptions about students’ intelligence inform whether they believe, or even check, the outputs of these tools. To prove Eva’s point, Erica recounts a time her teacher openly admitted to having low expectations for Students of Color. She reflects,

I have one teacher in particular… he automatically assumes that not even just the Black students but any Students of Color are not able to process what he’s saying… when we ask him for help, he’ll be like, “Oh, you guys are dumb” and “this is what’s wrong with you,” but then he’ll take extra time to help the white students that he feels actually deserve it.

According to Erica, this teacher uses AI-powered technologies that falsely flags his Black students more often than his non-Black ones. Rather than view these disparate outcomes as evidence of a faulty technology, this teacher likely had his deficit assumptions about Black students’ racial, cultural, or intellectual inferiority algorithmically confirmed. Leveraging her peers’ stories as an analytical point of entry, Leila challenges the popular claim that educators and administrators can operate as a ‘failsafe’ against racially biased AI systems, a concept known in the tech industry as “keeping humans in the loop.” She argues:

… the part where they were like, “it’s fine, because people are in the loop.” It’s like, yes, people are [in the loop], but also a lot of people in this world are biased towards our race. People think that we’re all sorts of different things that are negative and that we don’t impact the world in a beneficial way. Even though people are in the loop, they’re not in the loop to support us or to go against these [racist] AI outcomes.

Unsurprisingly, being presumed criminal, incompetent, and incapable by both teachers and artificial intelligence systems had both academic and socioemotional tolls on the youth in this study. According to Angelo, “it hurts my sense of pride. It makes me feel a certain way…like you take pride in everything you do. So when someone comes and claims [you’re using AI to cheat], and when it happens over and over again… you’re just defeated.”

In addition to hurt feelings, reduced pride and deflated sense of belonging, students also experienced heightened stress and anxiety. Malia describes anti-cheating technologies as “another layer of stress” for students, and as a result, she has become extremely worried about being falsely accused by the faulty tools. She states:

The worst part about writing a paper for me is turning it in, because it’s just the stress of not only turning in, but also, will they accuse me of AI? Will they accuse me of plagiarism? And will I be kicked out of this class without being able to defend myself because the technology said that I cheated?

Indeed, the academic consequences of being falsely accused using AI can be steep, resulting not only in failed assignments, but also failed classes and removal from AP pathways. As Eva explains:

Sometimes teachers don’t [listen], if they’re stern and they think because the system says you used [AI], they won’t change [your grade]. It’s kind of bad, because it really can affect your grade depending on what assignment they think you cheated on. Right now my current AP Lit class essays are worth 40% of our grade. So if I were to get flagged… my grade would just be a zero, and that’s like a whole 40% drop in my grade.

As a result, Black students are placed in an impossible bind: receive a failing grade that tarnishes their college prospects and academic record or go up against a teacher and their purportedly neutral AI technologies. Eva captures the added stress and emotional labor that comes with having to do just that, stating:

It was so awkward. I hate talking. I hate standing up to people. But I had to go to her class, and I was like, “hey, you flagged three of my assignments, and you didn’t give me full credit or any credit, because you said I used AI.” And she was like, “Oh, I put it in the system…and it was just too wordy and too “AI-y”… And then she pulled out poems and she was quizzing me on them, like “after you said this, what was the next line?” Or, “what word did you put here?” She even asked me if I knew what the word “fleeting” meant because I had that in my poem, so I think that also got flagged by AI.

As Eva’s example shows, being falsely accused of AI can not only result in emotional distress, but also additional work and stigmatization as teachers pressure students to prove their intelligence on the spot through additional tests and quizzes. When Ebony submitted a research paper on Frederick Douglas that was falsely flagged for cheating, she says “it was so scary, because for a second I was getting a zero, because if you get caught using AI for most teachers, it’s an automatic zero or fail. You can even get detention or suspended.”

As Ebony points out, false flags cause more than just socioemotional and academic harm; they have disciplinary consequences as well. Natalia explains, “If you get caught using ChatGPT, then you’re gonna get suspended, or you can even get expelled for cheating.” Unfortunately, these harsh disciplinary measures are not isolated events and were prevalent within all the schools that the study participants attended. As Amare states, “my school actually has zero tolerance on ChatGPT, a zero-tolerance policy on AI and different ChatGPT apps.” He then says, “you get a zero, and then you have to talk to the dean, and then it goes on your [permanent] record, so teachers can like see it like, for example, if you want to ask for a letter of recommendation, then they can see that, like you got a violation.” Importantly, it’s not lost on him that the “crime” of using AI (whether accurate or not) is given a carceral label, making the connections between AI, school discipline and the carceral apparatus even more apparent. As these narratives suggest, the carceral design and usage of AI in schools threaten to push more students out of schools and into more stringent containment pathways.

5.4. AI-Powered Technologies Exacerbate the Scope and Scale of the School-Prison Apparatus for Black Students

When asked to consider the larger implications of the rapid, wide-scale adoption of AI technologies for historically marginalized students, Black youth in our study cited socioemotional, academic, and carceral harms as their primary concerns. Using his personal experience as a point of entry, Malik writes, “Overall, I was dumbfounded by the sheer disparities in opportunity for minority students [to be surveilled] and it caused me to question the current technological makeup of schools.” Likewise, Saleni writes, “With the constant obstacles and discrimination most students of color face, utilizing AI to dehumanize them makes it worse. Ultimately, this can result in students feeling criminalized, mistrusted, and alienated in spaces that are meant for learning and growth.” Chelley echoes this in her own journal reflection, asserting, “There is no regulation on AI surveillance or the privacy of public-school children. I think it is detrimental to students’ motivation because the more they realize they are being discouraged from pursuing an education, the more they will believe they are not capable of gaining one.”

Importantly, students saw clear connections between socioemotional harms of carceral approaches to AI and devastating impacts on Black students’ academic performance, achievement, and overall motivation to succeed. Derrick wrote the following in his reflective journal:

First, the psychological implications that AI is employing by targeting these communities is dangerous. For example, the amount of schools that use a form of [AI] monitoring might set off a chain reaction: sending signals to teachers on “who to look out for,” leading to unjust treatment towards those students, finally resulting in demoralized attitudes and lower performances by the community.

Khelani feels similarly, writing:

… these tools are more prevalent in schools where there are mostly minorities, which can affect the academic performance of Black students. This stood out to me because in class we talked about how sometimes AI gives out wrong information and shows racism towards Black children. If the AI systems schools are using are not 100% correct it can harm the future of students.

Gabriel challenged the notion that AI closes ‘equity gaps’ for Black students and instead asserts that they work to reinforce existing disparities. He writes:

One effect this approach to AI use will have on marginalized students is it will discourage them when it comes to their education. Black students don’t receive the proper amount of support due to them going to schools with low funding, so AI being used to monitor their progress won’t make it easier to succeed.

Interestingly, the youth in this study drew additional connections between decreased motivations and feelings of belonging, to school pushout and carceral contact. As Chelly notes, “Before new technologies like artificial intelligence, Black and other students of color were already tracked for school dropout and criminality at high rates. With new algorithmic predictors, the issue seems exponentially worse.” Noriah sees similar issues around AI-mediated discipline, writing:

I attend a high school with one of the highest Black student populations. Over the years, I’ve seen an increasing number of school-based police sent by the district to “discipline” us. Overall, I think this approach to AI will only harm marginalized students further. The constant monitoring and surveillance may contribute to mental health struggles, ultimately leading to increased dropout rates.

In addition to fostering academic disengagement and school pushout, Jonathan felt that these tools may be contributing to early carceral contact, particularly for hyper-vulnerable students. He reflects:

AI-driven security systems are not perfect; they are prone to racial biases, which lead to students being labeled wrongly, and there have even been instances of students being wrongly arrested. If this were to happen to a student who was already wavering between the wrong and right path, if he were falsely arrested, he would most likely think that if that was already what people thought of him, he would head down the wrong path. This can create a vicious cycle in which students are pushed further from school and assistance. Instead of feeling safe, they feel targeted and hopeless.

Ultimately, the students in this study felt that educators’ use of AI for school safety and behavioral management, while well intentioned, risked the safety of Black children by emboldening the connections between schools and prisons. Saleni pens:

The use of AI surveillance in schools absolutely has negative effects on marginalized students… Rather than schools creating safer learning environments where their students can safely express themselves, this approach with the use of AI technology often reinforces existing biases/stereotypes and creates a systematic prison environment. Instead of utilizing these technologies for better uses… it is being utilized to treat students like prisoners… When algorithms flag students based on factors like attendance, grades, or behavior history, they can mislabel them as threats, leading to increased discipline, and sometimes even police involvement. Honestly, who wants to experience this at their school?

Kaiden echoes these sentiments, writing:

What stands out most is how AI is being used in schools under the mask of safety but is in reality enforcing harmful systems of control, especially for marginalized students. Certain technologies like facial recognition, predictive analytics, and some behavior monitoring software in schools target Black and low-income populations. Rather than addressing this, the tools unfortunately increase surveillance and punishment, often without accountability or oversight. This reminded me of many discussions I’ve had with my peers about systematic bias, where discipline and security measures criminalize, rather than support Black students. This for me raises lots of concerns. It assumes students that are in marginalized communities are threats to be managed.

Simone feels similarly, stating, “What stands out to me the most is the idea that AI, which is meant to be seen as a tool of progress, is actually a harmful system… This technology, labeled as a security measure may do more harm than good when it’s used to police rather than protect… It makes me wonder: are schools worried about students’ safety, or are they worried they’re suspects?” Keon writes, “This makes me truly think that AI is becoming more than a tool for homework, but another way to spy and invade our privacy, not just as students but future prisoners and criminals. Which only makes me believe that the system wants the school-to-prison pipeline to happen.”

Importantly, the critiques of AI by the students in this study were not solely focused on the antiblack and carceral uses of AI in their schools (e.g., the decision to adopt a slew of anti-cheating and surveillance technologies to monitor Black students deemed inherently criminal and inclined to cheat), but also the antiblack design of these tools. Chuck captures this dynamic succinctly when he writes:

What worries me is that this kind of technology can make mistakes and start to treat students like threats. When AI is being trained, that data is biased. It targets students who already face racism and other unfair treatment. This means that students who need support might get punished instead of receiving help. It reminds me of the school to prison pipeline.

Malik further captures the sinister interplay between schools, biased AI technologies, and the prison industrial complex, noting how datafied surveillance and algorithmic antiblackness help automate and normalize educational carcerality (Cabral, 2023), inequality, and school pushout for marginalized youth. He writes:

I have realized the unjustness of AI being used in the schooling system. Originally, I thought AI was only being used for creating assignments and checking for ChatGPT usage in homework. However, it is also being used for a larger, more harmful purpose behind the scenes. For instance, students are having their databases compromised (which includes their grades, attendance, histories of child abuse, etc.) by police officers, and their AI program is predicting who is most likely to become a criminal when they grow up. Therefore, they are able to surveil children and create a closer connection between school performance and incarceration. Granted, there are widespread inaccuracies and encoding bias that occurs within these systems regarding race, which makes the color of your skin a risk factor.

Chuck concluded his journal entry with a salient warning and call to action, having written, “If we don’t change this, AI could make schools even more harmful, especially for the students who are already struggling the most.” Kaiden agrees, stating, “the longer-term effect of this AI-driven monitoring may not just be increased policing in schools, but also lasting harm to students.”

6. Discussion

The sociotechnical insights and analyses offered by Black students in this study challenge mainstream assertions that AI-powered technologies are inherently capable of advancing equity, access, opportunity, and safety on behalf of and for Black youth. Instead, their testimonies illuminate how AI systems, implemented uncritically, race-evasively and at-scale, often create new algorithmically fortified barriers to high quality learning and educational opportunity for Black youth, particularly as it relates to discipline, surveillance, and school pushout. Further, while mainstream narratives would suggest the primary avenues by which AI comes into urban schools is via learning technologies, such as Khanmigo, ChatGPT, or Magic School AI, the youth in this study expose how the school discipline and carceral apparatus is one of the most pervasive avenues for AI adoption in schools that serve Black and Brown youth.

Importantly, the students’ astute assertion that AI-powered technologies, like weapons detection systems, content moderation systems, and anti-cheating technologies, are disproportionately placed in low-income Black and Brown schools due to logics of antiblackness that render Black children as inherently criminal, aggressive, and in need of constant surveillance, is confirmed by extant literature showing emerging relationships between the creation of high surveillance infrastructures and the over-representation of racially marginalized and low income students (Laird & Dwyer, 2023; Johnson & Jabbari, 2022; Feathers, 2023; Okoh, 2023). This macro-level analysis was perhaps most concretely evidenced when Khalid, Iyana, Tasha, and Noriah discussed the sociotechnical distinctions between their predominantly Black public high school and a predominantly white private charter school that shared the same campus. Following a lesson on historic redlining in Los Angeles, the youth in this study identified a phenomenon they described as ‘digital segregation’ and ‘algorithmic redlining,’ explaining that while low-income Black and Brown students are forced to overcome multiple algorithmic barriers in order to access the most basic learning opportunities, their wealthy non-Black counterparts have access to unrestricted, surveillance-free learning just a few feet away. When viewed through a CRTT lens, it becomes clear how antiblack carceral approaches to technology work to separate, stratify, and segregate Black youth via technologically fortified carceral enclosures, producing a new era of ‘separate but equal’ under the cloak of techno-solutionism and tech neutrality. As Crooks (2024) so aptly asserts, ‘access’ to technology for Black and Brown youth is often ‘capture’ by a different name.

Alongside these macro-structural manifestations, whereby schools, districts, and entire communities are not only cordoned off, but educationally suffocated by technologies meant to provide access and opportunity (Crooks, 2024; Benjamin, 2016), exists a slew of micro-level interactions that foster socioemotional, academic, and carceral harms for marginalized youth. As participants explained, having to constantly confront teachers and technologies that positioned them as inherently incompetent and incapable of academic excellence affected their sense of pride, feelings of belongingness, and socioemotional wellness. It also threatened their current and future academic opportunities, as multiple students endured failed assignments, dropped courses, and behavioral ‘violations’ that were added to their permanent records. As Axel, King, and Chuck note, schools not only submit disciplinary violations recorded on your permanent record to colleges when they send your transcripts, but they are also shared with high school teachers and used to inform the recommendation letters teachers write for college. Finally, and perhaps most saliently, Black youth navigated increasingly punitive learning conditions that resulted in higher levels of detention, suspension, and expulsion for everyone in the school, regardless of guilt or innocence, echoing scholarship documenting school-wide consequences of anti-black ontological (Tillman, 2025; Monroe, 2005; Coles & Powell, 2020) and socio-technical (Johnson & Jabbari, 2022) surveillance infrastructures. The youth in this study were all high achieving students, with every single participant enrolled in multiple AP courses throughout their high school career, including at the time of this study. Nonetheless, these students found themselves caught in a techno-educational carceral web that resulted in lowered teacher expectations, increased surveillance, reduced opportunities, and an uptick in disciplinary consequences for the entire school community.

Our participants’ narratives confirm scholarship positing that antiblackness not only allows for but requires schools to engage in hyper surveillance, criminalization, and punishment of Black culture, identity, language, and histories (Tillman, 2025; Dumas, 2014; Jenkins, 2021, 2022). CRTT further reveals how these historically anchored racial logics are now algorithmically encoded into the sociotechnical infrastructures (e.g., algorithms that flag AAVE and discussion of Black identity as dangerous or inappropriate) and architectures (e.g., biased AI tools that are directly connected to law enforcement) of AI-powered educational technologies. This perhaps best exemplified in our participants’ sustained inability to access content about salient historical events and political topics, including Black Lives Matter, LGBTQ+ rights, reproductive and sexual justice, and Civil Rights activists, on their school devices and using their school Wi-Fi. As Melody, a queer Black girl, surmises, “everything about my identity is blocked.” Though small, and seemingly harmless, CRTT allows us to name these sociotechnical invalidations as algorithmic microaggressions that (re)produce socioemotional distress, academic underachievement, and school push out and disengagement in new and familiar ways (T. Tanksley & Hunter, 2024). As research shows, the cumulative weight of microaggressions can be devastating, resulting not only in anxiety and depression, chronic illness, and cardiac strain, but also shortened life expectancy and premature death and dying (Pierce, 1995). Cases like those of Taki Allen, a teenage boy that was arrested on campus after a weapons detection system flagged his bag of Doritos as a gun; Lamaya Robinson, an adolescent girl that was misidentified by a facial recognition technology as a wanted suspect accused of committing a violent crime at a skating rink she had never visited; and Michael Oliver, a young man that was wrongfully arrested when an AI system falsely identified him as a student suspected of committing a violent assault against a teacher highlights not only the severity, but also the ubiquity of material violence meted out to Black adolescents and young adults by AI systems.

Without the mapping of a CRTT lens, these digitally mediated forms of violence could be reductively framed as random, unpredictable ‘glitches’ in otherwise objective and effective systems, whether it be the sociopolitical systems (e.g., where antiblackness quietly informs teachers’ disparate approaches to Black students’ use of AI) or technological ones (e.g., when content moderation systems block access to Black cultural content) that exist in schools. However, as articulated by Black youth in this study, carcerality and antiblackness exist as the ‘default settings’ and ‘organizing logics’ of both schools and school-based technologies, rendering Black youth hyper-visible in disciplinary scenarios while simultaneously reinforcing their invisibility in educational technologies and design features (Benjamin, 2019; see also Annamma et al., 2024; Wun, 2016b). In this way, AI technologies help to normalize schools as sites of Black suffering (Dumas, 2014) that deny Black students care, acceptance, safety, and opportunity via artificially reduced grades, heightened disciplinary action, automated carceral contact, strengthened surveillance cultures, and heightened racialized stress.

While our findings for this study provide a compelling answer to the ever relevant question posed by Du Bois, “How does it feel to be a problem?,” in the age of AI, we urge subsequent studies to concurrently answer the question posed by Love (2016): “How does a Black child live, learn, and grow when her spirit is under attack at school, and her body is in danger outside the classroom?” (p. 2). In order to answer this resounding call for abolitionist work within education (Cabral, 2024, 2025; Davis et al., 2025; Cortez et al., 2022), researchers and practitioners must extend these calls to the digital, working to develop, document, and deploy strategies of sociotechnical resistance that ensure Black youth can ‘do more than survive’ (Love, 2019) this new and renewed techno-educational carceral apparatus.

7. Conclusions

This paper leveraged CRTT (T. C. Tanksley, 2024; T. Tanksley, 2025a) to more closely examine the varying modes, forms, and styles of AI-powered technologies that function as surveillance mechanisms in the lives and schooling experiences of Black youth. Most saliently, the findings illuminated the role that AI plays in automating educational inequality, including denying access to high quality, culturally responsive learning; producing socioemotional and academic harms; and supercharging carcerality within the school-prison apparatus. In this way, AI becomes a form of carceral seepage (Serrano, 2024) and punitive inertia (Tillman, 2025), limiting students’ academic mobility and attainment, all while ensuring they are constantly surveyed, surveilled, stopped and stunted along their educational journey. Further, we showcased how AI-powered technologies encode antiblack biases that reproduced racialized harms across socioemotional, academic, and disciplinary domains. These insights overall demonstrate the persistent entrenchment of existing societal perspectives that yield longstanding deficit-oriented and dehumanizing assumptions about Blackness. After all, it is the hardwiring of antiblack logics into educational infrastructures and architectures that enable oppressive norms, policies, and ontological practices to exist unchallenged in both online and offline schooling contexts (Dumas, 2014).

Our aim through this article was primarily to signal awareness and continual urgency within growing interdisciplinary scholarship on the (mis)use and uncritical uptake of AI-powered technologies within and across schooling spaces. To do so, we took seriously the sociotechnical experiences, reflections, and articulations of Black youth situated in Southern California. Critically, the youth in this study expose, trace, and articulate how AI-powered technologies track and surveil them, including being monitored for their purported (mis)behaviors and race-specific knowledge production, stifling their attempts to meaningfully anchor their learning in Black and other marginalized racial/ethnic epistemologies, and having to navigate disciplinary policy landscapes that perpetually criminalize Blackness in both analog and digital contexts.

With this in mind, we encourage scholars and practitioners to continuously assess, audit, and interrogate how the ubiquitous and increasingly normalized presence of AI in schools—whether through the use of content moderation software and web filtering technologies, the installation of networked security systems and weapons detection systems, or the integration of predictive analytics and AI chatbots - is quietly exacerbating the scope and scale of the school-prison apparatus and the New Jim Code (Benjamin, 2019). We simultaneously urge scholars, educators, and other institutional actors to critically question the systematicity of corporate and governmental world-destructive practices where society is restructured by few elites into what Crawford (2021) delineates as an “atlas of AI.” Afterall, it is no coincidence that these attempts to algorithmically erase and technologically oppress marginalized youth in schools are occurring alongside efforts to abolish diversity, equity, and inclusion (DEI) programs, ban books on and courses about Black history, defund HBCUs, overturn race-conscious admissions, and overhaul hard won civil and human rights protections for Black students at the national level. When viewed through this lens, it becomes increasingly clear that AI is meant to play a key role in the success and rollout of white supremacy’s newest iteration: algorithmic oppression, digital segregation, and technological captivity.

Finally, we encourage readers to look toward Black youth’s development of critical sociotechnical literacies (T. Tanksley, 2023, 2025b; T. C. Tanksley, 2024; see also Arastoopour Irgens et al., 2025), as they will instruct and guide us toward abolitionist possibilities that are critical to our intellectual and material fight for a future sans digital carcerality and sociotechnical captivity. We must ensure that Black youth’s critical AI literacies no longer remain hidden or overlooked (Harvey & Howard, 2025). In effect, we must believe Black youth, such as Chuck, when they continually instruct us of the “kind of technology [that] can make mistakes and… treat students like threats.” These purported ‘mistakes’ are not worth the racialized, carceral, and death-making costs they are designed to produce.