Bias in Large Language Models: Origin, Evaluation, and Mitigation

Abstract

Large language models (LLMs) have revolutionized natural language processing, but their susceptibility to biases poses significant challenges. This comprehensive review examines the landscape of bias in LLMs, from its origins to current mitigation strategies. We categorize biases as intrinsic and extrinsic, analyzing their manifestations in various natural language processing (NLP) tasks. The review critically assesses a range of bias evaluation methods, including data-level, model-level, and output-level approaches, providing researchers with a robust toolkit for bias detection. We further explore mitigation strategies, categorizing them into pre-model, intra-model, and post-model techniques, highlighting their effectiveness and limitations. Ethical and legal implications of biased LLMs are discussed, emphasizing potential harms in real-world applications such as healthcare and criminal justice. By synthesizing current knowledge on bias in LLMs, this review contributes to the ongoing effort to develop fair and responsible artificial intelligence (AI) systems. Our work serves as a comprehensive resource for researchers and practitioners working towards understanding, evaluating, and mitigating bias in LLMs, fostering the development of more equitable AI technologies.

1. Introduction

The rapid development of large language models (LLMs) has transformed the field of natural language processing (NLP), introducing new possibilities and applications across various domains, including healthcare [1], finance [2], education [3], and entertainment [4]. These models, such as GPT-3 [5], BERT [6], and others, are designed to understand and generate human-like text by learning from vast corpora of text data. These models have become the backbone for tasks such as machine translation [7], text summarization [8], sentiment analysis [9], and automated question answering [10], reflecting their deep integration into various industries and day-to-day applications, and making them indispensable tools in the modern AI landscape.

Despite their immense potential and utility, LLMs have raised concerns due to their inherent biases that reflect societal prejudices present in their training data [11,12]. These biases, which manifest as gender, racial, cultural, and socioeconomic stereotypes, pose serious ethical and practical challenges, especially when LLMs are deployed in critical decision-making environments such as healthcare diagnostics [13], legal judgments [14], and hiring processes [15]. Such biases can lead to unequal treatment or skewed results that disproportionately affect marginalized groups, potentially exacerbating existing inequalities. Caliskan et al. [16] have noted that LLMs often mirror historical biases present in human language, risking the amplification of these biases within automated systems.

Understanding bias within LLMs requires insight into how biases traditionally arise in statistical models. In classical statistics, biases have been well studied in models for continuous, binary, and time-to-event endpoints [17]. These biases manifest due to several issues, such as selection bias [18,19,20], overfitting [19], handling of covariates [18,21], limited sample sizes [22], and even the randomization ratio [23], which can all lead to incorrect estimates and predictions. These statistical biases share important parallels with those encountered in LLMs. For instance, LLMs must handle incomplete or unbalanced datasets carefully to prevent skewed outputs. Similarly, the challenges of selection bias in classical models echo the biases found in LLMs trained on non-representative data sources, which can lead to harmful disparities in model predictions, especially for under-represented groups. Addressing these statistical biases requires a range of mitigation strategies, including a robust causal inference framework [24]. Similarly, bias mitigation in LLMs involves data-level interventions such as resampling and augmentation, model-level adjustments with fairness constraints, and post-processing corrections to fine-tune outputs [25].

LLMs also exhibit biases similar to those observed in traditional statistical models. Bias in LLMs can be broadly categorized into two types: intrinsic bias and extrinsic bias. Intrinsic biases originate from the training data, as well as the architecture and underlying assumptions made during model design [26]. LLMs, trained on vast datasets, often sourced from internet and textual repositories, inevitably inherit biases present in these sources. For example, language models may reinforce gender stereotypes in occupations, associating “doctor” with men and “nurse” with women [27,28]. Similarly, certain demographic groups may be either under-represented or misrepresented within training datasets, further compounding model biases [29]. This becomes particularly problematic when LLMs are applied in sensitive domains where accurate and unbiased information is crucial for decision making [11].

Extrinsic biases, on the other hand, emerge during the application of LLMs in real-world tasks. These biases are often more subtle as they manifest in the model outputs during specific tasks, such as sentiment analysis, content moderation, or automated decision-making systems. For example, Sap et al. [30] found that LLMs used in hate speech detection could disproportionately flag certain dialects or vernaculars, such as African-American Vernacular English (AAVE), as more offensive compared to standardized English, reinforcing societal stereotypes. Similarly, Kiritchenko and Mohammad [31] demonstrated that sentiment analysis systems could produce biased results when analyzing texts associated with different demographic groups, thereby perpetuating harmful stereotypes in their predictions.

The implications of biases in LLMs are far-reaching. In healthcare, for instance, biased models can lead to inappropriate treatment recommendations, potentially worsening existing health inequalities between demographic groups [32]. In legal settings, reliance on biased language models for risk assessment or sentencing decisions can result in discriminatory practices [14]. Moreover, biases embedded in LLMs affect everyday applications, such as search engines and social media platforms, where they shape public discourse, potentially reinforcing echo chambers and marginalizing minority voices [33,34].

These biases not only perpetuate inequality but also raise significant ethical and legal questions regarding the use of LLMs in decision-making processes. Given the critical nature of these challenges, there is an urgent need for comprehensive evaluation frameworks to detect, quantify, and mitigate these biases, ensuring that LLMs function as fair and equitable tools for all users. Sheng et al. [35] provide an overview of these efforts, noting that bias evaluation typically involves analyzing models at various stages of their lifecycle—from data preprocessing to model training and output generation. Common techniques include the use of benchmark datasets designed to highlight biases [36], probing models to understand their internal representations [37], and conducting user studies to assess the real-world impact of biased outputs [34].

In terms of bias mitigation, strategies are typically categorized into three broad approaches: data-level interventions, model-level techniques, and post-processing adjustments. Data-level interventions involve curating and balancing the training datasets to reduce the prevalence of biased content, often through data augmentation, filtering, or resampling techniques [38]. However, this approach is limited by the vast scale and inherent diversity of LLM training datasets, making it difficult to eliminate biases entirely. Model-level techniques may include altering the training objectives, incorporating fairness constraints, or modifying model architectures to minimize biased learning [39]. These methods often trade off some degree of model accuracy for fairness, presenting challenges in real-world deployment. Post-processing methods focus on adjusting the outputs of LLMs after generation, applying debiasing algorithms, controlled text generation, or reinforcement learning frameworks to reduce biased content [25]. While this approach offers flexibility, it may require substantial computational overhead and may not fully address the root causes of bias.

Despite these advancements, fully addressing bias in LLMs remains a complex and ongoing challenge. The complexity of language, the vastness of training datasets, and the massive architectures of modern LLMs—often comprising billions of parameters—make it difficult to identify and correct all instances of bias. Furthermore, as Mehrabi et al. [40] suggest, biases are often context-dependent and may evolve over time as societal norms and values change. This dynamic nature of bias necessitates continuous monitoring, evaluation, and refinement of both models and the methodologies used to assess their outputs.

To boost the reproducibility of this review and allow readers to audit the review’s coverage, the literature collection process is documented as follows. Specifically, we searched ACL Anthology, arXiv (cs.CL, cs.AI, cs.LG), Google Scholar, Semantic Scholar, and ACM Digital Library, covering major NLP and AI venues, including ACL, EMNLP, NAACL, NeurIPS, ICML, AAAI, and FAccT. Primary query strings included combinations of (“large language model” OR “LLM” OR “language model”) AND (“bias” OR “fairness” OR “stereotype” OR “debiasing”). The search covered publications from January 2016 through August 2024. Inclusion criteria required papers to directly address bias origins, evaluation, or mitigation in language models, be published in peer-reviewed venues or as substantive preprints, and be written in English. Approximately 200 papers were included in the final synthesis after title, abstract, and full-text screening.

This article provides a structured synthesis of current research on bias in LLMs, covering its origins, manifestations, evaluation, mitigation, and broader societal implications. While prior reviews such as Mehrabi et al. [40], Navigli et al. [41], and Gallegos et al. [42] have made important contributions to bias phenomena and mitigation methods, the present review distinguishes itself through five specific contributions. First, we propose and operationalize a precise two-level taxonomic framework distinguishing intrinsic bias, which is encoded within model representations during pretraining, from extrinsic bias, which emerges in downstream tasks and generated outputs, providing clearer conceptual boundaries than existing categorizations. Second, we review bias evaluation methods across data-, model-, and output-level analyses, while also discussing human-involved and domain-specific evaluation perspectives. Third, we provide a comparative synthesis of pre-model, intra-model, and post-model mitigation strategies, explicitly examining trade-offs among computational cost, generalizability, and fairness that have received less attention in prior surveys. Fourth, we examine the ethical and legal implications of biased LLMs through a structured taxonomy of representational and allocational harms, grounding abstract fairness concerns in documented real-world consequences. Fifth, we highlight cross-cultural and multilingual gaps that remain under-represented in the current literature, documenting bias phenomena and evaluation gaps across East Asian, Arabic, South Asian, African, and Latin American linguistic and cultural contexts. Figure 1 provides a high-level overview of the bias landscape in LLMs considered in this review, linking upstream sources of bias to intrinsic bias, extrinsic bias in downstream tasks, cross-cutting evaluation and mitigation perspectives, and broader societal harms.

The remainder of this paper is organized as follows. Section 2 and Section 3 discuss the sources and types of intrinsic and extrinsic bias, drawing on examples from various NLP tasks to illustrate how biases arise and persist. Section 4 reviews the methodologies used to detect and measure biases, including both quantitative and qualitative approaches. Section 5 compares mitigation strategies across different stages of the model development pipeline, assessing their effectiveness and limitations. Section 6 discusses the ethical considerations and legal implications of deploying biased AI systems, underscoring the importance of fairness, transparency, and accountability in AI development. Through these discussions, we aim to contribute to the broader discourse on responsible AI by offering insights and recommendations for researchers, practitioners, and policymakers working to ensure that LLMs serve as equitable and trustworthy tools in society.

2. Intrinsic Bias

The bias presented in large language models (LLMs) can be broadly categorized into intrinsic bias and extrinsic bias based on the different stages at which the biases manifest within the model’s lifecycle and the type of bias being measured [43]. Intrinsic bias refers to biases that are inherently within the internal representations or outputs of a trained or pretrained LLM and are independent of any specific downstream tasks. Extrinsic bias refers to the biases that manifest during the model’s performance on specific downstream tasks after training or fine-tuning. Intrinsic biases are typically encoded during the training or pretraining phase when the LLM learns from large-scale corpora. For instance, if a pretrained LLM consistently associates certain professions with specific genders (e.g., assuming all doctors to be male and assuming all nurses to be female) or racial stereotypes [44], this bias would be considered intrinsic bias. This type of bias is tied to the way the model has encoded relationships and patterns in its internal representations. In this section, we introduce the possible sources of intrinsic bias and present the manifestations of intrinsic bias.

Intrinsic biases are primarily introduced during the training/pretraining stages, where models learn patterns and representations from vast corpora. These biases can be traced to several aspects of data and model design, which are outlined below.

2.1. Bias in Training Data

Language models, especially LLMs that are trained on a large corpus, usually have intrinsic bias issues, since the training corpora often contain societal biases that are built into the model [45,46,47,48]. These biases can arise in several ways, as described below.

• Bias from over-representativeness and under-representativeness: Certain groups, such as gender, age, race, religion, ethnicity, culture, political, and socioeconomic classes, may be under-represented or over-represented in the corpus [49,50]. For example, men might be over-represented in datasets about leadership or science, while women may be more frequently mentioned in caregiving roles [51], thus leading to biased associations with demographic information. Another example is that the Latinx population, especially Mexican-American students, remains under-represented in U.S. higher education [50].
• Spatial and temporal bias: LLMs trained predominantly on a corpus from certain countries or geographic locations may absorb the cultural norms and values, hence building biases into the underlying LLMs. For instance, a model trained on Western-centric data may have a skewed understanding of non-Western cultures [52,53,54,55,56]. Prior studies have shown such effects in Arabic contexts, where models may prefer Western-associated entities over Arab ones [57], and in broader multilingual settings, where performance and cultural knowledge vary substantially across under-represented languages and regions [58]. Similarly, data collected over different periods may reflect outdated societal norms and values. LLMs trained on these data may be biased by those outdated norms and values. For instance, historical texts may exhibit racist or sexist terminology, which the model may absorb as part of its internal representations [59,60].

2.2. Bias in Data Collection Methods

The intrinsic biases can also stem from the methods that are used to collect the training corpus. A large training corpus of LLMs often comprises sub-datasets that are collected from different sources, including web scraping, social media and forums, books and literature, code repositories, scientific and academic papers, legal and finance documents, dialogue and transcriptions, government and public records, custom and proprietary databases, and model-generated datasets. These different types of sources have their own biases. For instance, internet-based datasets can include unmoderated, biased, or hateful content. Academic papers tend to be more neutral, but they often lack the diversity of language found in social media or everyday conversations [42,61,62]. Choosing which texts to include or exclude during the data collection phase can also lead to biases [41]. For instance, excluding certain groups or viewpoints, intentionally or unintentionally, can introduce selection bias, which can skew the model’s outputs toward the dominant narrative in the training data. Even with careful curation by LLM creators, biases can still be embedded in the model’s learning, as it is nearly impossible to fully eliminate inappropriate content given the vast scale of the training corpus.

2.3. Bias in Language Contexts

Biases can also stem from the context of language. Human languages often contain ambiguities (including lexical ambiguity, syntax ambiguity, semantic ambiguity, and context-dependent meaning) due to the flexible nature of words, syntax, and context, which can lead to model biases. For example, gender-neutral pronouns may be associated with one gender due to the patterns in the training dataset [63,64]. Certain linguistic constructions, such as figures of speech and rhetoric, add another layer of complexity and ambiguity to human language by using words and expressions in non-literal or persuasive ways and hence introduce biases [65].

2.4. Bias in Tokenization and Word Embeddings

Tokenization and the use of word embeddings can further introduce biases into LMs. Some tokenization techniques, such as WordPiece or Byte-Pair Encoding (BPE), may disproportionately split less frequently occurring words into smaller units. This splitting policy can result in fragmented representations of under-represented entities, names, or terminology, particularly affecting minority languages or groups. For instance, Petrov et al. show that biases can arise at the tokenization stage and that the same content can require substantially different token lengths across languages [66]. Ahia et al. demonstrate large cross-lingual token-count disparities with practical fairness consequences [67]. Ovalle et al. [68] discover that Byte-Pair Encoding (BPE) tokenization, the tokenizer powering many popular LLMs, influences LLM misgendering significantly. Similarly, biases can also arise from word embedding, where many language models are initialized with pretrained word embeddings [16,27,36,69,70].

3. Extrinsic Bias

Extrinsic bias refers to disparities in a model’s performance across different downstream tasks, also known as downstream bias or prediction bias [42,43,71]. This type of bias emerges when a model’s effectiveness varies among tasks or demographic groups, potentially leading to unequal outcomes in practical applications.

Depending on the downstream tasks, extrinsic bias manifests differently. We categorize these tasks into two main groups: natural language understanding (NLU) tasks and natural language generation (NLG) tasks. In NLU tasks, extrinsic bias affects how the model comprehends and interprets input text. In NLG tasks, it can result in the generation of biased or stereotypical language.

3.1. Natural Language Understanding (NLU) Tasks

NLU tasks aim to improve a model’s comprehension of input sequences beyond the literal meaning of words [72]. Extrinsic bias in these tasks can cause the model to misinterpret or unfairly process input text based on biased associations learned during training.

Common manifestations of bias in NLU tasks include the following:

• Gender bias: Models may incorrectly associate certain professions or roles with a specific gender, leading to errors in tasks like coreference resolution [28]. For example, assuming a doctor is male and a nurse is female, regardless of context.
• Age bias: Models might make assumptions about individuals based on age stereotypes [16]. For instance, associating technological proficiency only with younger people.
• Cultural or regional bias: Models could misinterpret idioms or expressions from different cultures or fail to recognize regional language variants [73]. This can result in misunderstandings in tasks like semantic textual similarity or natural language inference.

These biases can lead to unfair or inaccurate outcomes in various NLU tasks, such as coreference resolution, semantic textual similarity, natural language inference, classification, reading comprehension, and sentiment analysis. Recurrent issues include reinforcing stereotypes, misclassifying text due to dialects or language variants, and making incorrect inferences based on biased associations.

3.2. Natural Language Generation (NLG) Tasks

NLG tasks involve generating coherent and contextually relevant text based on input or instructions [72]. Extrinsic bias in these tasks can cause models to produce biased language or reinforce stereotypes in generated text.

Common manifestations of bias in NLG tasks include the following:

• Gender bias: Models might generate responses that align with gender stereotypes, such as using male pronouns for leaders and female pronouns for nurturing roles [28].
• Age bias: Models may produce content that reflects age-related stereotypes, like suggesting only sedentary activities for older adults [16].
• Cultural or regional bias: Models could favor content from dominant cultures or misrepresent cultural practices, leading to inappropriate or insensitive responses [74].

These biases affect tasks like question answering, sentence completion, conversational agents, recommender systems, machine translation, and summarization. Recurrent issues include reinforcing harmful stereotypes, under-representing minority cultures, and providing biased recommendations or translations.

In general, extrinsic biases in LLMs impact both understanding and generation tasks, leading to unfair or discriminatory outcomes. Addressing these biases is essential to ensure models perform fairly and accurately in real-world applications. An overview of these biases in NLU tasks is provided in

Table 1. Overview of extrinsic biases in natural language understanding (NLU) tasks. Each row corresponds to an NLU task, and each column summarizes representative examples of gender, age, and cultural or regional bias. The examples highlight typical patterns rather than exhaustive cases. This table can be used to compare how different types of bias manifest across tasks.

NLU Task	Gender Bias	Age Bias	Cultural or Regional Bias
Coreference resolution Identifying instances where different expressions refer to the same entity [75]	(1) Stereotypical occupation associations: Mislinking pronouns based on gender stereotypes in professions, e.g., doctor → male [28] (2) Gendered pronoun resolution: Incorrectly resolving pronouns for gender-neutral names, reflecting underlying gender biases [76]	(1) Assumptions about technological proficiency: Attributing technological skill to certain ages during pronoun resolution [77] (2) Bias in linking pronouns to age-related roles: Misassigning actions to younger or older individuals based on stereotypes [69]	(1) Regional variants and pronoun use: Difficulty resolving pronouns in pro-drop languages due to training on non-pro-drop data [77] (2) Cultural context in family roles: Misresolving pronouns based on cultural stereotypes about family responsibilities [76]
Semantic textual similarity Evaluating how similar the meanings of two texts are [78]	(1) Gendered language and pronoun resolution: Lower similarity scores for sentences differing only by gendered pronouns [79] (2) Gender-stereotyped professions: Overestimating similarity when sentences align with gender stereotypes [27]	(1) Age-based expectations in language use: Bias in similarity assessments due to age-related stereotypes [16] (2) Age stereotypes in sentiment and perception: Overestimating similarity based on stereotypes about age and wisdom [80]	(1) Cultural idioms and expressions: Underestimating similarity between culturally equivalent idioms [73] (2) Regional dialects and variants: Misjudging similarity due to differences in regional vocabulary [81]
Natural language inference Determining the relationship between a premise and a hypothesis [82]	(1) Stereotypical gender roles in professions: Incorrect inferences based on gender stereotypes in job roles [76] (2) Gendered language and assumptions: Bias in inference due to gender assumptions with neutral names [79]	(1) Bias toward younger individuals in dynamic roles: Overemphasizing youth in roles like entrepreneurship [16] (2) Age-related stereotypes in professional contexts: Assuming leadership roles are held by older individuals [28]	(1) Cultural norms and social roles: Inferences influenced by cultural stereotypes about professions [76] (2) Regional political and historical contexts: Difficulty with inferences requiring diverse historical knowledge [83]
Classification Assigning categories or labels to input text [84,85]	(1) Gendered language in job title classification: Associating certain professions with specific genders [27] (2) Gendered pronouns and name classification: Misclassifying names or pronouns based on gender stereotypes [79]	(1) Sentiment classification bias based on age: Misclassifying sentiment due to age-related stereotypes [86] (2) Bias in job application classification: Discriminating against older applicants in job screening [84]	(1) Misclassification due to cultural language variants: Incorrectly classifying non-standard dialects as negative [30] (2) Regional bias in political text classification: Bias toward dominant regional political ideologies [11]
Reading comprehension Answering questions based on a given text [87]	(1) Stereotypes in gendered activities: Assuming gender based on activities, e.g., cooking → female [88] (2) Assumptions about gender roles in family settings: Inferring roles based on traditional gender norms [16]	(1) Bias in health-related content: Associating certain health issues with age stereotypes [27] (2) Misinterpretation of age-related roles: Incorrectly assuming roles based on age [89]	(1) Cultural context misinterpretation: Misunderstanding culturally specific practices in texts [89] (2) Regional language varieties and dialects: Struggling with comprehension of non-standard dialects [12]
Sentiment analysis Identifying the emotional tone in text [90]	(1) Bias in sentiment toward gendered products or topics: Skewed sentiment analysis for gender-specific items [79] (2) Sentiment analysis in gendered contexts: Misclassifying sentiment in discussions challenging gender norms [91]	(1) Sentiment analysis on age-related topics: Assuming negative sentiment in texts by older individuals [92] (2) Stereotyping language use by older adults: Misinterpreting sentiment due to age-related language stereotypes [93]	(1) Cultural bias in sentiment toward social norms: Misclassifying sentiment across different cultural norms [94] (2) Sentiment in multilingual contexts: Incorrect sentiment assessment in code-switching texts [93]

, and in NLG tasks in

Table 2. Overview of extrinsic biases in natural language generation (NLG) tasks. Each row corresponds to an NLG task, while columns summarize representative examples of gender, age, and cultural or regional bias. These examples illustrate common bias patterns and are not exhaustive. Table can be used to compare how different types of bias manifest across tasks.

NLG Task	Gender Bias	Age Bias	Cultural or Regional Bias
Question answering Providing accurate answers based on a given text or knowledge base [95]	(1) Gender bias in answer generation: When asked, “What should a good leader do?”, the model might use stereotypically male attributes, implying leadership qualities are inherently male [28] (2) Bias in answering ambiguous gender questions: For questions without specified gender, the model might default to male pronouns, reinforcing gender assumptions [88]	(1) Stereotypical answers about aging: When asked about activities for elderly people, the model might focus on sedentary activities, neglecting active pursuits [16] (2) Negative bias toward youth: Suggesting that young people lack experience for roles like managing a company [27]	(1) Bias in answering culturally specific questions: Providing answers based on regional popularity rather than global knowledge, e.g., stating “American football” as the most popular sport [96] (2) Language and regional bias in answer accuracy: Better performance on questions related to well-represented regions and languages [97]
Sentence completion Predicting and generating the next word or sequence to complete a sentence [98]	(1) Gender bias in descriptions of physical appearance: Completing sentences in ways that reinforce stereotypes about women’s and men’s concerns [16] (2) Gender bias in personal attributes: Associating women with “emotional” and men with “strong” in sentence completions [79]	(1) Activity and lifestyle assumptions: Completing sentences with stereotypical activities based on age [69] (2) Learning and education stereotypes: Assuming older adults are “catching up” on education [26]	(1) Cultural stereotyping in sentence completion: Overemphasizing specific aspects of a culture, e.g., “In Japan, people often eat sushi” [74] (2) Regional bias in place-based completions: Reinforcing stereotypes about regions, e.g., “In Africa, many people live in villages” [99]
Conversational Generating coherent dialogue and maintaining conversation context [100]	(1) Stereotypical responses based on gendered prompts: Reinforcing gender stereotypes in descriptions, e.g., nurses as female and caring [27] (2) Bias in gendered interactions: Responding differently based on assumed user gender [101]	(1) Assumptions about being tech-savvy: Assuming tech-savvy users are younger and providing simplistic explanations to older users [93] (2) Bias in addressing age-related concerns: Discouraging older individuals from pursuing new careers [102]	(1) Culturally inappropriate responses: Failing to understand cultural norms in responses [103] (2) Bias in handling regional topics: Focusing on negative topics for certain regions [104]
Recommender systems Suggesting personalized content or items based on user data [105]	(1) Product recommendations based on gender stereotypes: Suggesting products reinforcing traditional gender roles [106] (2) Career and education recommendations: Suggesting STEM careers more to men and arts to women [107]	(1) Age-related product recommendations: Recommending products based on age stereotypes [108] (2) Media and entertainment recommendations: Assuming preferences based on age, limiting content diversity [109]	(1) Regional bias in news and information recommendations: Under-representing news from minority regions [110] (2) Bias in language and cultural content recommendations: Prioritizing content in dominant languages [111]
Machine translation Translating text from one language to another [112]	(1) Gendered language mismatch: Introducing gender bias when translating from gender-neutral to gendered languages [113] (2) Gender stereotyping in occupational translations: Assigning gendered pronouns based on stereotypes [114]	(1) Bias in addressing older adults: Translations that condescend to older adults [16] (2) Translation of age-related idioms: Reinforcing negative stereotypes in translations [93]	(1) Cultural nuance loss: Mistranslating idioms and expressions without cultural context [115] (2) Bias toward dominant cultures: Favoring translations aligning with dominant (e.g., Western) norms [116]
Summarization Generating concise summaries of longer texts [8]	(1) Differential emphasis on roles: Emphasizing traditional gender roles in summaries [89] (2) Selective emphasis on gendered information: Overemphasizing gender-specific details not central to the story [117]	(1) Bias in summarizing content for different age groups: Simplifying content in a condescending way [32] (2) Omission of contributions based on age: Highlighting contributions of younger people over older individuals [89]	(1) Omission of culturally significant details: Omitting culturally important information in summaries [118] (2) Bias toward western narratives: Prioritizing Western perspectives in global news summaries [119]

. Detailed examples can be found in Appendix A.

4. Bias Evaluation

As LLMs become increasingly integrated into various real-world applications, from healthcare decision making to legal judgments and everyday digital interactions, their potential to propagate biases has raised significant ethical and societal concerns [120]. These biases, deeply rooted in the data, model architecture, and even in post-processing steps, can lead to discriminatory outcomes against marginalized groups, amplifying existing inequalities [42]. Therefore, understanding and evaluating these biases are crucial steps in ensuring the fair and responsible deployment of LLMs.

This section focuses on the classification and evaluation of biases in LLMs, providing a comprehensive overview of the methods used to identify and assess biases at different stages of the model lifecycle [42]. We categorize bias evaluation methods into data-level, model-level, output-level, human-involved, and domain-specific approaches, each addressing distinct sources and manifestations of bias [43]. By systematically exploring these methods, we aim to equip researchers and practitioners with the tools necessary to detect and mitigate biases, ultimately enhancing the fairness and trustworthiness of LLMs in diverse applications [42].

Drawing on recent advancements in the field, we highlight key techniques such as demographic distribution analysis, fairness metrics, interpretability tools, and counterfactual fairness, among others [43,120]. Additionally, we discuss the role of human judgment and domain-specific evaluations in capturing context-dependent biases that automated methods may overlook [42]. Through this exploration, we underscore the importance of a multi-faceted approach to bias evaluation, one that combines technical rigor with contextual awareness to address the complex challenges of fairness in LLMs [42].

To avoid ambiguity, we distinguish the taxonomy of bias types from the taxonomy of bias evaluation methods. The earlier intrinsic–extrinsic distinction classifies where bias originates or manifests, whereas the present section classifies how bias is evaluated. Within this evaluation taxonomy, data-level, model-level, and output-level methods refer to the primary technical levels at which bias is assessed. In contrast, human-involved and domain-specific evaluations are best understood as cross-cutting evaluation modes: they can be applied to data-, model-, or output-level analyses rather than forming mutually exclusive categories of the same type.

4.1. Data-Level Bias Evaluation Methods

Data-level bias evaluation methods focus on identifying and quantifying biases inherent in the training data of LLMs. These biases, rooted in the data, can significantly impact model outputs and exacerbate societal stereotypes. Figure 2 illustrates the taxonomy for data-level bias evaluation that we discuss in this section.

• Data distribution analysis: Data distribution analysis is a critical step in identifying and understanding bias at the data level, as it provides insights into how different demographic groups or categories are represented in the dataset used to train LLMs. The primary focus of this analysis is to ensure that the data is balanced and representative, minimizing the risk of perpetuating or amplifying existing biases in the trained model.

  –

  Representation across demographics: Research has consistently demonstrated that the representation of demographics in training data can significantly influence the outcomes of LLMs. For instance, LLMs trained on datasets with disproportionate representation of certain demographic groups are prone to exhibit biases that favor those groups [121,122,123]. For example, an analysis of political biases in LLMs revealed that most conversational LLMs exhibit left-of-center political preferences when probed with politically charged questions or statements [124]. The study demonstrated that LLMs can be steered towards specific political orientations through supervised fine-tuning (SFT) with modest amounts of politically aligned data.

  To assess representation, statistical tools can be employed to measure the distribution of demographic attributes within the dataset. Key metrics might include the relative frequency of each demographic group or the variance in representation across different categories. Visual tools such as histograms, bar charts, or demographic distribution tables can be useful for identifying disparities, providing a clear picture of which groups are over-represented or under-represented in the data.

  –

  Imbalance and skewness detection: Imbalance and skewness in data distribution can lead to biased model behavior, where the model is overly influenced by the majority class or demographic group. Detecting these issues is crucial for ensuring that the model performs fairly across all segments of the population.

  Imbalances occur when certain classes or demographic groups are significantly more prevalent in the dataset than others. This can result in the model being biased towards the majority group, leading to poor performance for minority groups. Techniques such as Gini coefficients [125], entropy measures [126], vocabulary usage [127], and frequency analysis can be used to quantify imbalance. Skewness can be visualized using skew and stereotype quantifying metrics [128,129] box plots, histograms, or cumulative distribution functions (CDFs). These tools help identify the extent of skewness in the data and guide the selection of appropriate mitigation strategies.

  –

  Data source analysis: The quality and bias of a model are heavily influenced by the sources of the data used for training. Biases inherent in the data sources can significantly impact the model’s behavior. For instance, if a model is trained primarily on data from Western countries, it may perform poorly when applied in non-Western contexts.

  Evaluating the bias of data sources for LLM training involves analyzing the origins, diversity, and quality of the data, as well as its impact on model performance. This includes cataloging the data sources to ensure they cover a wide range of geographical, cultural, and linguistic contexts, and examining the credibility and inherent biases of each source. For instance, research [130] on the SlimPajama dataset, which includes a rigorously deduplicated combination of web text, Wikipedia, GitHub, and books, reveals how different data combinations affect LLM performance. The SlimPajama-DC study highlights two key aspects: the impact of global versus local deduplication on model performance and the importance of data diversity post-deduplication. The findings indicate that models trained with highly deduplicated, diverse datasets outperform those trained on less-refined data, underscoring the significance of comprehensive and balanced data sources. However, duplicate removal is not always neutral with respect to representation. If content from minority, low-resource, or less frequently documented communities is already sparse, aggressive deduplication may disproportionately remove repeated expressions of those voices, thereby reducing their visibility in the corpus and unintentionally increasing representativeness bias. Bias detection tools and adjustments in data weighting can further help in managing these biases, supported by transparent documentation and regular reviews to ensure fairness and accuracy in the model outputs.

• Stereotype and bias detection in text data: Stereotype and bias detection involves analyzing the content of the training data to identify and quantify biases related to stereotypes, offensive language, or prejudiced statements. This can be achieved through the following.

  –

  Lexical analysis: This method focuses on identifying specific words or phrases within the training data that are associated with bias or stereotypes. Lexical analysis relies on predefined lexicons, which are curated lists of terms known to carry biased or stereotypical connotations. Tools like Hatebase, an extensive repository of hate speech terms, provide a valuable resource for identifying harmful language in text data. Hate speech can also be measured by a large-scale study [131]. BiCapsHate [132] is a deep learning model used to detect hate speech in online social media posts. Similarly, the Dictionary of Affect in Language (DAL) [133] categorizes words based on their emotional connotations, offering insights into how language can perpetuate stereotypes. By scanning the training data for occurrences of these terms, researchers can quantify the prevalence of biased language and identify specific areas where the data may reinforce harmful stereotypes. For example, a study by [134] demonstrated how biased language in training data could lead to the propagation of racial and gender stereotypes in NLP models.

  –

  Contextual analysis: While lexical analysis identifies the presence of specific biased terms, contextual analysis delves deeper into how these terms are used within the text. This method employs advanced NLP techniques to examine the context in which potentially biased language appears, allowing for the identification of subtler forms of bias. For instance, the same word might be used neutrally in one context but carry a prejudiced meaning in another. Contextual analysis examines sentence structure, co-occurrence patterns, and the surrounding language to uncover these nuances. A similar study by [27] has shown that even when biased language is not overtly present, underlying patterns can still perpetuate stereotypes, such as gender biases, in word embeddings. This approach is crucial for identifying and mitigating biases that may not be immediately apparent but can significantly impact the behavior of LLMs. Zhao et al. [135] quantified and analyzed the gender bias exhibited in ELMo’s contextualized word vectors. Contextual analysis can be implemented by methods like template-based method [136], applied thematic analysis [137], and contextualized word embedding analysis [138].

  –

  Sentiment analysis: Sentiment analysis is another vital tool in detecting bias in text data. This technique assesses the sentiment—positive, negative, or neutral—associated with different demographic groups or topics within the training data. By analyzing how different groups are described, researchers can identify patterns of negative or biased portrayals that could influence the model’s outputs. For example, if certain demographic groups are consistently associated with negative emotions or connotations, the model might learn to replicate these biases in its predictions or interactions. Research by [31] has highlighted how sentiment analysis can be used to detect and address such biases in text data. This method provides a quantitative measure of bias, enabling targeted interventions to reduce the impact of these biases on the model’s behavior. Sentiment analysis can be implemented using various approaches, including traditional machine learning methods [139,140,141], frameworks based on preprocessed data [141], and transformer-based methods [6,142,143].

• Annotation bias analysis: Annotation bias analysis is a crucial component in evaluating and mitigating bias in LLMs. This process involves examining the biases introduced during the data annotation phase, where human annotators label or categorize training data. Since annotators bring their own biases and perspectives, their subjective decisions can inadvertently introduce skewed or biased annotations, which in turn affect the performance and fairness of the LLMs. To perform annotation bias analysis, researchers must review the annotation guidelines and the training process for annotators, ensuring that they are designed to minimize bias. Additionally, evaluating the consistency and fairness of annotations across different demographic groups and annotators helps identify any disparities. Tools such as inter-annotator agreement metrics [144] and statistical analysis [145] of annotated data can reveal potential biases. For example, research by [146] on gender bias in annotated datasets highlights how annotation practices can reinforce stereotypes and biases, emphasizing the need for rigorous analysis and correction methods. Another way to reduce annotation bias is a human–LLM collaborative approach [147]. By addressing annotation bias, researchers can enhance the quality and fairness of training data, leading to more balanced and unbiased LLMs.

4.2. Model-Level Bias Evaluation Methods

Model-level bias evaluation methods assess biases that arise during the training and prediction stages of LLMs, focusing on whether the model exhibits discriminatory behavior toward specific groups.

• Fairness metrics: These metrics are pivotal in evaluating the model’s output fairness across different groups. Common metrics include the following.

  –

  Equal opportunity: Ensures that the true positive rate (TPR) is similar across groups defined by sensitive attributes such as gender or race. For example, a sentiment analysis model should equally recognize positive sentiments across both male- and female-associated names [42,120].

  –

  Predictive parity: Focuses on the consistency of prediction accuracy (precision) across groups. This is particularly important in applications like credit scoring, where unequal predictive performance can lead to discriminatory outcomes against minority groups [42,43].

  –

  Calibration: This metric checks whether predicted probabilities align with actual occurrences, particularly across different demographic groups, ensuring that the model’s confidence in its predictions is justified [43,148].

• Interpretability tools: Tools such as SHAP (Shapley Additive Explanations) [149] and LIME (Local Interpretable Model-agnostic Explanations) [150] provide insights into how specific features influence model predictions, helping identify biases. For instance, SHAP values can reveal that gendered words heavily influence predictions, indicating potential gender bias in the model [120,151]. Similarly, LIME approximates complex models locally with interpretable models to highlight the impact of features on individual predictions, which is crucial for diagnosing bias [150]. However, these tools should be interpreted with caution in LLMs with billions of parameters. In such massive models, feature attributions are often approximate and local, and may be unstable across prompt wording, tokenization, sampling settings, or semantically similar inputs. They may also fail to capture complex high-order interactions and therefore should not be treated as definitive causal explanations of bias. For this reason, SHAP- and LIME-based analyses are best used as diagnostic signals that should be corroborated with counterfactual, robustness, and output-level evaluations.
• Counterfactual fairness: This method generates counterfactual examples by altering sensitive attributes (e.g., changing names from traditionally male to female) to see if the model’s outputs remain invariant. A fair model should ideally produce the same outcomes regardless of these attribute changes [43,71]. Here, counterfactual fairness is used in the model-level sense: it examines whether the model’s decision rule or prediction remains invariant when only a sensitive attribute is changed, rather than comparing differences in the wording or tone of free-form generated responses.

Table 3. Summary of model-level bias evaluation methods.

Method	Description
Fairness metrics	Equal opportunity: Ensures similar true positive rates (TPRs) across sensitive groups.
	Predictive parity: Checks for consistent prediction accuracy across groups.
	Calibration: Aligns predicted probabilities with actual outcomes for fairness.
Interpretability tools	SHAP: Explains the impact of individual features on model predictions.
	LIME: Provides local approximations of complex models for feature impact analysis.
Counterfactual fairness	Scenario testing: Alters sensitive attributes (e.g., gender) to test output consistency.
	Equity check: Verifies that changes do not affect model outcomes unfairly.

summarizes the three categories of model-level bias evaluation methods discussed above.

4.3. Output-Level Bias Evaluation Methods

Output-level bias evaluation methods assess how LLMs generate responses, especially with regard to fairness and neutrality across different demographic groups. These methods aim to identify whether the model’s predictions, recommendations, or generated content reflect biases that may have been learned during training, including those based on race, gender, socioeconomic status, political orientation, and more. Here, we explore four major metrics used to detect bias in LLM-generated text: counterfactual testing, stereotype detection, sentiment and toxicity analysis, and acceptance and rejection rates (see Figure 3). Unlike model-level counterfactual fairness, output-level counterfactual testing compares the generated outputs themselves under minimally modified prompts, focusing on whether the content, framing, sentiment, or recommendation changes across demographic substitutions.

• Counterfactual testing: Counterfactual testing involves modifying input prompts by altering specific attributes, such as gender, race, or ethnicity, while keeping the rest of the context unchanged. This method evaluates whether the LLM’s output varies based on these demographic changes, allowing researchers to isolate the effect of these attributes on the model’s behavior. For instance, swapping “John” with “Maria” in a sentence can reveal if the model responds differently due to the gender of the subject.
  Recent work [152] emphasizes the importance of robust causal reasoning in LLMs to enhance fairness, arguing that a strong understanding of causal relationships can mitigate biases and reduce hallucinations. Beyond fairness, this connection is also important for understanding hallucinations. When a prompt is ambiguous or underspecified, biased social associations learned from the training corpus may act as anchors, leading the model to generate stereotype-consistent details even when those details are not supported by the input. For example, the model may infer occupations, traits, intentions, or background facts from demographic cues because such associations are statistically frequent in the data, not because they are causally justified in the given context. In this sense, some hallucinations can be viewed not merely as factual errors, but as bias-driven distortions in which stereotype-consistent priors fill evidential gaps. Stronger causal reasoning may reduce this risk by helping the model distinguish socially correlated attributes from causally relevant information. Another study [153] introduces a dynamic framework that compares outputs across different demographic groups in the same context, improving the fairness of generated text without requiring costly model retraining. Additionally, new tools [154] for generating and analyzing counterfactuals allow users to explore LLM behavior interactively, ensuring the counterfactuals are both meaningful and grammatically accurate.
  These advancements underscore the power of counterfactual testing in identifying biases in LLMs. By systematically altering demographic features in prompts, researchers can evaluate how models treat different groups and ensure equitable outputs across all demographics.
• Stereotype detection in generated text: Stereotype detection in generated text focuses on identifying and mitigating harmful biases that LLMs may perpetuate in their outputs. Since LLMs are trained on vast amounts of publicly available data, which often contain stereotypical narratives related to race, gender, profession, and religion, the risk of these biases appearing in generated content is significant. This method assesses how models reproduce stereotypes, providing insights into implicit prejudices embedded in their responses.
  One approach [150] introduced a unified dataset combining multiple stereotype detection datasets. Researchers fine-tuned LLMs on this dataset and found that multi-dimensional classifiers were more effective in identifying stereotypes. This study also highlighted the use of explainable AI tools to ensure models align with human understanding. Another study [155] developed a dual framework that combines static evaluations with dynamic, real-world scenario simulations. This dynamic aspect is particularly effective in detecting subtle, context-specific biases that static tests might miss. A qualitative method presented in [156] uses prompting techniques to uncover implicit stereotypes in LLM-generated text. This approach, focusing on biases like gender and ethnicity, employs the Tree of Thoughts technique to systematically reveal hidden prejudices and provide a reproducible method for stereotype detection. These methods collectively help researchers detect and mitigate biases in LLM outputs, offering a deeper understanding of how stereotypes manifest in generated text.
• Sentiment and toxicity analysis: Sentiment and toxicity analysis is crucial for evaluating output-level biases in LLMs, specifically targeting harmful or offensive content and ensuring adherence to ethical standards. This area of evaluation not only addresses toxicity but also helps in identifying subtle biases that may emerge in generated content.
  Llama Guard [157] introduces a model-based approach to bias evaluation by using a safety risk taxonomy for classifying both prompts and responses in human–AI conversations. This model, fine-tuned on a carefully curated dataset, shows strong performance in detecting various forms of toxicity. It represents a practical application of bias evaluation methods by providing a dynamic tool for assessing and mitigating harmful content in real-world interactions. The definition-based toxicity metric [158] offers a flexible solution for bias evaluation by using LLMs to measure toxicity according to predefined criteria. This method outperforms traditional metrics, enhancing the F1 score significantly and addressing the limitations of existing models that rely on dataset-specific definitions. It exemplifies how bias evaluation methods can be refined to improve the detection of nuanced toxic content. Moderation Using LLM Introspection (MULI) [159] advances bias evaluation by analyzing internal model responses to detect toxic prompts. This approach leverages patterns in response token logits and refusal behaviors, providing a cost-effective way to assess biases without requiring additional training.
  The OpenAI Moderations Endpoint represents a practical implementation of bias evaluation methods for filtering potentially harmful content. This tool helps developers assess and moderate LLM outputs, ensuring they align with safety and ethical guidelines. It demonstrates how automated tools can be integrated into the bias evaluation process to manage and mitigate risks associated with model-generated text.
• Acceptance and rejection rates: Evaluating acceptance and rejection rates in LLMs is essential for identifying biases that may arise in decision-making tasks. Recent work has explored how these biases manifest, particularly in contexts such as hiring decisions or moral dilemmas, where the outputs of LLMs could reinforce existing societal inequalities.
  In one study [160], researchers assessed how LLMs respond to job applicants based on perceived race, ethnicity, and gender by manipulating first names. They found that acceptance rates were significantly higher for masculine White names compared to masculine Hispanic names. These acceptance and rejection rates were highly prompt-sensitive, meaning that variations in how the question was framed could lead to different outcomes. This suggests that LLMs can subtly reinforce biases depending on how inputs are structured. Another evaluation method [161] used to study LLM behavior in moral decision making showed that models are consistent in choosing commonsense actions in unambiguous scenarios, while in more ambiguous cases, LLMs exhibited uncertainty and greater variability. Closed-source models often displayed more consistent preferences in these ambiguous scenarios, which suggests that proprietary models might encode specific tendencies in moral decision making. BiasBuster [162] provides a systematic method for uncovering and evaluating cognitive biases in LLMs, particularly in high-stakes decision making. By using a dataset of 16,800 prompts designed to probe various types of biases (e.g., prompt-induced, sequential), this framework allowed researchers to assess how LLMs handled acceptance and rejection decisions under different bias conditions.

4.4. Human-Involved Bias Evaluation Methods

Human-involved evaluation incorporates human judgment to identify complex and context-dependent biases in LLM outputs. Human-involved evaluation is defined by the source of judgment (i.e., human assessors), whereas domain-specific evaluation is defined by application-specific norms, risks, and fairness criteria (e.g., healthcare, law, education); accordingly, a domain-specific evaluation may be human-involved, automated, or both.

• Human review and assessment: Experts or crowd-sourced reviewers manually assess model outputs for biases. This method is particularly effective in detecting nuanced biases that automated tools might miss, such as subtle stereotypes or culturally specific biases [43].
• Qualitative research methods: Interviews, focus groups, and case studies offer deep insights into how different communities perceive model outputs, providing a qualitative dimension to bias evaluation that complements quantitative approaches [120].

To reduce subjectivity in human-involved bias evaluation, the evaluation protocol (Figure 4) should be specified explicitly rather than left implicit. At a minimum, such a protocol should define: (1) the target population of evaluators and the rationale for their selection; (2) the diversity requirements of the evaluator pool, including demographic, cultural, linguistic, and professional diversity when the task involves socially situated judgments; (3) whether domain experts, affected-group members, crowd workers, or mixed panels are used; (4) standardized annotation guidelines, examples, and rating rubrics; and (5) the procedure for pilot calibration before formal evaluation. In practice, evaluators should assess outputs independently and, when possible, without access to model identity or study hypotheses, in order to reduce anchoring and confirmation effects. The manuscript should also report the number of evaluators, their relevant background characteristics, the assignment protocol, and inter-annotator agreement statistics when applicable [144]. For disagreement handling, a pre-specified rule should be provided, such as majority vote for straightforward labels, adjudication by a trained senior reviewer, or discussion-based consensus for context-sensitive cases; unresolved disagreements may also be retained and analyzed separately as indicators of ambiguity in the bias construct itself. Such protocol design helps ensure that human evaluation is systematic, transparent, and reproducible, rather than relying solely on subjective impressions.

4.5. Conclusions

Bias evaluation is critical for the responsible use of LLMs. By systematically categorizing and applying various evaluation methods, researchers can better understand and mitigate the biases that LLMs may propagate, thereby enhancing their fairness and societal impact.

5. Bias Mitigation

Biases are often present in unstructured data, and LLMs trained on such data can not only learn these biases but sometimes amplify them as well [27,163]. Various methods of bias mitigation exist under different contexts, such as social bias [164,165], political bias [166], and stereotype bias [167]. Instead of reviewing these methods for bias mitigation by bias contexts, we categorize them by the timing of debiasing. More specifically, methods for mitigating biases can be categorized into three distinct approaches: (1) preprocessing the input data prior to training LLMs; (2) changing the model’s architecture and/or picking the best model based on the predefined bias evaluation metrics; and (3) improving unbiasedness by calibrating the outputs at the decision-making stage. We call these three approaches pre-model debiasing, intra-model debiasing, and post-model debiasing, respectively.

Table 4. Overview of methods for bias mitigation.

Debiasing	Methods	Summary
Pre-model	Resampling; data augmentation; expert intervention	Time saving without model training; time consuming to annotate bias cases, ineffectiveness with a self-biased model, privacy issue of resampling
Intra-model	Equalization and declustering; movement pruning; transfer learning; dropout regularization; causal inference	Flexibility in mitigating various types of biases, strong performance in empirical studies; time consuming for modifying and training models
Post-model	Reinforced calibration; Self-Debias; projection-based methods; causal prompting	Time saving without model training; demand of a large amount of data

provides an overview of the advantages and disadvantages of each approach.

Figure 5 provides a visual summary of the three-stage debiasing pipeline and the representative techniques within each stage.

5.1. Pre-Model Debiasing

Pre-model debiasing approaches involve preprocessing the input data before training LLMs, so that the entire training data are balanced and representative of the entire population. Typical ways of balancing training data include oversampling, undersampling, and synthetic data generation [168]. We review the commonly used pre-model debiasing methods below.

One of the widely used methods is counterfactual data augmentation (CDA), which is often used to mitigate bias in the literature [169,170,171,172]. Counterfactual data substitution (CDS), a variant of CDA, is also proposed to randomly substitute potentially biased text to avoid duplication [173]. Generally speaking, CDA rebalances a corpus by swapping bias attribute words in a dataset. For example, we may swap the sentence

“The professor came to the classroom but he forgot to bring his laptop”

with

“The professor came to the classroom but she forgot to bring her laptop”.

Meade et al. [174] further empirically evaluate CDA for mitigating religious bias by swapping religious terms.

Serouis et al. [175] propose an iterative method in which a human can oversee the preprocessing step. That is, before training begins, a dataset report is sent to an LLM. An expert then decides the next step: either issuing a command to the LLM to create new training data or starting the training process. The decision is made by measuring the quality of the generated data. If a command is used to generate alternative training data, iterative feedback between the expert and the LLM continues until predefined criteria are satisfied, leading to a decision to proceed with model training. After training and experimentation, detailed results concerning protected groups are provided to the LLM for summarization (particularly for multiple groups and subgroups) or directly to the expert. Based on these comprehensive reports, the expert may choose to use the LLM to produce a revised dataset version to mitigate potential biases.

5.2. Intra-Model Debiasing

Intra-model debiasing works by selecting the model that prioritizes unbiasedness and fairness during the model fitting. For example, Ref. [164] proposes a method to mitigate explicit and implicit biases in BERT using existing and newly proposed loss functions. The proposed work focuses on gender bias and uses a predefined list of target word pairs and a text dataset as inputs. It takes the pretrained BERT and further trains it on the given input, while mitigating the existing social biases by equalizing and declustering. The equalizing stage aims to equalize the associations of neutral words by defining an equalizing loss [176]. Observing that words are stereotypically associated with a group (e.g., delicate, pink, beautiful), the authors develop a declustering stage to decluster the implicit clusters by defining a declustering loss. A similar technique is also applied to mitigate social stereotypes when exposed to different demographic groups [165]. Moreover, Ref. [177] presents an innovative framework for examining bias in pretrained transformer-based language models through movement pruning. It identifies a subset of the model that contains less bias than the original by pruning it while fine-tuning on a debiasing objective. It explores pruning attention heads, which are crucial components of transformers, by pruning square blocks and introducing a novel method for pruning entire heads. An interesting bias–performance trade-off for gender bias from this study is that the level of bias increases as the model’s performance improves.

Another line of studies focuses on the technique of transfer learning [178,179,180,181]. Transfer learning leverages knowledge learned from a pretrained model to improve the performance of a new model on a relevant task but with fewer data points. More specifically, it fine-tunes a pretrained model on the new task by updating the weights of neurons or modifying the network architecture. For example, FairDistillation [182], a cross-lingual method based on knowledge distillation, is used to build smaller language models while managing specific biases. It leverages existing language models as a teacher, providing a richer training signal without the need for retraining from scratch. To avoid transferring learned correlations to new language models, it further substitutes CDA’s augmentation strategy with probabilistic rules between tokens. Furthermore, we notice that a few works utilize multilinguality as bias mitigation [167,183,184]. These works advocate multilingual training to mitigate biases. They demonstrate the effectiveness of multilinguality in reducing bias and boosting prediction accuracy [167].

Another useful method is dropout regularization, which is studied in [170,185]. Briefly, they explore the impact of increasing dropout parameters for BERT’s and ALBERT’s attention weights and hidden activations, followed by an additional phase of pretraining. Their experiments reveal that dropout regularization reduces gender bias in these models. They propose that dropout disrupts the attention mechanisms in BERT and ALBERT, thereby preventing the models from learning undesirable word associations.

Grounded in the theory of causal inference, causality-guided methods have been investigated due to the theoretical guarantees and good generalization [186,187]. For instance, researchers have proposed a bias reduction method for sentiment classification [186]. This framework treats the neural network as a causal directed acyclic graph (DAG). Borrowing the concept of causal mediation analysis for neural networks [188], the authors study the direct and indirect effects of individual neurons to identify parts of a model where gender bias is causally implicated. Finally, the authors apply counterfactual training to only fine-tune the BERT target layers. Moreover, Ref. [187] uses causal mediation analysis to trace the causal effects of different components’ activation within an LLM. In this way, they were able to investigate the storage locations and generation mechanisms of gender bias in LLMs. The least square debias method (LSDM) was proposed to modify parameters to mitigate gender bias in models.

5.3. Post-Model Debiasing

Post-model debiasing mitigates bias by adjusting the output of LLMs. The benefits of post-model debiasing are that there are no changes in the training data and no changes in the architecture of the LLM. For example, reinforced calibration can be used to mitigate political bias in GPT-2 [166,189]. It keeps the architecture of GPT-2 unchanged but calibrates bias during the generation. More specifically, researchers have presented a reinforcement learning-based framework for political biases in two modes: word embedding debias and classifier-guided debias. For the word embedding debias, the researchers pick the unbiased token at each generation step by forcing neutral words to maintain equal distances from groups of sensitive words (e.g., male and female) in the embedding space [27,38,190]. The classifier-guided debiasing works when the bias is not purely word-level [191].

Self-Debias is another work that leverages the model’s internal knowledge to generate unbiased prompts [25]. The necessary condition for this approach is that if a model is able to discern the presence of undesired biases, it should be able to avoid them at test time. Self-Debias first detects when the models output undesirable attributes based on internal knowledge. It then reduces the probability of producing a biased test (such as sexist, racist, and violent) using a decoding algorithm. This has been empirically evaluated as a strong debiasing technique [174].

Projection-based methods have also been shown to be useful for mitigating biases. Notable methods such as SENT-DEBIAS [39] and INLP [192] work by projecting a model’s representations onto a subspace. More specifically, SENT-DEBIAS involves four steps: (1) identifying words that display biased attributes (e.g., he and she); (2) embedding these words within biased attribute sentences to generate their sentence representations by CDA; (3) estimating the bias subspace within these sentence representations by principle component analysis (PCA); and (4) debiasing general sentences by removing their projection onto this bias subspace. INLP roughly debiases a model’s representations by training a linear classifier to predict the protected attribute we want to remove (e.g., gender) from the representations. The representations are then debiased by projecting them into the nullspace of the trained classifier’s weight matrix, effectively eliminating all the information the classifier used to predict the protected attribute from the representation. This process can be applied iteratively to further debias the representation.

In alignment with causal methods in intra-model debiasing, researchers have also developed methods based on causal prompting [193,194]. Prompting methods have gained in popularity since the general public has no access to the model’s internal structure due to business interests. Specifically, Ref. [193] proposes causal prompting based on front-door adjustment [195]. The proposed method modifies prompts without access to the parameters and logits of LLMs. First, it queries LLMs to generate chains of thought (CoTs) m times with the input prompt (demonstration examples and a question of the test example). An encoder-based clustering algorithm is applied to these CoTs and the top K representative CoTs are selected. Next, it retrieves the optimal demonstration examples for each representative. Finally, the LLMs are queried T times to obtain T answers for each representative CoT, and the final answer is obtained by a weighted voting. Ref. [194] constructs detailed causal models for both the underlying data-generating process of the training corpus and the LLM’s reasoning process. They recognize that selection mechanisms are crucial in determining how the LLM’s output can be influenced by different prompts. Additionally, they propose a causality-guided debiasing framework. By gaining causal insights into the data-generating processes, the necessary conditions that prompt-design strategies must meet to achieve principled and effective debiasing are identified.

5.4. Critical Comparison of Mitigation Strategies

Having reviewed the three categories of debiasing approaches, we now offer a critical comparison of their effectiveness, computational requirements, and practical trade-offs.

Table 5. Estimated computational cost comparison of representative bias mitigation strategies. Costs are approximate and depend on model size and implementation.

Category	Method	Key Mechanism	Relative Cost	GPU Requirement
Pre-model	CDA/CDS	Augmenting training data	Low	Minimal
Pre-model	Expert curation	Human review of data	Medium	None
Intra-model	Equalization loss	Modified training objective	High	Full training
Intra-model	Movement pruning	Pruning biased subnetworks	High	Full training
Intra-model	FairDistillation	Knowledge distillation	Medium–high	Distillation run
Intra-model	Dropout regularization	Additional pretraining	Medium	Pretraining pass
Post-model	Self-Debias	Modified decoding	Low	Inference only
Post-model	SENT-DEBIAS/INLP	Subspace projection	Low–medium	Inference + PCA
Post-model	Reinforced calibration	RL-based generation	Medium	RL training
Post-model	Causal prompting	Prompt modification	Low–medium	Multiple queries

provides an estimated comparison of the computational costs across representative methods.

Several key trade-offs emerge from this comparison. Pre-model methods are conceptually straightforward and computationally inexpensive but cannot address biases that arise from model architecture or training dynamics. They are also limited in their ability to handle biases that are not explicitly encoded in individual words or phrases but emerge from complex distributional patterns. Intra-model methods offer the most direct intervention into the bias-producing mechanisms but require full model retraining or fine-tuning, which is computationally expensive and may not be feasible for the largest contemporary LLMs. Furthermore, intra-model methods are model-specific and may not generalize across architectures. Post-model methods are attractive for their flexibility and low computational overhead, as they do not require modifying the training data or model parameters. However, they address only the symptoms of bias rather than its root causes, and their effectiveness depends on the ability to detect bias at inference time. Self-Debias, for instance, relies on the assumption that the model can recognize its own biases, which may not hold for all bias types, especially subtle or intersectional biases.

In practice, a combination of methods across all three stages is likely to yield the most robust debiasing outcomes. Pre-model data curation can reduce the most egregious data-level biases, intra-model techniques can address biases encoded in model representations, and post-model methods can serve as a final safety net during deployment. Future research should focus on developing integrated debiasing pipelines that coordinate interventions across these stages.

6. Ethical Concerns and Legal Challenges

Due to rapid advancements and successful integration of LLMs into systems that are widespread throughout our society, more and more concerns are being raised that LLMs can learn and amplify societal biases (gender, race, religion, age, etc.) embedded in the enormous Internet data they are trained on [42]. As a result, LLMs can potentially generate biased outputs among different social groups, which falls under the ethical concerns of AI. Furthermore, disparate treatment caused by biased outputs in decision-making processes can result in legal challenges under anti-discrimination laws.

We categorize biased outputs and disparate treatment according to a pre-established taxonomy: representational harms and allocational harms [12]. Representational harms refer to when an automated system exhibits negative attitudes or misrepresents, even ignores existence of certain social groups. Allocational harms refer to when an automated system allocate resources in favor of some social groups over others.

After reviewing many types of representational harms described in the literature, we constrained our discussion to four categories that can cover most representational harms or ethical concerns:

• Stereotyping.

  –

  Definition: Negative abstractions about a labeled social group.

  –

  Result 1: Reinforced social bias against certain social groups.

  –

  Result 2: Toxic language towards certain social groups.

• Misrepresentation.

  –

  Definition: An incomplete sample generalized to a social group.

  –

  Result 1: Reinforced normativity of the dominant social group and implicit exclusion or devaluation of other groups.

  –

  Result 2: Biased outputs that mischaracterize certain social groups.

• Disparate system performance.

  –

  Definition: Degraded understanding or model performance in LLM language processing or generation between social groups.

  –

  Result 1: Allocational harms.

Stereotyping: LLMs can reinforce over-simplified and often negative labels attached to certain social groups. Abid et al. [196] demonstrate that LLMs capture Muslim-violence bias in various use cases and the Muslim-violence bias is more severe than commonly seen biases for other religious groups. Bender et al. [11] describe how LLMs tend to refer to “women doctors” since even in today’s world, doctor is a male-dominant occupation. Stereotypes perpetuated and even exacerbated by LLMs can further translate into offensive language that targets a social group. Gehman et al. [197] evaluate five language models (including GPT-1 to 3) using a “REALTOXICITYPROMPTS” machine and show that all five models degenerate into toxicity over 0.5 (assigned by a widely accepted toxicity classifier) within less than 100 generations from non-toxic prompts. Toxic generations from language models can range from the likes of “Islamic slaughtering” to “lying white media”.

Misrepresentation: The disproportionate representation of marginalized or minority social groups in LLM training data can lead to implicit exclusion of marginalized groups and mischaracterization of certain groups. Bender et al. [11] raise an example of ‘both genders’ that exclude non-binary identities from the context. Navigli et al. [41] provide several examples of misrepresentation across gender, age, nationality, ethnicity, etc. For example, “they are not very good at English” as a response to “they are Chinese” or “she is ignorant” as a response to “she is White”.

Disparate system performance: Koenecke et al. [198] reveal that all of the five state-of-the-art LLM-based automated speech recognition (ASR) systems show higher word error rates for black speakers than white speakers. Saunders and Byrne [199] demonstrate that the over-prevalence of male-centric forms in machine translation training data result in better translations for male-centric sentences and for sentences that contain male-dominant roles such as doctors and developers. Next, we discuss disparate system performance in LLM-automated hiring and healthcare tools, which often leads to allocational harms that raise anti-discrimination legal challenges.

• Allocational harms.

  –

  Definition: Disparate treatment due to membership of a social group or due to proxies associated with a social group.

  –

  LLM-based hiring tools: Resume and cover letter screening.

  –

  LLM-based healthcare tools: AI diagnostics.

Allocational harms: Social biases preserved by LLMs can translate into discriminatory allocation of opportunities and services—for example, by steering who gets shortlisted for jobs or which patients receive particular diagnostic workups—thereby creating disparate treatment or disparate impact in high-stakes settings.

In addition to ethical concerns, allocation harms intersect with legal and regulatory frameworks intended to prevent discrimination. In the U.S., federal employment discrimination protections apply to AI and other automated tools “just as they apply to other employment practices”, including recruitment and screening; employers remain responsible for ensuring that selection procedures do not unlawfully exclude protected groups, even when tools are provided by vendors [200]. At the subnational level, New York City’s Local Law 144 regulates “automated employment decision tools” by requiring recent bias audits, public summaries, and advance notice before use in hiring or promotion decisions [201]. In healthcare, the 2024 Section 1557 final rule prohibits discrimination through the use of patient care decision support tools and requires covered entities to make reasonable efforts to identify high-risk uses and mitigate discrimination when such tools are used in clinical decision making [202]. In the EU, the AI Act adopts a risk-based approach: “high risk” includes recruitment tools (e.g., CV sorting) and AI-based medical software, with obligations such as risk management, data quality to reduce discriminatory outcomes, documentation, and human oversight [203].

Despite these legal and regulatory frameworks, real-world examples demonstrate that allocation harms persist. Ferrara [168] argues that AI-automated tools may preserve and exacerbate societal biases embedded in the training data. LLM-aided resume-analyzing tools may disqualify people based on non-traditional backgrounds instead of job-related qualifications. In 2018, Amazon reportedly took an internal AI recruiting tool off the shelf after being accused of bias against women since the resumes in the training data are predominantly from white males. Another example is that LLM-aided healthcare tools may generate inaccurate predictions and treatment recommendations for under-represented groups, potentially influencing healthcare decision making and resulting in unfair medical resource allocation. Zack et al. [204] discovered that GPT-4 is less likely to recommend more expensive diagnostic procedures to Black people and more likely to predict Hispanic women are more likely to hide an alcohol-abuse history than Asian women.

7. Conclusions

This review has provided a comprehensive analysis of bias in LLMs, examining their sources, manifestations, evaluation methods, and mitigation strategies. Our categorization of intrinsic and extrinsic biases offers a structured approach to understanding their impact on various NLP tasks [42,43]. We have evaluated a range of bias assessment techniques, from data-level analysis to human-involved methods, providing researchers with a diverse toolkit for bias evaluation [120]. Our examination of mitigation strategies, encompassing pre-model, intra-model, and post-model approaches, highlights both promising developments and areas that require further research [166,168].

The ethical and legal implications of biased LLMs are substantial. The potential for both representational and allocational harms in real-world applications emphasizes the need for responsible AI development [12,204]. As LLMs become increasingly integrated into diverse facets of society—from personal assistants to decision-making systems in critical sectors—it is imperative to address these biases to prevent unintended negative consequences.

Looking forward, key research directions include developing context-sensitive bias evaluation metrics that can identify subtle biases across diverse cultural settings [150], advancing causal inference techniques for more effective debiasing methods [193], and investigating the long-term societal impacts of biased LLMs in critical domains such as healthcare and criminal justice [32]. Additionally, fostering interdisciplinary collaborations to address ethical challenges and develop adaptive regulatory frameworks [104], while optimizing the balance between model performance and fairness [177], remains crucial.

By thoroughly examining the complexities of bias in LLMs and highlighting avenues for ongoing research and collaboration, we aim to contribute to the development of AI systems that are both technologically advanced and socially responsible. Addressing these challenges is essential to harness the full potential of LLMs, fostering innovations that advance technology while upholding the principles of fairness and equality.

While much of the foundational work on LLM bias has focused on earlier models, such as GPT-2, GPT-3, and BERT, a critical question is whether the observed bias trends remain, are diminished, or have evolved in more recent, state-of-the-art models such as GPT-4, Llama 3, and Claude. Preliminary evidence suggests a mixed picture. On one hand, newer models benefit from more extensive alignment procedures, including reinforcement learning from human feedback (RLHF), constitutional AI training, and red-teaming exercises, which can reduce overt toxic outputs and explicit stereotyping. For instance, [204] found that GPT-4 exhibited more subtle, but still measurable, racial and gender biases in healthcare recommendations, suggesting that while the most egregious biases may be reduced, more nuanced forms persist. On the other hand, the increased scale and capability of newer models can introduce new challenges: larger context windows may amplify biases over extended interactions, and the ability to generate more fluent and persuasive text may make biased outputs harder to detect. Open-source models such as Llama 3 present additional considerations, as community fine-tuning may introduce or remove biases in ways that are difficult to track or audit. The rapid pace of model development underscores the need for continuous, longitudinal bias evaluation rather than one-time assessments, and for evaluation benchmarks that evolve alongside model capabilities.