Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies
Abstract
:1. Introduction
2. Sources of Bias in AI
2.1. Definition of Bias in AI and Its Different Types
2.2. Sources of Bias in AI, including Data Bias, Algorithmic Bias, and User Bias
2.3. Real-World Examples of Bias in AI
3. Impacts of Bias in AI
3.1. Negative Impacts of Bias in AI on Individuals and Society, Including Discrimination and Perpetuation of Existing Inequalities
3.2. Discussion of the Ethical Implications of Biased AI
4. Mitigation Strategies for Bias in AI
4.1. Overview of Current Approaches to Mitigate Bias in AI, Including Pre-Processing Data, Model Selection, and Post-Processing Decisions
4.2. Discussion of the Limitations and Challenges of These Approaches
5. Fairness in AI
5.1. Definition of Fairness in AI and Its Different Types
5.2. Comparison of Fairness and Bias in AI
5.3. Real-World Examples of Fairness in AI
6. Mitigation Strategies for Fairness in AI
6.1. Overview of Current Approaches to Ensure Fairness in AI, including Group Fairness and Individual Fairness
6.2. Discussion of the Limitations and Challenges of These Approaches
7. Conclusions
Funding
Acknowledgments
Conflicts of Interest
References
- Buolamwini, J.; Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, New York, NY, USA, 23–24 February 2018; pp. 77–91. [Google Scholar]
- Dastin, J. Amazon scraps secret AI recruiting tool that showed bias against women. In Ethics of Data and Analytics; Auerbach Publications: Boca Raton, FL, USA, 2018; pp. 296–299. [Google Scholar]
- Eubanks, V. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor; St. Martin’s Press: New York, NY, USA, 2018. [Google Scholar]
- Kleinberg, J.; Lakkaraju, H.; Leskovec, J.; Ludwig, J.; Mullainathan, S. Human decisions and machine predictions. Q. J. Econ. 2018, 133, 237–293. [Google Scholar] [PubMed]
- Kleinberg, J.; Ludwig, J.; Mullainathan, S.; Sunstein, C.R. Discrimination in the Age of Algorithms. J. Leg. Anal. 2018, 10, 113–174. [Google Scholar] [CrossRef]
- Kleinberg, J.; Ludwig, J.; Mullainathan, S.; Rambachan, A. Algorithmic fairness. AEA Pap. Proc. 2018, 108, 22–27. [Google Scholar] [CrossRef]
- O’Neil, C. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy; Broadway Books: New York, NY, USA, 2016. [Google Scholar]
- Asan, O.; Bayrak, A.E.; Choudhury, A. Artificial intelligence and human trust in healthcare: Focus on clinicians. J. Med. Internet Res. 2020, 22, e15154. [Google Scholar] [CrossRef] [PubMed]
- Berk, R.; Heidari, H.; Jabbari, S.; Kearns, M.; Roth, A. Fairness in Criminal Justice Risk Assessments: The State of the Art. Sociol. Methods Res. 2018, 47, 175–210. [Google Scholar] [CrossRef]
- Friedler, S.A.; Scheidegger, C.; Venkatasubramanian, S.; Choudhary, S.; Hamilton, E.P.; Roth, D. A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA, 29–31 January 2019; pp. 329–338. [Google Scholar]
- Yan, S.; Kao, H.T.; Ferrara, E. Fair class balancing: Enhancing model fairness without observing sensitive attributes. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Online, 26 June–31 July 2020; pp. 1715–1724. [Google Scholar]
- Caliskan, A.; Bryson, J.J.; Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 2017, 356, 183–186. [Google Scholar] [CrossRef] [PubMed]
- European Commission. Ethics Guidelines for Trustworthy AI. Commission Communication. 2019. Available online: https://op.europa.eu/en/publication-detail/-/publication/d3988569-0434-11ea-8c1f-01aa75ed71a1 (accessed on 15 December 2023).
- Ferrara, E. Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models. First Monday 2023, 28. [Google Scholar]
- Kleinberg, J.; Mullainathan, S.; Raghavan, M. Inherent trade-offs in the fair determination of risk scores. In Proceedings of the Innovations in Theoretical Computer Science (ITCS), Berkeley, CA, USA, 9–11 January 2017. [Google Scholar]
- Schwartz, R.; Vassilev, A.; Greene, K.; Perine, L.; Burt, A.; Hall, P. Towards a Standard for Identifying and Managing Bias in Artificial Intelligence; NIST Special Publication: Gaithersburg, MD, USA, 2022; Volume 1270, pp. 1–77.
- Crawford, K.; Calo, R. There is a blind spot in AI research. Nature 2016, 538, 311–313. [Google Scholar] [CrossRef]
- Selbst, A.D.; Boyd, D.; Friedler, S.A.; Venkatasubramanian, S.; Vertesi, J. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA, 29–31 January 2019; pp. 59–68. [Google Scholar]
- Angwin, J.; Larson, J.; Mattu, S.; Kirchner, L. Machine bias. In Ethics of Data and Analytics; Auerbach Publications: Boca Raton, FL, USA, 2016; pp. 254–264. [Google Scholar]
- Obermeyer, Z.; Powers, B.; Vogeli, C.; Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019, 366, 447–453. [Google Scholar] [CrossRef]
- Ferrara, E. GenAI against humanity: Nefarious applications of generative artificial intelligence and large language models. arXiv 2023, arXiv:2310.00737. [Google Scholar] [CrossRef]
- Ferrara, E. The butterfly effect in artificial intelligence systems: Implications for AI bias and fairness. arXiv 2023, arXiv:2307.05842. [Google Scholar] [CrossRef]
- Mittelstadt, B.D.; Allo, P.; Taddeo, M.; Wachter, S.; Floridi, L. The ethics of algorithms: Mapping the debate. Big Data Soc. 2016, 3, 2053951716679679. [Google Scholar] [CrossRef]
- Sweeney, L. Discrimination in online ad delivery. Commun. ACM 2013, 56, 44–54. [Google Scholar] [CrossRef]
- Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, Cambridge, MA, USA, 8–10 January 2012; pp. 214–226. [Google Scholar]
- Ananny, M.; Crawford, K. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media Soc. 2018, 20, 973–989. [Google Scholar] [CrossRef]
- Corbett-Davies, S.; Pierson, E.; Feller, A.; Goel, S.; Huq, A. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, 13–17 August 2017; pp. 797–806. [Google Scholar]
- Corbett-Davies, S.; Goel, S. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv 2018, arXiv:1808.00023. [Google Scholar]
- Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J.W.; Wallach, H.; Iii, H.D.; Crawford, K. Datasheets for datasets. Commun. ACM 2021, 64, 86–92. [Google Scholar] [CrossRef]
- Zafar, M.B.; Valera, I.; Gomez Rodriguez, M.; Gummadi, K.P. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, 3–7 May 2017; pp. 1171–1180. [Google Scholar]
- Kamiran, F.; Calders, T. Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 2012, 33, 1–33. [Google Scholar] [CrossRef]
- Barocas, S.; Selbst, A.D. Big data’s disparate impact. Calif. Law Rev. 2016, 104, 671–732. [Google Scholar] [CrossRef]
- Bolukbasi, T.; Chang, K.W.; Zou, J.Y.; Saligrama, V.; Kalai, A.T. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Adv. Neural Inf. Process. Syst. 2016, 29, 4349–4357. [Google Scholar]
- Ferguson, A.G. Predictive policing and reasonable suspicion. Emory LJ 2012, 62, 259. [Google Scholar] [CrossRef]
- Wachter, S.; Mittelstadt, B.; Russell, C. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. J. Law Technol. 2018, 31, 841–887. [Google Scholar] [CrossRef]
- Žliobaitė, I. Measuring discrimination in algorithmic decision making. Data Min. Knowl. Discov. 2017, 31, 1060–1089. [Google Scholar] [CrossRef]
- Crawford, K.; Paglen, T. Excavating AI: The politics of images in machine learning training sets. AI Soc. 2021, 36, 1105–1116. [Google Scholar] [CrossRef]
- Donovan, J.; Caplan, R.; Matthews, J.; Hanson, L. Algorithmic Accountability: A Primer; Data & Society: New York, NY, USA, 2018. [Google Scholar]
- Ezzeldin, Y.H.; Yan, S.; He, C.; Ferrara, E.; Avestimehr, S. Fairfed: Enabling group fairness in federated learning. In Proceedings of the AAAI 2023—37th AAAI Conference on Artificial Intelligence, Washington, DC, USA, 7–14 February 2023. [Google Scholar]
- Crenshaw, K. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In Feminist Legal Theories; Routledge: London, UK, 1989; pp. 23–51. [Google Scholar]
- Nicoletti, L.; Bass, D. Humans Are Biased: Generative AI Is Even Worse. Bloomberg Technology + Equality, 23 June 2023. [Google Scholar]
- Cirillo, D.; Catuara-Solarz, S.; Morey, C.; Guney, E.; Subirats, L.; Mellino, S.; Gigante, A.; Valencia, A.; Rementeria, M.J.; Chadha, A.S.; et al. Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. NPJ Digit. Med. 2020, 3, 81. [Google Scholar] [CrossRef] [PubMed]
- Noble, S.U. Algorithms of Oppression: How Search Engines Reinforce Racism; NYU Press: New York, NY, USA, 2018. [Google Scholar]
- Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 2017, 5, 153–163. [Google Scholar] [CrossRef] [PubMed]
- Huang, J.; Galal, G.; Etemadi, M.; Vaidyanathan, M. Evaluation and mitigation of racial bias in clinical machine learning models: Scoping review. JMIR Med. Inform. 2022, 10, e36388. [Google Scholar] [CrossRef] [PubMed]
- Park, J.; Arunachalam, R.; Silenzio, V.; Singh, V.K. Fairness in Mobile Phone-Based Mental Health Assessment Algorithms: Exploratory Study. JMIR Form. Res. 2022, 6, e34366. [Google Scholar] [CrossRef] [PubMed]
- Ricci Lara, M.A.; Echeveste, R.; Ferrante, E. Addressing fairness in artificial intelligence for medical imaging. Nat. Commun. 2022, 13, 4581. [Google Scholar] [CrossRef]
- Yan, S.; Huang, D.; Soleymani, M. Mitigating biases in multimodal personality assessment. In Proceedings of the 2020 International Conference on Multimodal Interaction, Utrecht, The Netherlands, 25–29 October 2020; pp. 361–369. [Google Scholar]
- Chouldechova, A.; Roth, A. The frontiers of fairness in machine learning. arXiv 2018, arXiv:1810.08810. [Google Scholar]
- Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. (CSUR) 2021, 54, 1–35. [Google Scholar] [CrossRef]
- Verma, S.; Rubin, J. Fairness definitions explained. In Proceedings of the International Workshop on Software Fairness, Gothenburg, Sweden, 29 May 2018; pp. 1–7. [Google Scholar]
- Lipton, Z.C. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 2018, 16, 31–57. [Google Scholar] [CrossRef]
- Mitchell, M.; Wu, S.; Zaldivar, A.; Barnes, P.; Vasserman, L.; Hutchinson, B.; Spitzer, E.; Raji, I.D.; Gebru, T. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA, 29–31 January 2019; pp. 220–229. [Google Scholar]
- Raji, I.D.; Buolamwini, J. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA, 29–31 January 2019; pp. 77–86. [Google Scholar]
- Chauhan, P.S.; Kshetri, N. The Role of Data and Artificial Intelligence in Driving Diversity, Equity, and Inclusion. Computer 2022, 55, 88–93. [Google Scholar] [CrossRef]
- Holstein, K.; Wortman Vaughan, J.; Daumé, H., III; Dudik, M.; Wallach, H. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; pp. 1–16. [Google Scholar]
- Stathoulopoulos, K.; Mateos-Garcia, J.C.; Owen, H. Gender Diversity in AI Research. 2019. Available online: https://www.nesta.org.uk/report/gender-diversity-ai/ (accessed on 15 December 2023).
Type of Bias | Description | Examples |
---|---|---|
Sampling Bias | Occurs when the training data are not representative of the population they serve, leading to poor performance and biased predictions for certain groups. | A facial recognition algorithm trained mostly on white individuals that performs poorly on people of other races. |
Algorithmic Bias | Results from the design and implementation of the algorithm may prioritize certain attributes and lead to unfair outcomes. | An algorithm that prioritizes age or gender, leading to unfair outcomes in hiring decisions. |
Representation Bias | Happens when a dataset does not accurately represent the population it is meant to model, leading to inaccurate predictions. | A medical dataset that under-represents women, leading to less accurate diagnosis for female patients. |
Confirmation Bias | Materializes when an AI system is used to confirm pre-existing biases or beliefs held by its creators or users. | An AI system that predicts job candidates’ success based on biases held by the hiring manager. |
Measurement Bias | Emerges when data collection or measurement systematically over- or under-represents certain groups. | A survey collecting more responses from urban residents, leading to an under-representation of rural opinions. |
Interaction Bias | Occurs when an AI system interacts with humans in a biased manner, resulting in unfair treatment. | A chatbot that responds differently to men and women, resulting in biased communication. |
Generative Bias | Occurs in generative AI models, like those used for creating synthetic data, images, or text. Generative bias emerges when the model’s outputs disproportionately reflect specific attributes, perspectives, or patterns present in the training data, leading to skewed or unbalanced representations in generated content. | A text generation model trained predominantly on the literature from Western authors may over-represent Western cultural norms and idioms, under-representing or misrepresenting other cultures. Similarly, an image generation model trained on datasets with limited diversity in human portraits may struggle to accurately represent a broad range of ethnicities. |
Approach | Description | Examples | Limitations and Challenges | Ethical Considerations |
---|---|---|---|---|
Pre-processing Data | Involves identifying and addressing biases in the data before training the model. Techniques such as oversampling, undersampling, or synthetic data generation are used to ensure the data are representative of the entire population, including historically marginalized groups. | 1. Oversampling darker-skinned individuals in a facial recognition dataset [1]. 2. Data augmentation to increase representation in underrepresented groups. 3. Adversarial debiasing to train the model to be resilient to specific types of bias [33]. | 1. Time-consuming process. 2. May not always be effective, especially if the data used to train models are already biased. | 1. Potential for over- or underrepresentation of certain groups in the data, which can perpetuate existing biases or create new ones. 2. Privacy concerns related to data collection and usage, particularly for historically marginalized groups. |
Model Selection | Focuses on using model selection methods that prioritize fairness. Researchers have proposed methods based on group fairness or individual fairness. Techniques include regularization, which penalizes models for making discriminatory predictions, and ensemble methods, which combine multiple models to reduce bias. | 1. Selecting classifiers that achieve demographic parity [31]. 2. Using model selection methods based on group fairness [11] or individual fairness [30]. 3. Regularization to penalize discriminatory predictions. 4. Ensemble methods to combine multiple models and reduce bias [34]. | Limited by the possible lack of consensus on what constitutes fairness. | 1. Balancing fairness with other performance metrics, such as accuracy or efficiency. 2. Potential for models to reinforce existing stereotypes or biases if fairness criteria are not carefully considered. |
Post-processing Decisions | Involves adjusting the output of AI models to remove bias and ensure fairness. Researchers have proposed methods that adjust the decisions made by a model to achieve equalized odds, ensuring that false positives and false negatives are equally distributed across different demographic groups. | Post-processing methods that achieve equalized odds [11]. | Can be complex and require large amounts of additional data [32]. | 1. Trade-offs between different forms of bias when adjusting predictions for fairness. 2. Unintended consequences on the distribution of outcomes for different groups. |
Type of Fairness | Description | Examples |
---|---|---|
Group Fairness | Ensures that different groups are treated equally or proportionally in AI systems. Can be further subdivided into demographic parity, disparate mistreatment, or equal opportunity. | 1. Demographic parity: Positive and negative outcomes distributed equally across demographic groups [31]. 2. Disparate mistreatment: Defined in terms of misclassification rates [30]. 3. Equal opportunity: True positive rate (sensitivity) and false positive rate (1-specificity) are equal across different demographic groups [11]. |
Individual Fairness | Ensures that similar individuals are treated similarly by AI systems, regardless of their group membership. Can be achieved through methods such as similarity-based or distance-based measures. | Using similarity-based or distance-based measures to ensure that individuals with similar characteristics or attributes are treated similarly by the AI system [25]. |
Counterfactual Fairness | Aims to ensure that AI systems are fair, even in hypothetical scenarios. Specifically, counterfactual fairness aims to ensure that an AI system would have made the same decision for an individual, regardless of their group membership, even if their attributes had been different. | Ensuring that an AI system would make the same decision for an individual, even if their attributes had been different [35]. |
Procedural Fairness | Involves ensuring that the process used to make decisions is fair and transparent. | Implementing a transparent decision-making process in AI systems. |
Causal Fairness | Involves ensuring that the system does not perpetuate historical biases and inequalities. | Developing AI systems that avoid perpetuating historical biases and inequalities [4,5,6]. |
Approach | Description | Examples | Limitations and Challenges |
---|---|---|---|
Group Fairness | Ensures that AI systems are fair to different groups of people, such as people of different genders, races, or ethnicities. Aims to prevent the AI system from systematically discriminating against any group. Can be achieved through techniques such as re-sampling, pre-processing, or post-processing the data. | 1. Re-sampling techniques to create a balanced dataset. 2. Pre-processing or post-processing to adjust AI model output. | 1. May result in unequal treatment of individuals within a group. 2. May not address systemic biases that affect individual characteristics. 3. Group fairness metrics may not consider intersectionality. |
Individual Fairness | Ensures that AI systems are fair to individuals, regardless of their group membership. Aims to prevent the AI system from making decisions that are systematically biased against certain individuals. Can be achieved through techniques such as counterfactual fairness or causal fairness. | 1. Counterfactual fairness ensuring the same decision regardless of race or gender. | 1. May not address systemic biases that affect entire groups. 2. Difficulty determining which types of fairness are appropriate for a given context and how to balance them. |
Transparency | Involves making the AI system’s decision-making process visible to users. | Making AI system’s decisions and processes understandable to users. | Different definitions of fairness among people and groups and changing definitions over time. |
Accountability | Involves holding the system’s developers responsible for any harm caused by the system. | Developers held responsible for unfair decisions made by AI systems. | Determining responsibility and addressing potential harm. |
Explainability | Involves making the AI system’s decisions understandable to users. | Providing clear explanations of AI system’s decisions. | Addressing the complexity of human behavior and decision-making. |
Intersectionality (not explicitly mentioned as an approach, but it is an aspect to consider) | Considers the ways in which different dimensions of identity (such as race, gender, and socioeconomic status) interact and affect outcomes. | Developing AI systems that consider the interaction of different dimensions of identity. | Addressing the complexity of intersectionality and ensuring fairness across multiple dimensions of identity. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ferrara, E. Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci 2024, 6, 3. https://doi.org/10.3390/sci6010003
Ferrara E. Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci. 2024; 6(1):3. https://doi.org/10.3390/sci6010003
Chicago/Turabian StyleFerrara, Emilio. 2024. "Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies" Sci 6, no. 1: 3. https://doi.org/10.3390/sci6010003
APA StyleFerrara, E. (2024). Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci, 6(1), 3. https://doi.org/10.3390/sci6010003