Next Article in Journal
Pruning Fuzzy Neural Network Applied to the Construction of Expert Systems to Aid in the Diagnosis of the Treatment of Cryotherapy and Immunotherapy
Next Article in Special Issue
The Supermoral Singularity—AI as a Fountain of Values
Previous Article in Journal
Mining Temporal Patterns to Discover Inter-Appliance Associations Using Smart Meter Data
Previous Article in Special Issue
Global Solutions vs. Local Solutions for the AI Safety Problem
Open AccessCommunication

Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence

Independent Researcher, Silver Spring, MD 20910, USA
Big Data Cogn. Comput. 2019, 3(2), 21; https://doi.org/10.3390/bdcc3020021
Received: 28 February 2019 / Revised: 24 March 2019 / Accepted: 29 March 2019 / Published: 5 April 2019
(This article belongs to the Special Issue Artificial Superintelligence: Coordination & Strategy)
An important challenge for safety in machine learning and artificial intelligence systems is a set of related failures involving specification gaming, reward hacking, fragility to distributional shifts, and Goodhart’s or Campbell’s law. This paper presents additional failure modes for interactions within multi-agent systems that are closely related. These multi-agent failure modes are more complex, more problematic, and less well understood than the single-agent case, and are also already occurring, largely unnoticed. After motivating the discussion with examples from poker-playing artificial intelligence (AI), the paper explains why these failure modes are in some senses unavoidable. Following this, the paper categorizes failure modes, provides definitions, and cites examples for each of the modes: accidental steering, coordination failures, adversarial misalignment, input spoofing and filtering, and goal co-option or direct hacking. The paper then discusses how extant literature on multi-agent AI fails to address these failure modes, and identifies work which may be useful for the mitigation of these failure modes. View Full-Text
Keywords: multi-agent systems; specification gaming; artificial intelligence safety; Goodhart’s Law multi-agent systems; specification gaming; artificial intelligence safety; Goodhart’s Law
MDPI and ACS Style

Manheim, D. Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence. Big Data Cogn. Comput. 2019, 3, 21.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop