Agents and Multi-Agent Systems

Journal Name	Impact Factor	CiteScore	Launched Year	First Decision (median)	APC
Applied Sciences applsci	2.5	5.3	2011	18.4 Days	CHF 2400	Submit
Electronics electronics	2.6	5.3	2012	16.4 Days	CHF 2400	Submit
Energies energies	3.0	6.2	2008	16.8 Days	CHF 2600	Submit
Modelling modelling	1.3	2.7	2020	18.9 Days	CHF 1000	Submit
Systems systems	2.3	2.8	2013	19.6 Days	CHF 2400	Submit

34 pages, 442 KiB

Open AccessReview

A Review of Multi-Agent Reinforcement Learning Algorithms

by Jiaxin Liang, Haotian Miao, Kai Li, Jianheng Tan, Xi Wang, Rui Luo and Yueqiu Jiang

Electronics 2025, 14(4), 820; https://doi.org/10.3390/electronics14040820 - 19 Feb 2025

Viewed by 3838

In recent years, multi-agent reinforcement learning algorithms have demonstrated immense potential in various fields, such as robotic collaboration and game AI. This paper introduces the modeling concepts of single-agent and multi-agent systems: the fundamental principles of Markov Decision Processes and Markov Games. The [...] Read more.

In recent years, multi-agent reinforcement learning algorithms have demonstrated immense potential in various fields, such as robotic collaboration and game AI. This paper introduces the modeling concepts of single-agent and multi-agent systems: the fundamental principles of Markov Decision Processes and Markov Games. The reinforcement learning algorithms are divided into three categories: value-based, strategy-based, and actor–critic algorithms, and the algorithms and applications are introduced. Based on differences in reward functions, multi-agent reinforcement learning algorithms are further classified into three categories: fully cooperative, fully competitive, and mixed types. The paper systematically reviews and analyzes their basic principles, applications in multi-agent systems, challenges faced, and corresponding solutions. Specifically, it discusses the challenges faced by multi-agent reinforcement learning algorithms from four aspects: dimensionality, non-stationarity, partial observability, and scalability. Additionally, it surveys existing algorithm-training environments in the field of multi-agent systems and summarizes the applications of multi-agent reinforcement learning algorithms across different domains. Through this discussion, readers can gain a comprehensive understanding of the current research status and future trends in multi-agent reinforcement learning algorithms, providing valuable insights for further exploration and application in this field. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

23 pages, 1254 KiB

Open AccessArticle

Event-Triggered MFAILC Bipartite Formation Control for Multi-Agent Systems Under DoS Attacks

by Han Li, Lixia Fu and Wenchao Wu

Appl. Sci. 2025, 15(4), 1921; https://doi.org/10.3390/app15041921 - 12 Feb 2025

Viewed by 666

Abstract

For multi-input multi-output (MIMO) nonlinear discrete-time bipartite formation multiagent systems (BFMASs) performing trajectory tracking tasks with unknown dynamics, a dynamic event-triggered model-free adaptive iterative learning control (DET-MFAILC) algorithm is proposed to address periodic denial-of-service (DoS) attacks. First, using the pseudo-partial derivative, a compact [...] Read more.

For multi-input multi-output (MIMO) nonlinear discrete-time bipartite formation multiagent systems (BFMASs) performing trajectory tracking tasks with unknown dynamics, a dynamic event-triggered model-free adaptive iterative learning control (DET-MFAILC) algorithm is proposed to address periodic denial-of-service (DoS) attacks. First, using the pseudo-partial derivative, a compact format dynamic linearization (CFDL) method is employed to construct an equivalent CFDL data model for the MIMO multi-agent system. A DoS attack model and its corresponding compensation algorithm are developed, while a dynamic event-triggered condition is designed considering both the consensus error and the tracking error. Subsequently, the proposed DoS attack compensation algorithm and the dynamic event-triggered mechanism are integrated with the model-free adaptive iterative learning control algorithm to design a controller, which is further extended from fixed-topology systems to time-varying topology systems. The convergence of the control system is rigorously proven. Finally, simulation experiments are conducted on bipartite formation multi-agent systems (BFMASs) under fixed and time-varying communication topologies. The results demonstrate that the proposed algorithm effectively mitigates the impact of DoS attacks, reduces controller updates, conserves network resources, and ensures that both the tracking error and consensus error converge to an ideal range close to zero within a finite number of iterations while maintaining a good formation shape. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

29 pages, 3720 KiB

Open AccessArticle

Modeling, Simulation, and Control of a Rotary Inverted Pendulum: A Reinforcement Learning-Based Control Approach

by Ruben Hernandez, Ramon Garcia-Hernandez and Francisco Jurado

Modelling 2024, 5(4), 1824-1852; https://doi.org/10.3390/modelling5040095 - 27 Nov 2024

Cited by 2 | Viewed by 2523

Abstract

In this paper, we address the modeling, simulation, and control of a rotary inverted pendulum (RIP). The RIP model assembled via the MATLAB (Matlab 2021a)^®/Simulink (Simulink 10.3) Simscape (Simscape 7.3)™ environment demonstrates a high degree of fidelity in its capacity to [...] Read more.

In this paper, we address the modeling, simulation, and control of a rotary inverted pendulum (RIP). The RIP model assembled via the MATLAB (Matlab 2021a)^®/Simulink (Simulink 10.3) Simscape (Simscape 7.3)™ environment demonstrates a high degree of fidelity in its capacity to capture the dynamic characteristics of an actual system, including nonlinear friction. The mathematical model of the RIP is obtained via the Euler–Lagrange approach, and a parameter identification procedure is carried out over the Simscape model for the purpose of validating the mathematical model. The usefulness of the proposed Simscape model is demonstrated by the implementation of a variety of control strategies, including linear controllers as the linear quadratic regulator (LQR), proportional–integral–derivative (PID) and model predictive control (MPC), nonlinear controllers such as feedback linearization (FL) and sliding mode control (SMC), and artificial intelligence (AI)-based controllers such as FL with adaptive neural network compensation (FL-ANC) and reinforcement learning (RL). A design methodology that integrates RL with other control techniques is proposed. Following the proposed methodology, a FL-RL and a proportional–derivative control with RL (PD-RL) are implemented as strategies to achieve stabilization of the RIP. The swing-up control is incorporated into all controllers. The visual environment provided by Simscape facilitates a better comprehension and understanding of the RIP behavior. A comprehensive analysis of the performance of each control strategy is conducted, revealing that AI-based controllers demonstrate superior performance compared to linear and nonlinear controllers. In addition, the FL-RL and PD-RL controllers exhibit improved performance with respect to the FL-ANC and RL controllers when subjected to external disturbance. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

20 pages, 538 KiB

Open AccessArticle

Learning-Based Multi-Stage Formation Scheduling with a Hybrid Controller

by Zhichao Zhang and Yao Li

Systems 2024, 12(11), 465; https://doi.org/10.3390/systems12110465 - 31 Oct 2024

Viewed by 719

Abstract

In the past decades, multi-agent systems have been a hot topic due to their wide applications, and the formation of multi-agent systems is a branch involving navigation, obstacle avoidance, controller design, and other issues. Due to the increasing requirements for accuracy and efficiency, [...] Read more.

In the past decades, multi-agent systems have been a hot topic due to their wide applications, and the formation of multi-agent systems is a branch involving navigation, obstacle avoidance, controller design, and other issues. Due to the increasing requirements for accuracy and efficiency, as well as for a bridge to link the sensing and control stages, the importance of transmission scheduling is gradually emerging, with the scheduling of limited resources under various constraints to better complete tasks becoming a focus of attention. However, most of the literature only considers the formation process as a whole, while overlooking the discrepancies in the formation process at different time stages. In this paper, a multi-stage formation scheduling problem with limited communication resources is studied. A multi-stage model has been proposed based on the different completion levels of formation. Compared to the single-stage model, the proposed multi-stage model reflects the different requirements during the formation process. Furthermore, in order to save communication energy, three transmission modes have been defined to reduce energy consumption in terms of communication frequency and communication radius. Considering the need for the dynamic scheduling of coupled parameters, we propose a reinforcement-learning-based hybrid controller which includes a basic controller and a fuzzy controller. The hybrid controller, which continuously adjusts parameters according to the requirements of each stage, conducts a trade-off between system performance and energy consumption. Additionally, the reinforcement learning ensures that all parameters are optimal in the corresponding situation. The simulation results show that our controller ensures both dynamic and steady-state performance with lower energy consumption. The comparison with other scheduling strategies demonstrates the optimality and effectiveness of our proposed framework and algorithms. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

19 pages, 1472 KiB

Open AccessArticle

Fault-Tolerant Cooperative Control of Multiple Uncertain Euler-Lagrange Systems with an Uncertain Leader

by Rongrong Gu and Xudong Sun

Electronics 2024, 13(11), 2068; https://doi.org/10.3390/electronics13112068 - 27 May 2024

Cited by 1 | Viewed by 802

Abstract

This paper explores the fault-tolerant cooperative control of multiagent systems, which are modeled via an uncertain leader system and multiple uncertain Euler–Lagrange systems with actuator faults. A self-adjusting observer is initially proposed to estimate the signal of the uncertain leader for different followers [...] Read more.

This paper explores the fault-tolerant cooperative control of multiagent systems, which are modeled via an uncertain leader system and multiple uncertain Euler–Lagrange systems with actuator faults. A self-adjusting observer is initially proposed to estimate the signal of the uncertain leader for different followers and compute the observer gain in real time. An adaptive fault-tolerant controller is designed based on the above observer and nonsingular fast terminal sliding mode surface. This controller estimates lumped uncertainty and ensures that tracking errors are ultimately bounded. The controller designed in this paper has the following three advantages. Firstly, the observer can estimate and transmit the leader’s state to each follower even without precise knowledge of the leader’s system matrix. Secondly, the controller is robust to actuator faults, uncertain parameters and external disturbances, the upper bounds of which can be arbitrarily large and unidentified. Thirdly, the controller has a simple structure and is also suitable for situations where the actuator is healthy. Lastly, simulations are provided to demonstrate the effectiveness of both the observer and the controller with or without actuator fault. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

25 pages, 30065 KiB

Open AccessArticle

Bio-Inspired Intelligent Swarm Confrontation Algorithm for a Complex Urban Scenario

by He Cai, Yaoguo Luo, Huanli Gao and Guangbin Wang

Electronics 2024, 13(10), 1848; https://doi.org/10.3390/electronics13101848 - 9 May 2024

Viewed by 1486

Abstract

This paper considers the confrontation problem for two tank swarms of equal size and capability in a complex urban scenario. Based on the Unity platform (2022.3.20f1c1), the confrontation scenario is constructed featuring multiple crossing roads. Through the analysis of a substantial amount of [...] Read more.

This paper considers the confrontation problem for two tank swarms of equal size and capability in a complex urban scenario. Based on the Unity platform (2022.3.20f1c1), the confrontation scenario is constructed featuring multiple crossing roads. Through the analysis of a substantial amount of biological data and wildlife videos regarding animal behavioral strategies during confrontations for hunting or food competition, two strategies are been utilized to design a novel bio-inspired intelligent swarm confrontation algorithm. The first one is the “fire concentration” strategy, which assigns a target for each tank in a way that the isolated opponent will be preferentially attacked with concentrated firepower. The second one is the “back and forth maneuver” strategy, which makes the tank tactically retreat after firing in order to avoid being hit when the shell is reloading. Two state-of-the-art swarm confrontation algorithms, namely the reinforcement learning algorithm and the assign nearest algorithm, are chosen as the opponents for the bio-inspired swarm confrontation algorithm proposed in this paper. Data of comprehensive confrontation tests show that the bio-inspired swarm confrontation algorithm has significant advantages over its opponents from the aspects of both win rate and efficiency. Moreover, we discuss how vital algorithm parameters would influence the performance indices. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

18 pages, 7360 KiB

Open AccessArticle

Dynamic Multi-Target Self-Organization Hunting Control of Multi-Agent Systems

by Shouzhong He, Liangshun Wang, Mingming Liu, Weifeng Liu and Zhihai Wu

Appl. Sci. 2024, 14(9), 3875; https://doi.org/10.3390/app14093875 - 30 Apr 2024

Cited by 1 | Viewed by 1487

Abstract

In this paper, we present a novel coordinated method tailored to address the dynamic multi-target hunting control problem in multi-agent systems, offering significant practical value. Our approach encompasses several key components: initially, we introduce a task allocation model that integrates a fuzzy inference [...] Read more.

In this paper, we present a novel coordinated method tailored to address the dynamic multi-target hunting control problem in multi-agent systems, offering significant practical value. Our approach encompasses several key components: initially, we introduce a task allocation model that integrates a fuzzy inference system with a particle swarm optimization algorithm. This hybrid model efficiently allocates hunting tasks for scattered evading targets, effectively transforming the dynamic multi-target hunting problem into multiple dynamic single-target-hunting problems. This transformation enhances the speed and efficacy of task allocation. Subsequently, we propose an attraction/repulsive model grounded in potential field theory. This model facilitates the coordinated hunting of each target by organizing agents into subgroups. Relying solely on relative position and velocity information between agents and targets, our model simplifies computation, while maintaining effectiveness. Furthermore, the coordination of hunting activities for each target is achieved through a series of agent subgroups, guided by our proposed motion model. This systematic approach ensures a cohesive and efficient hunting strategy. Finally, we validate the effectiveness and feasibility of our proposed method through simulation results. These results provide empirical evidence of the method’s efficacy and potential applicability in real-world scenarios. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

20 pages, 1360 KiB

Open AccessArticle

Scalable Multi-Robot Task Allocation Using Graph Deep Reinforcement Learning with Graph Normalization

by Zhenqiang Zhang, Xiangyuan Jiang, Zhenfa Yang, Sile Ma, Jiyang Chen and Wenxu Sun

Electronics 2024, 13(8), 1561; https://doi.org/10.3390/electronics13081561 - 19 Apr 2024

Cited by 3 | Viewed by 1953

Abstract

Task allocation plays an important role in multi-robot systems regarding team efficiency. Conventional heuristic or meta-heuristic methods face difficulties in generating satisfactory solutions in a reasonable computational time, particularly for large-scale multi-robot task allocation problems. This paper proposes a novel graph deep-reinforcement-learning-based approach, [...] Read more.

Task allocation plays an important role in multi-robot systems regarding team efficiency. Conventional heuristic or meta-heuristic methods face difficulties in generating satisfactory solutions in a reasonable computational time, particularly for large-scale multi-robot task allocation problems. This paper proposes a novel graph deep-reinforcement-learning-based approach, which solves the problem through learning. The framework leverages the graph sample and aggregate concept as the encoder to extract the node features in the context of the graph, followed by a cross-attention decoder to output the probability that each task is allocated to each robot. A graph normalization technique is also proposed prior to the input, enabling an easy adaption to real-world applications, and a deterministic solution can be guaranteed. The most important advantage of this architecture is the scalability and quick feed-forward character; regardless of whether cases have a varying number of robots or tasks, single depots, multiple depots, or even mixed single and multiple depots, solutions can be output with little computational effort. The high efficiency and robustness of the proposed method are confirmed by extensive experiments in this paper, and various multi-robot task allocation scenarios demonstrate its advantage. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

20 pages, 1916 KiB

Open AccessArticle

Autonomous Agent Navigation Model Based on Artificial Potential Fields Assisted by Heuristics

by Daniel Silva-Contreras and Salvador Godoy-Calderon

Appl. Sci. 2024, 14(8), 3303; https://doi.org/10.3390/app14083303 - 14 Apr 2024

Viewed by 1137

Abstract

When autonomous agents are deployed in an unknown environment, obstacle-avoiding movement and navigation are required basic skills, all the more so when agents are limited by partial-observability constraints. This paper addresses the problem of autonomous agent navigation under partial-observability constraints by using a [...] Read more.

When autonomous agents are deployed in an unknown environment, obstacle-avoiding movement and navigation are required basic skills, all the more so when agents are limited by partial-observability constraints. This paper addresses the problem of autonomous agent navigation under partial-observability constraints by using a novel approach: Artificial Potential Fields (APF) assisted by heuristics. The well-known problem of local minima is addressed by providing the agents with the ability to make individual choices that can be exploited in a swarm. We propose a new potential function, which provides precise control of the potential field’s reach and intensity, and the use of auxiliary heuristics provides temporary target points while the agent explores, in search of the position of the real intended target. Artificial Potential Fields, together with auxiliary search heuristics, are integrated into a novel navigation model for autonomous agents who have limited or no knowledge of their environment. Experimental results are shown in 2D scenarios that pose challenging situations with multiple obstacles, local minima conditions and partial-observability constraints, clearly showing that an agent driven using the proposed model is capable of completing the navigation task, even under the partial-observability constraints. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

21 pages, 1086 KiB

Open AccessArticle

Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning

by Qi Fang, Junjie Zeng, Haotian Xu, Yue Hu and Quanjun Yin

Appl. Sci. 2024, 14(8), 3209; https://doi.org/10.3390/app14083209 - 11 Apr 2024

Viewed by 1116

Abstract

When agents need to collaborate without previous coordination, the multi-agent cooperation problem transforms into an ad hoc teamwork (AHT) problem. Mainstream research on AHT is divided into type-based and type-free methods. The former depends on known teammate types to infer the current teammate [...] Read more.

When agents need to collaborate without previous coordination, the multi-agent cooperation problem transforms into an ad hoc teamwork (AHT) problem. Mainstream research on AHT is divided into type-based and type-free methods. The former depends on known teammate types to infer the current teammate type, while the latter does not require them at all. However, in many real-world applications, the complete absence and sufficient knowledge of known types are both impractical. Thus, this research focuses on the challenge of AHT with limited known types. To this end, this paper proposes a method called a Few typE-based Ad hoc Teamwork via meta-reinforcement learning (FEAT), which effectively adapts to teammates using a small set of known types within a single episode. FEAT enables agents to develop a highly adaptive policy through meta-reinforcement learning by employing limited priors about known types. It also utilizes this policy to generate a diverse type repository automatically. During the ad hoc cooperation, the agent can autonomously identify known teammate types followed by directly utilizing the pre-trained optimal cooperative policy or swiftly updating the meta policy to respond to teammates of unknown types. Comprehensive experiments in the pursuit domain validate the effectiveness of the algorithm and its components. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

26 pages, 10833 KiB

Open AccessArticle

Effective Evolutionary Principles for System-of-Systems: Insights from Agent-Based Modeling in Vehicular Networks

by Junjie Liu, Junxian Liu and Mengmeng Zhang

Systems 2024, 12(3), 98; https://doi.org/10.3390/systems12030098 - 15 Mar 2024

Cited by 1 | Viewed by 1888

Abstract

System-of-systems (SoS) evolution is a complex and unpredictable process. Although various principles to facilitate collaborative SoS evolution have been proposed, there is a lack of experimental data validating their effectiveness. To address these issues, we present an Agent-Based Model (ABM) for SoS evolution [...] Read more.

System-of-systems (SoS) evolution is a complex and unpredictable process. Although various principles to facilitate collaborative SoS evolution have been proposed, there is a lack of experimental data validating their effectiveness. To address these issues, we present an Agent-Based Model (ABM) for SoS evolution in the Internet of Vehicles (IoV), serving as a quantitative analysis tool for SoS research. By integrating multiple complex and rational behaviors of individuals, we aim to simulate real-world scenarios as accurately as possible. To simulate the SoS evolution process, our model employs multiple agents with autonomous interactions and incorporates external environmental variables. Furthermore, we propose three evaluation metrics: evolutionary time, degree of variation, and evolutionary cost, to assess the performance of SoS evolution. Our study demonstrates that enhanced information transparency significantly improves the evolutionary performance of distributed SoS. Conversely, the adoption of uniform standards only brings limited performance enhancement to distributed SoSs. Although our proposed model has limitations, it stands out from other approaches that utilize Agent-Based Modeling to analyze SoS theories. Our model focuses on realistic problem contexts and simulates realistic interaction behaviors. This study enhances the comprehension of SoS evolution processes and provides valuable insights for the formulation of effective evolutionary strategies. Full article

(This article belongs to the Topic Agents and Multi-Agent Systems)

► Show Figures

Figure 1

Topic Menu

Topic Editors

Agents and Multi-Agent Systems

Topic Information

Keywords

Participating Journals

Published Papers (11 papers)

Further Information

Guidelines

MDPI Initiatives

Follow MDPI