A Multi-Agent Based Intelligent Training System for Unmanned Surface Vehicles

Abstract

The modeling and design of multi-agent systems is imperative for applications in the evolving intelligence of unmanned systems. In this paper, we propose a multi-agent system design that is used to build a system for training a team of unmanned surface vehicles (USVs) where no historical data concerning the behavior is available. In this approach, agents are built as the physical controller of each USV and their cooperative decisions used for the USVs’ group coordination. To make our multi-agent system intelligently coordinate USVs, we built a multi-agent-based learning system. First, an agent-based data collection platform is deployed to gather competition data from agents’ observation for on-line learning tasks. Second, we design a genetic-based fuzzy rule training algorithm that is capable of optimizing agents’ coordination decisions in an accumulated manner. The simulation results of this study demonstrate that our proposed training approach is feasible and able to converge to a stable action selection policy towards efficient multi-USVs’ cooperative decision making.

1. Introduction

The modeling and design of multi-agent systems for applications in the evolving intelligence of unmanned systems is interesting and promising [1,2,3,4], especially in situations where traditional methods can be costly, dangerous, or even impossible to realize. Several applications can be found in a very broad range of domains such as energy [5,6,7], security [8], robotics [9], and resource management [10]. A multi-agent system consists of autonomous agents that interact in an environment to achieve specific goals [11]. An autonomous agent, in this sense, is able to perceive its environment and perform actions using actuators.

Over the years, there have been several studies that have proposed principles for designing multi-agent systems, as well as approaches to coordinate the individual behavior of agents [12,13,14,15,16]. In most multi-agent application domains, a priori specification of the optimal agents’ behavior is difficult due to the complexity and/or dynamics of the environment [11]. In such environments, it is natural for agents to adapt or learn optimal actions that maximize performance on-line. However, one of the key challenges is the need to build a simulation platform that can be used for fast training so as to gather enough training data to promote the intelligent development process.

In the context of multi-surface vehicles’ modeling and design [17,18,19], an unmanned surface vehicles (USVs) system is one of the development trends of modern weapon equipment [20]. The unmanned surface vehicle system has become an important means of information confrontation, precision strikes, and special combat tasks in future war [21]. Over the years, unmanned surface vehicles have long been applied to varying kinds of military applications in the real world. Some of the application domains include anti-mine warfare, submarine warfare, reconnaissance, and surveillance [22,23,24]. Due to the open and complex dynamic combat environment, the USV combat system must develop in the direction of autonomy and synergy, and the future surface unmanned combat should have a strong autonomous capacity, be able to be controlled autonomously, and complete complex and diverse combat tasks independently or collaboratively in a complex uncertain environment [25]. However, human knowledge is difficult to apply directly to the coordination of USVs.

We propose a multi-agent-based intelligent training system for unmanned surface vehicles (USVs). We focus on the problem of training agents of a multi-agent-based unmanned surface vehicles system where no historic data concerning the agent behavior is available. Using the team learning framework, we provide approaches for learning the rule base for multi-agent systems, designing the learning environment for simulating cooperative and competitive agents’ behavior, and gathering competition data from agents’ observation for off-line learning tasks. To this end, this paper first presents a decentralized coordination platform and simulator design based on a multi-agent architecture. Secondly, a USV-based agent model is presented. The paper also proposes a data collection platform for agent learning and USV training. Lastly, a fast learning and training algorithm design based on the agents’ rule base is presented.

To evaluate our approach, we used an island-conquering scenario where two teams of unmanned surface vehicles compete to conquer islands in an environment. We model this case study as a partially-observable stochastic game where one team has to learn the behavior that maximizes their returns against a human-controlled team. Our empirical evaluations show that our approach to learning the knowledge base of a multi-USV system, when applied to the trained team, was able to find a knowledge base (KB) to achieve better performance.

We first present the problem this study seeks to address in Section 2. Section 3 discusses the multi-surface vehicle training system design. Finally, in Section 4, we discuss our experiments and analysis of the simulation results followed by the conclusion of our study in Section 5.

2. Multi-Agent-Based USVs’ Training Problem

We first discuss the multi-agent-based intelligent training system learning problem of this study.

In general, a stochastic game is an extension of the Markov decision process (MDP) to the multi-agent context. In such a game, agents may have conflicting goals, and their joint actions determine rewards and transition between states. Stochastic games usually assume agents as having a complete view of their states, whereas the partial observation case is discussed under the more general partially-observable stochastic games (POSGs) domain. Our study is performed under the POSG case.

Theorem 1.

A POSG is a tuple $<X,U_1\dots U_n,O_1\dots O_n,f,R_1\dots R_n>$ where:

• n is the number of agents
• X is the finite set of states
• $U_i,i=1,\dots ,n$ is the finite set of actions available to the agents, which form the joint action set $\mathbf{U}=U_1\times \dots \times U_n$
• $O_i,i=1,\dots ,n$ is the finite set of observations of the agents. The joint observation is denoted as $\mathit{o}=\{o_1,\dots ,o_n\}$
• $f:X\times \mathbf{U}\times X\to \left[0,1\right]$ is the Markovian state transition function where $f\left(x^{\prime }\right|x,\mathit{u})$ denotes the probability of reaching state $x^{\prime }$ after the joint action $\mathit{u}$ in state x. The joint action of all agents $\mathit{u}$ at a stage t is denoted as ${\mathit{u}}_t={[u_{1,t},\dots ,u_{n,t}]}^{\top }{u}_t\in \mathbf{U},u_{i,t}\in U_i$
• $R_i:X\times \mathbf{U}\times X\to \mathbb{R}$, $i=1,\dots ,n$ are the reward functions of the agents.

In this study, we consider a team of multiple unmanned surface vehicles that must be trained to perform a given mission m in a mission space $\mathcal{D}\subset {\mathbb{R}}^n$ consisting of opposite forces. Let $H\left(m\right)$ be the objective function dependent on the mission. Given that a team T consists of N USVs, the dynamics of the $i^{\mathrm{th}}$ USV is given by:

$$\begin{array}{cc}\hfill \dot{x}{}_i& =f_i(x_i\left(t\right),u_i\left(t\right))\hfill \\ \hfill i& =1,2,3\dots ,N\hfill \end{array}$$

(1)

where $x_i\left(t\right)\in {\mathbb{R}}^n$ and $u_i\left(t\right)\in {\mathbb{R}}^m$ are the state and control inputs (available actions) of the USV, respectively, at time t. Depending on the USV model and mission type, a set of constraints (such as obstacle avoidance and the physical limitations of the USV) on the state $x_i\left(t\right)$ of a USV are imposed. The control inputs of a USV are bounded according to the limits of the USV’s actuators $|u_i\left(t\right)|<u_{max}$.

The reward of an USV i after taking action $u_{i,t}\in U_i$ in stage t is denoted as $r_{i,t+1}$. The individual policies $h_i:X\times U_i\to \left[0,1\right]$ of the USVs in a team form the joint policy $\mathbf{h}$. In the case of team learning, the joint policy is defined by the single learner. Since the reward of each USV in a team depends on the joint action, the respective USV rewards depend on the joint policy:

$$R_i^{\mathbf{h}}\left(x\right)=\mathbf{E}\left\{\sum _{t=0}^T{\gamma }^t{r}_{i,t+1}|x_0=x,\mathbf{h}\right\}$$

(2)

where $\gamma$ is the discounting factor and T is the length of the horizon. The training objective for the intelligent training system is to find a policy that maximizes Equation (2) such that the trained team of USVs can outperform its rival USVs, while adapting to the changing dynamics of it’s operating environment.

3. Multi-Agent-Based Training System Design

In this section, we discuss the multi-agent-based training system design of our study, as well as the ideas involved in training the agents. To start with, the section first presents the architecture of the multi-agent-based intelligent training system. Next, we present the USV agent model for cooperative training and competition. Finally, the section presents the agent data collection platform for gathering competition for the agent information sharing and learning algorithm.

3.1. System Architecture

The architecture of the system used in this study is presented in Figure 1. We decouple the agents’ control from the main environment in order to allow different implementations of controllers. In this case, we consider the multi-agent simulation platform as a server, and any attached controller becomes a client. Each USV agent selects its action based on the observed information from the environment. Based on the data visualization model of the USV, the data collection platform forms and mounts the corresponding resource tree structure of the USV. Historical data are provided to the upper machine learning algorithm by the data collection platform after data fusion. The obtained algorithm controller is then loaded into the USV.

The multi-agent cooperative reasoner component serves as the policy used to select an action for a controlled agent using the agent’s local observation. In this study, we model the reasoner as an FISdecomposed into a genetic fuzzy trees (GFT). Thus, the reasoner maintains a KB, which is used for the mapping fuzzy values of the observation of an agent to an action for execution in the POSG.

The learner component of the intelligent control model facilitates KB learning, tuning, or optimizing. It seeks to find the policy that exhibits the designed behavior of the controlled team as specified by the reward function. Thus, gradient-based approaches such as neural networks (NN) [26], as well as non-gradient-based approaches can be employed for this purpose. As already indicated in our usage of GFT for the reasoner component, we use the GA in this study to learn the fuzzy rules and tune the MFsof a GFT. Each candidate KB is applied to the control problem for the whole of a simulation episode (we use round and episode interchangeably) as found in the Pittsburgh approach. Hence, the rewards that are received each time step are aggregated and assigned as the fitness value of the candidate KB used at the end of the round/episode.

3.2. Agent Design

The USV agent model that is used by the intelligent training system for cooperative training and competition is equipped with a radar and weapon system. The radar of a USV can detect a hostile threat within it’s detection range. In this work, the weapon system of a USV consists of a calibrated gun with limited forward turning angle and firing range. A USV can turn in both directions (left or right) and with limited speed and turning radius r. In this study, only surge, sway, and yaw are used to describe the USV’s movement at sea, as shown Figure 2). Therefore, the kinematic relationship between the USV position in the global inertial frame XYZ and the boat body-fixed frame xyz can be defined as:

$$\begin{array}{cc}\hfill x& =ucos\left(\psi \right)-vsin\left(\psi \right)\hfill \\ \hfill y& =vsin\left(\psi \right)+vcos\left(\psi \right)\hfill \\ \hfill \psi & =\lambda \hfill \end{array}$$

(3)

The location and orientation of USVs in the coordinates of the environment are represented by $(x,y)$ and $\psi$ in the Earth-fixed reference frame, while $u,v,$ and $\lambda$ represent the velocity of surge, sway, and yaw in the body-fixed reference frame, respectively.

3.3. Data Collection Model

Agents’ observation can persist in the course of training to enable off-line operations. Data collected during training can be useful for agents’ cooperative decision making. Furthermore, the gathering of training data means that different agents’ learning algorithm can be employed both on-line and off-line for agents’ performance improvement. Moreover, the data can be organized to enable communication between agents. The gathered domain data (in resource forms) can be used for purposes such as improving agents’ performance using state-of-the-art deep reinforcement learning (DRL) algorithms. Hence, the data collector component is not only necessary, but also can extend the scope of learning algorithms that can be employed for intelligent training.

The data collection model is designed using a training and control platform consisting of loosely-coupled components and a common resource model. One such paradigm for implementing modular services is the Future Airborne Capability Environment’s (FACE (http://www.opengroup.org/face)) common operating environment approach. In Figure 3, we show the components of FACE with regards to our study. Their descriptions, in the context of our study, are as follows:

• Historical training data level (HTDL): The HTDL layer consists of the entities involved in learning the KB or agents’ policies for a POSG. In situations where the reasoner or policy output corresponds to composite tasks, planning and agent scheduling services may be implemented as modular components in this layer. This layer contains the historical resource data that are leveraged by the underlying learning algorithm to learn the KB.
• Cooperative decision data level: This level consists of entities that handle individual agent-level data for agent decision making. Data at this level are real-time data and observations from the agent.
• Individual agent decision model (IADM): A resource management service for hosting agents’ observations composes this layer. Such a service exposes a uniform set of APIs that can be consumed by the training and control platform or the agents for reasoning or cooperation operations. Modeling the observations as resource data enables the data elements to be uniquely addressed through the APIs.
• Data collection services: Services in this layer receive data from the agents in their raw form and then perform adaptation to a common resource model for use in the training and control platform. Thus, they abstract the heterogeneous forms that agents may report from their local observation.

3.4. Multi-Agent-Based USV Training Algorithm Design

The intelligent control model, as depicted in Figure 4, is responsible for performing learning and reasoning. It controls a team of agents in the simulation environment by sending agent actions to the environment every time step. Thus, at time step t, it sends the joint actions $u_t$ to the environment for execution where the action for agent $i,i\le n$, is $u_{i,t}\in u_t$. The intelligent control model consists of a learner and a reasoner that employs fuzzy logic and genetic algorithms (GA) known as genetic fuzzy systems (GFS) [27]. In this technique, GA is used to learn and tune the rule base and membership function of an FIS, respectively. To do this, an initial population of solutions, or strings, is created to encode the rule base and membership functions.

Classical approaches such as Michigan [28], Pittsburgh [29], and iterative rule learning [30] are mostly used to derive GFS fuzzy rules. We give an instance to illustrate how the rule base is encoded in this study using the Pittsburgh approach. Suppose $X_1$ and $X_2$ are input variables each with linguistic terms $\{ter{m}_1,ter{m}_2\}$ and output variable Y with terms $\{a_1,a_2\}$ with an arbitrary rule base of an FIS as follows:

• IF X1 IS term1 AND X2 IS term1 THEN Y IS a1
• IF X1 IS term1 AND X2 IS term2 THEN Y IS a2
• IF X1 IS term2 AND X2 IS term1 THEN Y IS a2
• IF X1 IS term2 AND X2 IS term2 THEN Y IS a1

Assume we assign codes zero and one to $a_1$ and $a_2$ respectively. The chromosome 0110 is obtained. This approach can represent an if-then rule in a single digit. This implies that each chromosome encodes possible outputs of a set of rules.

Similarly to tuning membership functions, each digit in the string corresponds to some endpoint of a membership function. GA can then be used to tune the parameters as part of the evolution process; such as found in [31], where initial parameters of a triangular MF $\mathbb{T}\left(\alpha ,\beta ,\gamma \right)$ are tuned using:

$${\alpha }_{i+1}\leftarrow ({\alpha }_i+{\delta }_i)-{\eta }_i$$

(4)

$${\beta }_{i+1}\leftarrow ({\beta }_i+{\delta }_i)$$

(5)

$${\gamma }_{i+1}\leftarrow ({\gamma }_i+{\delta }_i)-{\eta }_i$$

(6)

where $\delta$ and $\eta$ are the tuning coefficients, whereas $\alpha ,\beta$ and $\gamma$ parameterize $\mathbb{T}$. $\delta$ shifts the MF to the right or left, whereas $\eta$ shrinks or expands the MF. Therefore, every MF has two tuning parameters that can be optimized using GA.

Although a GFS can be used to find KBs that optimize the fitness function, it is inefficient to use a single GFS for a complex control problem. In such a case, there is an increase in GA search space complexity, the tendency to have redundant rules, and KB size. Therefore, using genetic fuzzy trees (GFT) in [2] helps to mitigate this problem. A GFT is, essentially, an ensemble of GFSs arranged in a tree structure according to a logical and conditional sequence of execution. A GFT enables the decomposition of a complex fuzzy control system (FCS) or GFS into logical sections with each node in the tree focusing on an aspect of the control problem. Each GFS in a GFT defines its own KB. Hence, the KB of a GFT consisting of $m\in \mathbb{R}$ GFSs $G=\{gf{s}_1,gf{s}_2,\dots ,gf{s}_m\}$ may have the genetic structure (represented as a concatenated string) $gf{s}_1^{rb}gf{s}_2^{rb}\dots gf{s}_n^{rb}gf{s}_1^{mf}gf{s}_2^{mf}\dots gf{s}_n^{mf}$ where $gf{s}_i^{rb}$ and $gf{s}_i^{mf}$, $i\le m$, denote the RB and MF string/genome, respectively, of a given GFS. This structure can be seen in Figure 5.

Algorithm 1 shows how beginning with an initial population, the KB of a GFT can be learned/optimized using the GA process. In Line 2, the initial population of the GFT KB is generated as the current population. This can be based on a predefined KB set or randomly generated. In Lines 3–4, the current population is passed on to the the FIS to be used for the control task. After all these KBs have taken turns performing the control task and have been evaluated, they are then subjected to the GA operators (Lines 6–17). This operation continues till the termination condition is reached. The resulting candidate KBs are returned as the best performing or most suitable of the KBs.

Algorithm 1 Procedure: GFS procedure.

Input:  GA hyperparameters Output:  Best set of GFT KBs 1: Initialize $t=0$ 2: Generate initial population as ${\mathcal{P}}_0$ 3: Set current population ${\mathcal{C}}_t:={\mathcal{P}}_0$ 4: Run the fuzzy control system using ${\mathcal{C}}_t$ 5: Evaluate the members of ${\mathcal{C}}_t$ 6: whileNot termination conditiondo 7:  ${\mathbb{P}}_t:=selection\left({\mathcal{C}}_t\right)$ 8:  ${\mathcal{O}}_t:=crossover\left({\mathbb{P}}_t\right)$ 9:  ${\mathcal{C}}_{temp}:=mutate\left({\mathcal{O}}_t\right)$ 10:  if elitism then 11:   ${\mathcal{C}}_{temp}:=applyElitism\left({\mathcal{C}}_{temp}\right)$ //keeps a percentage of previous chromosomes 12:  end if 13:  $t:=t+1$ 14:  ${\mathcal{C}}_t:={\mathcal{C}}_{temp}$ 15:  Run the fuzzy control system using ${\mathcal{C}}_t$ 16:  Evaluate the members of ${\mathcal{C}}_t$ 17: end while 18: Return best-performing candidate solutions

Non-stationarity is an inherent problem in the multiple learning agents’ environment. In this section, we present an approach for incorporating agent-induced non-stationarity awareness into a GFT based on the framework of [32].

Since the root of the GFT has the strongest impact on the learning process or action selection, we consider it to be the main component for addressing this problem. Unlike the other GFSs in the GFT, which can be modeled to address specific aspects of a control problem (domain dependent), we consider the root GFS to be a special case.

We regard the sub-trees that are formed under the root GFS as the elements of the BR function co-domain. Thus, a BR function, in the GFT case, selects a sub-policy to be used for selecting an action for the agent. The influence function is also represented as an FIS (or GFS in our case, since GA is applied to learn the KB), which takes PGFsas input variables and a BF as output the variable. This is motivated by the idea that an influence function maps beliefs to possible best responses and can be designed to use deductive reasoning. These are properties that an FIS exhibits due to fuzzy logic. Thus, the root of the GFT takes incomplete/partial observations and produces its belief of another agent’s policy. This connotes that the terms of the output variables then serve as possible opponent strategies as perceived by the reasoning agent.

In the case of the PGFs that serve as input variables to the GFT root FIS, the history of observations concerning the opponent or another agent is maintained and used by each PGF variable to produce an input value. This PGF input value can then be fuzzified and used in the inference process. The problem that arises is then how to design a PGF to capture the adaptation dynamics of another agent. In the work of [33], an agent i estimated the model of agent j as:

$${\widehat{\sigma }}_j^i\left(u_j\right)=\frac{C_j^i\left(u_j\right)}{{\sum }_{\overline{u}\in U_j}{C}_j^i\left({\overline{u}}_j\right)}$$

(7)

where $C_j^i\left(u_j\right)$ is the frequency that agent i observed agent j taking action $u_j$. Therefore, given the history of observations, Equation (7) can be extended to include the observation of agent i as:

$${\widehat{\sigma }}_j^i(o_i,u_j)=\frac{C_j^i(o_i,u_j)}{{\sum }_{\overline{u}\in U_j}{C}_j^i(o_i,{\overline{u}}_j)}$$

(8)

where $C_j^i(o_i,u_j)$ is the frequency that agent i observed agent j taking action $u_j$ when its local observation was $o_i$. ${\widehat{\sigma }}_j^i(o_i,u_j)$ becomes the input value of agent i’s PGF for monitoring a specific action $u_j$ of agent j. With the input variables (PGFs) and output variable (BF) determined for the root FIS, GA can be used to find the best response for agent-induced non-stationarity and thereby stabilize learning in the POSG.

4. Multi-Agent-Based Simulator Design

The simulator used for the experiment was developed using the Java programming language. The fuzzy control library [34] was extended for the design and encoding of fuzzy rules for learning purposes as implemented by the intelligent control model of our architecture. In this section, we present the experiment conducted for this study and follow it up with the analysis of the simulation results.

4.1. Environment Setup

To evaluate the effectiveness of the proposed learning system, a scenario of multiple boats competing for conquering more islands while engaging in combat was adopted. In this scenario, the environment set in a maritime setting consisted of N islands and two teams of unmanned surface vehicles (boats) referred to as blue and red forces. The boats were equipped with radar and guns for detecting and shooting enemies, respectively. The guns were set to have a fixed left-to-right traversal angle and shooting range, as shown in Figure 6. Since both teams had conflicting goals, a team achieved its goal by contending with opponent boats. Each team had information of the location and number of islands and their states, whether conquered or unconquered by the team. An island is said to be conquered by a team if a member of the team moves to the coordinate of the island and stays there for that time step or no opponent boats move to that particular island conquered by the team. If two opponent boats occupy an island at the same time, the island is not awarded to any team for the elapsed time steps. For our experiment, we implemented two controllers for both teams. The controller for the blue force was made to use fixed rules provided by humans, whereas the red force controller, which we sought to train, had to learn the best rules that maximized their performance. At the beginning of each time step, the simulation environment received a batch of commands from the (ally and enemy) controllers of both teams and updated the environment.

The possible actions of each agent are also explained below:

• retreat: the agent moves towards its base.
• left: turn left and move.
• right: turn right and move.
• straight: move forward at the current heading.
• stop: stop moving.
• assist: move to help a teammate that is under enemy fire; a boat can assist a random or the closest teammate.
• units: causes a boat to team up with other ally boats to conquer an island.
• closest: the boat conquers an unconquered island that is closest to it.

4.2. Competition Objective

The goal of each team was to conquer all islands and destroy opponents, while staying alive. The performance of each team was evaluated after each time step using the function:

$$R_t^f=A_t^f\times p+I_t^f-D_t^{f^{\prime }}\times p$$

(9)

where $A_t^f$ is the destruction suffered by the opponent team $f^{\prime }$ as a result of attacks from team f in time step t, $I_t^f$ the conquered island points received by team f, $D_t^f$ the damaged caused by the opponent $f^{\prime }$ to the team being assessed f, and p the points awarded for boat attacks/destruction. The winner of an encounter is decided after the end of the episode. The team with the highest score is declared a winner of that encounter.

4.3. Data Collector for Training and Learning

As mentioned earlier, gathering of training data is useful in a number ways. The cooperation of agents can be enhanced when agents share their observation. Furthermore, data gathered during training can be used later by other learning algorithms such as deep learning methods for analysis and performance improvement. The resource model shown in Figure 7 is used for on-line data gathering during training. The data that are sent to a controller as feedback take this form. The main components are as follows:

• Simulation information: This node provides general information about the simulation for a given time step.
• Observation: This provides the observation made by all agents in the team. Each agent also reports opponents that have been detected, as well as the observable properties of each detected opponent.
• System data: All miscellaneous data concerning the simulation platform (e.g., time, CPU consumption, memory usage, etc.) may be provided under this resource element.

4.4. Training and Learning Algorithm

In order to provide a realistic virtual environment and opponent for the red force to train against, the blue force control algorithm uses human-defined rules.

Table 1. Example encoded task selection rules used by the enemy team.

Unconquered	DetectedEnemies	TaskOutput
None	None	Retreat
None	Moderate	Intercept
None	Many	Intercept
Moderate	None	Conquer
Moderate	Moderate	Conquer
Moderate	None	Conquer
Moderate	Many	Intercept
Many	Many	Intercept
Many	Moderate	Conquer

shows an example encoding rules used for the blue force task selection during simulation. The GFT of the red force control algorithm is illustrated in Figure 8. The details of each of the GFSs of the GFT can also be seen in

Table 2. Details of the GFSsthat constitute the GFT used for training the ally team.

GFS	Input Variables	Output Variable	Output Variable Terms	RBSize	No. of MF Tuning Parameters
Task	UnconquredIslands	BeliefFunction	conquering Intercepting	243	30
	PGF_attacked
	PGF_moved
	PGF_conquered
	PGF_retreated
Intercept	DetectedEnemies	InterceptOutput	Assist Operation	6	6
	InEnemyFiringRange
Assist	DetectedEnemies	InterceptOutput	assistClosest assistRandom	9	12
	FactionDetectedEnemies
Operation	StayingPower	OperationOutput	track retreat	9	12
	DetectedEnemies
Track	DistanceToEnemy	TrackOutput	left, right stop, straight	21	20
	HeadingDifference
Conquer	UnconqueredIslands	UnconqueredIslands	conquer in units conquer closest	3	6

.

The Assignment GFS was designed to consider opponent-induced non-stationarity with the following PGFs:

• PGF_attacked: reports the confidence level that the detected enemy boats are attacking
• PGF_moved: reports the confidence level that the detected enemy boats are moving towards an island or the agent whose local observations are being used for reasoning
• PGF_conquered: computes the confidence level that the detected enemies are conquering an island
• PGF_retreated: evaluates the confidence level that the detected enemies are withdrawing to their initial positions or base.

The linguistic terms of each PGF are low, moderate, and high. Each PGF first computes the action probabilities for each detected agent. A softmax is then computed over the reported action probabilities, and the maximum is selected as the PGF input value. The belief function output variable selects the perceived opponent strategy in each IF-THEN rule. The other input variables are:

• UnconqueredIslands: The number of islands not conquered by the team to which a boat belongs.
• DetectedEnemies: The number of detected enemies by a boat.
• FactionDetectedEnemies: The number of detected enemies by a team.
• InEnemyFiringRange: Returns 1 if this boat is in the enemies’ firing range, and 0 otherwise.
• HeadingDifference: The difference in heading between this boat and the target boat.
• DistanceToEnemy: The distance between this boat and the target boat.
• StayingPower: The current strength of a boat to sustain enemy attacks.
• TeammatesUnderFire: Reports the number of teammates a boat has observed to be under attack by opponent boats.

Triangular MFs are used to define the semantics of the fuzzy rules. The initial MF tuning parameters were sampled uniformly from $[-1.5,1.5]$ for each input variable MF. The GA parameters we set for training the ally team controller are presented in

Table 3. The GA parameters used for training the ally team.

Parameter	Value
Population size	20
Number of generations	101 (initial population included)
Crossover probability	0.7
Mutation probability	0.1
Mutation decay Schedule	Exponential decay (decay rate = 0.15)
Crossover operator	BLXcrossover (alpha = 0.15)
Mutation operator	Adaptive non-uniform mutation (b = 5)
Selection operator	Tournament selection

.

As indicated in Table 3, we combined a mutation probability schedule with an adaptive non-uniform mutation to control exploration and exploitation. The ally team learner was set to maintain the last 5 observations of each detected enemy boat.

4.5. Experiment Results and Validation Analysis

We have evaluated our intelligent training system using four distinct scenarios of the island conquering case study. The parameters used in Scenario 1 for both team are shown in

Table 4. The fixed simulation parameters.

Parameter	Value
Number of boats per team	6
Number of islands	5
Simulation steps per episode	30
Time per step	3 s
Radar radius	140
Forward firing angle	20
Boat’s velocity	4/system iteration
Firing range	100

. Furthermore, in Scenario 1, sinking an opponent and conquering islands were both worth two points each. In Scenario 2, the number of USVs in a team was reduced to three, and sinking an opponent point was reduced to one. The simulation for Scenarios 1 and 2 was run concurrently on two different machines for 2020 from 20 randomly-generated chromosomes as the initial rules for the blue force against the enemy team, which used a human-sampled rules sets. Figure 9 shows the snapshots of the competition as simulation progresses.

If the blue force began an episode knowing exactly what to do in the environment and the red force was yet to learn, then the blue force should perform better than the red force in the initial stages of the simulation. However, as the simulation progresses, the performance of the red force should be improving since the red force learns, while the blue force uses the static rule. Furthermore, if the blue blue force is using a different set of rules even though a static set of rules, then there is a tendency for a dynamic rise and fall in the performance measure by the strength of a simulating rule set.

The observed simulation results of both teams during training are presented in Figure 10. As can be seen in Figure 10a,c, the blue force was performing better than the red force during the first generation of rules. However, after 10 and 13 generations, respectively, the red force began to outperform the blue force in the environment, and this is what caused the red force performance to rise in Figure 10a,c. In Figure 10b,d, we compare the highest score attained by both teams in each generation. Furthermore, the same analogy can be drawn from this graph, as in the the initial generations, the blue force seemed to be scoring high as compared to the counterpart red force.

In the third simulation, the capabilities of the red force was constrained below that of its counterpart blue force. With each team consisting of 10 boats, the firing range, detection range, and firing radius of the ally team was set to $80\%$ of the opponent team, respectively. Furthermore, we set the initial population of rules to that of the rule base of Simulation 1. Figure 10e,f shows the performance graphs of both teams.

Finally, a simulation of 15 boats with the objective of destroying more opponent boats while still countering islands was conducted. To this end, we assigned three points for destroying opponent boats and a point for conquering an island. All other parameters were reset to the initial fixed parameters with the exception of the firing range, which was maintained as $80\%$ of the opponents’ firing range. The initial population of rules in this scenario was the final population rules’ set from the scenario. The performance graph is shown in Figure 10g,h.

The observed performance of these KBs run over one hundred episodes against the enemy team is presented in Figure 11. The blue line in Figure 11a represents the best performance of the enemy team in the three cases against the ally team’s performances in all cases. In Figure 11b, the average scores over 100 episodes with increasing number of boats in a team is illustrated. The blue bar is the average scores of the the enemy teams, while the rest of the bars represent the performance of the ally team with varied team size and point allocation.

Whilst the blue force’s performance degraded, the red force’s reasoner was able to maintain a set of KBs that achieved relatively high performance for most parts of the entire simulation of all training scenarios. Comparing the corresponding episode scores of the final KB of the red force against the blue force rule base used during training revealed that the trained red force was better in at least 82% of all cases, as can be seen in Figure 11. The simulation results demonstrate the feasibility of our approach for multi-agent KB learning and the ability to converge to a stable action selection.

5. Conclusions and Future Work

In this study, we have presented an unmanned multi-surface vehicle training approach for complex control. We used team learning where a central learner controls the agents in an environment modeled as a partially-observable stochastic game. We modeled the control problem as a decomposed FIS and have provided practical ways for constructing and learning the FIS KB. Our proposed framework enables the usage of different FIS decompositions for a complex control problem with minimal or no modification to the reasoner implementation. It also enables the incorporation of agent-induced non-stationarity awareness in the learning process and a resource model for gathering agents’ local observations for off-line learning tasks. Our contributions enable multi-agent control to be performed in domains where no historic data are unavailable for training, but the desired system behavior can be specified as a function of the agents’ performance. The multi-surface vehicle island-conquering case study demonstrates the feasibility and convergence properties of this study. Our future works will focus on improving the performance of our approach using multi-agent deep reinforcement learning techniques with the gathered agents’ observations.