Decentralized Policy Coordination in Mobile Sensing with Consensual Communication

Abstract

In a typical mobile-sensing scenario, multiple autonomous vehicles cooperatively navigate to maximize the spatial–temporal coverage of the environment. However, as each vehicle can only make decentralized navigation decisions based on limited local observations, it is still a critical challenge to coordinate the vehicles for cooperation in an open, dynamic environment. In this paper, we propose a novel framework that incorporates consensual communication in multi-agent reinforcement learning for cooperative mobile sensing. At each step, the vehicles first learn to communicate with each other, and then, based on the received messages from others, navigate. Through communication, the decentralized vehicles can share information to break through the dilemma of local observation. Moreover, we utilize mutual information as a regularizer to promote consensus among the vehicles. The mutual information can enforce positive correlation between the navigation policy and the communication message, and therefore implicitly coordinate the decentralized policies. The convergence of this regularized algorithm can be proved theoretically under certain mild assumptions. In the experiments, we show that our algorithm is scalable and can converge very fast during training phase. It also outperforms other baselines significantly in the execution phase. The results validate that consensual communication plays very important role in coordinating the behaviors of decentralized vehicles.

1. Introduction

Over the past decade, the ubiquitous adoption of mobile vehicles has greatly enhanced the flexibility and convenience of environment sensing. When equipped with sensors, multiple vehicles can autonomously navigate to different locations to collect distributed environmental data. This paradigm, often referred to as mobile sensing, has attracted attention from a variety of disciplines, such as air quality sensing [1], traffic monitoring [2], fire detection [3], etc. For example, in a smart home, multiple devices (e.g., sweeping robots) can cooperate to sense the environment and perform related tasks [4], such as cleaning and tidying.

In a typical mobile sensing scenario, multiple events (e.g., fire, traffic jam, and pollution emission) may occur randomly and dynamically (depicted in Figure 1). Detecting such events in time is crucial for the mobile sensing application. However, since each vehicle can only observe the local environment within a limited radius, one central problem emerging is how to navigate the decentralized vehicles to maximize the spatial–temporal coverage of the events. As the vehicles need to make sequential navigation decisions, reinforcement learning (RL), in particular, multi-agent reinforcement learning (MARL) methods, have become a promising approach. RL methods can be model-free to optimize the navigation policies through exploration and exploitation. They are, therefore, applicable in different scenarios, even when the environmental model is not assumed [5,6].

Figure 1. An illustration of the mobile sensing, where multiple vehicles cooperate to monitor the random events. The blue dash lines represent the moving trajectory of each vehicle. Events with a redder color imply higher intensities.

Despite the progress made in recent years, one critical challenge that has been largely overlooked is the decentralized coordination of the vehicles. As illustrated in Figure 1, the events are mostly distributed at the left and right sides of the map. It could be better if one of the right vehicles moves to the left area for sensing. However, without coordination, the right vehicles may compete to sense nearby events, leading to wasted sensing efforts. One possible direction to tackle this challenge is to use a centralized controller that manages the policies of all vehicles. However, centralized approaches may face the problem of “single point of failure” and low scalability.

To navigate multiple vehicles in an open, dynamic environment, we adopt the MARL as the basic solution. However, in the execution phase, the vehicles may still have uncoordinated behaviors due to the lack of common consensus [7,8,9]. Inspired by the recent advances of learning to communicate [10,11], we can also introduce the communication mechanism in the cooperative navigation. On one hand, the common signal can provide global information from all the vehicles. On the other hand, the other vehicles’ moving actions can also be inferred if there is positive correlation with the signal.

Our Method In this paper, we consider the decentralized management of the mobile vehicles, and introduce a communication-based framework to coordinate the behaviors of the vehicles. At each step before moving, the vehicles should first broadcast communication messages to others to share information. Afterwards, when receiving the communication messages from others, each vehicle can be conditioned on the received messages to take navigation actions. By adopting this communication framework, the vehicles can share information with each other to break through the dilemma of local observation. In particular, the communication message is also learned via reinforcement learning with the aim to maximize the spatial–temporal coverage of the events. This learning to communicate framework is flexible, and can be applicable in different dynamic environments.

One major concern in the communication framework is that the vehicles may simply ignore the communication message and focus only on local observations. To deal with this problem, in this paper, we try to maximize the mutual information between the received messages and the vehicles’ navigation policies. By maximizing this term, the mobile vehicles can correlate their policies with the received messages. Intuitively, a positive correlation implies that other vehicles’ policies can be inferred based on the received message. Therefore, the vehicles can achieve consensus implicitly. Theoretical analysis shows that this regularized algorithm can converge to equilibrium points under certain mild conditions.

In the experiment part, we implement and evaluate the proposed algorithm in a simulation environment built from a real-world data set. We first validate the decentralized algorithm in both the training and execution phases. The results show that the consensual communication framework can successfully coordinate the behaviors of the decentralized vehicles. The mutual information term plays an important role in the coordination. Our method can also adapt to multiple scenarios with different hyper-parameters. In different settings, our algorithm can consistently outperform other baselines. Our work can be widely adopted in different fields, such as smart homes, smart city, agriculture, etc.

1.1. Contributions

Our key contributions are listed as follows:

• We model the mobile sensing problem as a decentralized sequential optimization problem, where the vehicles navigate to maximize the spatial-temporal coverage of the events in the environment.
• A communication framework is proposed for cooperative navigation. In particular, the communication protocol is learned by model-free reinforcement learning methods.
• We explicitly correlate the vehicles moving policies with the communication messages to promote coordination. The regularized algorithm can be proved to converge to equilibrium points under certain mild assumptions.
• Extensive experiments are conducted to show the effectiveness of our approach.

1.2. Organizations

The rest of the paper is organized as follows. We first introduce the related work in Section 2. Next, we formulate the system model and the optimization objective in Section 3. Section 4.1 presents the framework of learning to communicate. We then present how to enforce positive communication in Section 5. Evaluation is given in Section 6. We conclude the paper in Section 7.

2. Related Work

In this section, we first introduce the recent advances in reinforcement learning, which is the main technical solution in this work. Next, we will review the related works of mobile sensing, with a focus on how to navigate the mobile vehicles in the environment to maximize the event coverage.

2.1. Reinforcement Learning

Reinforcement learning (RL) has achieved great success in wide areas, such as Game of Go [12], Atari [13], Starcraft [8], etc. The problem of RL can generally be modeled as a Markov decision process (MDP) $⟨\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma ⟩$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, and $\mathcal{T}:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$ is the transition model for generating the next state. $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is the reward function. $\gamma \in (0,1]$ is a discount factor. At each step t, when an agent observes the state $s^t\in \mathcal{S}$ and executes an action $a^t\in \mathcal{A}$, it will then be transitioned into a new state $s^{t+1}$ and receive an immediate reward $r^t$, with probability $p(s^{t+1},r^t\mid s^t,a^t)\in \mathcal{T}$. Let $R^t$ denote the cumulative return at time t. In an infinite horizon MDP, the cumulative return can be represented as

$$R^t=\underset{h\to \infty }{lim}\frac{1}{h}\sum _{t^{\prime }=t}^h\mathbb{E}\left[r^{t^{\prime }}\right]$$

(1)

The goal of reinforcement learning is to find the optimal policy ${\mu }^{*}$ to maximize the return: ${\mu }^{*}=arg{max}_{\mu }{\mathbb{E}}_{\mu }\left[R^0\right]$, where policy $\mu (a_t\mid s_t)$ is a function which maps the state $s_t$ to a distribution of actions $a_t$. MDP has the property of the Bellman equality:

$$Q(s^t,a^t)=r^t+\gamma \sum _{s^{t+1}}p(s^{t+1}\mid s^t,a^t)v\left(s^{t+1}\right)$$

(2)

where $Q(s,a)=\mathbb{E}[R^t\mid s^t=s,a^t=a]$ is the state-action value function and $v\left(s\right)=\mathbb{E}[R^t\mid s^t=s]$ is the value function of state s.

The process of RL can be generally divided into training and execution phases. In the training phase, the RL agent uses exploration and exploitation in the environment to optimize the policy. While in the execution phase, the agent will fix the policy parameters in the environment. In this paper, as the vehicles need to move in a continuous space, we focus on DDPG [14,15], which can generate continuous actions. In DDPG, there is a critic function to evaluate the state-action value by following a deterministic policy $\mu$ as $Q^{\mu }(s^t,a^t)$, and an actor function which maps the state $s^t$ to a deterministic action, $a^t=\mu \left(s^t\right)$.

Recently, multi-agent reinforcement learning (MARL) has also been a hot research topic. MARL models the environment as a decentralized partially observable Markov decision process (Dec-POMDP) [8,9] as a tuple $⟨\mathcal{S},\mathcal{T},\mathcal{A},\mathcal{R},\mathcal{O},\mathcal{I}⟩$, where $\mathcal{O}$ is the set of local observations and $\mathcal{I}$ is the set of agents. The agents that make decisions are based on the observations. Let $o_i^t\in \mathcal{O}\subseteq \mathcal{S}$ be the local observation of agent i at step t. Each agent i can choose an action $a_i^t\in \mathcal{A}$, forming a joint action ${\mathbf{a}}^t\in {\mathcal{A}}^n$, and transition to the next state $s^{t+1}\in \mathcal{S}$ according to the function $p(s^{t+1},r^t\mid s^t,{\mathbf{a}}^t)\in \mathcal{T}$, where the reward function $r^t\in \mathcal{R}$ is shared by all the agents.

To optimize the policies of the agents in MARL, previous works, such as COMA [8], MADDPG [9], QMIX [7], etc., mainly adopted the “centralized training, decentralized execution” (CTDE) mechanism: during training, global state information can be used to train the policy network; and during execution, the agents can only condition on local observations. In the execution phase, the agents could still change their policies dynamically, leading to incoordination of the decentralized policies. However, we address that such a CTDE mechanism may not be applicable in decentralized environments where the agents can only be trained separately. Recent works are considering methods of learning to communicate [16,17,18,19], where the communication policy is learned via RL. We will also adopt this mechanism in our work. In comparison to previous works [16,17,19] that mostly use lazy communication, we propose to enforce positive communication so that the messages can be utilized more efficiently. Moreover, most of previous works only used ungrounded, cheap talk for communication [10]. We address that such cheap talk communication may not be effective in coordination.

2.2. Mobile Sensing

Mobile sensing has been extensively studied with the emergency of autonomous vehicles. One of the main problem is maximizing the coverage of events in the environment. Earlier works mostly assumed that the environment model is a prior and proposed combinatorial optimization method. For example, Karaliopoulos et al. [20] modeled the problem as a cover problem and proposed the approximation ratio algorithm. Hu et al. [21] also proposed mobile sensing methods with spatial–temporal awareness. The paper adopted a combinatorial pinning zero-determinant (ZD) strategy to find a cost-efficient mobile sensing strategy. In comparison, our work addresses the dynamics of the environment, and the coordinated policies of different mobile users are learned via repeated interactions.

As the users make independent decisions, decentralized algorithms based on game theory were also considered. Rahili et al. [22] designed a rule-based communication protocol in which agents can communicate with local neighbors and use their local information make decisions. Esch et al. [23] depicted a distributed algorithm where the agents can communicate with one another wirelessly within a fixed communication radius. Li et al. [24] modeled the mobile crowdsourcing as a Stackelberge game, and proposed a three-party evolutionary game model for task allocation. However, most previous methods are hard to generalize to unseen scenarios. In an open environment, it is critical for the agents to adapt to dynamic environment events. Data privacy is also important in mobile sensing and has been a hot research topic very recently [25,26,27,28,29]. In comparison, we focus more on the navigation of the mobile vehicles instead of the data-collecting process.

When the environment model is unknown, machine learning approaches attract attention [30,31]. In particular, as the environment is often dynamic [32,33], online learning or RL-based algorithms are widely considered, which are sequential and model-free. An et al. [34] adopted the multi-armed bandits method to select users to improve service quality. However, bandit algorithms neglect the sequential behavior of agents and may not be feasible for mobile sensing problems. As RL uses deep learning to extract the representation of the environment for exploration and exploitation, it can be naturally applicable in the dynamic environment. For example, Zhang et al. [35] adopted RL for a coarse-to-fine deep scheme to address the aspect ratio variation in UAV tracking. Liu et al. [36,37] used deep RL for high quality data collection. The main idea is to employ multiple mobile vehicles to schedule their paths independently to maximize the coverage of distributed POIs (point of interests). Zeng et al. [38] divided the problem into four sub-optimal problems, and used an iterative algorithm solve the optimal problem. Liu et al. [5] proposed a multi-UAV mobile sensing framework based on multi-agent reinforcement learning (MARL), and utilized “centralized training decentralized execution” (CTDE) for cooperation. Wei et al. [6] considered the multi-robot informative path planning problem and proposed independent learning through credit assignment for cooperative sensing. Samir et al. [39] leveraged unmanned aerial vehicles (UAVs) for mobile sensing and proposed an RL approach to maximize the sensing coverage. A major challenge in these works is to coordinate the policies of different mobile vehicles for cooperation. While most previous works implicitly learn the cooperation policies for each agent, in our work, we addressed that coordination is crucial and explicitly proposed policy coordination methods based on consensual communication.

3. System Model

In this paper, we consider a mobile sensing problem where a set of mobile vehicles $\mathcal{I}=\{1,2,\dots ,N\}$ cooperate to maximize the spatial–temporal sensing coverage of the events in the environment. Suppose the time horizon is divided into infinite discrete intervals as $\{0,1,2,\dots ,\infty \}$. At each interval t, each vehicle $i\in \mathcal{I}$ at position $(x_i^t,y_i^t)$ need to decide the moving action $a_i^t$, which can be represented as a tuple of speed ${\nu }_i^t\in [0,S_{max}]$ and angle ${\varphi }_i^t\in [0,2\pi )$, i.e., $a_i^t=({\nu }_i^t,{\varphi }_i^t)$. After moving, the new position will be $(x_i^t+{\nu }_i^t\sin {\varphi }_i^t,y_i^t+{\nu }_i^t\cos {\varphi }_i^t)$. Meanwhile, vehicle i is associated with a battery capacity $b_i^t\in [0,b_{max}]$. The battery has a consumption rate ${\Delta }_i^t$ that is linear with the vehicle speed, i.e., ${\Delta }_i^t=\beta {\nu }_i^t+{\Delta }_0$, where $\beta$ is a coefficient and ${\Delta }_0$ is a constant intrinsic battery consumption. The battery capacity will be updated as $b_i^{t+1}=b_i^t-{\Delta }_i^t$ each time. To avoid running out of power, the vehicles should regularly move to the charging station, in which the battery will be recharged for a fixed number of units $b_0$ at each interval.

In the environment, random events may happen at different positions with time-varying intensities. Let $\mathcal{E}$ be the set of events. We use ${\tau }_e^t$, $e\in \mathcal{E}$ to represent the event intensity of e at step t. The event e at position $(x_e^t,y_e^t)$ is sensed/covered by vehicle i if it is within a limited radius of i. Let ${𝟙}_{ie}^t$ be an indicator function to represent if the event is covered by vehicle i:

$${𝟙}_{ie}^t=\left\{\begin{array}{cc}\hfill 1,& \mathrm{if}\sqrt{{(x_i^t-x_e^t)}^2+{(y_i^t-y_e^t)}^2}\le l_i,\hfill \\ \hfill 0,& \mathrm{o}.\mathrm{w}.\hfill \end{array}\right.$$

(3)

where $l_i$ is the sensing radius of vehicle i. The benefit will be ${\tau }_e^t$ if the event e is covered by at least one of the mobile vehicles. Note that if multiple vehicles cover the same event e simultaneously, the benefit is still ${\tau }_e^t$. Therefore, the vehicles should cooperate to avoid repeatedly sensing the same event. We use ${𝟙}_e^t$ as an indicator function that the event e is covered by at least one vehicle at interval t, i.e., ${𝟙}_e^t=max\{{𝟙}_{1e}^t,{𝟙}_{2e}^t,\dots ,{𝟙}_{Ne}^t\}$. The problem can then be formulated as finding the joint moving policies for the vehicles, so that the cumulative sensing coverage of the events is maximized:

$$\begin{array}{cc}\hfill & max\sum _{t=0}^{\infty }\sum _{e\in \mathcal{E}}{\tau }_e^t{𝟙}_e^t\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.b_i^t\ge 0,\forall i\in \mathcal{I},\forall t\in \{0,1,2,\dots \infty \}.\hfill \end{array}$$

(4)

The inequality constraint in the objective means that the mobile vehicles could no longer move or sense when running out of battery. According to the objective, the vehicles need to make sequential navigation decisions to cover the dynamic events. However, as the vehicles make decentralized decisions, it could be difficult for the vehicles to acknowledge others’ observations or intentions. This brings the dilemma of local observation and will be the main focus of this paper. Table 1 summarizes the key parameters in this paper.

Table 1. Key parameter table of system model.

Notation	Definition
$\mathcal{I}$	the set of mobile vehicles: $\mathcal{I}=\{1,2,\dots ,N\}$
$\mathcal{E}$	the set of events
t	the time step $t\in \{0,1,\dots ,\infty \}$
$(x_i^t,y_i^t)$	the position of vehicle i at step t
$({\nu }_i^t,{\varphi }_i^t)$	moving speed and angle of vehicle i at step t
$b_i^t$	battery capacity of vehicle i at step t
${\Delta }_i^t$	battery consumption rate of vehicle i at step t
$b_0$	battery charging rate at the charging station
$l_i$	the sensing radius of vehicle i
${\tau }_e^t$	the event intensity of event $e\in \mathcal{E}$ at step t
c	the penalty for running out of battery
$r_i^t$	the reward of vehicle i at step t
${\mu }_i(·)$	the moving policy function of vehicle i
${\mu }_i^m(·)$	the communication policy function of vehicle i
$Q_i(·)$	the action–value function of vehicle i
$v_i(·)$	the state value function of vehicle i
$q(·)$	proxy for the posterior function
$I(·)$	mutual information function
$\rho$	the weight of the MI reward

4. Learning to Communicate

To break through the dilemma of local observation, in this section, we first formulate the problem as a Markov game. Then we formally introduce the communication framework, where the vehicles can share information with each other. Finally, we will show how to optimize the moving policies of each vehicle under this framework.

4.1. Mobile Sensing as a Markov Game

According to the system model, we can formulate the mobile sensing problem as a Dec-POMDP with tuples of $⟨\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{I}⟩$, where the set of agents $\mathcal{I}$ represent the mobile vehicles. Now we give the definitions of other elements as follows:

• State: In the mobile sensing problem, at each interval t, the system state $s^t\in \mathcal{S}$ includes the global information of the environment.
• Observation: In the environment, each vehicle i can only partially observe the state. The observation $o_i^t\in \mathcal{O}$ is the subset of the environment state: $o_i^t\subseteq s^t$. We assume that each vehicle can observe the environment information within the sensing radius $l_i$, including its own position, last moving action, remaining battery capacity and sensed events.
• Action: The action of the mobile vehicle i is a continuous tuple $a_i=({\nu }_i^t,{\varphi }_i^t)\in {\mathcal{A}}_i$, where ${\nu }_i^t$ is the speed, and ${\varphi }_i^t$ represents the moving angle. At each interval, all the vehicles will take the moving action to form a joint action ${\mathbf{a}}^t=(a_1^t,a_2^t,\dots ,a_N^t)$.
• Transition: Given the joint actions of the vehicles, the environment will transit to the next state $s^{t+1}$ according to the transition function:

  $$p(s^{t+1}\mid s^t,{\mathbf{a}}^t):\mathcal{S}\times {\mathcal{A}}_1\times \dots \times {\mathcal{A}}_N\to \mathcal{S}$$

  (5)
  Note that this function is not known to be used, and can only be inferred through repeated interaction with the environment.
• Reward: As the mobile vehicles cooperate to maximize the spatial–temporal coverage of the environment, we define a global reward as the sensed events intensities:

  $$r^t=\sum _{e\in \mathcal{E}}{\tau }_e^t{𝟙}_e^t$$

  (6)
  However, for each vehicle, it is intricate to infer its contribution to the global reward. Therefore, we decompose the reward function and define the individual reward for each vehicle i as

  $$r_i^t=\sum _{e\in \mathcal{E},{𝟙}_{ie}^t=1}\frac{{\tau }_e^t{𝟙}_{ie}^t}{{\sum }_{k\in \mathcal{I}}{𝟙}_{ke}^t}$$

  (7)
  The reward function indicates that the reward of sensing event e is averaged by the number of vehicles that cover e at this step. It is obvious to see that $r^t={\sum }_{i\in \mathcal{I}}{r}_i^t$. To take the battery capacity into account, we relax the constraint in Equation (4) with an additional term c when the vehicle runs out of battery power. The vehicles will receive this penalty when the capacity is below zero, i.e., $c\left(b_i^t\right)=c$ if $b_i^t<0$; otherwise, $c\left(b_i^t\right)=0$. The value of c balances the preference between sensing a reward and penalty of battery loss. The relaxed version of the reward can be formulated as

  $$r_i^t=\sum _{e\in \mathcal{E},{𝟙}_{ie}^t=1}\frac{{\tau }_e^t{𝟙}_{ie}^t}{{\sum }_{k\in \mathcal{I}}{𝟙}_{ke}^t}-c\left(b_i^t\right)$$

  (8)

4.2. The Communication Framework

As the vehicles only have limited observation, we introduce a communication framework to share information among the vehicles. Figure 2 presents an illustration of the communication procedure. We now separately describe how to broadcast and receive the messages.

Figure 2. An illustration of the procedure. With local observation, the vehicles first exchange their messages by broadcasting. Afterwards, they make the moving decision based on the received message and local observation. This figure shows the communication process of the ego vehicle i in a single step t: The vehicle first obtains the observation $o_i^t$ in the environment. The vehicle then broadcasts a message $m_i^t$ to other vehicles based on the observation. The messages $(m_1^t,\dots ,m_N^t)$ from other vehicles will also be aggregated as part of the observation. Vehicle i will finally make the moving decision $a_i^t$ based on the environment observation and the aggregated message.

Communication Broadcasting As presented in Figure 2, at each step t before moving, each vehicle i first broadcasts a message $m_i^t$ to other vehicles. When broadcasting the message, an intuitive idea is to send the observation $o_i^t$ and the intended action $a_i^t$ to other vehicles. However, this is not possible since the vehicle will also be conditioned on the received messages from others to take action $a_i^t$. Moreover, the dimensions of the observation may be large with high overhead. Instead, we introduce the mechanism of learning to communicate. Suppose vehicle i uses a communication policy network ${\mu }_i^m\left(o_i^t\right)$ parameterized by ${\theta }_i^m$ to output the message content $m_i$, which can be a fixed-size continuous vector. In particular, the communication policy network can be optimized via the RL-based algorithm, where the goal is the long-term cumulative sensing coverage of the events. By learning to communicate, the vehicles can encode the observations and intentions into a compact embedding, which significantly reduces the transmission cost. Moreover, it can be flexible to deal with different scenarios and environments. More details on how to optimize the communication policy network will be introduced in Section 4.3.

Communication Receiving After broadcasting, each vehicle can also receive the messages from other vehicles: ${\mathbf{m}}^t=(m_1^t,m_2^t,\dots ,m_N^t)$. The messages can be aggregated with different operators, such as mean, max, or neural networks, such as recurrent neural networks (RNN). The aggregated message can be represented as $m_g=\mathrm{AGG}\left({\mathbf{m}}^t\right)$, where AGG is the aggregator of the received messages. Suppose the moving policy of vehicle is represented as ${\mu }_i(·)$. It can be formulated as conditioning on the local observation and received messages for moving: $a_i^t={\mu }_i^t(o_i^t,m_g)$.

4.3. Policy Optimization

With the communication framework, we can now optimize the moving policy networks ${\mu }_i(·)$ and communication policy networks ${\mu }_i^m(·)$ for each vehicle $i\in \mathcal{I}$. As the moving action of each vehicle is a continuous vector, we use DDPG for policy optimization. Let $Q_i(·)$ be the action value function (critic) parameterized by ${\theta }_i^Q$. (We temporarily abbreviate the time indicator t. The sign ${}^{-}$ indicates $t-1$ and ${}^{\prime }$ indicates $t+1$.) The policy functions ${\mu }_i^m(·)$, ${\mu }_i(·)$ and the critic function $Q_i(·)$ can all be implemented with neural networks. The parameters ${\theta }_i$ of the moving policy network ${\mu }_i(·)$ can be updated according to the deterministic policy gradient theorem [14]:

$${\nabla }_{{\theta }_i}J\left({\mu }_i\right)={\mathbb{E}}_{o_i,\mathbf{m},a_i\sim \mathcal{D}}\left[{\nabla }_{{\theta }_i}{\mu }_i(o_i,m_g){\nabla }_{a_i}{Q}_i(o_i,a_i,m_g){\mid }_{a_i={\mu }_i(o_i,m_g)}\right]$$

(9)

where $J(·)$ is the return of the policy and $\mathcal{D}$ is the set of historical data samples. Similarly, we can also update the parameters of the communication policy network ${\mu }_i^m$ as

$${\nabla }_{{\theta }_i^m}J\left({\mu }_i^m\right)={\mathbb{E}}_{o_i,\mathbf{m},a_i\sim \mathcal{D}}\left[{\nabla }_{{\theta }_i^m}{\mu }_i^m\left(o_i\right){\nabla }_{m_i}{Q}_i(o_i,a_i,m_g){\mid }_{m_i={\mu }_i^m\left(o_i\right)}\right]$$

(10)

where ${\theta }_i^m$ represents the parameters of the communication policy network. The action value network can be updated by minimizing the temporal difference (TD) error:

$$\begin{array}{cc}\hfill \mathcal{L}\left({\theta }_i^Q\right)=& {\mathbb{E}}_{o_i,m_g,a_i,r_i,o_i^{\prime },m_g^{\prime }}[Q_i(o_i,a_i,m_g)-\hfill \\ \hfill & {(r_i+\gamma Q_i(o_i^{\prime },a_i^{\prime },m_g^{\prime }){\mid }_{m_i^{\prime }={\mu }_i^m\left(o_i^{\prime }\right),a_i^{\prime }={\mu }_i(o_i^{\prime },m_g^{\prime })}]}^2\hfill \end{array}$$

(11)

According to the above formulations, we can update the parameters of the policy networks and action value networks simultaneously. Compared to the CTDE framework, which requires centralized training, in our framework, the networks can be optimized independently based on the local observation and communication messages. Therefore, this framework can be applicable in decentralized training scenarios.

5. Consensual Communication

By learning to communicate, the mobile vehicles can share local information with each other. However, previous works have shown that selfish agents do not learn to use this type of ungrounded, cheap talk communication channel effectively [11]. In this section, we first try to enforce the mobile vehicles to have consensual communication, i.e., the communication will indeed influence the vehicles’ behaviors. Next, we show that the algorithm can converge under the communication framework.

5.1. Mutual Information for Consensual Communication

To enforce positive communication, we maximize the mutual information between the moving policy ${\mu }_i$ and the aggregated message from i’s neighbors: $m_g$. Intuitively, by maximizing the mutual information, the vehicle can correlate its moving policy with the messages from neighbors. This can also be regarded as reducing the uncertainty of vehicles’ moving policy after receiving the messages. Formally, we augment the reward function as follows:

$${\widehat{r}}_i=(1-\rho )r_i+\rho I({\mu }_i;m_g)$$

(12)

where $\rho \in [0,1]$ is a hyper-parameter that controls the importance of the mutual information term $I({\mu }_i;m_g)$. The mutual information item can be expressed in terms of entropy and conditional entropy:

$$\begin{array}{cc}\hfill I({\mu }_i;m_g)& =\mathcal{H}\left(m_g\right)-\mathcal{H}(m_g\mid {\mu }_i)\hfill \\ \hfill & =\mathcal{H}\left({\mu }_i\right)-\mathcal{H}({\mu }_i\mid m_g)\hfill \end{array}$$

(13)

where $\mathcal{H}(·)$ is the entropy function. The mutual information will become zero if the communication message does not influence the moving policy. In this case, $\mathcal{H}\left(m_g\right)$ equals $\mathcal{H}(m_g\mid {\mu }_i)$. Maximizing the mutual information indicates that we enforce all the vehicles to correlate their policies with the message. Thus, the vehicles can infer other neighbors’ behaviors by acknowledging the broadcast message, which implicitly promotes coordination among the vehicles. However, directly maximizing the MI is intractable. We instead introduce the variational distribution $q(m_g\mid {\mu }_i)$ as a proxy for the posterior over ${\mu }_i$. Learning a neural network to predict the messages based on the policy ${\mu }_i$ provides a lower bound on MI:

$$\begin{array}{cc}\hfill & I({\mu }_i;m_g)=\mathcal{H}\left(m_g\right)-\mathcal{H}(m_g\mid {\mu }_i)\hfill \\ \hfill & =\mathcal{H}\left(m_g\right)+{\mathbb{E}}_{m_g,{\mu }_i}[logq(m_g\mid {\mu }_i)]\hfill \\ \hfill & -{\mathbb{E}}_{m_g}\left[D_{KL}(p(m_g\mid {\mu }_i)\mid \mid q(m_g\mid {\mu }_i))\right]\hfill \\ \hfill & \ge \mathcal{H}\left(m_g\right)+{\mathbb{E}}_{{\mu }_i,m_g}[log\left(q(m_g\mid {\mu }_i)\right]\hfill \end{array}$$

(14)

where $D_{KL}$ is the KL divergence between two probabilities. The establishment of inequality is because the KL-divergence distance is non-negative. In practice, as the policy ${\mu }_i$ is a network, we use historical observation–action trajectories $tra{j}_i$ to represent the policy.

The network structure of our framework is presented in Figure 3. For each vehicle, there are four neural networks associated, including one critic network, two actor networks, and an additional variation network which is used for policy coordination. The output of the critic network can be used to update the actor networks during training. For the variation network, even though the gradient cannot be backpropagated to update the actor–critic networks, the augmented reward function can guide the mobile vehicles to generate coordinated behaviors. In the network structures, FC means fully connected, and GRU is gated recurrent unit. GRU is used to extract information from the sequential observations. More details of the network parameters will be introduced in the experiment part. As the network parameters for each vehicle can be optimized in a decentralized way, this framework can be scalable to a large number of mobile vehicles.

Figure 3. The network structure of policy coordination with consensual communication. FC represents fully connected layer, and GRU represents gated recurrent unit layer. AGG is the aggregator operator for the received messages. According to the network structure, each mobile vehicle needs to maintain 4 networks: moving actor network (the green part), critic network (the blue part), communication actor network (the orange part) and the variation network (the purple part).

Algorithm Now we formally present the algorithm in Algorithm 1 for an ego vehicle i. In this algorithm, we first initialize the parameters of the networks for the ego agent i. At each step, we generate the broadcast message $m_i$ based on the current observation $o_i$. The agent will then receive and aggregate messages from others and execute actions $a_i$. The tuples will be stored into the replay buffer $\mathcal{D}$. During training, we sample a mini-batch of tuples from the buffer and perform gradient back propagation to update the critic network and actor networks. Finally, the variation network is also trained by maximizing the mutual information.   

Algorithm 1: Policy optimization for ego vehicle i

Complexity According to the above algorithm, we give a formal analysis of the time complexity of the training phase for each ego vehicle. At each step of training, the vehicle need to sample K tuples and update the networks. The update of the networks takes $O\left(1\right)$ complexity for gradient descent. Suppose the convergence takes C steps. The time complexity of the algorithm will be $O\left(KC\right)$. In the experiments, we will show that when choosing the batch size $K=256$, the algorithm takes about $C=100,000$ steps to converge. In fact, this algorithm can be computed on a cuda device very quickly. During execution, the policy can be computed in $O\left(1\right)$ time.

5.2. Convergence Analysis

Given the above algorithm, in this section, we formally show that the value functions $Q_i,i\in \mathcal{I}$ can converge to an equilibrium point under certain assumptions:

Assumption 1.

Every state $s\in \mathcal{S}$ and action $a_i\in \mathcal{A}$, for $i\in \mathcal{I}$, is visited infinitely often.

Assumption 2.

The critic learning rates ${\alpha }_t$ for optimizing Equation (11) satisfy ${\sum }_{{\alpha }_t=0}^{\infty }{\alpha }_t(s,\mathbf{a})=\infty$, and ${\sum }_{{\alpha }_t=0}^{\infty }{\left[{\alpha }_t(s,\mathbf{a})\right]}^2<\infty$ holds uniformly with probability 1.

Assumption 3.

The aggregated message $m_g$ is a representation of the global state information s and action $\mathbf{a}$.

Assumption 4.

The stage game at each interval t has a global optimal point. The global points are selected by our algorithm to update the critic functions with probability 1.

Assumptions 1 and 2 are weak ones that are easy to meet. Assumption 3 is met if (1) the communication message $m_g$ can encode the entire state without information loss; (2) every other vehicle’s policy can be inferred based on $m_g$. The two conditions are reasonable according to our communication-based framework. Assumption 4 is a strong assumption. It may not be easily met. However, our empirical experiments demonstrate that this assumption is satisfied mostly since the algorithm can converge in different scenarios. The convergence result mainly originates from the following lemma [40]:

Lemma 1. (Szepesvari and Littman (1999), Corollary 5)

Assume ${\rho }_t$ satisfies Assumption 2 and the mapping $P^t:\mathbb{Q}\to \mathbb{Q}$ has the following condition: there exists a number $0<\gamma <1$ and a sequence ${\lambda }_t\ge 0$ converging to zero with probability 1 such that ${∥P^{t}Q-P^t{Q}^{*}∥}_{\infty }\le \gamma {∥Q-Q^{*}∥}_{\infty }+{\lambda }_t$ for all $Q\in \mathbb{Q}$ and $Q^{*}=\mathbb{E}\left[P^t{Q}^{*}\right]$, then the iteration defined by

$$Q^{\prime }=(1-{\rho }_t)Q+{\rho }_t\left[P^{t}Q\right]$$

(15)

converges to $Q^{*}$ with probability 1.

According to Assumption 3, the messages $m_g$ is a compact representation of the global state s and actions $\mathbf{a}$. Therefore, there is $Q_i(s,\mathbf{a})=Q_i(o_i,a_i,m_g)$. Define the transition function $P^t$ and the convergence point $Q^{*}$ as

Definition 1.

Let $P^t:\mathbb{Q}\to \mathbb{Q}$ be a mapping on the complete metric space $\mathbb{Q}\to \mathbb{Q}$, $P^{t}Q=(P^t{Q}_1,P^t{Q}_2,\dots ,P^t{Q}_N)$, where

$$\begin{array}{c}\hfill P^t{Q}_i(s,\mathbf{a})=r_i+\gamma Q_i(o_i^{\prime },{\mu }_i(o_i^{\prime },{\mu }^m\left(s^{\prime }\right)),{\mu }^m\left(s^{\prime }\right)))\end{array}$$

(16)

for $i\in \mathcal{I}$, where ${\mu }^m(·)=({\mu }_1^m,\dots ,{\mu }_N^m)$.

Definition 2.

$Q^{*}$ is the convergence point if it satisfies

$$\begin{array}{cc}\hfill Q_i^{*}(o_i,& o_i,m_g)=r_i+\gamma \sum _{s^{\prime }\in S}p(s^{\prime }\mid s,\mathbf{a})\hfill \\ \hfill & Q_i^{*}(o_i^{\prime },{\mu }_i(o_i^{\prime },{\mu }^m\left(s^{\prime }\right),{\mu }^m\left(s^{\prime }\right)))\hfill \end{array}$$

(17)

With the above definitions, we show that the transition function $P^t$ is a “contraction mapping” with the fixed point at $Q^{*}$.

Lemma 2.

The convergence point is a fixed point: $\mathbb{E}\left[P^t{Q}^{*}\right]=Q^{*}$.

Proof. 

Since $Q^{*}$ is a convergence point in the game, the vehicles will still follow the current policy ${\mu }^{*}$. According to the Bellman equation (Equation (1)), there is

$$\begin{array}{cc}\hfill & Q_i^{*}(o_i,a_i,m_g)\hfill \\ \hfill & =r_i+\gamma \sum _{s^{\prime }\in S}p(s^{\prime }\mid s,\mathbf{a})Q_i^{*}(o_i^{\prime },{\mu }^m\left(s^{\prime }\right),{\mu }_i(o_i^{\prime },{\mu }^m\left(s^{\prime }\right)))\hfill \\ \hfill & =\sum _{s^{\prime }\in S}p(s^{\prime }\mid s,\mathbf{a})(\gamma Q_i^{*}(o_i^{\prime },{\mu }^m\left(s^{\prime }\right),{\mu }_i(o_i^{\prime },{\mu }^m\left(s^{\prime }\right)))+r_i)\hfill \\ \hfill & =\mathbb{E}\left[P^t{Q}_i^{*}(o_i,x_i,m_g)\right]\hfill \end{array}$$

(18)

where the forth line takes the expectation from $p(s^{\prime }\mid s,\mathbf{a})$ and the Bellman equation. □

Next, we show that $P^t$ is a “contraction mapping”. According to Assumption 3, there is ${\mu }_i(o_i,m_g)={\mu }_i\left(s\right)$. Similar to [41], the max-norm of the mapping operator can be defined as

$$\begin{array}{cc}\hfill & {∥Q-\widehat{Q}∥}_{\infty }\equiv \underset{i}{max}\mid Q_i-{\widehat{Q}}_i\mid \hfill \\ \hfill & \equiv \underset{i,s}{max}\mid \gamma Q_i\left(s\right)-\gamma {\widehat{Q}}_i\left(s\right)\mid \hfill \\ \hfill & \equiv \underset{i,s,\mathbf{a}}{max}\gamma \mid Q_i(s,\mathbf{a})-{\widehat{Q}}_i(s,\mathbf{a})\mid \hfill \end{array}$$

Lemma 3.

${∥P^{t}Q-P^t\widehat{Q}∥}_{\infty }\le \gamma ∥Q-\widehat{Q}∥\infty ,\forall Q,\widehat{Q}\in \mathbb{Q}$.

Proof. 

According to the transition function $P^t$, there is

$$\begin{array}{cc}\hfill & {∥P^{t}Q-P^t\widehat{Q}∥}_{\infty }\hfill \\ \hfill & =\underset{i,s}{max}\gamma \mid Q_i(s,\mathbf{a})-{\widehat{Q}}_i(s,\mathbf{a})\mid \hfill \\ \hfill & =\underset{i,s}{max}\gamma \mid \prod _{j=1}^N{\mu }_j^a\left(s\right)Q_i\left(s\right)-\prod _{j=1}^N{\widehat{\mu }}_j^a\left(s\right){\widehat{Q}}_i\left(s\right)\mid \hfill \\ \hfill & \le \underset{i,s}{max}\gamma \mid \prod _{j=1}^N{\mu }_j^a\left(s\right)[Q_i\left(s\right)-{\widehat{Q}}_i\left(s\right)]\mid \hfill \\ \hfill & \le \underset{i,s}{max}\gamma \mid Q_i\left(s\right)-{\widehat{Q}}_i\left(s\right)\mid \hfill \\ \hfill & =\gamma {∥Q_i-{\widehat{Q}}_i∥}_{\infty }\hfill \end{array}$$

(19)

The fourth line of equality comes from our Assumption 3 that the message $m_g$ is a compact representation of s. The fifth line of inequality is from Assumption 4 that the vehicles play the best response with respect to the broadcast message $m_g$. □

Summarizing the above two lemmas, it is proved that $P^t$ is a “contraction mapping” with the fixed point at $Q^{*}$. Thus, according to Lemma 1, there is the following.

Theorem 1.

Under Assumption 1-4, the sequence $(Q_1,\dots ,Q_N)$ updated by Algorithm 1 converges a fixed value $Q^{*}=(Q_1^{*},\dots ,Q_N^{*})$.

6. Evaluation

In this section, we first introduce the experiment setup, including the description of the environment, the baselines, and the model parameters. Next, we will show the performance of our algorithm with comparisons with other baselines. In particular, the results validate the importance of the consensual communication framework.

6.1. Experiment Setup

The Environment To validate the effectiveness of our algorithm, we manually construct a mobile sensing simulation environment based on real historical data set. The data set is collected from a road network from Google Map (Google Map: https://www.google.com/maps, accessed on 10 March 2022), which has the traffic volume at the road network across different hours (the data sets generated during the current study are available in the following https://www.dropbox.com/s/42cl68ns2fud5yk/GOOGLETraffic.zip?dl=0, accessed on 10 March 2022). We focus on an area of 10 km × 10 km square area centered at ($48.16,16.33$). In this map, we uniformly sample $40\times 40$ points as the locations of events. For each position, the traffic volumes are extracted as the event intensities. An illustration of the event map at a given time is presented in Figure 4. The dots represent the events happening at different locations. The events have 5 levels of intensities as $0,1,2,3,4$. We also add random uniform noise $(0,1)$ to the event intensities for randomness. Dots with darker colors have higher event intensities. In this map, there assumed to be 5 charging stations at locations of $(8,32)$, $(32,8)$, $(8,8)$, $(32,32)$ and $(20,20)$.

Figure 4. A snapshot of the event intensities in the target.

By default, we suppose the max speed of each vehicle is $S_{max}=2$, and the sensing radius is $l_i=2$. Therefore, each vehicle can cover multiple events at the same time. The battery capacity of each vehicle is $b_{max}=40$. During moving, the coefficient of battery consumption is $\beta =1$, ${\Delta }_0=1$. The vehicles can regularly navigate to the charging station, where they will be recharged $b_0=20$ units of battery at each time step. The penalty of running out of power is set as $c=40$. We will also try other values to validate the effectiveness of our algorithm. A small size of the replay buffer is set as ${10}^5$, since the vehicles policies may be dynamic.

Baselines We name our algorithm as ConComm (CONsensual COMMunication), and compare with the following baselines which can generate continuous actions.

• ConComm (no MI): In this algorithm, we implement the ConComm algorithm without the mutual information item. This comparison is to demonstrate the effectiveness of the mutual information item.
• DDPG [15]: In this algorithm, each mobile vehicle independently learns a policy to schedule the sensing path. The main drawback is that the multi-agent environment does not follow the Markov property, which may lead to the failure of this algorithm.
• MADDPG [9]: MADDPG uses the CTDE framework, where there is a global critic function that has access to the historical samples from all mobile vehicles. However, the policies of the vehicles are not coordinated explicitly during execution.
• MAPPO [42]: This algorithm is a multi-agent version of PPO. It has achieved state-of-the-art performance in many scenarios.

Model Parameters For different algorithms, we use similar critic network structures with an FC layer with 64 hidden units. The FC layer is followed by a ReLU activation layer for non-linear activation. The output is connected with a GRU layer with 64 hidden units and then fed into another FC layer to output the critic value. The actor networks have a similar structure. The only difference is the output of the networks. The communication actor network outputs a message with size 6 followed by a sigmoid layer to restrict the message in the range $(0,1)$. The messages are aggregated with a MEAN operator, i.e., $m_g=\frac{1}{N}{\sum }_{i\in \mathcal{I}}{m}_i$. The moving actor network outputs a vector of size 2, followed by a sigmoid layer to restrict the range of the speed and angle. Maximum speed and angle are used to project the outputs into new ranges. For the variation network, the input is the embedding after the FC layer. It is then fed into two FC layers with 64 hidden units to predict the aggregate message. Mean squared error is used as the loss function for the variation network. The weight of the MI item $\rho$ is set as $0.5$ so that different parts of the reward function are comparable.

6.2. Performance Analysis

Convergence of Training In the first experiment, we assume there are $N=12$ mobile vehicles, and examine the convergence of the algorithms during training in Figure 5. The average step reward is evaluated every 200 steps. We assume different vehicles share the same network parameters. Nonetheless, the vehicles can still behave differently with local observations. The y-axis represents the average step reward for each vehicle $r=\frac{1}{N}{\sum }_{i\in \mathcal{I}}{r}_i$. Each of the RL-based algorithms is trained 3 times. The shaded area represents one standard deviation. As presented, our proposed ConComm achieves the highest performance at most of the time. The average step reward of ConComm can converge to around 17 after about only $50,000$ steps. The performance then stabilizes around at this level. Moreover, the variance of ConComm is also more stable compared to others. This is because the vehicles are more likely to have coordinated behaviors. ConComm (no MI) is the algorithm without explicit policy coordination. The result can be relatively high due to the communication among the mobile vehicles. However, the performance is worse than ConComm, which validates the effectiveness of the MI item. DDPG has the worst performance among the algorithms. This is mainly due to the fact that the vehicles make decisions independently. Therefore, there may be lots of repeated sensing efforts among the vehicles. MADDPG and MAPPO have similar performances that are slightly better than DDPG. The main reason is that they adopt the “centralized training, decentralized execution” mechanism. However, in the execution phase, there may still be uncoordinated behaviors with unseen environment states. Different vehicles may not achieve consensus before making decisions. The above comparisons show that communication plays an important role in coordinating the vehicles’ behaviors.

Figure 5. The performance of different algorithms in the execution phase.

Performance during Execution After training, we fix the network parameters and compare the performance of different algorithms in the simulation environment without exploration. The results are shown in Figure 6. In this figure, the height of each bar represents the sensing reward, where the red part is the battery penalty, and the blue part is the true average reward, which equals the sensing reward minus the battery penalty. The algorithm with the highest blue bar has the best performance.

Figure 6. The performance of different algorithms in the execution phase.

As presented, our proposed ConComm achieves the best performance (the blue part) among the algorithms. In particular, the sensing reward (the blue+red part) also outperforms other algorithms significantly. This is because the vehicles in ConComm can avoid repeated sensing through communication. The ConComm (no MI) can also have high performance. It achieves lower battery penalty because the vehicles’ behaviors will not be affected by the communication messages explicitly. DDPG also performs well in charging since each vehicle only cares about its own reward. However, the global sensing reward can be quite limited, which may be caused by the lack of coordination. For the MADDPG and MAPPO algorithms, as they lack the mechanism of coordination in the execution phase, they may not perform as well as our ConComm algorithm. In summary, to achieve high performance, the vehicles should not only try to sense more events with the limited battery, but they need also coordinate with others to avoid repeated sensing.

We also investigate the trajectories of the vehicles in our ConComm to show the effectiveness. We collect the vehicles’ trajectory in the execution phase for 1000 steps and obtain the appearance count in the map. The appearance counts are normalized and plotted as a heatmap. The result is presented in Figure 7. In the heatmap, areas with a redder color are visited more often by the mobile vehicles, and the blue areas are visited less often. Compared with Figure 4, the areas where the event intensities are higher also have more vehicle appearances. These areas are dispersed since the vehicles can cooperate to maximize the coverage and reduce repeated sensing. Moreover, the areas near the charging stations also have redder colors; this is because the vehicles regularly moves to the stations for charging. Above all, the heatmap validates that the vehicles of ConComm can not only navigate back for charging, but also properly move to the areas with high event intensities. This heatmap illustrates that our proposed ConComm can properly coordinate the navigation of the vehicles.

Figure 7. The heatmap of the vehicles trajectories of ConComm.

Policy Coordination via Communication The above two experiments have already shown that explicitly coordinating the policies of different mobile vehicles is crucial for cooperative sensing. In this part, we investigate the effect of coordination by adjusting the weight of the MI item. In addition to the default value $\rho =0.5$, we change the weight $\rho$ to different values from 0 and 1 and observe the convergence process during training. Note that when $\rho$ is 0, the algorithm degrades to the case of ConComm (no MI). When $\rho$ is 1, the vehicles neglect the sensing reward and battery penalty, and focus only on coordinating with others.

The results are shown in Table 2. As presented, introducing the policy coordination can significantly improve the performance when $\rho$ is non-zero. This validates that positive communication is necessary for coordinating the decentralized vehicles. Meanwhile, when the coefficient is too large, the performance may decrease since the vehicles care more about coordination and less about sensing reward. When the coefficient reaches 1, the vehicles focus only on the coordination and thus the sensing reward is very poor. The results show that the vehicles need to balance between coordination and sensing. The performance will degrade if focusing on only one of them.

Table 2. Different parts of the performance of ConComm. The average step reward can be described as the difference between the sensing reward and the battery penalty.

$\mathit{\rho }$	Sensing Reward	Battery Penalty	Average Step Reward
0	$17.33$	$2.14$	$15.19$
0.25	$20.83$	$3.20$	$17.35$
0.5	$20.76$	$3.34$	$17.42$
0.25	$19.58$	$4.20$	$15.38$
1	$7.87$	$37.36$	$-29.49$

Validating the Variation Network In this part, we show that the communication message indeed influences the vehicles’ moving policy. More concretely, we compute the cross entropy between the policy ${\mu }_i$ and the neighbors’ aggregated message $m_g$ as $\mathcal{H}({\mu }_i,m_g)$. The policy ${\mu }_i$ is represented as the historical trajectories $tra{j}_i$. Cross entropy measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution, rather than the true distribution. It can also be regarded as the distance between the two probability distributions. A low cross entropy distance indicates that the two probability distributions could have high correlation.

We present the dynamics of the cross entropy during training in Figure 8. As presented, the cross entropy is high at the beginning. This is because the vehicles have not learned to correlate with the communication message. As the training proceeds, the cross entropy value becomes lower and stabilizes at about $1.0$. This validates that the policy ${\mu }_i$ becomes more correlated with the communication.

Figure 8. The cross entropy between the vehicles aggregated message $m_g$ and moving policy ${\mu }_i$ represented as $tra{j}_i$. The value is lower when the message is more correlated with the vehicles’ moving policies.

Validation the Penalty of Battery Next, we investigate the effect of the hyper-parameter c in shaping the battery penalty. Generally, with a larger value of c, the vehicles will navigate to the charging station more frequently to avoid running out of battery. In practice, this parameter can be set freely by the vehicles and our algorithm can adapt to different values of c. In this experiment, we train the algorithms with different values of c and validate the performance with the default value $c=40$. Figure 9 presents the results.

Figure 9. The average step reward (orange part), sensing reward (red part) and battery penalty (blue part) in the execution phase with respect to different values of c.

As presented, when c is 0, the vehicles will not care about the battery penalty and focus only on sensing events. Therefore, there will be high sensing reward, but the battery penalty will also be very high, leading to low average step reward. When the value of c increases, the vehicles will be more conservative to run out of power. They will have low battery penalty. However, the sensing reward will also decrease. In general, choosing a proper value of c can balance the preference of sensing and battery. In practice, we can set the value of c as the cost of reclaiming the vehicles when they run out of power. If this is unacceptable, we can also enforce the vehicles to navigate back to the charging station if needed.

Scalability In this last experiment, we validate the scalability of ConComm. We increase the number of mobile vehicles to 128 and charging stations to 16. The map is divided into $80\times 80$ grid space with charging stations randomly and uniformly distributed. Similar to that above, we assume the mobile agents share the same network parameters. The algorithms of MADDPG and MAPPO would take too much time, so we only present the result of ConComm, ConComm (no MI) and DDPG. As shown in Figure 10, the ConComm algorithm can still achieve better performance. When there are more agents, they may become more easy to coincide. So the average step reward will be lower than before. Nonetheless, ConComm can still successfully coordinate the behaviors of the agents and achieve high performance. In this case, as there is no explicit coordination, the variance of ConComm (no MI) will be larger. The result of DDPG is also not stable since the vehicles’ policies are mostly dynamic, leading to low efficiency of coordination. Moreover, the performance of DDPG will even degrade after about 600,000 steps. This may result in the DDPG agents being not coordinated and falling into local optimal points.

Figure 10. The scalability.

7. Conclusions

This paper studies the problem of mobile sensing in an open, dynamic environment. To maximize the long-term spatial–temporal coverage of the events, we propose a decentralized policy coordination framework. The main idea is to introduce a communication mechanism among the mobile vehicles. On one hand, the vehicles can share local information with each other to break through the dilemma of decentralized execution; on the other hand, the vehicles can have coordinated behavior with enforced positive communication. In particular, the consensual communication is achieved by maximizing the mutual information between the received message and the policy. We conduct extensive experiments to validate the performance of our algorithm. The results show that our algorithm can converge very fast in the training phase, and outperforms other baselines significantly in the execution phase. Moreover, the experiments show that the consensual communication mechanism plays an important role in coordinating the behaviors.

For future works, we aim to extend the current method from two aspects. First, the battery constraints in this paper are relaxed as part of the objective, and may lead to violations. Therefore, we need to devise method with “hard” constraints. Second, we will improve the interpretability of the communication messages to understand the internal mechanism that promotes the cooperation among the vehicles.