Distributed and Multi-Agent Reinforcement Learning Framework for Optimal Electric Vehicle Charging Scheduling

Abstract

The increasing number of electric vehicles (EVs) necessitates the installation of more charging stations. The challenge of managing these grid-connected charging stations leads to a multi-objective optimal control problem where station profitability, user preferences, grid requirements and stability should be optimized. However, it is challenging to determine the optimal charging/discharging EV schedule, since the controller should exploit fluctuations in the electricity prices, available renewable resources and available stored energy of other vehicles and cope with the uncertainty of EV arrival/departure scheduling. In addition, the growing number of connected vehicles results in a complex state and action vectors, making it difficult for centralized and single-agent controllers to handle the problem. In this paper, we propose a novel Multi-Agent and distributed Reinforcement Learning (MARL) framework that tackles the challenges mentioned above, producing controllers that achieve high performance levels under diverse conditions. In the proposed distributed framework, each charging spot makes its own charging/discharging decisions toward a cumulative cost reduction without sharing any type of private information, such as the arrival/departure time of a vehicle and its state of charge, addressing the problem of cost minimization and user satisfaction. The framework significantly improves the scalability and sample efficiency of the underlying Deep Deterministic Policy Gradient (DDPG) algorithm. Extensive numerical studies and simulations demonstrate the efficacy of the proposed approach compared with Rule-Based Controllers (RBCs) and well-established, state-of-the-art centralized RL (Reinforcement Learning) algorithms, offering performance improvements of up to 25% and 20% in reducing the energy cost and increasing user satisfaction, respectively.

1. Introduction

The recent advances in battery technologies, alongside the rise in fuel prices, are increasing the importance of EVs. EVs introduce various benefits in comparison to fossil fuel vehicles and can significantly reduce transport-related pollution [1,2]. Moreover, the cost of use of an EV for the customer is cheaper, since charging an EV by using electricity is less costly than filling up a vehicle with gasoline [3]. Finally, Vehicle-to-Grid (V2G) capabilities allow EVs to discharge their batteries, offering energy back to the power grid and allowing operators to optimize grid operation and market scheduling and EV owners to participate in profit-earning Demand–Response (DR) schemes [4,5]. All of these reasons led to a big increase in the numbers of EVs and of plug-in electric vehicles (PEVs). Sales of electric vehicles are seeing exponential growth, as sales exceeded 10 million in 2022. In 2022, electric vehicles accounted for 14% of all new automobile sales, up from 9% in 2021 and less than 5% in 2020 [6].

However, the integration of such vehicles is significantly affecting the power system, creating different load profiles and new demands for electricity production and storage. That is why researchers have tried to tackle these problems and provide innovative solutions for the reduction in energy consumption [7,8]. Moreover, the stochasticity of EV arrival and departure times, user preferences and commuting behavior transform the EV charging/discharging problem into a dynamic, time-varying solution. Thus, in order to support further marketability and utilization of EVs, it is critical to develop suitable control and management algorithms that will satisfy the charging needs to contribute to optimized grid operation, to reduce the charging costs and to satisfy user needs and preferences.

This paper addresses the critical issue of optimizing energy consumption in electric vehicle charging stations (EVCSs), with the dual objective of minimizing overall energy usage while maximizing user satisfaction regarding the charging levels of their vehicles. By implementing advanced optimization techniques, the proposed solution aims to enhance the efficiency and effectiveness of EV charging infrastructure, thereby contributing to sustainable energy management and improved user experience.

1.1. Related Work

During the last years, various EV charging management and load-scheduling schemes have been proposed, aiming to tackle different problems. In [9,10,11,12], the authors presented comprehensive surveys on the charging scheduling algorithms’ enhancement in power grid performance as well as the improvement in grid reliability. In [13,14], EVs offer auxiliary services to the power grid (such as peak shaving, grid backup and load balancing) and are used to promote the integration and storage of renewable energy sources in microgrids. The literature is vast on such examples, and a thorough review of similar works can be found in [15]. On the other hand, focusing on EV scheduling for reducing the charging operation costs, a vast number of different frameworks and charging policies can be found in the literature. In general, we can divide charging/discharging frameworks into two major categories: (a) model-based (where a model of the problem is constructed or the dynamics of the problem are known) and (b) model-free approaches.

Regarding the first category, that of model-based solutions, in [16], a model predictive control approach was proposed, introducing random EV arrivals and departures and constructing a low-level computational complexity problem. In [17], a photovoltaic-equipped charging station was presented by using a mixed-integer programming approach to reduce the cost of energy trading. The works [18,19,20] presented robust optimization approaches that dealt with uncertainties in electricity prices, weather conditions and user behavior to provide intraday and day-ahead EV charging scheduling. The authors in [21] formulated the charging problem as an open-loop cost minimization problem. However, apart from the above optimization approaches, stochastic optimization is also widely used to handle the uncertainty that is introduced by user preferences, grid conditions and weather forecasts. A lot of effort was put into designing and developing real-time scheduling strategies to respond to the changing and time-varying dynamics of an EV charging station. For example, in [22], a binary programming strategy was developed in order to coordinate multiple EV charging spots in a parking station, aiming to provide curtailment capabilities to utility requests. In [23], the authors developed a two-stage stochastic approach to maximize the profit in intraday, balancing market and day-ahead situations, taking into account the uncertainties on market prices and EV fleet mobility. Finally, we have already developed a two-stage stochastic approach based on approximate dynamic programming [24,25,26] for the control of a grid-connected EV charging station equipped with a photovoltaic panel, aiming to reduce the cost of operation under dynamic pricing and diverse weather conditions. However, despite the fact that all the presented works have obtained remarkable results on the scheduling problem, they rely on the use of models or known dynamics of the problem in order to deliver robust and acceptable solutions, and they are often tailored to specific charging scenarios.

Recently, developments in RL and Machine Learning (ML) have led to algorithms that can undertake complex decision-making problems with great success [27]. The big advantage of these methods, compared with those already presented, is that RL methods usually do not need any system model information. Some first attempts to use RL methods for a charging station problem were presented in [28,29], where a tabular Q-learning approach was used. Although both works offered acceptable results and performance, the studied problems only considered a small number of states and actions. To overcome such problems, recent works use deep neural networks in order to learn complex problems from high-dimensional data [30,31]. However, it is only in recent years that deep RL methods which tackle the charging scheduling problem have been presented in the literature [32,33,34,35]. Moreover, refs. [36,37] used Deep Q-Learning (DQN) and State–Action–Reward–State–Action (SARSA) algorithms, two Q-Learning-based approaches to solve the optimal scheduling problem, whereas in [38,39], DDPG, Soft Actor–Critic (SAC) and Twin Delayed DDPG (TD3) were utilized. In addition to single-agent approaches, a few Multi-Agent Reinforcement Learning (MARL) strategies were presented in [40,41] to address the challenging problem of applying RL to coordinate the charging of multiple EVs in a charging station. A thorough review of RL implementations in EV management and control can be found in [42]. Although RL approaches have shown promising results, there are several challenges that make it difficult to apply them to real-world problems. One of the main challenges is that deep RL algorithms typically require a large number of interactions with the environment in order to produce robust policies. This can translate into hundreds of days of data that are needed in real-world problems. Additionally, dynamic programming approaches like RL algorithms are known to suffer from the “curse of dimensionality” [43] where they are unable to converge if the size of the state–action vectors is too large. This is particularly problematic for popular algorithms like DDPG [44] and SAC [45], which are widely used in real-world continuous setups.

1.2. Main Contributions

In this paper, the EV charging/discharging scheduling problem is formulated as a Markov Decision Process (MDP) from a single charging station owner’s perspective. The goal is to find the optimal charging/discharging scheduling of EVs to take advantage of real-time fluctuations in electricity prices (which are assumed to be known on a day-ahead basis) and available solar energy, all while respecting the EV owner’s demands with respect to cost effectiveness, departure schedules and general user satisfaction. To achieve generalization to different charging station configurations, a model-free approach is proposed to determine the optimal policy in a real-world scenario. To tackle the problems identified above, we propose a novel innovative Multi-Agent Reinforcement Learning framework, the Difference Reward Multi-Agent Reinforcement Learning (DR-MARL) algorithm, where each agent controls (charges/discharges) a single charging spot of the charging station. Instead of using one agent to control every charging spot, in this work, we use one agent per charging connection. By combining the advantages of RL, specifically DDPG, the proposed MARL methodology is highly scalable and sample-efficient without simplifying either the states or the actions of the environment and by using low-level continuous space charging/discharging actions rather than simplistic discrete-based solutions. Overall, the main contributions of the DR-MARL algorithm are the presented in the next paragraph.

All the decision variables’ updates can be made in a (parallel) distributed fashion, ensuring that no private information will be shared among different charging spots. It is protected against malfunctions, having the ability to change the number of supported charging stations on the fly without the need to start the whole optimization process from scratch. A thorough experimental study establishes the superiority of DR-MARL over Rule-Based (without learning) Controllers and state-of-the-art baseline algorithms. DR-MARL shows significant improvement in terms of sample efficiency and number of environment interactions during training while maintaining the same or even a better level of performance compared with the current state-of-the-art methods (as shown in Section 4, where DR-MARL is compared with DDPG and SAC). A cornerstone result is that although the local controllers of DR-MARL only have access to the local measurements related with the local control parameters along with their evaluation, the overall performance is on par with powerful centralized RL algorithms. On top of that, due to the distributed nature, the overall performance of DR-MARL seems not to be affected by the number of charging spots (overall number of decision variables). Such result renders DR-MARL suitable for charging stations with a large number of charging spots, cases that are really challenging for the most advanced state-of-the-art RL algorithms.

The rest of this paper is organized as follows: We introduce the system model and the Chargym environment and formulate the problem as an MDP problem. Afterwards, we propose the DR-MARL algorithm. The results and comparison studies are presented, and finally, in the last section, we conclude the paper and present the final outcomes of our work.

2. Materials and Methods

In this subsection, we present the charging station custom-built environment used for the evaluation of our work. The electric vehicle charging station (EVCS) used in this paper, was presented by the authors of [46] (It is publicly available at https://github.com/georkara/Chargym-Charging-Station, accessed on 1 July 2023). The EVCS environment is available online and can be used by interested researchers to replicate the presented results or compare new proposed methods with state-of-the-art algorithms. In a nutshell, here, we briefly present how we build upon this environment to construct the test cases for the evaluation of DR-MARL.

The EVCS architecture is presented in Figure 1 and showcases the basic details of the station. (i) The station comprises N charging spots; (ii) a photovoltaic (PV) generation system is connected directly to the station, offering “free” energy to the station; (iii) a power grid connection is available, enabling the station to absorb any amount of energy necessary; (iv) the charging station has V2G capabilities, allowing the charging policy to use the stored energy of the vehicles when necessary. The owner of the station is aiming to minimize the cost of the charging of the connected vehicles while ensuring that all EVs will reach the desired level of state of charge (SOC) by the departure time.

The remarks below describe the operation of the EVCS, providing a clear picture concerning the implementation of the environment’s operating framework.

Remark 1.

The time horizon is divided into 1-h time slots. Each EV arrival, departure and charging/discharging action is updated every hour.

Remark 2.

The station can use the produced energy from the PV installation or the stored energy in the vehicles (under V2G operation). The station can absorb energy from the grid, at any timestep, under fluctuating pricing schemes.

Remark 3.

The decisions made for each charging spot at the electric vehicle charging station (EVCS) may impact the overall operation due to the shared nature of resources. The EVCS features a shared photovoltaic (PV) unit that provides energy to the charging vehicles and a virtual common storage unit created by the discharging vehicles. This interdependence highlights the need for coordinated charging and discharging management.

Remark 4.

Due to simplicity, all vehicles that request connection to the station share the same technical properties (battery capacity, charging/discharging rates, etc.). The details can be found in [46].

Remark 5.

Each charging spot can be used more than once per day, without a constraint regarding the minimum or maximum times of usage.

Remark 6.

The arrival of EVs to the station is random and not known a priori to the station owner. If all charging spots are full, the new arrival is not connected to the station.

Remark 7.

The departure time and the desired state of charge for every EV at departure time are selected by the EV owners and is shared with the station upon EV arrival.

2.1. Markov Decision Process Framework

In this subsection, we introduce the MDP framework, which corresponds to the problem of the presented charging station. Our goal is to formulate the real-time charging/discharging scheduling problem as an RL problem, where an agent (or a set of agents) interact with the environment over a sequence of discrete timesteps. At the beginning of each timestep t, the charging station calculates the charging or discharging rate (action) of each vehicle spot based on the information (state) of the current events gathered and the available predictions, including the weather conditions, the energy pricing, the SOC of each connected vehicle and the departure times. Thus, we construct an MDP framework with a tuple containing 4 major elements: $(\mathbf{S},\mathbf{A},P,R)$, where $\mathbf{S}$ is the set of observations/states, with states $s\in S$; A is the set of actions/charging set-points, with actions $a\in A$; $P\subset \left(\right(S\times A\times S)\to [0,1\left]\right)$ is the probabilistic state transition set; and R is the transition response set defined as $P:S\times A\times S\to [0,1]$ $R:S\times A\times S\to \mathbb{R}$. $P(s_1,a,s_2)=p\left(s_2\right|s_1,a)$ is the probability that state $s_1$ becomes state $s_2$ after taking action a, and $R(s_1,a,s_2)=r(s_1,a,s_2)$ is the reward given to the algorithm based on the accuracy of its predictions. In the following sections, we discuss the construction of the state vector, the calculation of the action set-points, the reward function and the transition dynamics of the environment that lead to the state of the next timestep $t+1$.

2.2. State

The state vector of the charging station at timestep t is described as

$$\begin{array}{c}\hfill s_t=\left(\begin{array}{c}{G}_t,G_{t+1},\dots ,G_{t+3},p{r}_t,p{r}_{t+1},\dots ,p{r}_{t+3},\end{array}\right.\cdots \\ \hfill \cdots \left.\begin{array}{c}So{C}_t^1,\dots ,So{C}_t^N,T{l}_t^1,\dots ,T{l}_t^N\end{array}\right)\end{array}$$

(1)

The vector encapsulates four different types of information: (1) $G_t$ and $(G_{t+1},\dots ,G_{t+3})$ represent the current and three-timestep-ahead predictions of the solar radiation values. (2) $p{r}_t$ and $(p{r}_{t+1},\dots ,p{r}_{t+3})$ represent the electricity prices for the current timestep and again the three next timesteps. The idea behind including only 3-timestep-ahead predictions for both solar radiation and electricity prices reflects our intention to restrict the agent’s information only to estimations with high accuracy. The pricing is not affected by the demand of the station; rather, it is a value that is dynamically defined by the grid operator (e.g., electricity power distributor). (3) $So{C}_t^i$ denotes the state of charge of the EV at the ith charging spot at timestep t. (4) $T{l}_t^i$ represents the remaining hours until departure for the EV at the ith charging spot. Figure 2a presents the four pricing profiles that are used for simulations, whereas Figure 2b shows a typical day of electricity production. Pricing models 1 and 2 were obtained from [47], while pricing model 3 is taken from [48], with more complex pricing levels than the previous two models. Finally, model 4, taken from NordPool [49], considers late night hours as high-demand hours. These profiles have also been used in former works by the authors [46,50].

Since the state contains information regarding the SoC and the remaining hours of connection for each charging spot, no additional parameter is needed to indicate whether an EV is connected to the charging spot or not, while if the $So{C}_t^i$ of the ith charging spot is equal to 0, this means that this specific charging spot is unoccupied, i.e., there is no vehicle.

2.3. Action

Having described the state $s_t$, the actions $a_t^i$ that are available to the user represent the charging/discharging power of the 10 charging spots. These 10 action set-points ($a^i$) are defined as continuous variables, which are constrained in the $[-1,1]$ space. The charging/discharging power for each vehicle i at timestep t is defined as

$$M{P}_t^i=\left\{\begin{array}{cc}(1-SO{C}_t^i)·B_{max}\hfill & a_t^i>=0\hfill \\ \left(SO{C}_t^i\right)·B_{max}\hfill & a_t^i<0\hfill \end{array}\right.$$

(2)

$$M{P}_t^i<=P_{ch,max}·{\eta }_{ch}$$

(3)

$$P_t^{dem,i}=actio{n}_t^i·M{P}_t^i$$

(4)

where $M{P}_t^i$ is the maximum charging/discharging power for each vehicle i at timestep t, $So{C}_t^i$ is the state of charge of each EV (i) at time t and $B_{max}$, $P_{ch,max}$, ${\eta }_{ch}$ are the EV maximum battery capacity, the maximum charging output of each charging spot and the charging/discharging efficiency, respectively, and are considered constant over the evaluation horizon.

All three equations above present the calculation of the electricity demand of each spot at any given timestep t by incorporating the selected action. Note that the action is positive when the EV is charging and negative when discharging. The value of $actio{n}^i$ directly affects the power demand $P_t^{dem,i}$ of each electric vehicle spot at timestep t as shown in Equation (4). Equations (2) and (3) describe the constraints on the maximum charging/discharging power that can be allocated in one timestep.

2.4. Transition Function

The transition probability $p\left(s^{\prime }\right|s,a)$ represents the likelihood of moving from one state s to another state $s^{\prime }$ given an action a. This probability depends on the $N=10$ charging/discharging power $actio{n}_t^i$, the physical and mathematical dynamics of the EVCS, the random nature of customer-selected arrival and departure times and the fluctuations in electricity prices and solar radiation. Considering that the exact transition probability is unknown, the state transition from state $s_t$ to state $s_{t+1}$ can only be calculated for the already connected vehicles:

$$So{C}_{t+1}^{connected,i}=So{C}_t^{connected,i}+P_t^{dem,i}$$

(5)

On the other hand, the overall requested power from the grid, for timestep t, is given as

$$P_t^{dem,grid}=\sum _{i=1}^{10}{P}_t^{dem,i}-P_t^{PV}$$

(6)

where $P_t^{PV}$ is the mean production of solar power at timestep t.

2.5. Reward

The goal of the EVCS’s controller and owner is to use a charging/discharging scheduling policy that will minimize the cost of the grid-absorbed electricity. Therefore, the main part of the reward function encapsulates the cost of the electricity bill that is paid to the grid/energy company. On the other hand, as presented in the Section 1, an additional term is incorporated in order to penalize situations where the departing EVs are not fully charged (since this is a common request of vehicle owners). The equation describing this specific formulation is the following:

$$r_t(S_t,A_t)=\sum _{i\in {\Omega }_t}(p{r}_t·P_t^{dem,grid})+\sum _{i\in {\Psi }_t}{[2·(1-So{C}_t^i)]}^2$$

(7)

where $P_t^{dem}\in {\Omega }_t$ denotes the total energy demand of the EVCS to the energy company, $\Omega$ is the set of all time periods during which the energy demand is measured and accounted for and $\Psi$ is the set of all EVs for which the $SoC$ is evaluated. The electricity price, $p{r}_t$, fluctuates in every timestep h, and as in real life, the information is shared to the EVCS by the energy company. The billing is presented in Figure 2a. The second cost term measures the level of customer satisfaction. The EVCS’s goal is to reach 100% $SoC$ for each vehicle by the given vehicle’s chosen departure time. The second term in the reward function penalizes situations where departing EVs are not fully charged. The factor of 2 within the square amplifies the penalty for not meeting the target SoC. The squaring of $(1-So{C}_t^i)$ ensures that the penalty increases more significantly as the SoC deviates further from 100.

A Pareto frontier, also known as a Pareto front or Pareto boundary, represents the set of all Pareto optimal solutions in a multi-objective optimization problem. In such problems, we aim to optimize multiple conflicting objectives simultaneously. A solution is considered Pareto optimal if no other solution can improve one objective without worsening at least one other objective [51]. In order to visually illustrate the inherent multi-objective nature of the problem, Figure 3 presents a Pareto analysis chart depicting the trade-off between cost reduction and user dissatisfaction. The horizontal axis (X-axis) represents the degree of cost reduction achieved through our proposed EV charging strategy, while the vertical axis (Y-axis) quantifies the level of user dissatisfaction attained. Each point on the chart corresponds to a specific charging profile generated by the RL state-of-art methodology presented in the next section. As we move along the Pareto frontier depicted in Figure 3, we observe the delicate balance between minimizing costs and maximizing user satisfaction. This graphical representation serves as an intuitive demonstration of the complex relationship between these two objectives, highlighting the diverse range of solutions available to decision makers. It becomes evident that any substantial improvement in one objective comes at the expense of the other, reinforcing the need for a multi-objective approach that accommodates varying priorities.

3. Proposed DR-MARL Method

3.1. Problem Transformation

Having presented the real-world details that define the EVCS, we proceed to first transform the problem into an equivalent MDP setup; then, in the subsequent subsection, we propose an algorithm that builds upon a distributed, multi-agent algorithm [52] and delivers a MARL dedicated algorithm for the problem at hand.

The EVCS consists of N charging spots interacting with each other and a common energy source (solar panel and grid connection) for achieving a global objective (minimize the cost of operation of the charging station). The matrix containing all the decision variables is set to be

$$\mathbf{x}\left(k\right)={\left[x_1^{\tau }\left(k\right),x_2^{\tau }\left(k\right),\dots ,x_N^{\tau }\left(k\right)\right]}^{\tau }$$

(8)

where $x_i\left(k\right)\in {\mathbb{R}}^m$ represents the decision set-point vector of the ith charging spot at the kth timestep, with m being the dimensionality of the vector. These decision set-points represent the controllable charging/discharging rates of the ith charging spot. The MARL framework proposes a setup where the local agents utilize knowledge only from the respective charging spot, and the global coordinator calculates the discrepancy ${\Delta }_i\left(k\right)$, which is fed back to local agents.

Furthermore, the observation/state matrix which represents the available vehicle measurements is defined to be

$$\mathbf{y}\left(k\right)={\left[y_1^{\tau }\left(k\right),y_2^{\tau }\left(k\right),\dots ,y_N^{\tau }\left(k\right)\right]}^{\tau }$$

(9)

where $y_i\left(k\right)\in {\mathbb{R}}^m$ denotes the state transition of the ith vehicle at the k timestep. It is important to mention the information contained in $y_i\left(k\right)$ and highlight the difference between distributed and centralized methods. The information shared in $y_i\left(k\right)$ is the following: $y_i\left(k\right)=\left(\begin{array}{c}{G}_t,p{r}_t,G_{t+1},G_{t+2},G_{t+3},p{r}_{t+1},p{r}_{t+2},p{r}_{t+3},So{C}_t^i,T{l}_t^i\end{array})\right.$.

On the other hand, for centralized implementations, $y\left(k\right)$ is as described in Equation (1). Therefore, in charging station implementations that utilize 10 charging spots, the $y_i\left(k\right)$ distributed information vector utilizes 10 values, whereas the centralized implementations utilize 28 values. On larger implementations (20-35-50 charging spot implementations), DR-MARL still utilizes the same 10-value information vector for each charging spot, whereas the centralized approaches utilize vectors with up to 108 values (50 spots). The results of this difference will be analyzed in the next section.

The measurement vector’s evolution can be defined as

$$y_i\left(k\right)=h_i(k,x_i\left(k\right))$$

(10)

where $h_i(·)$ denotes an unknown, nonlinear function that depends on both $x_i\left(k\right)$ and the specific problem characteristics (presented previously in the Transition State subsection). The aim of the multi-vehicle system’s objectives (e.g., minimization of the charging cost and respect of the user preferences/leaving hours) can be translated into the maximization of a specifically defined global cost function $j_k$ and set to be

$$j_k=\mathrm{J}\left(x_1\left(k\right),x_2\left(k\right),\dots ,x_N\left(k\right)\right)$$

(11)

where $\mathrm{J}(·)$ is a non-negative, nonlinear, scalar function that depends, apart from the charging-spot decision variables, on the particular dynamics of the problem (e.g., the EVCS environment).

3.2. DR-MARL—Global Coordination

Step 1. For each timestep k, the charging spots deliver the measured state observations, after the appliance of the $\mathbf{x}\left(k\right)$ decision action/set-point.

Step 2. The cost function associated with the whole EVCS operation can be directly derived from (see (11))

$$j_k=\mathrm{J}\left(y_1\left(k\right),\dots ,y_N\left(k\right)\right)$$

In addition, for each ith agent/vehicle, we calculate the following index:

$${\Delta }_i\left(k\right)=j_k-\mathrm{J}\left(y_1\left(k\right),\dots ,y_{i-1}\left(k\right),y_i(k-1),y_{i+1}\left(k\right),\dots ,y_N\left(k\right)\right)$$

(12)

${\Delta }_i\left(k\right)$ describes the contribution of that $x_i\left(k\right)$ on the current problem for the kth timestamp.

Note that since the second part of (12) is analytically available, we can calculate this term without the use of a new experiment or the use of a system model/simulator, although the resulting calculation does not necessary match the actual value when the charging spots utilize the following decision variables:

$$\left\{x_1\left(k\right),\dots ,x_{i-1}\left(k\right),x_i(k-1),x_{i+1}\left(k\right)\dots ,x_N\left(k\right)\right\}$$

The convergence of the proposed methodology is not affected by the difference between the calculation of $J(·)$ and its actual value. The resulting discrepancy is based on the application and the environment operation and describes the effect of other agent’s actions to each vehicle’s measurements.

Step 3. Next, the calculated discrepancy ${\Delta }_i\left(k\right)$ is sent to the ith charging spot/agent.

The diagram of the algorithm described is presented in Figure 4.

3.3. Local Agent—DDPG

In each charging spot/agent, an instance of Deep Deterministic Policy Gradient [44] algorithm is utilized to find the best policy that maximizes the previously defined difference reward. The main advantage of the DDPG algorithm is the ability to directly deal with continuous state and control variables, as well as utilizing a policy that maximizes the return/reward/cost by maintaining a set of estimates of expected returns for the optimal policy (off-policy function). The EVCS environment is a fully continuous environment, as most of the energy/smart grid-related problems where all the state and control variables are in continuous form.

The most crucial parameters of the EVCS, namely, the SOC of each vehicle connected to the station, the electricity–energy pricing and the departure time of each vehicle are used for action selection for achieving the minimization of the cost (or maximization for the reward). By utilizing these input parameters, the learning agent decides whether the vehicle should be charged or discharged and chooses the charging/discharging set-point, as shown in Figure 5. This figure showcases a generic description of the local agent operation. The actual output of the local agent is a continuous parameter from −100 to +100.

The actor of our formulation is defined as $\mu \left(s_t\right|{\theta }^{\mu })$, where $s_t$ is the input vector state we mentioned before and ${\theta }^{\mu }$ stands for the weights of the actor neural network, while $a_t$ is the control action. The actor network links an action to each vector state, while the critic network calculates the expected reward according to the policy using the action proposed by the actor. The critic network is defined as $Q(s_t,a_t|{\theta }^Q)$, where $S_t$ is the state vector, $a_t$ the control action by the actor and ${\theta }^Q$ the weights of the critic. The Q function describes the expected reward by taking the specific control action $a_t$ at state $s_t$ with the policy $\pi$. To improve training stability, we use two target networks, which are the copies of each critic $Q^{\prime }(s_t,s_t,|{{\theta }^Q}^{\prime })$ and actor ${\mu }^{\prime }(s_t,{{\theta }^{\mu }}^{\prime })$. Moreover, to ensure robustness at both actor and critic levels, a finite-sized replay buffer is defined. By interacting with the EVCS, the DDPG agent network gets trained. Specifically, the $SOC$ of each connected vehicle, the present electricity pricing and the departure time of each vehicle every hour h are collected and fed into the policy network, which then returns the action. Due to the deterministic nature of the action selected, noise is added to the actions at training time to improve DDPG’s exploration performance:

$$a_k=\mu \left(s_k\right|{\theta }^{\mu })+N_k$$

(13)

where $N_k$ is the exploration Ornstein–Uhlenberk (O-U) noise. In order to explore effectively in momentum-rich physical situations, we employed this temporally correlated noise for the exploration noise process. The Ornstein–Uhlenbeck [53] procedure simulates the speed of a Brownian particle subjected to friction, yielding values that are temporally coupled and cluster around 0.

In every transition, the 4-tuple $(s_t,a_t,r_t,s_{k+1})$ is saved and stored in batches in the replay buffer and used for actor training.

Regarding the update of the actor–critic networks, the first step is to calculate the immediate reward by using the following equation:

$$y_i{=}_i+\gamma Q^{\prime }(s_{i+1},{\mu }^{\prime }(s_{i+1}|{{\mathsf{\theta }}^{\mu }}^{\prime })|{{\mathsf{\theta }}^Q}^{\prime })$$

(14)

where $\gamma$ is a discount factor, with $\gamma \in [0,1]$.

The critic network in each transition will be updated by minimizing the MSE (mean square error) between the estimated reward and the expected reward from the critic.

$$L={\displaystyle \frac{1}{N}}\sum _i^N{(y_i-Q(s_i,a_i|{\theta }^Q))}^2$$

(15)

The actor network will be updated by a Deep Deterministic Policy Gradient:

$${\nabla }_{{\theta }^{\mu }}\mathrm{J}\approx {\displaystyle \frac{1}{N}}\sum _i^N{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|s_i$$

(16)

where J is

$$J_i=\sum _{t=1}^{T_{max}}{\gamma }^{t-1}{\Delta }_i\left(t\right)$$

(17)

Target actor and target critic network are smoothly updated by using the following expressions:

$${{\theta }^Q}^{\prime }\leftarrow t{\theta }^Q+(1-t){{\theta }^Q}^{\prime }$$

$${{\theta }^{\mu }}^{\prime }\leftarrow t{\theta }^{\mu }+(1-t){{\theta }^{\mu }}^{\prime }$$

We should note that after the training process, only the actor network is necessary and is stored.

4. Results

In this section, we evaluate the performance of the proposed MARL algorithm compared with a Rule-Based Controller and two state-of-the-art, centralized, off-policy RL algorithms: DDPG and SAC. Both these algorithms are among the most used and successful approaches in off-policy Reinforcement Learning strategies. Note that based on the description provided in the previous section, DDPG is used as the local-agent decision-making tool in our MARL framework. We assess the scalability and optimality of these methods in managing the EVCS’s operations under various scenarios and conditions, including different weather and pricing situations.

4.1. Setup of Numerical Study

The following analysis provides a detailed overview of the operational and experimental framework that was used during the validation study. The simulations were based on the following key attributes.

Day = episode: Each simulation day corresponds to an RL episode. For an episode in RL frameworks, one sequence of states, actions and rewards, which ends with a terminal state, is defined. Therefore, our terminal state is the end of day.

Diverse EV scheduling: The overall dynamics and EV scheduling for each episode are decided by a particular automated and random procedure. These values reflect the stochastic nature of the EVs’ numbers, arrival and departure rates, timetables, initial SoCs upon arrival and the varying and fluctuating nature of electricity pricing and solar energy every day. This methodology aims for the control strategies to be trained and evaluated under diverse conditions, resulting in robust solutions under unpredictable and random real-world conditions.

Real-world application: Since the EVCS formulation is based on common and applicable EV knowledge and assumptions, any acquired policy may be easily applied to an appropriate EVCS instance. Our objective is to develop strategies that determine the best charging/discharging schedule based on a general sense of the environment. However, in this work, we mainly focus on online setups, where the tested algorithms try to optimize their performance by interacting with the environment, rather than by training on historical or previous data.

Diverse conditions: In order to validate the performance of the proposed methodologies under different conditions and scenarios, we created five different pricing cases with the four different pricing tariffs presented in Figure 2. The first four cases correspond to keeping the four pricing schedules constant from day to day, whereas in the fifth case, each day/episode introduces one of the four main pricing schedules in a random order. It is crucial to mention that the weather conditions are different each day.

As mentioned in the Section 1, the proposed algorithm (DR-MARL), as well as state-of-the-art RL algorithms, were compared with a Rule-Based Controller (RBC), which emulates basic rules (human-like decisions) on how to charge and discharge the connected EVs. The RBC provides straightforward decision making; however, it is far from the optimal choice. It is used in [46]. The following equation presents the function of the RBC:

$$actio{n}_t^i=\left\{\begin{array}{cc}1\hfill & T{l}_t^i<=3\hfill \\ {\displaystyle \frac{(G_t+G_{t+1})}{2}}\hfill & T{l}_t^i>3\hfill \end{array}\right.$$

(18)

Equation (18) presents the following logic: If a vehicle is bound to leave during the next three hours (the time needed by the EVCS parameters to fully charge a vehicle from 0 to 100), the charging set-point is set to maximum (1). If a vehicle is not leaving in the next three hours, the charging set-point is defined by the current available and the one-hour-ahead forecasted available solar energy. It is obvious that the RBC does not take advantage of the fluctuating electricity prices and the V2G functionality. However, it is a good baseline with which to compare the performance of the proposed framework.

4.2. Cost Comparison with Baseline Methods

As mentioned earlier, we compared the performance of the proposed DR-MARL method with the two centralized approaches, as well as the RBC, which was also used as the baseline. It is crucial to mention that since we are presenting an RL formulation, the performance index is a reward index (maximization problem) and that sample efficiency is another factor to be considered. This is because we are proposing a method that has direct, real-world implementation and fast convergence to a superior solution leads to a greater cost reduction.

In Figure 6, we present the learning performance of the proposed DR-MARL algorithm compared with DDPG and SAC. The figure showcases the evolution of the average daily reward. The MARL algorithm converges significantly faster than the centralized approaches of DDPG and SAC while achieving similar levels of reward by the end of the training process. More specifically, DR-MARL achieves a similar or greater level of performance compared with the RBC in less than 20k interactions with the environment, while both SAC and DDPG require more than 100k.

Moreover, in order to showcase that DR-MARL is not only faster than the state-of-the-art approaches but also converges to the same level of performance, we present

Table 1. Mean reward over 10 different random days/episodes.

	RBC	DDPG	SAC	DR-MARL
Price 1	−35.29	−29.45	−31.41	−30.79
Price 2	−36.96	−25.91	−27.11	−26.09
Price 3	−36.06	−29.53	−31.43	−30.34
Price 4	−34.50	−29.58	−30.59	−30.86
Price Random	−35.88	−30.21	−30.84	−30.94

. As shown in this table, each algorithm was trained under diverse weather conditions and random EV schedules, in five different experiments. In the first four experiments, the EVCS was constantly operating (every day) under the four pricing models presented in Section 3, whereas in the fifth experiment, the EVCS was operated each day under a different and randomly selected pricing scheme (as in the case presented in Figure 6). For each different pricing profile, we simulated 10 days with random EV arrival scheduling and weather conditions, and we tested the algorithms on these days. To evaluate the algorithms, we simulated 10 episodes (days) under diverse conditions. Each day presented a random car arrival schedule, diverse weather conditions and a random pricing profile. So, each day was unique and different both from the previous and the next one. However, in order to successfully compare the algorithms and provide meaningful results, the algorithms were tested on the same set of 10 episodes. We collected the rewards for each day and presented the mean of the episodic reward of each algorithm (the average daily reward). We considered that 10 episodes are adequate to showcase valid results about the performance of the algorithms. As a reward, we define the index presented in Equation (7), which is a metric for both energy cost minimization and user satisfaction. It is clear (and expected) that each algorithm outperformed the RBC. However, DR-MARL achieved similar (yet slightly worse) performance levels with the regular version of DDPG and in most cases was better than the SAC algorithm. We should highlight that the DR-MARL method is a distributed approach, not sharing information between the different charging spots, whereas centralized DDPG and SAC operate at the station level, utilizing knowledge and data from all charging spots.

Even if utilizing data from all charging spots may seem to make centralized DDPG converge to slightly better solutions for the presented 10-spot EVCS , in the next subsection, we will showcase that handling all this information is a heavy duty in more complex and large cases.

4.3. Scalability Analysis

As shown in this subsection, we tested the scalability of the proposed method in contrast with centralized SAC and DDPG. We evaluated the performance of the presented algorithms in three different EVCS cases, one with 20 available charging spots, one with 35 spots and one with 50 spots. The EV arrival pattern was used and scaled linearly with the number of chargers, as well as the available solar/produced energy. In Table 1, we present the mean reward over 10 different random days/episodes for a 10-spot EVCS. In

Table 2. Standard deviation of the mean reward presented in Table 1.

	RBC	DDPG	SAC	DR-MARL
Std	-	2.84	2.33	3.32

, we present the standard deviation to convey the variability in the data. The standard deviation provides insights into how much the results differ from the mean value. Comparing the results in Table 1 and

Table 3. Comparison results over 10 different random days/episodes.

	RBC	DDPG	SAC	DR-MARL
$EVC{S}_{20}$	−71.83	−66.76	−61.24	−61.69
$EVC{S}_{35}$	−141.45	−250.89	−152.33	−123.45
$EVC{S}_{50}$	−207.34	−380.76	−344.69	−175.12

, one can notice that as the charging station size increases, the average daily performance of DR-MARL increases compared with the centralized RL algorithms. A comparison of the algorithms’ performance for the EVCS use case of 20 charging spots is also shown in Figure 7. The MARL algorithm not only minimizes the energy needed from the grid but also maximizes the user satisfaction. These algorithms, after some point, fail to handle the large number of state–actions and do not converge to an acceptable solution. More importantly, as shown in Figure 8a–c, DR-MARL achieved better performance than the RBC baseline quite early on in the training process.

Finally, in

Table 4. Number of iterations required to overcome RBC performance.

	DDPG	SAC	DR-MARL
$EVC{S}_{10}$	110k	60k	20k
$EVC{S}_{20}$	400k	200k	20k
$EVC{S}_{35}$	-	-	25k
$EVC{S}_{50}$	-	-	27k

, the number of iterations that were required from each method to achieve a similar performance to the RBC are presented. We note that DR-MARL in every case needed around 20k–30k iterations to achieve a performance similar to that of the RBC. On the other hand, DDPG and SAC needed significantly more iterations; alternatively, in some cases, they failed to converge to an acceptable solution altogether. These results showcase that the proposed DR-MARL methodology could scale to large numbers of the action/state dimensions, in contrast with the state-of-the-art RL algorithms, which, in the two last cases (35 and 50 charging spots), failed to converge.

5. Discussion

5.1. Limitations

Our study recognizes several limitations that could impact the effectiveness and applicability of the proposed MARL framework. Firstly, the availability and quality of data are crucial for the performance of our algorithm. Insufficient or inaccurate data can lead to suboptimal decision making, potentially affecting the efficiency of the charging station operations. Additionally, user willingness to adopt and interact with the algorithm plays a significant role. Reluctance or lack of engagement from users could hinder the algorithm’s practical implementation and effectiveness. From a technical standpoint, the computational intensity of the MARL algorithm is another limitation. To address this, future research could explore the utilization of parallel computing algorithms, which would significantly enhance the speed of computational calculations and improve the algorithm’s scalability. Furthermore, our current framework operates on an hourly timestep, which may not capture the finer dynamics of EV charging behaviors. Future studies could investigate the use of shorter timesteps to increase the granularity and accuracy of the scheduling process, thereby enhancing the generalizability and robustness of the results across different scenarios and environments.

5.2. Conclusions and Future Work

In this paper, a distributed RL framework, the DR-MARL algorithm, was presented and managed the operation of a charging station with multiple charging spots considering V2G operation, a connected renewable energy system unit and fluctuating electricity prices. The objective of the proposed control policy is to maximize the reward of operation of the station by minimizing the cost of absorbed grid energy and maximizing user satisfaction. Extensive numerical and scalability studies show that the proposed MARL framework offers equal or (even better) results compared to state-of-the-art baseline methods, but with much fewer iterations needed. The scalability analysis shows that the proposed framework is able to find satisfactory policies in challenging setups (35 and 50 spots) where traditional state-of-the-art RL algorithms fail to converge. This underscores the robustness and scalability of our approach, making it a significant advancement in smart grid management, because of the decentralized control approach, which is a less common yet highly effective alternative to traditional centralized methods. Future work will focus on enhancing the exploration attributes of the DR-MARL framework by integrating state-of-the-art exploration algorithms, implementing reward-shaping techniques that incentivize exploration, optimizing the learning rate parameters to strike a balance between exploration and exploitation and using its distributed capabilities with parallel computing implementation. Additionally, research can build on these findings by exploring the scalability of the MARL framework in larger and more complex environments, experimenting with different cost criteria and environments that also incorporate diverse energy sources. Furthermore, in our prospective endeavors, a central emphasis will be placed on conducting a comparative analysis between our method and other notable approaches, including but not limited to Multi-Agent Proximal Policy Optimization MAPPO [54], Heterogeneous-Agent Trust Region Policy Optimisation HATRPO [55] and Multi-Agent DDPG MADDPG [56].