Online EVs Vehicle-to-Grid Scheduling Coordinated with Multi-Energy Microgrids: A Deep Reinforcement Learning-Based Approach

Abstract

The integration of electric vehicles (EVs) into vehicle-to-grid (V2G) scheduling offers a promising opportunity to enhance the profitability of multi-energy microgrid operators (MMOs). MMOs aim to maximize their total profits by coordinating V2G scheduling and multi-energy flexible loads of end-users while adhering to operational constraints. However, scheduling V2G strategies online poses challenges due to uncertainties such as electricity prices and EV arrival/departure patterns. To address this, we propose an online V2G scheduling framework based on deep reinforcement learning (DRL) to optimize EV battery utilization in microgrids with different energy sources. Firstly, our approach proposes an online scheduling model that integrates the management of V2G and multi-energy flexible demands, modeled as a Markov Decision Process (MDP) with an unknown transition. Secondly, a DRL-based Soft Actor-Critic (SAC) algorithm is utilized to efficiently train neural networks and dynamically schedule EV charging and discharging activities in response to real-time grid conditions and energy demand patterns. Extensive simulations are conducted in case studies to testify to the effectiveness of our proposed approach. The overall results validate the efficacy of the DRL-based online V2G scheduling framework, highlighting its potential to drive profitability and sustainability in multi-energy microgrid operations.

1. Introduction

Electric vehicles (EVs) are increasingly being recognized not just as modes of transportation but also as valuable energy storage resources within smart grid contexts [1,2]. Vehicle-to-grid (V2G) technology enables bidirectional energy flow between EVs and the grid, allowing EVs to serve as mobile energy storage units that can both charge from and discharge to the grid [3,4]. This concept has garnered significant attention for its potential to enhance grid stability, facilitate renewable energy integration [5], and provide additional revenue streams for EV owners and grid operators [6,7].

A substantial body of literature exists on V2G technology, exploring various aspects such as its technical feasibility, economic viability, and environmental impacts [8,9,10,11,12]. Researchers have investigated different V2G control strategies, charging/discharging algorithms, and optimization techniques to maximize the benefits of V2G integration while minimizing its drawbacks. Reference [8] investigates the approach to reducing the cost of charging for electric utilities, thus increasing profits for EV owners. Reference [9] designs a two-stage optimization formulation to determine the charging and discharging schedule for EVs participating in a V2G service at an office building. A principles-based techno-economic model in [10] is developed to estimate the levelized cost of storage (LCOS) of V2G technology for energy arbitrage and frequency regulation. A two-stage stochastic optimization framework in [11] is built to derive the optimal power management strategy for the V2G systems, aiming to minimize operating costs. Reference [12] assesses the feasibility of the hybrid backup system within the microgrid that incorporates V2G technology to minimize the operational costs, the wasted energy in the dummy load, and the loss of power supply probability. Additionally, studies [13,14] have examined the potential challenges and barriers to widespread V2G adoption, including interoperability issues [15,16], regulatory hurdles [17], and consumer acceptance [18].

Despite the considerable body of research on V2G systems, a notable gap persists in the literature concerning the coordinated integration of V2G with multi-energy microgrids [19,20,21]. While existing studies have predominantly examined V2G in isolation, scant attention has been directed toward exploring its synergistic interactions with other distributed energy resources within the framework of multi-energy microgrids. This gap in the literature is particularly noteworthy given the escalating complexity of contemporary energy systems and the imperative for holistic approaches to energy management. The challenges inherent in this integration stem from the myriad sources of uncertainties in V2G online scheduling and the management of multi-energy flexible loads.

Deep Learning (DL) and Reinforcement Learning (RL) algorithms, while powerful in various domains, face specific limitations when applied to Vehicle-to-Grid (V2G) scheduling problems. DL algorithms, known for their ability to learn complex patterns from data, often struggle with the dynamic and stochastic nature of V2G environments. They may require extensive training data and struggle to adapt to real-time changes in EV charging patterns and grid conditions. On the other hand, RL algorithms, while effective in learning decision-making policies using trial and error, can cause challenges in V2G scenarios due to the high-dimensional action spaces and continuous state transitions involved. Additionally, RL algorithms may exhibit slow convergence rates and struggle with reward function design complexities in V2G scheduling, where the optimization goals are multifaceted and dynamic. These limitations underscore the need for more sophisticated approaches, such as Deep Reinforcement Learning (DRL) [22,23,24,25], which can leverage the strengths of DL for feature learning and RL for decision-making to effectively tackle the intricate challenges of online V2G scheduling and optimize multi-energy microgrids utilization while ensuring operator profitability.

DRL has demonstrated remarkable success in various domains of smart energy management, including demand response (DR), EV charging management, and frequency regulation. In reference [26], the authors implement a DRL-based approach to facilitating the pricing strategy of demand response providers (DRPs). The objective is to maximize their profits by strategically scheduling DR activities to enhance system reliability. In reference [27], a DR scheme is introduced for virtual power plants, aimed at minimizing difference penalties incurred from participation in the electricity market. Notably, the novel DRL-based framework is adept at handling inherent randomness within the system model. In addition, references [28,29] design a single EV V2G scheduling problem as a constrained Markov Decision Process (MDP). The aim is to find a constrained charging/discharging scheduling strategy to minimize the charging cost and guarantee the EV can be fully charged. Reference [30] formalizes the scheduling problem of EV charging as an MDP. It utilizes DRL algorithms to minimize the total charging time of EVs and maximal reduction in the origin-destination distance. References [31,32] apply advanced DRL algorithms to jointly optimize vehicle path planning and energy usage schemes for IoT networks. Finally, references [33,34] propose DRL-based approaches to enhance the overall operation efficiency in the coordinated power and transportation networks. However, the potential application of DRL in the context of V2G and multi-energy microgrids remains largely unexplored. By leveraging the power of DRL algorithms, it may be possible to develop intelligent and adaptive online V2G scheduling frameworks that can dynamically respond to changing grid conditions and user preferences in real time, thereby maximizing the economic and environmental benefits of V2G integration within multi-energy microgrids.

In this paper, we propose an innovative framework for online scheduling of V2G and multi-energy microgrids, leveraging DRL techniques. We begin by developing an integrated scheduling model that combines V2G management with the optimization of multi-energy flexible demands within microgrids. This model is formulated as an MDP, considering the uncertain preferences of end-users and fluctuating electricity prices. Our key contribution lies in the application of a novel Soft Actor-Critic (SAC) algorithm tailored specifically for dynamic V2G scheduling in response to real-time grid conditions and energy demand patterns. Using extensive simulations and case studies, we demonstrate the effectiveness of our proposed approach in maximizing the total profits of multi-energy microgrid operators (MMOs) while ensuring operational constraints and satisfying user requirements. Additionally, we highlight the scalability and practical implications of deploying our V2G scheduling framework in real-world microgrid deployments, paving the way for more efficient and sustainable energy management practices.

Our main contributions are twofold:

(1)

Integration of V2G and multi-energy DR: Unlike existing literature, which often focuses solely on V2G scheduling or multi-energy DR in isolation, our framework integrates both aspects into a unified scheduling model. This integration allows for coordinated management of V2G and multi-energy flexible loads, maximizing the overall profitability of MMOs while adhering to operational constraints.

(2)

DRL-based online scheduling framework with novel SAC algorithm: A novel online scheduling framework is proposed that leverages DRL to optimize the utilization of EV batteries within multi-energy microgrids. By formulating the scheduling problem as an MDP and employing a SAC algorithm, our framework dynamically schedules V2G activities in response to real-time grid conditions and energy demand patterns.

The remaining sections of the paper are structured as follows: Section 2 outlines the formulation of a V2G scheduling coordinated with microgrids. Section 3 details the proposed online V2G framework based on DRL with a Soft Actor-Critic (SAC) algorithm. Section 4 covers the simulation experiments and their corresponding results. Lastly, Section 5 provides conclusions and explores avenues for future research.

2. Formulation of V2G Scheduling Coordinated with Multi-Energy Microgrids

In this section, the V2G Scheduling Coordinated with Multi-energy Microgrids is formulated using the bilevel model. Specifically, the lower level of the bi-level model is the EVs with uncertain arrival and departure patterns and action responses by users in microgrids with different energy sources. On the other hand, the upper level of the bilevel model tackles the efficient coordination of multi-energy microgrids, integrating dynamic pricing strategies established by operators to manage multi-energy microgrid operations effectively.

2.1. The Overall Framework

Figure 1 depicts the proposed bi-level model of online V2G scheduling, strategically formed to optimize the operations of microgrids with the primary objective of minimizing total operational costs. The microgrids with different energy sources encompass three fundamental energy flows. Firstly, the power flow encompasses various components such as electricity transactions with the upper-level grid (either purchase or sale), interactions with EVs, the unpredictable output of Photovoltaic systems (PVs), production from a gas turbine, usage by electric heat pumps, and end-user consumption. Notably, EVs participate in bi-directional power flow within the microgrid, facilitated by V2G scheduling capabilities.

Secondly, the heat flow component represents heat generation from the electric heat pump and gas turbine, along with heat consumption by end users. Additionally, the gas flow component delineates the balance between gas supply, gas utilization by the gas turbine, and gas consumption by end-users. Energy prices established by operators influence end-users. End-users adapt their energy consumption patterns based on personal preferences, which are not disclosed to operators. Consequently, operators encounter two primary challenges in resolving the proposed model: (1) devising an online V2G scheduling strategy that navigates uncertainties in electricity prices and EV arrival/departure patterns and (2) orchestrating the harmonious integration of diverse energy sources within the microgrid to minimize overall operational costs. The formulation of the proposed model employs bilevel programming techniques, which will be elaborated upon in subsequent sections.

2.2. The Upper-Level Problem

The upper level pertains to the optimal operations of microgrids with electricity, heat, and gas demands. The dynamic pricing set by operators from the perspective of MMOs. Specifically, the objective function aims to minimize the total operational cost and is defined as follows:

$$\min {\displaystyle \sum _{t\in T}\left({\lambda }_t^{e,in}{p}_t^{in}+{\lambda }_t^{g,in}{g}_t^{in}-{\lambda }_t^{e,out}{p}_t^{out}-{\displaystyle \sum _{k\in K}\left({\lambda }_t^{k,e}{p}_t^k+{\lambda }_t^{k,g}{g}_t^k+{\lambda }_t^{k,h}{h}_t^k\right)}\right)}$$

(1)

$${\lambda }_t^{e,in}\le {\lambda }_t^{k,e}\le {\lambda }_t^{e,\max },\forall t,k$$

(2)

$${\lambda }_t^{g,in}\le {\lambda }_t^{k,g}\le {\lambda }_t^{g,\max },\forall t,k$$

(3)

$${\lambda }_t^{h,in}\le {\lambda }_t^{k,h}\le {\lambda }_t^{h,\max },\forall t,k$$

(4)

where ${\lambda }_t^{e,in}$, ${\lambda }_t^{e,out}$ and ${\lambda }_t^{g,in}$ represent the electricity purchase, selling prices, and gas purchase price at period $t$. $p_t^{in}$, $g_t^{in}$ and $p_t^{out}$ denote the purchasing power, the generation power of the gas turbine, and the selling power to the upper-level grid. In addition, $p_t^k$, $g_t^k$ and $h_t^k$ are the electricity, gas and heat demands at period $t$ of end user $k$. These parameters are the feedback from the end users at the lower level based on different preferences for the prices of electricity ${\lambda }_t^{k,e}$, gas prices ${\lambda }_t^{k,g}$ and heat prices ${\lambda }_t^{k,h}$ set by MMOs. The relationships among the prices above can be formulated as constraints (2)–(4), which impose limits on the various demand pricing ranges set by MMOs.

Consequently, end-users will respond to these prices according to their preferences, adjusting their consumption behavior accordingly. Thus, the formulation of balance constraints of different energy sources can be expressed as follows:

$$p_t^{in}-p_t^{out}+p_t^g+p_t^{PV}+{\displaystyle \sum _i{p}_t^{i,\mathrm{V}2\mathrm{G}}}={\displaystyle \sum _k{p}_t^k}+p_t^h,\forall t$$

(5)

$$h_t^p+h_t^g={\displaystyle \sum _k{h}_t^k},\forall t$$

(6)

$$g_t^{in}=g_t^g+{\displaystyle \sum _k{g}_t^k},\forall t$$

(7)

where constraints (5)–(7) ensure the balances between demands and supply of electricity, heat, and gas, respectively. Specifically, $p_t^{i,\mathrm{V}2\mathrm{G}}$ is the V2G power of EV i. $p_t^g$ and $p_t^{PV}$ are the generation power of the gas turbine and the uncertain output of PV. In addition, $p_t^h$ is the consumption power of electric heat pump. As for the balance constraints of heat and gas, $h_t^p$ and $h_t^g$ describe the heat output by the gas turbine and the electric heat pump, while $g_t^{in}$ and $g_t^g$ are the procured and used gas. The decision variables are subject to the following constraints:

$$p_t^{g,\min }\le p_t^g\le p_t^{g,\max },\forall t$$

(8)

$$0\le p_t^{PV}\le {\overline{p}}_t^{PV},\forall t$$

(9)

$$0\le p_t^h\le p_t^{h,\max },\forall t$$

(10)

$${\eta }^h{p}_t^h=h_t^p,\forall t$$

(11)

$${\eta }_g^h{g}_t^g=h_t^g,\forall t$$

(12)

$${\eta }_g^p{g}_t^g=p_t^g,\forall t$$

(13)

where constraints (8) describe the upper and lower limits of gas turbine generation $p_t^{g,\min }$ and $p_t^{g,\max }$. Constraints (9) describe the PV actual output ${\overline{p}}_t^{PV}$. Constraints (10) limit the limits of generation out of electric heat pump. Constraints (11)–(13) describe the relation between the energy conversion process, where ${\eta }^h$, ${\eta }_g^h$ and ${\eta }_g^p$ denote the corresponding conversion efficiency. In specific, ${\eta }^h$ is the efficiency between electricity converting into heat, while ${\eta }_g^h$ and ${\eta }_g^p$ denote the conversion ratio of different energy sources of the gas turbine.

In addition, EV i is required to charge its battery to a predetermined State of Charge (SoC) level before departing. It is assumed that during the parking intervals, MMOs have access to chargers and EVs, enabling them to regulate the charging power and switch between charging and discharging modes to facilitate V2G scheduling. This capability allows MMOs to achieve additional reductions in operational costs and enhance overall energy efficiency. The corresponding SoC constraints can be formulated as follows:

$$L_t^i=L_{t-1}^i+{\eta }^{i,ch}{p}_t^{i,ch}-p_t^{i,dis}/{\eta }^{i,dis},\forall i,t$$

(14)

$$L_t^{i,\min }\le L_t^i\le L_t^{i,\max },\forall i,t$$

(15)

$$0\le P_t^{i,ch}\le P_t^{ch,\max },\forall i,t$$

(16)

$$0\le P_t^{i,dis}\le P_t^{dis,\max },\forall i,t$$

(17)

where $L_t^i$ denotes the SoC level of EV i at period $t$. $L_t^{i,\min }$ and $L_t^{i,\max }$ denote the minimum and maximum levels of SoC, respectively. ${\eta }^{i,ch}$ and ${\eta }^{i,dis}$ are the charging and discharging efficiency of EV i. $p_t^{i,ch}$ and $p_t^{i,dis}$ are the charging and discharging power of EV i at period $t$, which are bounded by the charging and discharging power maximum limits $P_t^{ch,\max }$ and $P_t^{dis,\max }$. Constraints (14) describe the relationship between the SoC level and charging/discharging power. Constraints (15)–(17) denote the upper and lower limits of the EV states. Based on that, the total power of V2G can be represented as follows:

$$p_t^{i,\mathrm{V}2\mathrm{G}}=p_t^{i,dis}-p_t^{i,ch},\forall t,i$$

(18)

2.3. The Formulation of Lower-Level Problem

After MMOs establish the V2G scheduling and set prices for multi-energy demands, end-users would respond to these price signals by adapting their consumption patterns according to their preferences, aiming to maximize their welfare. Subsequently, MMOs monitor the multi-energy demands and make additional operational decisions. The specifics of the reaction of the kth end users in the lower-level problem can be formulated as follows:

$$\underset{p_t^k,g_t^k,h_t^k}{\max }{\displaystyle \sum _{t\in T}\left(U\left(p_t^k,g_t^k,h_t^k\right)-{\lambda }_t^{k,e}{p}_t^k-{\lambda }_t^{k,g}{g}_t^k-{\lambda }_t^{k,h}{h}_t^k\right)}$$

(19)

$$U\left(p_t^k,g_t^k,h_t^k\right)=a_t^{k,p}{\left(p_t^k\right)}^2+b_t^{k,p}{p}_t^k+a_t^{k,g}{\left(g_t^k\right)}^2+b_t^{k,g}{g}_t^k+a_t^{k,h}{\left(h_t^k\right)}^2+b_t^{k,h}{h}_t^k,\forall t,k$$

(20)

$$0\le p_t^k\le p_t^{k,\max },\forall k,t$$

(21)

$$0\le g_t^k\le g_t^{k,\max },\forall k,t$$

(22)

$$0\le h_t^k\le h_t^{k,\max },\forall k,t$$

(23)

where constraints (20) represent the welfare expression of kth end user, and $a_t^{k,p}$, $b_t^{k,p}$, $a_t^{k,g}$, $b_t^{k,g}$, $a_t^{k,h}$, and $b_t^{k,h}$ denote the preference parameters that significantly impact the responses of end-users to the prices of different energy prices. Furthermore, constraints (21)–(23) delineate the upper limits on the energy consumption of various energy sources. Notably, parameters such as preferences information of end-users and energy prices remain private and undisclosed to MMOs. This confidentiality constraint underscores the necessity for developing an MDP-based formulation, which will be discussed in subsequent sections.

3. MDP Formulation for MMOs Online V2G Scheduling in Multi-Energy Microgrids

To devise the scheduling strategy for MMOs, the previously discussed bi-level programming is restructured as an MDP problem. This transformation facilitates the development of a DRL-based framework aimed at learning an effective online V2G scheduling strategy, considering the uncertainties surrounding EVs’ arrival and departure patterns, electricity prices, and the undisclosed end-users’ preference behavior patterns. The proposed framework is formulated within a finite MDP framework, empowering the DRL process to devise an online V2G scheduling that is synchronized with multi-energy microgrids.

Properly selecting states in an MDP is crucial, particularly in the context of online V2G scheduling, to achieve optimal performance and devise effective strategies. This importance becomes particularly pronounced when considering the substantial presence of EVs as distributed resources within the system. The choice of states in an MDP significantly influences the decision-making process of the reinforcement learning algorithm, impacting the quality of the resulting V2G scheduling strategy. Specifically, in the case of a large number of EVs acting as distributed resources, the states should adequately capture the dynamic nature of EV-related parameters such as arrival and departure patterns, battery state of charge, charging and discharging rates, and EV owners’ preferences and behaviors. By accurately defining these states, the MDP-based framework can learn and adapt to the complex and uncertain environment of V2G scheduling, leading to improved performance, enhanced energy efficiency, and better coordination of multi-energy microgrids with EVs as integral components.

In our proposed MDP formulation, the MMOs receive information regarding the remaining SoC EVs required $L_t^{i,re}$, remaining charging time $T^{i,re}$, electricity and gas prices ${\lambda }_t^{e,in}$, ${\lambda }_t^{e,out}$ and ${\lambda }_t^{g,in}$. Given this information and the state at previous time periods, the DRP constructs the tensor of $s_t$. On that basis, the neural network of the MMOs determines its action $a_t$. Subsequently, this information is sent to each end-user at the corresponding time period. Upon receiving the dynamic pricing information, each end-user formulates its own optimization problem and adjusts its response accordingly. MMOs then calculate the total operation costs based on end users’ responses. Subsequently, MMOs observe the next state and generate an action. This iterative process continues in an online fashion until MMOs’ scheduling and pricing strategy terminates.

3.1. States

The upper level of the bi-level model describes optimal online scheduling of V2G in microgrids with different energy sources and time-varying price settings of an MMO. In specific, the objective function is to minimize the total operation cost as follows:

The states of MMOs include remaining SoC EVs required $L_t^{i,re}$, remaining charging time $T^{i,re}$, the selling and purchase of electricity prices and gas prices ${\lambda }_t^{e,in}$, ${\lambda }_t^{e,out}$ and ${\lambda }_t^{g,in}$, at previous $N$ and $M$ periods, listed below:

$$s_t=\left[{\left(L_t^{i,re},T^{i,re}\right)}_{i=1}^I,{\left(p_{t-n}^k,g_{t-n}^k,h_{t-n}^k\right)}_{k=1,n=1}^{\#K,\#N},{\left({\lambda }_{t-m+1}^{e,in},{\lambda }_{t-m+1}^{e,out},{\lambda }_{t-m+1}^{g,in}\right)}_{m=1}^{\#M}\right],t\in T$$

(24)

3.2. Actions

Receiving the states $s_t$, the MMOs determine an action, denoting the online V2G scheduling and time-varying prices of different energy demands, which are sent to each end user $k$:

$$a_t=\left[{\left(p_t^{i,\mathrm{V}2\mathrm{G}}\right)}_{i=1}^{\#I},{\left({\lambda }_t^{k,e},{\lambda }_t^{k,g},{\lambda }_t^{k,h}\right)}_{k=1}^{\#K}\right],\hspace{1em} \forall t$$

(25)

3.3. Reward

In response to the action of MMOs, EVs perform the charging and discharging orders, and end users determine the energy usage plan, resulting in the total operation cost. The reward signals can be designed and expressed as follows:

$$r_t=-\left({\lambda }_t^{e,in}{p}_t^{in}+{\lambda }_t^{g,in}{g}_t^{in}-{\lambda }_t^{e,out}{p}_t^{out}-{\displaystyle \sum _{k\in K}\left({\lambda }_t^{k,e}{p}_t^k+{\lambda }_t^{k,g}{g}_t^k+{\lambda }_t^{k,h}{h}_t^k\right)+C_t^{EV}}\right),\forall t$$

(26)

where $C_t^{EV}$ is associated with the penalty of not satisfying the required SoC when EV leaves at period $t$, which can be formulated as follows:

$$C_t^{EV}={\displaystyle \sum _i{\lambda }^{pe}}\max \left\{0,\left(L_t^i-L_t^{set}\right)\right\},\forall t$$

(27)

3.4. Transition Function

States transition from $s_t$ to $s_{t+1}$ is determined by action $a_t$ of MMOs and inaccessible information of end users $k$ as follows:

$$s_{t+1}=f\left(a_t,s_t,\omega \right)$$

(28)

where $\omega$ denotes the exogenous randomness information. In particular, the MMOs’ scheduling will explicitly influence the EVs’ SoC.

4. SAC Algorithm for Solving MDP Formulation

In this section, the SAC algorithm is introduced as a method to approximate the solution to the online V2G scheduling challenge. This approach enables the algorithm to obtain an optimized pricing strategy adaptively.

4.1. Preliminaries

The SAC algorithm represents a maximum-entropy off-policy DRL approach renowned for its enhanced sample efficiency and robust training outcomes. Diverging from conventional deep Q networks (DQN), SAC implements soft-version updates to the Q-network, fostering exploration across the action space. Moreover, the agent also receives an entropy bonus that encapsulates the policy’s stochastic nature, mitigating premature convergence towards suboptimal solutions. Consequently, the training objective aims to maximize the cumulative reward (or minimize the total cost) expressed as:

$${\pi }_{\theta }^{*}=\arg \underset{\theta }{\max }{\displaystyle \sum _{t=1}^T{\gamma }^t\mathbb{E}}\left[r_t\left(s_t,a_t\right)+\alpha H\left({\pi }_{\theta }\left(s_t\right)\right)\right]$$

(29)

where ${\pi }_{\theta }$ denotes the policy network $\theta$, and $\gamma$ indicates the discount factor. $H(\cdot )$ is the entropy expression and $\alpha$ is the temperature factor of the entropy term.

4.2. Training Process

Figure 2 illustrates the structural layout of the SAC algorithm: the actor network, the critic network, and the temperature parameter network. The critic network is characterized by $\phi$ and is trained to implement the soft Bellman optimality equation. In order to mitigate the issue of overestimation of Q values, the clipped double DQN approach is implemented, utilizing target networks parameterized by $\overline{\phi }$.

$$y=\underset{i=1,2}{\min }{r}_t+\gamma \left(Q^{{\overline{\phi }}_i}\left(s_{t+1},a_{t+1}\right)-\alpha \log {\pi }_{\theta }\left(a_{t+1}|s_{t+1}\right)\right)$$

(30)

$$J\left({\phi }_i\right)=\underset{(s_t,a_t,r_t,s_{t+!})~B}{\mathbb{E}}\left[{\left(Q^{{\phi }_i}\left(s_t,a_t\right)-y\right)}^2\right],i=1,2$$

(31)

where $y$ is the update value of two q functions, and $\log {\pi }_{\theta }(\cdot )$ indicates the policy entropy:

$$a_{\theta }=\tanh \left({\mu }_{\theta }+\epsilon {\sigma }_{\theta }\right),\epsilon ~N$$

(32)

$$J\left(\theta \right)=\underset{\theta }{\max }\underset{\epsilon ~N}{\mathbb{E}}\left[\underset{i=1,2}{\min }{Q}^{{\phi }_i}\left(s_t,a_{\theta }\right)-\alpha \log {\pi }_{\theta }\left(a_{\theta }|s_t\right)\right]$$

(33)

Hence, the entropy automated adjusting method is applied to adaptively adjust the $\alpha$ via minimizing the expression as follows:

$$J\left(\alpha \right)=\underset{\epsilon ~N}{\mathbb{E}}\left[-\alpha \log {\pi }_{\theta }\left(a_{\theta }|s_t\right)-\alpha \overline{H}\right]$$

(34)

where the target entropy $\overline{H}$ is set as the negative of the action dimension. In addition, target networks are applied to smooth the approximation of q functions. Based on that, the detailed training process of the SAC algorithm is presented in Algorithm 1.

Algorithm 1 The Proposed DRL-based Online V2G in Multi-energy Microgrids with SAC
1:	Initialize replay buffer $B$
2:	Initialize actor $\theta$, critic ${\phi }_i$, $\alpha$ and target network ${\overline{\phi }}_i$
3:	for each epoch do
4:	for each state transition step do
5:	Given $s_t$, take actions $a_t$ based on (32)
6:	Observe the multi-energy demands (19) with $a_t$ as V2G scheduling and       multi-energy prices
7:	Solve the scheduling model and obtain operation costs
8:	Receive $r_t,s_{t+1}$ and record them in buffer $B$
9:	end for
10:	for each gradient step do
11:	$\theta \leftarrow \theta -{\lambda }_{\theta }{\nabla }_{\theta }J\left(\theta \right)$
12:	${\phi }_i\leftarrow \theta -{\lambda }_{{\phi }_i}{\nabla }_{{\phi }_i}J\left({\phi }_i\right)$
13:	$\alpha \leftarrow \alpha -{\lambda }_{\alpha }{\nabla }_{\alpha }J\left(\alpha \right)$
14:	${\overline{\phi }}_i\leftarrow \tau {\phi }_i-(1-\tau ){\overline{\phi }}_i$
15:	end for
16:	end for

5. Case Studies and Discussion

In this section, we conducted different case studies to verify the effectiveness of the proposed DRL-based framework for online V2G scheduling in microgrids with different energy sources. First, the training process of the policy network in SAC that approximates the optimal online V2G scheduling and multi-energy pricing strategy is shown. Then, we compare the V2G power of the proposed approach and optimal values solved in hindsight. Finally, we illustrate the ability of the proposed approach to approximate the SoC constraints given different arrival and departure times to showcase its huge capacity in the real-world V2G application. The case studies are composed of two parts with five EVs and ten EVs in microgrids which are denoted as Case I and Case II, respectively. In specific, the patterns of EVs’ arrival and departure are sampled from the given distributions, as shown in

Table 1. Probability of EVs’ hourly arrival.

Hour	Probability of Arrival
1	0.070
2	0.070
3	0.062
4	0.060
5	0.023
6	0.033
7	0.050
8	0.060
9	0.060
10	0.050
11	0.040
12	0.030
13	0.030
14	0.040
15	0.040
16	0.060
17	0.040
18	0.060
19	0.040
20	0.040
21	0.030
22	0.005
23	0.005
24	0.002

and

Table 2. Probability of EVs plug-in intervals.

Lasting Hours	Probability
1	0.00
2	0.10
3	0.15
4	0.20
5	0.15
6	0.15
7	0.13
8	0.05
9	0.05
10	0.02

. Additionally, the electricity prices are selected from the open data of the New York Independent System Operator (NYISO). Specifically, the electricity prices of the NYC zone on 1 March 2023 are used.

5.1. Training Process and Results of Case I

This part presents the outcomes derived from the extensive training regimen of neural networks involving 5 EVs conducted over a span of 5000 epochs, as delineated in Figure 3. The efficacy of the DRL-based online V2G framework underwent scrutiny via the computation of the ratio between the optimal returns yielded by the bi-level model and those achieved by the neural network. It is imperative to note that the bi-level model benefitted from complete access to EV arrival and departure times as well as end users’ private preferences, contrasting with the proposed DRL-based approach that operates without any prior information.

The graphical representation illustrates a substantial variance in the neural network’s ratio, ranging between 0.20 and 0.65 during the initial 1500 training epochs owing to the Stochastic Actor-Critic (SAC) algorithm. This algorithm, designed to foster exploration, inclines the neural network towards selecting actions with elevated entropy to traverse the action space comprehensively. Such actions, albeit infrequently chosen, contribute to refining the neural network’s accuracy in estimating global values. Consequently, the policy neural network initiates the adoption of an optimal strategy grounded on the current fitting of the Q-function. As a result, the ratio rapidly increases to 0.90 between the 3500th and 5000th training epochs, indicating the neural network’s remarkable ability to approximate the optimal online V2G scheduling. After the 4000th training epoch, the neural network’s performance stabilizes, converging around 0.95.

The effectiveness of the proposed approach in approximating optimal values for V2G power management is evident from the charging and discharging strategy shown in Figure 4. The analysis of calculated mean absolute error (MAE) values reveals that the proposed approach approximates optimal values, showcasing its robustness and accuracy in handling charging and discharging power operations. Specifically, the MAE for charging power approximates 0.1029, indicating a relatively small deviation from the optimal values. Similarly, the MAE for discharging power is even lower at approximately 0.0701, underscoring the proposed approach’s exceptional ability to approximate optimal values with high precision. This close alignment is further evidenced by the shape of the charging and discharging power curves, which exhibit a significant resemblance to the optimal curves. The negligible discrepancies observed in the MAE values reinforce the notion that the proposed approach accurately captures the intricacies of V2G power dynamics, resulting in effective and reliable management strategies for energy systems optimization.

The analysis of the net purchase data for gas and electricity reveals distinct patterns indicative of strategic decision-making in energy procurement, as shown in Figure 5. Notably, during periods of low electricity prices, such as between 0:00 and 3:00, the net purchase of electricity peaks while gas purchases decline. Conversely, during daytime hours, when electricity prices are higher, there is a corresponding increase in gas purchases and a reduction in electricity procurement. This strategic shift in purchasing behavior aligns with optimizing the utilization of gas turbine power generation, effectively leveraging the complementary characteristics of multi-energy microgrids. By prioritizing electricity purchases during off-peak hours and strategically adjusting gas procurement based on market dynamics, the system efficiently meets the demand for power generation while minimizing costs. These findings underscore the adaptive and responsive nature of energy procurement strategies, showcasing the effectiveness of leveraging diverse energy sources within microgrid systems.

The analysis of State of Charge (SoC) constraints for EVs under the proposed approach reveals a commendable performance in meeting desired SoC levels, as shown in Figure 6. The data from 5 EVs, with SoC values ranging from approximately 0.793 to 0.810, showcases the effectiveness of the proposed approach in maintaining SoC close to the optimal target of 0.8. The proximity of the observed SoC values to the desired 0.8 thresholds indicates a high degree of compliance with SoC constraints, reflecting the precision and accuracy of the proposed approach in managing EV battery states. This capability is crucial for optimizing EV operations and ensuring optimal battery utilization while meeting performance requirements. Overall, the results highlight the effectiveness of the proposed approach in successfully managing SoC constraints for EVs, contributing to efficient and reliable EV operation within energy systems.

5.2. Training Process and Results of Case II

This part presents the outcomes derived from the extensive training regimen of neural networks involving 10 EVs conducted throughout 5000 epochs, as shown in Figure 7. The training trajectory of the proposed methodology, under the guidance of the SAC algorithm, manifests a gradual evolution towards heightened performance, substantiated by meticulous ratio curve analysis. Throughout the exploration phase, the ratio demonstrates a dynamic variation ranging from approximately 0.1 to 0.7, portraying the algorithm’s active engagement in exploratory learning processes. However, a discernible upward surge in the ratio becomes apparent with progressive training, achieving notable peaks reaching up to 0.90 between the 3500th and 5000th training epochs. This steep ascent underscores the algorithm’s adeptness in refining policies and executing judicious decisions, resulting in a substantial performance enhancement. After the 4000th training epoch, the ratio stabilizes at approximately 0.90, denoting a consistent and elevated level of performance closely aligned with optimal objectives. Compared with the training process in Case I, the neural network exhibits the ability to adapt effectively to the scenario with more EVs. The discerned pattern within the ratio curve serves as a testament to the DRL-based algorithm’s efficacy in fostering efficacious learning processes, adaptive behaviors, and performance amplification within the proposed approach.

In addition, the SAC algorithm is compared with Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) to illustrate its superiority in solving online V2G problems, as shown in Figure 7. It can be observed that SAC achieves a higher ratio than DDPG and TD3 thanks to its exploration policy, which is driven by the maximum entropy mechanism. In online V2G scheduling, the policy is challenged by the high dimensions of action space, and SAC is capable of efficiently updating the temperature factors and fine-tuning the policy network. Specifically, the ratio achieved by SAC is 103.5% and 66.1% higher than that of DDPG and TD3.

The efficacy of the proposed methodology in approximating optimal values for V2G power management is demonstrated in Figure 8. The MAE assessment reveals an effective approximation between the proposed approach and optimal values, highlighting its robustness and precision in managing charging and discharging power operations. Specifically, the MAE for charging power approximates 0.1382, indicating a minor deviation from optimal values. Similarly, the MAE for discharging power is even lower at approximately 0.1014, underscoring the remarkable accuracy of the proposed approach in approximating optimal values with exceptional precision. This alignment is further substantiated by the striking similarity between the charging and discharging power curves and the optimal curves, further reinforcing the accuracy of the proposed approach in capturing the complexities of V2G power dynamics. These findings underscore the effectiveness of the proposed methodology in devising efficient and reliable management strategies for optimizing energy systems.

Examining gas and electricity net purchase data reveals discernible patterns that reflect strategic decision-making in energy procurement, as depicted in Figure 9. Particularly noteworthy is the observed trend during periods of low electricity prices, notably between 0:00 and 3:00, where electricity procurement peaks while gas purchases diminish. In contrast, as daytime ensues and electricity prices rise, there is a corresponding surge in gas procurement coupled with a reduction in electricity acquisition. This strategic realignment of procurement strategies aligns with optimizing the utilization of gas turbine power generation, effectively capitalizing on the synergistic attributes of multi-energy microgrids. The strategic emphasis on electricity procurement during off-peak intervals, alongside the adept adjustment of gas acquisition in response to market dynamics, ensures efficient meeting of power generation demands while mitigating costs. These observations underscore the adaptive and responsive nature inherent in energy procurement strategies, highlighting the efficacy of integrating diverse energy sources within microgrid systems.

The assessment of SoC constraints for EVs within the proposed framework demonstrates commendable performance, as illustrated in Figure 10. The dataset comprising SoC values from 10 EVs, ranging between approximately 0.783 and 0.819, underscores the effectiveness of the proposed approach in maintaining SoC levels near the optimal target of 0.8 at the period of departure. This proximity to the desired SoC threshold signifies a high level of adherence to SoC constraints, indicative of the precision and accuracy embedded in the proposed methodology for managing EV battery states. This capability is of significant importance in optimizing EV operations and ensuring the optimal utilization of batteries while meeting performance benchmarks. In summary, these findings affirm the effectiveness of the proposed approach in adeptly managing SoC constraints for EVs, thus enhancing the efficiency and reliability of EV operations within energy systems.

5.3. Tests on the Robustness and Efficiency of the SAC Algorithm

In this section, we further testify to the robustness and efficiency of SAC algorithms under different levels of uncertainties. We first define the level of uncertainties as the ratio of variances and mean of the random variables. Specifically, the uncertainties of electricity prices and the various demands of end users are considered. We note that only the uncertain patterns of EV arrivals and departures are included in the training. The detailed performance of trained SAC under different levels of uncertainties is shown in Figure 11. The range of total profits becomes wider when the levels of uncertainties grow. Nonetheless, the average total profits slightly drop from $318.3 to $313.6 by 1.45%, which demonstrates the robustness and efficiency of the SAC algorithm.

5.4. Tests on the Scenarios with Different Numbers of EVs

In this section, we increase the number of EVs that participate in the online V2G scheduling to further validate the effectiveness of the SAC algorithm, as illustrated in Figure 12. It can be observed that the increase in the number of EVs will have an impact on the performance of the SAC algorithm due to the growing dimension of action space. With the same number of training epochs, the policy network has less chance to find the optimal V2G scheduling strategy. Hence, the ratio which represents the effectiveness of SAC will decrease with the rise in the number of EVs. Specifically, compared with the scenario with 10 EVs, the ratios drop by 5.1%, 6.6%, 11.4%, and 18.9% in the scenarios with 15, 20, 25, and 30 EVs, respectively. We note that the above comparison is conducted with the same size of policy neural network. With the larger capacities of neural networks, the performance will become better.

5.5. The Daily Battery Profiles of EVs

In this section, we testify the daily battery profiles of EVs to illustrate the effectiveness of the proposed approach. In specific, the daily battery profiles of five and ten EVs are shown in Figure 13 and Figure 14, respectively. It can be observed that, on the one hand, the online V2G scheduling controlled by the SAC algorithm will adapt to the EVs’ arrival and departure patterns and regulate the SoC levels to satisfy the charging requirements. In addition, SAC will utilize the capability of V2G to increase profits via charging at periods with lower electricity prices and discharging at periods with higher prices. When the departure periods are close, SAC will adjust SoC to the predefined level (80%).

6. Conclusions

In this paper, we introduce a novel approach for online Electric Vehicles (EVs) Vehicle-to-Grid (V2G) scheduling coordinated with multi-energy microgrids, leveraging Deep Reinforcement Learning (DRL) techniques. Our proposed framework effectively tackles challenges arising from uncertainties such as fluctuating electricity prices and EV arrival/departure patterns. By formulating the scheduling problem as a Markov Decision Process (MDP) and employing a Soft Actor-Critic (SAC) algorithm, we achieve dynamic and optimized scheduling of EV charging and discharging activities in response to real-time grid conditions and energy demand fluctuations. The results from extensive simulations in the case studies demonstrate the efficacy of our approach. We observed significant cost reductions for Multi-Model Operators (MMOs) due to improved coordination of V2G scheduling and multi-energy flexible loads. The average deviation from optimal V2G schedules was minimized, leading to enhanced profitability and sustainability in microgrid operations. Furthermore, our approach exhibited robustness and adaptability in handling uncertainties, showcasing its ability to dynamically adjust scheduling strategies based on evolving grid conditions and energy demand patterns. Overall, our work contributes to advancing smart grid technologies and promoting efficient utilization of renewable energy resources. The intelligent and adaptive online V2G scheduling strategies proposed in our framework have the potential to revolutionize energy management practices, unlocking the potential for more resilient and environmentally sustainable energy infrastructure.

For future work, we aim to expand our research in several directions. Firstly, we plan to investigate the integration of advanced forecasting techniques, such as machine learning-based prediction models, to enhance the accuracy of demand forecasting and improve the adaptability of our scheduling framework. Additionally, we will explore the incorporation of demand response mechanisms and energy storage technologies to further optimize energy utilization and grid stability. Moreover, we seek to evaluate the scalability and deployment feasibility of our approach in real-world multi-energy microgrid environments, considering factors like communication protocols, computational efficiency, and scalability.