Deep Q-Network Based Optimal Charging Coordination of Electric Vehicles Considering Vehicle-to-Grid Technology

Highlights

• V2G technology is employed in optimal charging coordination.
• MDP is employed for optimal scheduling of charging coordination of electric vehicles.
• DQN is employed for multi-cycle global decision optimization.

Abstract

To further enhance the active participation of electric vehicles in grid interaction and reduce the decision-making costs for electric vehicle aggregators, this paper addresses the challenges in current EV charging and V2G (Vehicle-to-Grid) management. Considering the owners’ willingness to participate, an optimal charging and V2G model for EV charging stations based on a Deep Q-Network is established. The paper analyzes in detail the mutual influence between the level of EV owner participation and the strategies of EV aggregators. Based on the owners’ willingness and the physical constraints of the EVs, an evaluation metric for EV participation in charging scheduling is developed. The Deep Q-Network is employed to make decisions regarding EV participation, thereby enhancing the decision-making capability of the EV aggregator, reducing the instability of its scheduling plans, and improving the reliability of these plans. Simulation results demonstrate that this method can dynamically consider EV owners’ willingness to participate, adaptively optimize the scheduling margin ratio, make global decisions across multiple time periods, and formulate charging and V2G scheduling plans for the EV aggregator.

1. Introduction

In recent years, the surge in the number of electric vehicles (EVs) and their large-scale integration into the power grid have imposed significant demands on grid operation and control. Within this context, optimal control technologies for EV charging and discharging have garnered considerable attention. Among these, private EVs, characterized by their short daily usage time and relatively fixed routines [1], have emerged as excellent controllable resources. Therefore, by controlling the intraday V2G of private EVs, the pressure exerted on the power grid by their large-scale integration can be alleviated.

Reference [2] proposes that although the battery capacity of a single EV is relatively small, the aggregation of multiple EVs can create substantial energy resources. Consequently, an EV aggregator can be considered as a single scheduling entity, establishing a hierarchical dispatch system from the “secondary dispatch center to the aggregator to the EVs,” thereby enabling centralized and refined management of EV charging and discharging. In this model, the EV aggregator acts as the core bridge connecting the grid and EV owners. The scientific rigor and flexibility of its scheduling decisions directly impact the stability of grid operation and the V2G experience of owners. Simultaneously, they determine the enthusiasm of EVs to participate in grid interaction and the aggregator’s own operational costs.

Currently, most research on EV charging and V2G scheduling is confined to upper-level dispatch, focusing on the coordinated control between EV aggregators and the power grid. Reference [3] proposes that EV aggregators, acting as auxiliary service providers, can offer frequency regulation services to the grid, thereby improving communication delays with the upper-level dispatch and enhancing their coordinated control capabilities with the grid. Reference [4] comprehensively evaluates the role of EV aggregators in promoting renewable energy consumption under grid congestion caused by renewable energy generation, integrating the entire EV aggregator as an energy storage facility into grid dispatch control. References [5,6] utilize the flexible scheduling of EV charging and discharging by aggregators to smooth supply–demand fluctuations and increase the scale of grid demand response. Reference [7] leverages the flexibility of EV aggregators in charging and discharging to address congestion and voltage issues in distribution networks, validated using various node models. Building upon the physical operational requirements, references [8,9] further emphasize the critical importance of cyber-physical security within the V2G communication infrastructure, specifically highlighting how vulnerabilities in EV-to-aggregator channels can be exploited through sophisticated false data injection attacks. Reference [10] primarily focuses on EV aggregators engaging in energy arbitrage in the energy market, providing auxiliary services from aggregated EVs while offering charging services to EV owners. References [11,12] investigate the stochastic nature of traffic conditions, the flexibility of EV drivers, and the presence of the wholesale electricity market as constraints for EV aggregators. They formulate dynamic electricity prices for aggregators, treating the aggregator as a holistic entity to prevent excessive pressure on the grid during peak periods.

However, the aforementioned studies all focus on the EV aggregator as a whole and do not analyze the relationship between the aggregator and individual EV owners. In reality, however, an EV aggregator must decompose its control down to each electric vehicle. As shown in Figure 1, EV owners and the aggregator exchange information, collectively responding to grid dispatch instructions. Given that the schedulable time windows and capacities submitted by each EV vary, the absence of a scientific scheduling mechanism can easily lead to situations where some vehicles are not utilized during their available periods, while other periods face resource scarcity due to concentrated vehicle responses, and remaining periods suffer from capacity idleness. Therefore, it is necessary to delve into the charging and discharging priority among the EV fleet. By optimizing the response scheduling for each vehicle, an efficient allocation of available capacity can be achieved, thereby effectively supporting the attainment of EV charging and V2G dispatch objectives.

Figure 1. Schematic diagram of electric vehicle-grid interaction classification.

References [13,14] model the interaction between individual EV owners and the aggregator as a game, using profit as the primary metric, but they fail to consider the proactive declarations of EV owners. However, current charging and discharging scheduling at EV charging stations still faces numerous pressing issues, with the core contradiction lying in the imbalance between the safe operation requirements of the power grid and the personalized charging/discharging preferences of owners. On one hand, the grid side requires aggregators to ensure grid stability through strategies such as peak-load shifting and load smoothing. On the other hand, owners’ charging and V2G action needs are dynamically influenced by factors like travel plans, charging costs, and willingness to participate. If scheduling strategies only consider physical constraints such as battery capacity and charging/discharging power, while neglecting the dynamic fluctuations in owners’ willingness to participate, it can easily lead to a disconnect between the scheduling plan and the actual needs of owners. This results in low owner participation and poor execution rates of the scheduling plan, subsequently increasing the aggregator’s decision-making costs, destabilizing the scheduling plan, and reducing its credibility. Furthermore, the EV V2G scheduling scenario is characterized by its multi-period, multi-vehicle, and multi-constraint complexity. Although traditional optimization methods (e.g., linear programming, genetic algorithms) have been applied in simple scenarios, they struggle to handle dynamically changing complex scheduling problems and cannot achieve globally optimal decisions across multiple time periods.

Existing scheduling strategies based on reinforcement learning are mostly developed from the aggregator’s perspective, focusing on cluster-level energy optimization and grid interaction objectives. Reference [15] proposes an optimal charging/discharging scheduling strategy for an EV cluster based on reinforcement learning, aimed at enhancing the aggregator’s economic benefits and regulation response capability in grid interaction. However, such methods typically treat the aggregated EVs as homogeneous resources during modeling, neglecting the dynamic changes in individual owners’ willingness to participate and their impact on the executability of the scheduling plan. Xi et al. [16] conducted research on the assessment and enhancement of EV hosting capacity in distribution networks, which provides a core basis for optimizing the grid carrying capacity under the large-scale integration of EVs and clarifies the basic constraints of distribution network-side on EV scheduling. Wang et al. [17] proposed a service-quality-based EV charging pricing strategy from the perspective of smart energy communities, integrating user experience with pricing mechanisms and offering a market-oriented solution to stimulate the enthusiasm of vehicle owners to participate in grid interaction. Ge et al. [18] designed a flexible energy scheduling strategy for fast charging stations considering EV vehicle-to-grid (V2G) operation and photovoltaic (PV) output uncertainty, which provides an important reference for multi-source coordinated scheduling at the charging station level and also highlights the application value of scheduling methods accounting for multiple uncertainties in practical scenarios. Consequently, the generated scheduling strategies struggle to accurately reflect the coupling relationship between user behavioral intentions and aggregator decisions, and their decision-making accuracy and adaptive capacity still require improvement.

Reference [19] presents a probabilistic assessment method to estimate the aggregated household charging load of an EV fleet at the sub-transmission level under a 100% electrification scenario. It evaluates the overload risk of primary substation transformers through probabilistic power flow simulations. Using an actual power grid in southern Sweden as a case study, combined with optimal available vehicle and traffic data, the authors conduct an empirical analysis to quantify the potential impact of large-scale home charging on the upstream grid. However, its focus is on the physical impact assessment of load aggregation, and further exploration is needed into optimal control strategies that leverage aggregator mechanisms to guide users towards orderly charging and alleviate transformer overloading pressure. To further address the aforementioned gaps—specifically the precise matching of V2G aggregation with dynamic owner willingness—contemporary studies have recently proposed sophisticated economic and physical boundary models. For instance, recent research [20] leverages option-based demand response contracts to economically manage the aggregation of EVs under uncertain conditions. Similarly, reference [21] formulates strategic siting and incentive pricing models to dynamically guide EV scheduling and stimulate participation based on quantified willingness. Furthermore, to bridge subjective user behavior with grid physical limits, methodologies such as those in [22] mathematically quantify the operational flexibility regions of fast-charging stations by explicitly incorporating EV user discharge willingness. While these advanced methodologies effectively capture the economic rationale and probabilistic behavior of EV owners, they often rely on indirect price stimulation or soft statistical boundaries, which may face execution uncertainty during real-time, critical grid scheduling.

Although existing research has preliminarily explored optimization methods for aggregated EV scheduling, significant gaps remain, which can be explicitly summarized into three aspects:

The gap between macroscopic grid dispatch and microscopic individual control: Most studies fail to simultaneously consider “grid physical constraints + owner willingness to participate + global optimization”.

The neglect of individual willingness and the assumption of EV homogeneity: There is a lack of an efficient scheduling strategy that can dynamically adapt to owners’ varying willingness and reduce aggregators’ decision-making costs.

The methodology has limitations in handling multi-period global optimization with strict discrete constraints; moreover, many studies utilize the entropy weight method for scheduling optimization, which incurs high computational costs and requires manual assistance.

In light of this, this paper addresses the optimal control problem for charging and discharging in electric vehicle charging stations. Considering the owners’ willingness to participate, a charging and discharging optimal control strategy based on a Deep Q-Network (DQN) is proposed. Firstly, the paper analyzes the interaction mechanism between EV owners’ willingness to participate and the aggregator’s scheduling strategy. By integrating the physical constraints of EVs with the owners’ willingness, a comprehensive evaluation metric for EV charging and V2G scheduling is constructed. Subsequently, intelligent decision-making for EV participation in charging and discharging is realized using a Deep Q-Network, enhancing the aggregator’s decision-making capability, reducing the instability of scheduling plans, and improving the reliability of the scheduling scheme. Finally, the effectiveness of the proposed strategy is validated through case study simulations. The method achieves global multi-period decision-making and dynamically optimizes the V2G scheduling margin ratio. It assists EV aggregators in formulating scientific, efficient, and executable charging and discharging schedules, facilitates the large-scale participation of electric vehicles in grid interaction, reduces the decision-making costs for aggregators, and promotes the coordinated and sustainable development of both the power grid and electric vehicles. The complete structural diagram is shown in Figure 2.

Figure 2. System optimization framework.

2. Factors Influencing Scheduling

2.1. Declared Information by EV Owners

In actual operation, when electric vehicle users submit their interaction plans, they need to specify the time of connection to and disconnection from the grid, the battery State of Charge (SOC) range upon disconnection, and their desired participation status (charging or discharging) during each time period. Theoretically, a single vehicle could declare both statuses for the same period. However, considering that grid dispatch has a clear directional requirement—meaning the vehicle fleet needs to uniformly execute either a charging or discharging instruction within a given period—this paper introduces a reasonable simplification to the declaration rules: A single electric vehicle can only choose to declare one status, either charging or discharging, for each time period. Nevertheless, within its entire connection period, the vehicle is allowed to switch its status across different time periods, and participation in every period is not mandatory. Furthermore, to ensure user travel convenience, vehicles can connect to the grid multiple times, further enhancing the flexibility of the interaction mode.

Based on the interaction plans submitted by users, the schedulable time periods and corresponding capacities for each electric vehicle can be determined: for charging, the schedulable capacity is the difference between the set upper limit of electricity and the current remaining energy; for discharging, it is the difference between the current remaining energy and the set lower limit of electricity. The upper and lower limits of electricity are jointly determined by the user-set off-grid SOC range and the battery capacity. Additionally, the EV aggregator needs to collect information on charging losses (these losses increase as the battery ages, leading to a decrease in schedulable capacity, as well as line losses during the actual charging process). During the declared schedulable time periods, users must keep their vehicles connected to the grid to be available for dispatch. The aggregator should establish an interaction history database for each vehicle, recording their historically declared schedulable time periods, schedulable capacities, and actual dispatch status, to provide data support for subsequent optimization.

To this end, this paper introduces the concept of electric vehicle user credibility from reference [18] to measure the extent to which EV users adhere to their declared scheduling plans, reflecting the randomness of user scheduling behavior. The calculation formula is:

$$\eta ={\displaystyle \frac{{\overline{C}}_b-{\overline{C}}_w}{{\overline{C}}_h}}$$

(1)

where ${\overline{C}}_h$ denotes the average historically declared capacity of the electric vehicle; ${\overline{C}}_b$ denotes the average actual dispatched capacity; and ${\overline{C}}_w$ denotes the average unscheduled capacity, i.e., the available capacity lost when the vehicle disconnects from the grid.

To reflect the urgency with which the remaining usable energy of an electric vehicle needs to be dispatched within its remaining declared time period, this paper establishes an available capacity ratio for it. The calculation formula is:

$$\lambda =min\left({\displaystyle \frac{C_0-C_1}{\left(T_0-T_1\right)P}},1\right)$$

(2)

where ${\mathrm{C}}_0$ and ${\mathrm{T}}_0$ represent the initial available capacity and the total number of schedulable time periods for the electric vehicle, respectively; $C_1$ and $T_1$ represent the dispatched capacity and the number of elapsed time periods, respectively; $P$ is the rated charging/discharging power of the electric vehicle.

This paper further considers the losses incurred during the electric vehicle charging process, for which a battery loss indicator, $\zeta$, is established. This indicator includes the line losses from the charging pile and the aging level of the EV battery. A smaller total charge throughput results in lower actual dispatchable capacity and power transfer efficiency, making it a reverse indicator. To facilitate calculation, it is converted into a positive indicator, $1-\zeta$, during modeling.

Declared Time Period Mask $\Gamma$ Mathematical Model: When EV owners declare their schedulable charging and discharging time periods, based on the principle of user convenience, they only submit scheduling plans for their idle periods. Undeclared periods may correspond to travel needs; therefore, it is necessary to impose hard constraints to prohibit scheduling during these times, thereby avoiding interference with users’ normal vehicle usage. In this paper, a declared time period mask is generated. Its core function is to filter out invalid decisions, ensuring that scheduling actions comply with users’ declared intentions and actual vehicle usage requirements.

$${\Gamma }_{i,t}=\left\{\begin{array}{ll}1,\hfill & \mathrm{if}\mathrm{EV}i\mathrm{submits}\mathrm{a}\mathrm{scheduling}\mathrm{plan}\mathrm{for}\mathrm{time}\mathrm{period}t\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

$$a_{i,t}°{\Gamma }_{i,t}=a_t$$

(4)

where $\Gamma \in {\{0,1\}}^{N\times T}$ is the declared time period mask matrix, $N$ s the total number of electric vehicles in the cluster (in this paper’s case study, N = 20), $T$ is the total number of scheduling time periods (in this paper’s case study, $T=3$, corresponding to 16:00, 17:00, and 18:00). The matrix element ${\Gamma }_{i,t}$, which takes a value of either 0 or 1, intuitively characterizes whether the $i$-th electric vehicle has declared a scheduling plan for the $t$-th time period. $a_t\in {\{0,1\}}^N$ is the scheduling decision vector for the $t$-th time period, where its element $a_{i,t}=1$ indicates scheduling the $i$-th electric vehicle, $a_{i,t}=0$ indicates not scheduling. The symbol $°$ denotes the Hadamard product (element-wise product), used to filter the scheduling decision vector on an element-by-element basis.

2.2. Response of the Electric Vehicle Aggregator

When considering multi-period scheduling, the dispatch priority of an electric vehicle depends not only on its remaining schedulable capacity and remaining schedulable time periods but also needs to be integrated with the grid’s demand during each period, charging costs, and the uncertainty of user behavior. This leads to the construction of a dynamic priority model. Such a model can reflect the time value of vehicle resources in real-time, prioritizing vehicles that are “about to leave” or have “tight capacity,” while avoiding situations where no vehicles are available for subsequent periods due to premature scheduling. This achieves a temporally balanced utilization of cluster resources.

To ensure that the indicators used by the EV aggregator during the scheduling process are not inconsistent, this paper standardizes all indicators. This eliminates differences in dimensions and magnitudes among various indicators, prevents model learning biases caused by inconsistent indicator scales, and ensures a balanced contribution of each indicator to the decision-making process. To this end, this paper employs the linear extremum method for standardization. A larger indicator value signifies a higher scheduling value. Standardization is performed using Equation (5), making the standardized value positively correlated with the original indicator. For computational convenience, this paper utilizes only positive indicators.

$$x_{i,k}^{*}={\displaystyle \frac{x_{i,k}-\min \left\{x_{:,k}\right\}}{\max \left\{x_{:,k}\right\}-\min \left\{x_{:,k}\right\}}}$$

(5)

where $x_{i,k}$ is the original value of the $k$-th indicator for the $i$-th electric vehicle.

2.3. State Transition Process

To comprehensively articulate the sequential decision-making mechanism, the state transition process of the MDP from time step $t$ to $t+1$ must be explicitly formulated. Upon the execution of the scheduling action ${\mathit{a}}_{\mathit{t}}$, the environment undergoes a deterministic state transition, denoted as $S_{t+1}=\mathcal{T}\left(S_t,{\mathit{a}}_{\mathit{t}}\right)$. This transition process governs the dynamic, real-time updates of both the individual electric vehicles’ physical states and the overarching cluster parameters. Specifically, for each EV $i$ selected to participate in the grid interaction (where $a_{t,i}=1$, its remaining schedulable capacity is mathematically updated based on the actual charging or V2G discharging power executed during the current time step, strictly bounded by the predefined upper and lower State of Charge limits. Concurrently, the temporal state variables are updated; the elapsed connection time for each vehicle increases, which directly triggers the recalculation of the dynamic evaluation metrics for the next step, most notably the available capacity ratio. By strictly defining these physical energy transfers and temporal updates, the transition function ensures that the agent accurately captures the evolving realities of the scheduling environment, linking current decisions ${\mathit{a}}_{\mathit{t}}$ with future payoffs $G_t$, thereby providing a rigorous theoretical foundation for the subsequent process transitions.

3. Markov Decision Process (MDP)

3.1. Action Space: The Agent’s Scheduling Decision

The action space defines a binary decision of “whether to schedule a specific EV or not.” The dimension of the action vector corresponds to the number of EVs. For example, $a_t=[1,0,1]$ signifies “schedule the 1st and 3rd EVs, do not schedule the 2nd EV.” This decision is concise and can be directly translated into scheduling instructions.

$$a_t=[a_{t,1},a_{t,2},\dots ,a_{t,N}]\in {\{0,1\}}^N$$

(6)

$$a_{t,i}=\left\{\begin{array}{ll}1,\hfill & \mathrm{if}\mathrm{EV}i\mathrm{is}\mathrm{dispatched}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

3.2. Reward Function

The reward function integrates grid demand, scheduling reliability, and constraint compliance. The formulation is designed to balance multi-objective optimization.

$$R_t=-k_1\cdot {\xi }_t+k_2\cdot {\rho }_t-k_3\cdot \psi$$

(8)

where $k1=0.1,k2=100,k3=50$; ${\xi }_t$ is the grid plan tracking error; ${\xi }_t={(P_{\mathrm{act}}(t)-P_{\mathrm{grid}}(t))}^2$, $P_{\mathrm{act}}(t)={\displaystyle \sum _{i=1}^N{a}_{t,i}}\cdot P_{\mathrm{rat},i}$ a larger deviation incurs a heavier penalty, ensuring that scheduling aligns with grid demand. $R_t^{\mathrm{rel}}$; ${\rho }_t$ represents scheduling reliability ${\rho }_t={\displaystyle \prod _{i:a_{t,i}=1}\mathcal{N}}({\eta }_{i,t};0.85,{0.1}^2)$, $\mathcal{N}$ which serves as the normal distribution probability density function. A higher degree of user integrity yields a more substantial reward, thereby incentivizing the algorithm to prioritize the dispatch of highly reliable users; ${\psi }_t$ which serves as the normal distribution probability density function, ${\psi }_t\in \{0,1,2\}$. A higher degree of user integrity yields a more substantial reward, thereby incentivizing the algorithm to prioritize the dispatch of highly reliable users.

3.3. Discount Factor and Cumulative Reward

$$G_t={\displaystyle \sum _{m=0}^{T-t-1}{\beta }^m}\cdot R_{t+m}$$

(9)

where $\beta =0.95$ is the discount factor, which encourages the agent to value future rewards, preventing resource depletion caused by pursuing single-period optimality; ${\mathrm{G}}_{\mathrm{t}}$ is the cumulative discounted reward, linking current decisions with future payoffs to guide multi-period global optimization.

4. DQN Decision Network

By leveraging its concise network structure and efficient learning mechanism, DQN can autonomously uncover optimal scheduling strategies within a high-dimensional state space. Its core training logic can be summarized as follows: it breaks data correlations through experience replay and stabilizes the learning process using a target network, ultimately achieving end-to-end optimization of the scheduling strategy. The key formulas are as follows:

4.1. State-Action Value Function: The Value Judgment of Decisions

In the context of EV scheduling, the agent needs to evaluate the long-term payoff of taking a specific scheduling action given the current system state in real time. To this end, we construct a two-layer fully connected neural network to approximate the state-action value function $Q_{\omega }\left(s_t,a_t\right)$:

$$Q_{\omega }\left(s_t,a_t\right)={\sigma }_2\left(W_2\cdot {\sigma }_1\left(W_1\cdot s_t+b_1\right)+b_2\right)\cdot a_t$$

(10)

where $\omega =\left\{W_1,b_1,W_2,b_2\right\}$: learnable parameters of the network, with $W_1\in {ℝ}^{d_h\times d_s}$ as the weight matrix from the input layer to the hidden layer, $b_1\in {ℝ}^{d_h}$ as the hidden layer bias; $W_2\in {ℝ}^{1\times d_h}$ as the weight vector from the hidden layer to the output layer, and $b_2\in ℝ$ as the output layer bias. ${\sigma }_1={\sigma }_2=\mathrm{ReLU}$: Rectified Linear Unit activation function, which introduces nonlinear representation capability while mitigating the vanishing gradient problem. $s_t\in {ℝ}^{d_s}$: system state vector at time $t$, containing multivariate information such as real-time grid demand, SOC distribution of the EV cluster, and remaining schedulable capacity. $a_t\in ℝ$: scheduling action at time $t$, i.e., the number of electric vehicles actually participating in the response or the total charging/discharging power.

$Q_{\omega }\left(s_t,a_t\right)\in ℝ$: the expected value of the cumulative discounted reward obtainable by continuing to follow the optimal policy after executing action $a_t$ in state $s_t$.

4.2. Dual-Network Mechanism: Stabilizing the Training Process

In traditional Q-learning, the target value calculation and the current Q-value estimation share the same set of parameters. This causes the target value to fluctuate frequently as the parameters are updated during training—akin to “chasing a moving target”—which can easily lead to oscillations or even divergence. To address this challenge, DQN introduces a dual-network mechanism that decouples the online network from the target network:

$$y_t=R_t+\beta \cdot \underset{a_{t+1}}{\max }{Q}_{{\omega }^{\prime }}\left(s_{t+1},a_{t+1}\right)$$

(11)

where $y_t$ is the target Q-value, serving as the anchor for updating the current network parameters; $R_t$ is the immediate reward obtained after executing the action at time $t$, comprehensively reflecting grid demand response accuracy, scheduling reliability, and constraint compliance; $\beta \in [0,1)$ is the discount factor, balancing the importance of immediate and future rewards; ${\max }_{a_{t+1}}{Q}_{{\omega }^{\prime }}\left(s_{t+1},a_{t+1}\right)$ is the optimal action value evaluated by the target network in the next state $s_{t+1}$; ${\omega }^{\prime }$ denotes the target network parameters, which are synchronized every 10 training steps by setting ${\omega }^{\prime }\leftarrow \omega$ and remain fixed otherwise.

4.3. Loss Function and Optimization: Enhancing Decision Accuracy

To quantify the discrepancy between the current network’s Q-value and the target value, and to guide the network parameters toward an optimal update, we define the mean squared error loss function:

$$L\left(\omega \right)={\mathbb{E}}_{\left(s_t,a_t,R_t,s_{t+1}\right)\sim \mathcal{R}}\left[{\left(Q_{\omega }\left(s_t,a_t\right)-y_t\right)}^2\right]$$

(12)

where, $\mathcal{R}$ denotes the experience replay buffer, with a capacity of 10,000, used to store historical transition samples $(s_t\left|a_t\right|R_t|s_{t+1})$ generated from the interaction between the agent and the environment. Batch sampling: 32 samples are randomly drawn from $\mathcal{R}$ to form a mini-batch for each training step, effectively breaking the temporal correlations between consecutive samples. Adam optimizer: An adaptive learning rate adjustment mechanism is employed, with the initial learning rate set to $5\times {10}^{-4}$, dynamically adjusting the step size during gradient descent. Training objective: The gradient of the loss with respect to the parameters $\omega$ is computed via backpropagation, and the Adam update rule is applied to minimize $L\left(\omega \right)$.

5. Optimization Objectives and Constraints: Constructing a Safe and Efficient Scheduling Framework

In the process of an electric vehicle cluster participating in grid demand response, the optimization objectives and constraints together form the core framework of the scheduling strategy. This section proposes a dual-layer collaborative optimization mechanism. At the physical layer, it achieves a balance between grid power tracking and scheduling reliability; at the network layer, it drives policy evolution by minimizing the loss function. Concurrently, multiple constraints are introduced to ensure the safety, feasibility, and user rights throughout the scheduling process from the three dimensions of quantity, capacity, and margin.

To clarify how the mathematical constraints and objectives are embedded within the iterative process of the Markov Decision Process (MDP), the proposed Dual-Layer Collaborative Optimization design functions as a dynamic bridge across each time step. During the environment interaction phase at the physical layer, safety boundaries and user willingness limits are strictly enforced prior to action execution via the declaration mask mechanism, ensuring the agent only explores valid state-action spaces. Following the execution of a valid action, the physical optimization objectives—namely, minimizing the grid power tracking error and maximizing scheduling reliability—are instantly quantified and returned to the agent as the immediate reward. Subsequently, during the policy evolution phase at the network layer, this immediate physical reward serves as the critical feedback signal. As the MDP iterates, the network utilizes the experience replay buffer to minimize the prediction loss of the Q-values, leveraging the physical reward to continuously update its algorithmic weights and drive the neural network to approximate the globally optimal scheduling policy.

5.1. Optimization Objectives: Dual-Layer Collaborative Optimization

The effectiveness of a scheduling strategy needs to be comprehensively evaluated from both the physical execution outcome and the algorithmic learning effectiveness. To this end, we construct a dual-layer optimization objective that nests the physical layer within the network layer:

$$\left\{\begin{array}{l}min{J}_1={{\displaystyle \sum }}_{t=0}^{T-1}{\left(P_{\mathrm{act}}\left(t\right)-P_{\mathrm{grid}}\left(t\right)\right)}^2\\ max{J}_2={{\displaystyle \prod }}_{t=0}^{T-1}{{\displaystyle \prod }}_{i:a_{t,i}=1}\mathcal{N}({\eta }_{i,t};0.85,{0.1}^2)\end{array}\right.$$

(13)

$$\underset{\omega }{\min }L\left(\omega \right)$$

(14)

where $P_{\mathrm{act}}\left(t\right)$ is the actual total charging/discharging power of the EV cluster during time period $t$ (in kW); $P_{\mathrm{grid}}\left(t\right)$ is the dispatch power expected by the grid during time period $t$ (in kW), determined by the retailer based on supply–demand balance requirements; ${\eta }_{i,t}$ is the reliability coefficient for the actual participation of the $i$-th electric vehicle in scheduling during time period $t$, defined as the ratio of the actual response power to the declared available power; $\mathcal{N}\left({\eta }_{i,t};0.85,{0.1}^2\right)$ is the probability density function of a normal distribution with a mean of 0.85 and a standard deviation of 0.1, used to quantify the degree to which the reliability coefficient approaches the ideal value; $a_{t,i}$ is the scheduling decision variable (1 indicates scheduling the vehicle, 0 indicates not scheduling); $L\left(\omega \right)$ is the loss function of the DQN network (see Equation (16)), which measures the Q-value prediction error.

5.2. Core Constraints: Ensuring Scheduling Safety and Feasibility

While pursuing the optimization objectives, a series of constraints must be strictly adhered to. This ensures that the scheduling scheme does not infringe upon user rights, exceed equipment capabilities, and possesses robustness against uncertainties during actual operation.

(1)

Charging/Discharging Quantity Constraints: Respecting User Declarations

$$\left\{\begin{array}{l}{N}_c\left(t\right)\le N_c^{\max }\left(t\right)\\ N_d\left(t\right)\le N_d^{\max }\left(t\right)\end{array}\right.$$

(15)

where $N_c\left(t\right)$ and $N_d\left(t\right)$ are the actual number of electric vehicles participating in charging and V2G during time period $t$, respectively; $N_c^{\max }\left(t\right)$ and $N_d^{\max }\left(t\right)$ are the total number of electric vehicles that have declared charging and V2G plans during time period $t$, respectively.

(2)

Capacity Constraints: Preventing Equipment Overload

$$\left\{\begin{array}{l}{{\displaystyle \sum }}_{i=1}^{N_c\left(t\right)}{P}_{i,c}\left(t\right)\le {{\displaystyle \sum }}_{i=1}^{N_c^{\max }\left(t\right)}{P}_{i,c}\left(t\right)\\ {{\displaystyle \sum }}_{i=1}^{N_d\left(t\right)}{P}_{i,d}\left(t\right)\le {{\displaystyle \sum }}_{i=1}^{N_d^{\max }\left(t\right)}{P}_{i,d}\left(t\right)\end{array}\right.$$

(16)

where $P_{i,c}\left(t\right)$ and $P_{i,d}\left(t\right)$ are the rated charging and V2G power of the $i$-th vehicle during time period $t$ (determined by the characteristics of the vehicle and the charging pile). The left-hand side summation represents the total power demand of the vehicles actually dispatched, while the right-hand side summation represents the maximum total power that can be provided by all vehicles that declared availability.

(3)

Margin Constraints: Addressing User Default Risk

$$N_{r,c}\left(t\right)=\alpha \cdot N_c\left(t\right),N_{r,d}\left(t\right)=\alpha \cdot N_d\left(t\right)$$

(17)

where $N_{r,c}\left(t\right)$ and $N_{r,d}\left(t\right)$ are the actual number of reserved standby vehicles for charging and discharging during time period $t$, respectively; $\alpha \in \left[1.1,1.3\right]$ is the margin coefficient, indicating a reserved redundancy of 10% to 30%; $N_c\left(t\right)$ and $N_d\left(t\right)$ are the theoretically calculated minimum number of vehicles required for scheduling (based on power demand and estimated average power per vehicle).

In summary, the optimization objectives and constraints together form the decision-making framework for EV scheduling: The dual-layer objectives guide the agent toward accurate and reliable evolution, while the multiple constraints define the safety boundaries, ensuring all scheduling actions remain within physically feasible and user-approved limits. This framework provides a clear basis for designing the reward function and defining the action space constraints for the subsequent DQN algorithm training, serving as the cornerstone for achieving intelligent scheduling.

6. Case Study Analysis

This paper utilizes a four-node, two-generator system as a benchmark, as shown in Appendix A. The topological structure of this system includes a complete “power source—grid—load” link. Node information for the system is referenced from [23]. To enhance decision-making transparency, the DQN model used in this paper accepts a 62-dimensional state vector from the environment at its input layer; this vector integrates nonlinear preference features—such as driver credibility and battery degradation—in real time. The output layer employs a masking mechanism to ensure that scheduling actions are generated only during the user’s declared idle periods. The model is trained using the Adam optimizer, which updates the weights by minimizing the mean squared error (MSE) between the target Q-value and the predicted Q-value. This paper conducts optimal scheduling based on three time periods, namely 16:00–18:00. Based on the calculation of system load and power source output balance, the power gaps (scheduling targets) that need to be supplemented by the discharge of the EV cluster for each time period are as follows, consistent with the measured data. The final scheduling gaps for periods 16:00–18:00 are shown in Table 1.

Table 1. The final scheduling gaps for periods 16:00–18:00.

Time Period	Grid Planned Power Gap (kW)	Capacity Margin Requirement (%)	Core Scheduling Objective
T1 (16:00)	72.31	≥5	Smooth Peak Shaving
T2 (17:00)	49.03	≥25	High-Reliability Assurance
T3 (18:00)	11.45	≥20	Precise Matching

To clearly illustrate the execution logic of the proposed Deep Q-Network (DQN)-based optimal scheduling strategy for EV clusters, a detailed step-by-step pseudocode is provided in Algorithm 1. This algorithm explicitly defines the inputs of the decision model—a comprehensive state vector and a declaration mask matrix reflecting the owners’ willingness—and the output, which is the optimized combination of EVs participating in the response. It is worth noting that, unlike traditional supervised learning methods that rely on statically divided datasets (i.e., fixed training, validation, and testing sets), the DQN agent dynamically generates and stores training data through continuous interaction with the environment over thousands of episodes. During the action generation phase, the algorithm strictly applies a mask filtering mechanism to eliminate invalid scheduling actions. Furthermore, by integrating experience replay and a dual-network synchronization mechanism, it ensures the stable convergence and global optimization of the scheduling policy under complex, high-dimensional constraints. Additionally, all simulation models in this study were implemented using Python 3.10 and the PyTorch 2.10.0 framework, and executed on a hardware platform equipped with an Intel Core Ultra 7 265K processor, an NVIDIA RTX 5070 GPU, and 128 GB of RAM, ensuring high computational efficiency during both the training and testing phases.

Algorithm 1: Deep Q-Network (DQN) Based Optimal Scheduling for Electric Vehicle Clusters

Inputs: State Vector ${\mathit{S}}_{\mathit{t}}$: Multidimensional data including real-time grid demand, State of Charge (SoC) distribution of the EV cluster, and remaining schedulable capacity; Declaration Mask Matrix ${\mathit{M}}_{\mathit{t}}$: Binary mask representing the availability of EVs in specific time slots; Hyperparameters: Adam optimizer learning rate, discount factor $\gamma$, and exploration rate $𝜖$. Outputs: Action Vector ${\mathit{A}}_{\mathit{t}}$: Optimized combination of EV units participating in the response. 1:    Initialize replay memory D with capacity 10,000 2:    Initialize online Q-network Q(s, a; θ) with random weights θ 3:    Initialize target Q-network $\hat{\mathrm{Q}}$(s, a; θ−) with θ− ← θ 4:    Initialize global step counter step ← 0 5:    for episode = 1 to 10,000 do 6:            state S1 ← env.reset() 7:            for t in {16:00, 17:00, 18:00} do 8:                    Generate declaration mask M_t based on EV user plans 9:                    With probability ε: 10:                          Select random action A_t 11:                  Otherwise: 12:                          Select A_t = argmax_a Q(S_t, a; θ) 13:                  Apply mask to filter invalid actions: A_t ← A_t ⊙ M_t 14:                  Execute action A_t in environment 15:                  Observe reward R_t and next state S_{t + 1} 16:                  Store transition (S_t, A_t, R_t, S_{t + 1}) in D 17:                  If |D| ≥ 32: 18:                          Sample mini-batch of 32 transitions (S_j, A_j, R_j, S_{j + 1}) from D 19:                          For each sampled transition: 20:                                  Compute target: Y_j = R_j + γ * max_{a′} $\hat{\mathrm{Q}}$(S_{j + 1}, a′; θ−) 21:                          End For 22:                          Perform gradient descent on loss L using Adam optimizer 23:                  End If 24:                  If step mod 10 == 0: 25:                          Update target network: θ− ← θ 26:                  End If 27:                  S_t ← S_{t + 1} 28:                  step ← step + 1 29:          end for 30:          Decay exploration rate ε ← max(0.01, ε × 0.9995) 31:  end for 32:  Return trained Q-network as optimal scheduling policy

While the four-node system with 20 EVs demonstrates the fundamental mechanism of the proposed DQN strategy, its scalability to larger power systems is a key consideration for practical engineering. The proposed framework maintains high scalability through a hierarchical dispatch structure, which allows an EV aggregator to act as a single scheduling entity, bridging the gap between the grid and individual owners.

The capacity margin, serving as a preset constraint for grid scheduling, has its core purpose in ensuring operational safety. By reserving standby capacity to address sudden situations such as EV discharge power fluctuations or load forecast deviations, it prevents the power gap from remaining unfulfilled due to insufficient scheduled power. On this basis, the margin requirements are set differently according to the scheduling objectives for each period: During the T1 period, the load is at the early stage of ramping up with relatively small fluctuations, hence the minimum margin requirement (≥5%) is set to achieve smooth peak shaving. The T2 period is the load peak, demanding the highest reliability for discharge; sufficient margin needs to be reserved to cope with fluctuations, therefore the highest margin requirement (≥25%) is set. During the T3 period, the load tends to stabilize, and a medium margin (≥20%) is set under the principle of balancing precise matching with safety. The relationship between capacity margin and scheduling reliability is illustrated in Figure 3. The aforementioned values are comprehensively determined by referencing the load characteristics of Node 2 and the measured discharge stability data of the EV cluster, combined with power system scheduling specifications, ensuring that the margin requirements are both scientifically sound and aligned with engineering practice. Source of the power gap: during these periods, the load at Node 2 increases rapidly (1000 → 1100 → 1200 kW), and the generator output, along with power support from the upstream grid, cannot fully cover this increase, necessitating supplementation through the discharge of the EV cluster.

Figure 3. The relationship between capacity margin and scheduling reliability.

This paper selects 20 electric vehicles participating in the dispatch. The core parameters and priority data are derived from actual measurement results, as detailed below in Table 2:

Table 2. The core parameters and priority data.

EV No.	Declared Period (1 = Declared, 0 = Not Declared)	Declared Power (kW)	T1 (16:00) F1/F2/F3	T2 (17:00) F1/F2/F3	T3 (18:00) F1/F2/F3	Model Self-Learning Priority
1	T1 = 1, T2 = 0, T3 = 0	7	1.0/0.67/0.12	0.0/0.0/0.0	0.0/0.0/0.0	0.82
2	T1 = 1, T2 = 0, T3 = 0	7	1.0/0.45/0.05	0.0/0.0/0.0	0.0/0.0/0.0	0.79
3	T1 = 1, T2 = 0, T3 = 0	7	1.0/0.19/0.16	0.0/0.0/0.0	0.0/0.0/0.0	0.70
4	T1 = 1, T2 = 0, T3 = 0	7	1.0/0.64/0.12	0.0/0.0/0.0	0.0/0.0/0.0	0.81
5	T1 = 0, T2 = 1, T3 = 0	7	0.0/0.0/0.0	1.0/0.72/0.21	0.0/0.0/0.0	0.80
6	T1 = 1, T2 = 1, T3 = 0	7	0.5/0.74/0.08	0.5/0.74/0.08	0.0/0.0/0.0	0.83
7	T1 = 1, T2 = 1, T3 = 0	7	0.5/0.39/0.14	0.5/0.39/0.14	0.0/0.0/0.0	0.72
8	T1 = 1, T2 = 1, T3 = 0	7	0.5/0.66/0.06	0.5/0.66/0.06	0.0/0.0/0.0	0.81
9	T1 = 0, T2 = 1, T3 = 1	7	0.0/0.0/0.0	0.5/0.48/0.23	0.5/0.48/0.23	0.70
10	T1 = 0, T2 = 1, T3 = 1	7	0.0/0.0/0.0	0.33/0.48/0.14	0.33/0.48/0.14	0.71
11	T1 = 0, T2 = 1, T3 = 1	7	0.0/0.0/0.0	0.33/0.92/0.02	0.33/0.92/0.02	0.85
12	T1 = 0, T2 = 1, T3 = 1	7	0.0/0.0/0.0	0.33/0.84/0.03	0.33/0.84/0.03	0.83
13	T1 = 1, T2 = 1, T3 = 0	14	1.0/0.55/0.11	0.5/0.55/0.11	0.0/0.0/0.0	0.82
14	T1 = 1, T2 = 1, T3 = 0	14	1.0/0.80/0.04	0.5/0.80/0.04	0.0/0.0/0.0	0.89
15	T1 = 0, T2 = 1, T3 = 1	14	0.0/0.0/0.0	0.5/0.60/0.13	0.5/0.60/0.13	0.78
16	T1 = 0, T2 = 1, T3 = 1	14	0.0/0.0/0.0	0.5/0.76/0.05	0.5/0.76/0.05	0.84
17	T1 = 1, T2 = 1, T3 = 1	14	0.67/0.85/0.14	0.33/0.85/0.14	0.33/0.85/0.14	0.85
18	T1 = 1, T2 = 1, T3 = 1	14	0.67/0.48/0.04	0.33/0.48/0.04	0.33/0.48/0.04	0.79
19	T1 = 1, T2 = 1, T3 = 1	21	1.0/0.35/0.02	0.33/0.35/0.12	0.33/0.35/0.12	0.78
20	T1 = 1, T2 = 1, T3 = 1	21	1.0/0.66/0.03	0.33/0.66/0.03	0.33/0.66/0.03	0.84

In this case study, the DQN model serves as the core decision-making module for the discharge scheduling of the EV cluster. Its role is to achieve optimal decisions for the scheduling objectives of different time periods within the predefined constraint framework. This function operates across three interrelated levels: constraint adaptation, target achievement, and efficiency optimization. First, in terms of constraint adaptation, the model embeds hard constraints such as capacity margin requirements, user-declared time periods, and power ranges directly into the decision-making process, automatically avoiding actions that violate the rules. For example, the margin for the T1 period must be no less than 5%. The model outputs a total power of 77.0 kW by selecting a combination of EVs, corresponding to a margin of 6.49%. For the T2 period, the margin requirement increases to 25%; the model schedules 63.0 kW, achieving a margin of 28.49%. This satisfies the constraints while avoiding power redundancy, demonstrating a flexible adaptability to differentiated constraints. Second, at the level of precise target achievement, DQN guides decision-making through the design of the reward function: The T1 period focuses on smooth peak shaving, prioritizing EVs with stable power and low losses (e.g., EV1, EV4). The T2 period emphasizes high reliability; the average completeness (F2) of the vehicles scheduled by the model reaches 0.75, while the average battery loss (F3) is only 0.07. The T3 period pursues precise matching, controlling the power tracking error to 2.55 kW, achieving a tracking accuracy of 77.7%. Overall, in this case study, DQN assumes the dual role of a constraint enforcer and a target achiever. Its decision-making logic is deeply coupled with the requirements of the case, demonstrating its engineering practicality in the EV cluster scheduling scenario. The model training parameters were determined based on the characteristics of the measured data (the DQN Scheduling Model Parameter is shown Table 3), ensuring training convergence and scheduling accuracy.

Table 3. DQN scheduling model parameter.

Parameter Category	Parameter Name	Value	Setting Rationale
Network Structure	Input Layer/Hidden Layer/Output Layer Dimensions	62/128 × 2/20	Covers 3 types of indicators, time periods, and planned power characteristics.
Training Optimization	Optimizer/Learning Rate	Adam/5 × 10−4	Adapts to high-dimensional state space for fast convergence.
Reinforcement Learning	Discount Factor/Exploration Rate Range	0.95/0.01~0.9	Balances immediate and future rewards, enables sufficient policy exploration.
Reward Function	Tracking Error/Reliability/Penalty Coefficient	0.1/100/50	Aligns with measured power range and constraint requirements.

Based on the above data, the simulation process in this paper is configured as follows: First, the credibility, available capacity ratio, and battery loss index for each electric vehicle are calculated based on measured data. After standardization, a 62-dimensional state space is constructed. Simultaneously, a mask matrix is generated based on the users’ declared time periods to filter out invalid scheduling actions. During the model training phase, dual Q-networks and an experience replay buffer with a capacity of 10,000 are initialized. The ε-greedy strategy is employed to interact with the environment and collect experiences. The target network parameters are synchronized every 10 steps. After 10,000 training episodes, the fluctuation of the cumulative reward converged to within 5%. For scheduling decisions, the trained model (with the exploration rate set to 0) is loaded, the state vector for each time period is input, and the optimal scheduling combination is output based on priority value sorting. Finally, the scheduling performance is validated using tracking error, capacity margin, and reliability indicators. The results show that all constraint conditions are satisfied, confirming the rationality and effectiveness of the proposed method, as illustrated in Figure 4.

Figure 4. Power comparison before and after scheduling.

The above simulation experiments (shown in Table 4) validated the effectiveness of the proposed scheduling method for three typical time periods. In the T1 period (16:00) (shown in Figure 5), aiming for smooth peak shaving, vehicles EV 1, 2, 3, 4, 6, 7, 8, 13, and 14 were selected in descending order of priority from the declared vehicles to participate in scheduling, totaling 9 vehicles. Their priority ranking is shown in Figure 4, with a total power of 77.0 kW. The power tracking error for this period was 4.69 kW, corresponding to a tracking accuracy of 93.5%. The capacity margin of 6.49% met the constraint requirement of no less than 5%. The mean integrity of the dispatched vehicles was 0.61, and the mean battery loss was 0.08, exhibiting low-loss characteristics. The goal of smooth peak shaving was effectively achieved. In the T2 period (17:00), focusing on high-reliability assurance, vehicles EV 12, 13, 14, 16, and 17 were selected, totaling 5 vehicles, with a total scheduled power of 63.0 kW. The power tracking error was 13.97 kW (accuracy 71.5%). The capacity margin of 28.49% met the constraint of no less than 25%. During this period, the mean integrity increased to 0.75, and the mean battery loss decreased to 0.07, both being the optimal values among the three periods, thus achieving the high-reliability scheduling objective. In the T3 period (18:00), with the goal of precise matching, only the highest-priority vehicle, EV 17, was scheduled, with a power of 14.0 kW. The power tracking error was 2.55 kW (accuracy 77.7%), and the capacity margin of 22.27% met the requirement of no less than 20%. The scheduling results for the three periods were highly consistent with the simulation data. The margin for each period strictly satisfied the preset thresholds, and the integrity and battery loss characteristics of the dispatched vehicles matched the objectives of each respective period. This validates the decision-making effectiveness of the model under differentiated scheduling requirements.

Table 4. The scheduling decision.

Time Period	Scheduling Target Power (kW)	Scheduled EV Nos. (Descending Priority)	Number of Scheduled Vehicles	Actual Scheduled Power (kW)
T1	72.31	14, 2, 1, 4, 13, 3, 8, 6, 7	9	77.0
T2	49.03	12, 13, 14, 16, 17	5	63.0
T3	11.45	17	1	14.0

Figure 5. EV Priority Value in T1.

To further validate the effectiveness of the proposed method, an analysis is conducted from three dimensions: capacity margin, EV priority, and indicator weight distribution. Regarding the capacity margin, the actual values for each time period strictly satisfy the preset thresholds: the margin in the T1 period is 6.49%, slightly above the lower limit requirement of 5%; in the T2 period, it is precisely 28.49%, meeting the 25% target; and in the T3 period, it is 22.27%, satisfying the 20% constraint. These values are entirely consistent with the data presented in the table, demonstrating the model’s robust capability in adhering to constraints. In terms of EV priority, taking the T1 period as an example, the top three EVs ranked by priority based on the entropy weight method are EV14 (0.91), EV2 (0.78), and EV1 (0.72). All three exhibit the characteristics of a “high available capacity ratio (F1) + high integrity (F2) + low battery loss (F3),” reflecting a strong alignment between the model’s selection results and the scheduling reliability objectives.

From the perspective of indicator weight distribution (Table 5), the results calculated by the entropy weight method show that the weight of F2 (integrity) progressively increases over the time periods: it is 0.58 in T1, rises to 0.63 in T2, and further increases to 0.69 in T3. Conversely, the weight of F1 (available capacity ratio) decreases from 0.32 to 0.26, while the weight of F3 (battery loss) remains consistently low, between 0.07 and 0.10. This trend aligns perfectly with the scheduling objectives of high-reliability assurance in T2 and precise matching in T3. The weight calculations are based on the actual indicator values for each period, consistent with the underlying code logic. The above analysis, from different perspectives, validates the unity between the model’s decision-making mechanism and the scheduling objectives.

Table 5. Indicator weight distribution.

Time Period	F1 (Available Capacity Ratio)	F2 (Integrity)	F3 (Battery Loss)
T1	0.32	0.58	0.10
T2	0.30	0.63	0.08
T3	0.26	0.69	0.07

7. Conclusions

This paper addresses the challenges in managing charging considering V2G at electric vehicle charging stations, such as high decision-making costs for aggregators and insufficient stability of scheduling plans. By considering the willingness of EV owners to participate, an optimal charging considering V2G control strategy for EV charging station based on a Deep Q-Network (DQN) is proposed. Through theoretical modeling, indicator construction, and case study simulations, the following main conclusions are drawn:

• The constructed comprehensive evaluation index system for EV charging considering V2G scheduling effectively integrates owners’ willingness to participate with the physical constraints of the vehicles. By employing three core indicators—credibility, available capacity ratio, and battery loss—combined with a declared time period mask mechanism, it achieves an accurate characterization of owners’ declared behavior and filters out invalid scheduling actions. This provides scientific data support for subsequent intelligent decision-making, addressing the issue of low execution rates of scheduling plans caused by neglecting individual owner willingness in traditional research.
• The DQN decision network, built upon a Markov Decision Process (MDP), stabilizes the training process through a dual-network mechanism (online network and target network). Combined with a reward function that integrates grid demand, scheduling reliability, and constraint compliance, it achieves globally optimal decisions across multiple time periods. This network can adaptively accommodate the dynamic changes in owners’ willingness to participate, optimize the scheduling margin ratio, effectively enhance the aggregator’s decision-making capability, and reduce both the instability of scheduling plans and the aggregator’s decision-making costs.
• The case study simulation results validate the effectiveness and engineering practicality of the proposed strategy. Taking 20 electric vehicles and three typical time periods (16:00–18:00) as the research objects, simulations conducted on a four-node, two-generator system demonstrate that the strategy can strictly satisfy the capacity margin constraints for each period, accurately match the grid power gap, and achieve the differentiated scheduling objectives of smooth peak shaving, high-reliability assurance, and precise matching. The power tracking accuracy and scheduling reliability meet the expected requirements, and the indicator characteristics of the dispatched vehicles are highly consistent with the objectives of each period, confirming the application value of the strategy in practical scenarios.

In summary, the DQN-based optimal charging considering V2G control strategy proposed in this paper successfully balances “grid physical constraints + owner willingness to participate + global optimization.” It effectively resolves the core contradictions in current EV charging considering V2G scheduling, providing new ideas and methods for EV aggregators to formulate scientific, efficient, and executable charging schedules. This facilitates the large-scale participation of electric vehicles in grid interaction and promotes the coordinated and sustainable development of both the power grid and electric vehicles.

8. Future Outlook

The research in this paper provides an effective solution for the optimal control of charging and discharging in EV charging stations, but it still has certain limitations. Considering the current development trends of electric vehicles and smart grids, further research can be conducted from the following aspects in the future:

• Expand multi-scenario adaptability. The case study in this paper only investigates fixed time periods, a fixed vehicle scale, and a single grid scenario. Future research could consider complex scenarios such as extreme weather (high temperature, severe cold), grid faults, and fluctuations in renewable energy output. Introducing an integrated wind–solar–storage coordination scheduling mechanism and optimizing the state space and reward function design of the DQN model can enhance the robustness and adaptability of the strategy under complex operating conditions.
• Improve the multi-agent collaborative scheduling mechanism. Current research primarily focuses on the bidirectional interaction between the aggregator, owners, and the grid. Future work could incorporate multiple agents such as charging pile operators and energy storage equipment operators. Constructing a multi-agent benefit coordination optimization model to balance the benefits of all parties, while introducing dynamic incentive mechanisms, can further enhance owners’ willingness to participate in charging and discharging scheduling and optimize the overall efficiency of cluster scheduling.
• Optimize the model structure and training efficiency. This paper employs a traditional DQN network for decision optimization. Future research could introduce improved reinforcement learning algorithms (e.g., Double DQN, Deep Deterministic Policy Gradient) to optimize the network structure, reducing oscillations and convergence time during training. Simultaneously, integrating edge computing technology could offload some decision-making tasks to local charging stations, reducing data transmission latency, enabling real-time dynamic scheduling, and improving scheduling response speed.
• Promote engineering implementation and empirical research. Future work could select actual EV charging stations for empirical testing, optimizing model parameters based on measured data to resolve discrepancies between theoretical modeling and practical operation. Through these upcoming actual engineering projects, we will further reveal and validate the practical novelty and real-world applicability of the proposed V2G mechanisms. Concurrently, exploring integration methods with existing grid dispatching systems and developing a scheduling decision platform adapted to real-world scenarios would promote the engineering application of the proposed strategy, providing technical support for the high-quality development of smart grids.
• Consider multi-dimensional uncertainty factors. Current research still has certain errors in predicting owners’ travel plans and grid power demand. Future research could introduce probabilistic prediction models to quantify the impact of uncertainties such as travel randomness and load fluctuations on scheduling performance. This would further enhance the credibility and executability of scheduling plans, enabling more precise global optimization decisions.

Despite the effectiveness of the proposed DQN-based framework in translating subjective behavioral traits into deterministic action masks for grid-side operational security, several limitations remain to be addressed. First, as a fundamentally data-driven approach, the model assumes high initial data observability. This heavy reliance on the accuracy of state observations makes the framework potentially vulnerable to severe environmental noise, False Data Injection Attacks (FDIA), or strategic manipulation such as falsified capacity declarations. While the current model attempts to mitigate participation bias and normal behavioral randomness through linear extremum standardization, dynamic state updates, and historical credibility evaluations, extreme information asymmetry could still degrade its multi-period optimization efficiency. Second, because our methodology prioritizes absolute physical dispatch boundaries over market mechanism design, it inherently differs from traditional game-theoretic models and currently lacks the formal mathematical guarantees of economic incentive compatibility and Nash equilibrium. Therefore, crucial avenues for future research include integrating multi-agent mechanism design to explicitly ensure strict incentive compatibility, as well as transitioning toward robust reinforcement learning or Partially Observable Markov Decision Processes (POMDPs). These future integrations will help dynamically filter noisy signals, handle information asymmetry, and maintain optimal V2G coordination under practical uncertainties.