Deep Reinforcement Learning-Based Optimization of Mobile Charging Station and Battery Recharging Under Grid Constraints

Abstract

With the rise in traffic congestion, time has become an increasingly critical factor for electric vehicle (EV) users, leading to a surge in demand for fast and convenient charging services at locations of their choosing. Mobile Charging Stations (MCSs) have emerged as a new and practical solution to meet this growing need. However, the limited energy capacity of MCSs combined with the increasing volume of charging requests underscores the necessity for intelligent and efficient management. This study introduces a comprehensive mathematical framework aimed at optimizing both the deployment of MCSs and the scheduling of their battery recharging using battery swapping technology, while considering grid constraints, using the Deep Q-Network (DQN) algorithm. The proposed model is applied to real-world data from Chattanooga to evaluate its performance under practical conditions. The key goals of the proposed approach are to maximize the profit from fulfilling private EV charging requests, optimize the utilization of MCS battery packages, manage MCS scheduling without causing stress on the power grid, and manage recharging operations efficiently by incorporating photovoltaic (PV) sources at battery charging stations.

1. Introduction

1.1. Motivation and Problem Statement

The increasing demand for EVs is reshaping the transportation landscape by offering a cleaner and more sustainable alternative to traditional fossil fuel-powered vehicles. The electrification of the transportation sector has become a global priority as nations seek to reduce their carbon footprints and mitigate the effects of climate change. EVs have been widely adopted as an alternative to internal combustion engine vehicles, offering significant reductions in greenhouse gas emissions and fossil fuel dependency. However, with the rapid growth of EV adoption, several critical challenges have emerged, particularly concerning the charging infrastructure. Issues such as limited access to charging stations, long waiting times, high operational costs, congestion during peak demand time, the inability to service remote or populated urban areas, and uneven distribution of charging locations have made it difficult for EV users to fully embrace the technology. These challenges also contribute to range anxiety, a major concern for EV drivers [1,2,3,4]. One promising solution to these challenges is the development of MCSs. Unlike fixed charging stations (FCSs), which are geographically bound and can become congested during peak hours, MCSs offer flexibility by providing on-demand charging at various locations. MCSs are particularly valuable in urban areas where space is limited for permanent charging infrastructure and for addressing the spatiotemporal heterogeneity of EV charging demand [5,6,7,8]. These MCSs can be dispatched to specific areas based on real-time demand, offering an adaptable and scalable solution to EV charging, thus reducing charging times and alleviating range anxiety. Ref. [9] presents a framework for optimizing the operation of MCS for EVs, using a mixed-integer linear programming (MILP) model and a genetic algorithm (GA) to maximize profits and enhance user satisfaction. The authors in [10] introduce a profit-maximizing assignment strategy for idle MCSs, using heat maps to track EVs in need of charging. This approach enhances MCS assignment efficiency and increases profitability by targeting high-demand areas. A self-scheduling model for smart parking lots with MCSs is developed in [11], prioritizing EV charging demands based on social equity factors and optimizing energy generation and storage.

Although the allocation of MCSs to EVs for charging addresses many challenges related to energy supply, incorporating precise planning for charging the batteries of these MCSs, as well as scheduling timely battery replacements, can significantly enhance this allocation process. To achieve optimal allocation and maximize the profit for MCSs, it is essential to develop an efficient schedule and location plan for charging the batteries. Additionally, forecasting suitable times and locations for battery replacements when necessary can prevent time and cost waste, thereby enhancing the system’s overall stability and profitability. The integration of renewable energy resources, such as solar with EV charging infrastructure, has been recognized as a critical step toward creating sustainable charging systems. FCSs can incorporate photovoltaic (PV) systems along with battery energy storage systems (BESSs) to store excess energy and provide power during peak demand periods. BESSs enable FCSs to store excess PV energy generated during off-peak hours, which can then be used for MCS charging during high-demand periods, thereby reducing the strain on the power grid. This approach not only supports the use of clean energy but also improves the resilience of the energy supply [12,13,14,15].

1.2. Related Work

Ref. [16] proposes an optimal power dispatch model for a grid-connected EV charging station with renewable energy and battery storage to enhance reliability and reduce costs. Ref. [17] addresses the optimization of mobile renewable energy charging station locations, aiming to support EV adoption and reduce carbon emissions.

Time-of-Use (ToU) pricing is widely adopted in EV charging management studies as it provides an effective incentive to shift demand from peak to off-peak hours, thereby reducing operational costs and alleviating stress on the distribution grid [6,18].

Robust planning of MCSs for EV charging requires advanced optimization methods. Classical optimization methods are limited in addressing complex, dynamic problems. They rely on fixed objective functions and lack adaptability to changing environments. Additionally, they cannot dynamically interact with or learn from the environment and are unable to utilize neural networks for multidimensional problem-solving, often focusing only on short-term gains. In recent years, MCSs have evolved with the integration of advanced optimization techniques such as Deep Reinforcement Learning (DRL) to improve their scheduling and routing. DRL allows MCSs to make real-time decisions based on the locations of EVs with low battery levels, dynamically dispatching MCSs to areas with higher demand. This not only improves charging efficiency but also ensures better utilization of MCSs, reducing idle times and maximizing service coverage. The use of DRL has shown significant promise in managing complex EV charging scenarios, especially in smart cities where the charging demand fluctuates based on the time and location [19,20,21]. Ref. [22] proposes a multi-agent reinforcement learning framework to optimize EV charging in smart cities, enhancing coordination among stations, reducing wait times and failure rates, and improving user satisfaction. Ref. [23] presents a DRL model to solve the EV routing problem with time windows, optimizing routes effectively for large-scale cases.

The rapid expansion of smart distribution networks, increased utilization of PV systems, and the growing presence of EV charging stations have significantly increased the complexity of power network management. Due to their variable nature and high power demand, charging stations contribute to increased losses, voltage drops, and fluctuations within the network. On the other hand, PV systems, due to their intermittent generation and dependence on weather conditions, lead to voltage fluctuations and power backflow into the grid. Energy storage systems, with their controllable charging and discharging capabilities, play a crucial role in voltage regulation, load balancing, and improving network efficiency. Therefore, the coordinated integration of these resources and the application of effective management strategies are essential for maintaining stability and enhancing the reliability of power systems. Load flow monitoring is one of the fundamental issues in power systems, aimed at determining the optimal distribution of active and reactive power within a network. The primary objective of load flow analysis is to determine the voltage, phase angle, current, and power at each point in the network to ensure optimal and stable system performance [24,25,26,27,28].

Various studies have proposed effective solutions to enhance power flow management in distribution networks with high penetration of EVs and renewable energy sources. The authors in [29] utilized probabilistic power flow analysis in combination with machine learning, enabling the accurate prediction of network congestion and improved power management. The authors in [30] applied an optimal power flow formulation, which increased the EV hosting capacity and reduced voltage fluctuations and power losses by controlling power injection at the connection points. The Newton–Raphson (NR) load flow method that incorporates Weak Bus Placement is presented in [31] to identify the optimal locations for installing distributed generators, leading to improved network stability and reduced voltage fluctuations. As presented in [32], power flow management in DC fast charging stations integrated with the power grid, PV systems, and BESS has been optimized using linear active disturbance rejection control and a modified Maximum Power Point Tracking algorithm. Using the Enhanced Coati Optimization Algorithm, the probabilistic optimal power flow problem has been solved in [33], considering uncertainties in solar and wind energy generation as well as EV charging. Power flow management in distribution networks with high penetration of EVs and renewable energy sources has been improved in [34,35,36,37,38] through the use of optimization methods and control strategies.

To the best of our knowledge, none of the existing studies simultaneously address profit maximization, battery swap technology, PV integration, and demand uncertainty management for MCSs within a unified framework. Unlike the studies in [2,9], which primarily focus on optimizing MCS scheduling to maximize profits, this study expands its scope by integrating PV, as highlighted in [14,17], to reduce reliance on the grid and operational costs. Unlike [11], which emphasizes social equity in EV charging, this study adopts a broader perspective by combining economic, environmental, and technological dimensions to create a unified, sustainable framework for MCS operations. Furthermore, while ref. [17] employs the differential evolutionary Q-learning (DEQL) method to tackle demand uncertainty, this study extends the approach by exploring the potential application of the DQN algorithm to enhance real-time adaptability. Ref. [39] studies the coordinated operation of multi-energy microgrids with hydrogen integration and congestion management through a safe policy learning approach. While their work addresses congestion management in multi-energy microgrids, our study focuses on MCSs for EVs, where the challenge lies in the real-time scheduling of services and managing the impact of MCS battery charging on the distribution grid. Moreover, in contrast to [6], which focuses on urban EV management through MCS scheduling, and [10], which uses profit-maximizing strategies for idle MCSs, this study also evaluates the feasibility of battery swap technology as a solution to improve operational efficiency.

1.3. Proposed Approach and Contributions

To address the research gaps, this paper proposes a scheduling model for MCSs as a complex multi-agent scheduling problem, aiming to maximize profits, optimize the EV charging process, and reduce stress on the power grid during peak hours. Given the uncertainty in charging demand, the DQN algorithm is applied to provide optimal responses and effective scheduling to address fluctuations. Additionally, the use of battery swap methods to meet a higher number of charging requests and incorporating PV as an auxiliary source enhance the efficiency of these stations. The major contributions of this paper are summarized as follows:

• A decision-making framework that schedules MCSs and EVs charging to maximize profits, reduce grid stress at peak hours, and address demand uncertainty using the DQN algorithm.
• Integration of PV with MCS as an auxiliary source to increase income and minimize reliance on the grid during peak consumption hours.
• Implementation of MCS battery swapping to efficiently respond to more EV charging requests and improve overall service performance.
• Analysis of the impact of MCS battery charging on grid constraints, and the introduction of a scheduling approach to ensure safe and efficient operation.

2. Overview

As a new component in the transportation system, MCSs play a key role in EV charging schedules. To meet demand efficiently, a strong mathematical model based on customer history is essential. MCSs face two main challenges: optimizing service timing to cover maximum EV requests and refilling energy storage promptly. This study explores two scheduling stages: first, optimal MCS scheduling for EV requests, and second, managing battery recharges in BESS stations to meet MCS energy needs. Then, the limitations of the power grid are considered for this scheduling. Figure 1 shows the proposed scheduling model. The scheduling framework is built on a few simplifying assumptions. We limit the analysis to charging requests from private passenger EVs. MCSs are modeled as plug-in service units that can handle only one vehicle within each decision interval. Incoming requests are assumed to be known slightly in advance, which enables timely scheduling decisions. The power network is represented by a standard radial distribution test system, and electricity costs follow a time-varying tariff structure.

3. Mathematical Model

3.1. Optimizing the Distribution of MCSs to EV Charging Requests

The objective function of this research consists of two parts: the first part maximizes the MCS revenue from delivering energy to EV requests, and the second part is the optimal managing and charging of portable battery packages for MCSs.

$$F_1=max\sum _{j\in {\mathrm{\Omega }}_{mcs}}\sum _{i\in {\mathrm{\Omega }}_{ev}}\sum _{t\in T}{C}_{i,j}^{total}$$

(1)

Equation (1) aims to maximize the income generated from charging EVs by MCSs.

The income generated from charging EVs is shown as follows:

$$C_{i,j,t}^{total}=P_{i,j,t} \ast {\lambda }_{j,t}^{mcs}+{\alpha }_j \ast P_{i,j,t}^2+{\lambda }_{i,j,t}^{del} \ast U_{i,j,t}^{mcs to ev}$$

(2)

In Equation (2), each part respectively determines the income generated from the power sold by each MCS to the EVs, the battery depreciation cost, and the delivery cost; ${\alpha }_i$, $P_{i,j,t}$, ${\lambda }_{i,j,t}^{del}$ and $U_{i,j,t}$ represent the Battery Depreciation Cost (BDC) coefficient, the requested power of each EV at time t, the delivery cost considering the relocating cost of MCS, and the connection binary variable of $MC{S}_j$ to $E{V}_i$ at time t, respectively. It should be noted that these terms are not internal operating expenses of the MCS but rather the service charges that are collected from the EVs in exchange for receiving energy, compensating battery usage, and covering transportation.

The constraints related to connecting the EV to the MCS are shown as follows:

$$U_{i,j,t}^{mcs to ev}\le {\overline{A}}_{i,t}\forall i,j,t$$

(3)

$$P_{i,j,t}=U_{i,j,t}^{mcs to ev} \ast {\displaystyle \frac{P_{i,t}^{Request}}{{\eta }_j^{tr}}}\forall j,t$$

(4)

Equation (3) states that each connection status of MCS must be based on the request from an EV during the day (${\overline{A}}_{i,t}$). Equation (4) states that the delivered power from MCS to EV must be based on the requested power ($P_{i,t}^{Request}$), the efficiency coefficient (${\eta }_j^{tr}$) and MCS status at that time interval.

$$W_{j,t}\le 1-\sum _{i\in {\mathrm{\Omega }}_{ev}}{U}_{i,j,t}^{mcs to ev}$$

(5)

$$U_{j,b,t}^{mcs to b}\le W_{j,t}$$

(6)

Equations (5) and (6) show that the MCS can proceed with battery replacement ($U_{j,b,t}^{mcs to b}\in \{0,1\}$), and the variable $W_{j,t}\in \{0,1\}$ is set to 1 only if no EV service has been scheduled during that time interval.

$$U_{j,b,t}^{mcs to b}\le SO{C}_{k,b,t-1}$$

(7)

Equation (7) shows that the possibility of replacing the MCS battery with battery b is allowed when the state of charge related to the battery b in the storage station k ($SO{C}_{k,b,t-1}$) is equal to 1. This constraint does not govern the overall SOC evolution; rather, it serves as a binary signaling condition to ensure that only fully charged batteries are eligible for swapping, since the fully charged packs deliver only to MCSs in practice.

$${\underline{E}}_{j,t}^{mcs}\le E_{j,t}^{mcs}\le {\overline{E}}_{j,t}^{mcs}$$

(8)

$$E{N}_{j,t}^{mcs}\le {\overline{E}}_{j,t}^{mcs}-E_{j,t-1}^{mcs}$$

(9)

$$E{N}_{j,t}^{mcs}\le ({\overline{E}}_{j,t}^{mcs}-{\underline{E}}_{j,t}^{mcs}) \ast \sum _{b\in {\mathrm{\Omega }}_B}{U}_{j,b,t}^{mcs to b}$$

(10)

$$E_{j,t}^{Request}=\mathrm{\Delta }t \ast \sum _{i\in {\mathrm{\Omega }}_{ev}}{P}_{i,j,t}$$

(11)

$$E_{j,t}^{mcs}=E{N}_{j,t}^{mcs}+E_{j,t-1}^{mcs}-E_{j,t}^{Request}$$

(12)

Equations (8)–(11) represent the constraints related to the amount of energy reduction from the MCS’s total energy. Equation (12) shows the final energy level of the MCS at the end of the time interval t, considering whether energy has been delivered to the vehicle or the battery has been replaced.

3.2. Charging Optimization at Storage Station

Figure 2 shows that the batteries placed at each station can obtain energy based on their current energy levels from the upstream grid, the available PV power, and the energy discharged from neighbor batteries. The optimal power scheduling is aimed at quickly preparing a fully charged BESS. Simultaneously, the charging schedule aims to minimize storage station costs while maximizing energy exchange between BESSs, considering the grid exchange price. The objective functions of battery storage stations are shown as follows:

$$F_2=min[\sum _{k\in {\mathrm{\Omega }}_K}\sum _{t\in T}{\rho }_t \ast (PG2S_{k,t}-PS2G_{k,t}-PV2G_{k,t})-\sum _{k\in {\mathrm{\Omega }}_K}\sum _{t\in T}PS2B{P}_{k,t}]$$

(13)

In this equation, ${\rho }_t$ represents the exchange energy price of the grid at each time t.

3.2.1. Grid Constraints

The grid constraints are shown as follows:

$$P{G}_{k,t}=PG2S_{k,t}-PS2G_{k,t}-PV2G_{k,t}$$

(14)

$$|P{G}_{k,t}| \le P{G}_k^{Max}$$

(15)

Equations (14) and (15) show the maximum exchangeable power with the upstream grid. In these equations, $P{G}_{k,t}$, $PG2S_{k,t}$, $PS2G_{k,t}$ and $PV2G_{k,t}$ represent the exchanged power by each charging station to the upstream grid, the power purchased from the upstream grid by each storage station, the power sold to the upstream grid by each storage station, and the power sold to the upstream grid by the PV, respectively.

3.2.2. PV Constraints

The constraints related to PV production are shown as follows:

$$PV2S_{k,t}+PV2G_{k,t}\le PP{V}_k^{Max}$$

(16)

In this equation, $PV2S_{k,t}$ and $PP{V}_k^{Max}$ represent the power transferred to the storage station by the PV and the maximum generated power of the PV at each time t, respectively.

3.2.3. Storage Constraints

The constraints related to exchanging power in the storage system are shown as follows:

$$PG2S_{k,t}+PV2S_{k,t}+PS2B{P}_{k,t}\le PCH{S}_{k,t}$$

(17)

Equation (17) shows the total power provided to charge the batteries. In this equation, $PCH{S}_{k,t}$ and $PS2B{P}_{k,t}$ represent the charging power of the storage station and the power exchange between BESSs, respectively.

$$PS2G_{k,t}+PS2B{P}_{k,t}\le PDCH{S}_{k,t}$$

(18)

Equation (18) shows the total power delivered from the discharging of the batteries. In this equation, $PDCH{S}_{k,t}$ represents the discharging power of the storage station.

$$PS2G_{k,t}\le Z_{k,t} \ast PS{G}_k^{Max}$$

(19)

$$PG2S_{k,t}+PV2S_{k,t}\le V_{k,t} \ast P{S}_k^{Max}$$

(20)

$$Z_{k,t}+V_{k,t}\le 1$$

(21)

Equation (19) shows the maximum power that can be delivered from the storage system to the upstream grid ($PS{G}_k^{Max}$). Equation (20) shows the maximum energy that can be provided to the storage system ($P{S}_k^{Max}$). Equation (21) shows that each set of batteries either receives power from the system or delivers it to the system during a time interval, and these two states do not occur simultaneously ($Z_{k,t} ,V_{k,t}\in \{0,1\}$).

$$PCH{S}_{k,t}\le \sum _{b\in {\mathrm{\Omega }}_B}BPC{H}_{k,b,t}$$

(22)

$$PDCH{S}_{k,t}\le \sum _{b\in {\mathrm{\Omega }}_B}BPDC{H}_{k,b,t}$$

(23)

$$BPC{H}_{k,b,t}\le X_{k,b,t} \ast {\delta }_{ch} \ast CA{P}_{k,b}$$

(24)

$$BPDC{H}_{k,b,t}\le Y_{k,b,t} \ast {\delta }_{dch} \ast CA{P}_{k,b}$$

(25)

Equations (22) and (23) illustrate that the total charge and discharge of the storage system are the sum of the individual charges and discharges of each battery package. Equations (24) and (25) represent the maximum power at which each BESS can be charged and discharged, respectively. In these equations, ${\delta }_{ch}$, ${\delta }_{dch}$, and $CA{P}_{k,b}$ represent the maximum allowable charging and discharging percent and battery package capacity, respectively.

$$X_{k,b,t}+Y_{k,b,t}\le (1-U_{j,b,t}^{mcs to b})$$

(26)

Equation (26) shows that the BESS is not being charged or discharged when called for replacement. Each BESS is either in charge or discharge state at a time interval and these two states do not happen simultaneously ($X_{k,b,t} ,Y_{k,b,t}\in \{0,1\}$).

$$\begin{array}{cc}SO{C}_{k,b,t}=max\hfill & (U_{j,b,t}^{mcs to b}·{\displaystyle \frac{E_{j,t-1}^{mcs}}{CA{P}_{k,b}}},\hfill \\ \hfill & (1-U_{j,b,t}^{mcs to b})·(SO{C}_{k,b,t-1}+{\displaystyle \frac{BPC{H}_{k,b,t}·{\eta }_{k,b}^{ch}-\frac{BPDC{H}_{k,b,t}}{{\eta }_{k,b}^{dch}}}{CA{P}_{k,b}}}))\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill & SO{C}_{k,b,t}\le {\displaystyle \frac{E_{j,t-1}^{mcs to b}}{CA{P}_{k,b}}}+\left[SO{C}_{k,b,t-1}-U_{j,b,t}^{mcs to b}\right]+{\displaystyle \frac{BPC{H}_{k,b,t}·{\eta }_{k,b}^{ch}-\frac{BPD{C}_{k,b,t}}{{\eta }_{k,b}^{dch}}}{CA{P}_{k,b}}}\hfill \end{array}$$

(27b)

$${\displaystyle \frac{E_{j,t-1}^{mcs to b}}{CA{P}_{k,b}}}\le U_{j,b,t}^{mcs to b}$$

(27c)

$$E_{j,t}^{mcs to b}\le E_{j,t-1}^{mcs}$$

(27d)

Equation (27) shows the SOC of the BESS at each time interval depending on the SOC from the previous interval. Equation (27a), which is nonlinear, is reformulated using the auxiliary linear constraints (27b–27d) to ensure compatibility with the MILP framework. It is possible to replace the battery if the SOC is equal to 1. If the SOC reaches 1, the battery can be replaced. When a battery is replaced, its status is updated to match that of the most recently replaced battery.

3.3. DQN Approach with New Decision-Making Process and Reward Design

DQN extends Q-learning, a widely used model-free reinforcement learning algorithm, by approximating Q-values through a neural network, which enables the agent to learn effective policies in complex environments [40]. Although actor–critic and policy gradient methods are well-suited for continuous action spaces, the inherently discrete nature of the MCS scheduling problem makes DQN a more efficient and effective choice for this study. The proposed Multi-Agent MCS Scheduling Problem (MAMCSP) is formulated such that each agent (each MCS is considered as an independent agent) executes an action at each time step according to its local state, obtains an immediate reward, and then transfers to the next scheduling time interval.

3.3.1. State Design

The state vector at time t consists of the EV request, the status of each battery package, and the status of the local MCS, which is expressed as follows:

$$S^t \Rightarrow [\underset{[E_i, x_i, y_i]}{\underbrace{ {\mathrm{EV}}_i^t }}, \underset{[{\mathrm{SOC}}_b, R_b]}{\underbrace{ B_b^t }},\underset{[E_j, x_j, y_j]}{\underbrace{ {\mathrm{MCS}}_j^t }}]$$

(28)

The ${\mathrm{EV}}_i^t$ vector consists of the needed energy of the EV and the requested location, the $B_b^t$ vector consists of the state of charge of the battery storage and the ready-for-change status, and the ${\mathrm{MCS}}_j^t$ vector consists of the remaining energy of the MCS and the MCS location.

3.3.2. Action Configuration

The action vector ${\mathrm{Act}}^t$ for the $MC{S}_j$ at time t selects one of the three possible actions, either not acting, delivering energy to one of EV requests, or taking action, to replace the battery, illustrated as follows:

$${\mathrm{Act}}_j^t \Rightarrow [{\mathrm{Act}}_{\mathrm{N}}^{\mathrm{t}}, {\mathrm{Act}}_{{\mathrm{EV}}_{\mathrm{i}}}^{\mathrm{t}}, {\mathrm{Act}}_{{\mathrm{B}}_{\mathrm{b}}}^{\mathrm{t}}]$$

(29)

Here, ${\mathrm{Act}}_N^t$ corresponds to the no-action (idle) decision, where the $MC{S}_j$ remains inactive during time t. ${\mathrm{Act}}_{E{V}_i}^t$ represents the charging action, in which the $MC{S}_j$ delivers energy to the selected EV request $E{V}_i$, subject to capacity and time constraints. ${\mathrm{Act}}_{B_b}^t$ denotes the battery replacement action, where the $MC{S}_j$ returns to the depot to swap its depleted battery with a fully charged one, ensuring continued service in subsequent time steps.

3.3.3. Reward Design

The reward function is the immediate benefit obtained by $MC{S}_j$ at the time t to take the action ${\mathrm{Act}}_j^t$ based on the state $S^t$, which is expressed as follows:

$$r\left(r^{\tau }\right) = \sum _{t\in \tau }[(\sum _{j\in {\mathrm{\Omega }}_{\mathrm{mcs}}}\sum _{i\in {\mathrm{\Omega }}_{ev}}{C}_{i,j,t}^{\mathrm{total}})+{\beta }_{A,C}^t]$$

(30)

In this equation, ${\beta }_{A,C}^t$ is related to the judgment term, which is determined based on the criteria in each action connection.

In order to model DQN, it is necessary to design a reward and penalty structure for the performance of MCS as the problem agent. Each MCS in each time step takes the appropriate decision according to the requests of EVs in that time step (the EV request was sent to the control center one time step earlier) and the information received from the charging batteries.

• Action 1: MCS does nothing to service the EV or replace its battery (no action).
  In this situation, according to the system conditions of this decision, the agent has a possible reward or penalty as follows: If the service request is from the EV side, no battery is ready to be replaced and the energy stored in the MCS is able to supply the vehicle’s energy. In this case, the agent will be fined for not providing the power of the EV. If the battery is ready for replacement and the energy stored in the MCS is less than its maximum possible energy, failure of the MCS to refer for battery replacement will result in a penalty. There is a service request from the EV, the battery is ready for replacement, and the energy stored in the MCS is less than the energy requested by that EV, but the MCS does not replace the battery. Failure of the vehicle to refer for battery replacement will result in a penalty; this penalty is twice the first and second cases. If there is no service request from the EV, the battery is not ready for replacement, or there is a service request from the EV, but the amount of energy stored in the MCS is less than the requested energy of the EV, and no battery has announced that it is ready for replacement. In this case, the no agent’s act is correct and it will receive a reward.

  $${\beta }_{A1,C}^t=\left\{\begin{array}{cc}-{\beta }_{A1,C1}^t\hfill & \mathrm{if} [E_{ev}>0 \& {\mathrm{R}}_b=0 \& E_{\mathrm{MCS}}>E_{ev}]\hfill \\ -{\beta }_{A1,C2}^t\hfill & \mathrm{if} [{\mathrm{R}}_b=1 \& E_{\mathrm{MCS}}<E_{\mathrm{MCS}}^{\mathrm{Max}}]\hfill \\ -{\beta }_{A1,C3}^t\hfill & \mathrm{if} [E_{ev}>0 \& {\mathrm{R}}_b=1 \& E_{\mathrm{MCS}}<E_{ev}]\hfill \\ {\beta }_{A1,C4}^t\hfill & \mathrm{if} [E_{ev}>0 \& {\mathrm{R}}_b=0 \& E_{\mathrm{MCS}}<E_{ev}]\hfill \\ \hfill & \mathrm{or} [E_{ev}=0 \& {\mathrm{R}}_b=0]\hfill \end{array}\right.$$

  (31)
• Action 2: The agent selects one of the EVs for servicing (${\mathrm{EV}}_i$).
  If the energy available in the MCS is less than the requested energy of the EV, in this case, the agent will be subject to a penalty. If the EV-requested energy is zero, in fact, the EV has no demand. In this case, the agent will be fined. If it is none of the above cases, in this case, the agent will receive a reward due to the correct action.

  $${\beta }_{A2,C}^t=\left\{\begin{array}{cc}-{\beta }_{A2,C1}^t\hfill & \mathrm{if} E_{\mathrm{MCS}}<E_{ev}\hfill \\ -{\beta }_{A2,C2}^t\hfill & \mathrm{if} E_{ev}=0\hfill \\ {\beta }_{A2,C3}^t\hfill & \mathrm{if} E_{ev}>0\hfill \end{array}\right.$$

  (32)
• Action 3: The agent refers to one of the batteries to replace the battery ($B_b^t$).
  If the battery is not ready for replacement or the energy stored in the MCS is equal to the maximum possible energy, the agent will be fined. If the battery has announced its readiness for replacement (battery readiness has been sent to the control center one time step earlier) and the MCS energy is lower than the maximum MCS value, in this case, the agent will receive a reward due to the correct action.

  $${\beta }_{A3,C}^t=\left\{\begin{array}{cc}-{\beta }_{A3,C1}^t\hfill & \mathrm{if} {\mathrm{R}}_b=0 \mathrm{or} E_{\mathrm{MCS}}=E_{\mathrm{MCS}}^{\mathrm{Max}}\hfill \\ {\beta }_{A3,C2}^t\hfill & \mathrm{if} {\mathrm{R}}_b=1 \& E_{\mathrm{MCS}}<E_{\mathrm{MCS}}^{\mathrm{Max}}\hfill \end{array}\right.$$

  (33)

4. Branch Flows Model

The branch flow model represents the distribution power flow problem by the following conic program. The objective function in this power flow model is minimizing the total active power injection imported from the upstream grid at the substation. The constraints of the power flow are given in Equation (34a–34h) [41].

$${\left({\underline{V}}_m\right)}^2\le V_{m,t}^{sq}\le {\left({\overline{V}}_m\right)}^2,\forall m\in \mathcal{M}$$

(34a)

$$\sum _{f:f\to m}\left(P_{f,m,t}-r_{f,m}{I}_{f,m,t}^{sq}\right)-\sum _{n:m\to j}{P}_{m,n,t}=P_{m,t}^d, \forall m\in \mathcal{M}$$

(34b)

$$\sum _{f:f\to m}\left(Q_{f,m,t}-x_{f,m}{I}_{f,m,t}^{sq}\right)-\sum _{n:m\to n}{Q}_{m,n,t}=Q_{m,t}^d, \forall n\in \mathcal{M}$$

(34c)

$$P_t^{\mathrm{grid}}=\sum _{n:1\to n}{P}_{1,n,t}$$

(34d)

$$Q_t^{\mathrm{grid}}=\sum _{n:1\to n}{Q}_{1,n,t}$$

(34e)

$$V_{n,t}^{sq}=V_{m,t}^{sq}-2(r_{\ell }{P}_{m,n,t}+x_{\ell }{Q}_{m,n,t})+(r_{\ell }^2+x_{\ell }^2)I_{\ell ,t}^{sq},\forall mn=\ell \in \mathcal{L}$$

(34f)

$$I_{\ell ,t}^{sq}\ge 0,\forall \ell \in \mathcal{L}$$

(34g)

$$P_{m,n,t}^2+Q_{m,n,t}^2\le V_{m,t}^{sq}{I}_{\ell ,t}^{sq}, \forall \ell =mn\in \mathcal{L}$$

(34h)

Disciplined convex programming syntax does not allow for the product of two optimization variables. The rotated conic constraint in Equation (34h) can be implemented as follows:

$$\underset{\mathrm{norm}2\left(\left[P_{m,n,t}{Q}_{m,n,t}(V_{m,t}^{sq}-I_{\ell }^{sq})/2\right]\right)}{\underbrace{\sqrt{P_{m,n,t}^2+Q_{m,n,t}^2+{\left({\displaystyle \frac{V_{m,t}^{sq}-I_{\ell ,t}^{sq}}{2}}\right)}^2}}}\le {\displaystyle \frac{V_{m,t}^{sq}+I_{\ell ,t}^{sq}}{2}},\forall \ell =mn\in \mathcal{L}$$

(35)

Regardless of the implementation, commercial solvers such as MOSEK will work with the equivalently converted quadratically second order cone.

5. Numerical Results and Discussion

In order to simulate the vehicle request time and the amount of energy requested by private EVs, according to Chattanooga Area Regional Transportation Authority (CARTA) EV request information, the probability distribution function has been obtained as a Normal Distribution. Given the uncertainty in EV requests and the need for training the DQN model, the CARTA data have been utilized to improve the accuracy and reliability. MCSs are prioritized in high-demand urban zones where user time is valuable.

Table 1. Simulation parameters [6].

${\mathbf{CAP}}_{\mathbf{k},\mathbf{b}}$	${\underline{\mathit{E}}}_{\mathit{j},\mathit{t}}^{\mathbf{mcs}}$	${\overline{\mathit{E}}}_{\mathit{j},\mathit{t}}^{\mathbf{mcs}}$	${\mathit{\delta }}_{\mathbf{ch}/\mathbf{dch}}$	${\mathit{\eta }}_{\mathit{k},\mathit{b}}^{\mathbf{ch}/\mathbf{dch}}$
100 KW	20 KW	100 KW	30%	90%
${\lambda }_{j,t}^{\mathrm{mcs}}$	${\lambda }_{i,j,t}^{\mathrm{del}}$	${\alpha }_j$	${\rho }_t$	$\mathrm{\Delta }t$
$0.12	$30	$0.0004	[6]	30-min

shows the simulation parameters.

The EV requests are sent to a control center, and based on the location of the request within each MCS zone, the corresponding requests are forwarded to the relevant MCS. Through optimal agent scheduling, the MCS swaps its battery at the charging station for a full recharge. The charging station is equipped with two pieces of charging equipment for MCS batteries, with the capability to obtain power from both a PV system and the upstream grid. The power exchange scheduling of the internal components within this charging station is formulated as an MILP problem and optimized using Python Optimization Modeling Objects (Pyomo, version 6.9.2) with the GLPK solver. For the parameter settings of the DQN algorithm, the custom policy network consists of four hidden layers with sizes set to 64, 128, 128, and 64, respectively. Each hidden layer uses the Rectified Linear Unit (ReLU) activation function. The Adam optimizer is employed for updating the network with a learning rate of $\alpha ={10}^{-4}$.

A sample scenario involving EV requests is implemented to assess the proposed model’s performance. Figure 3 illustrates the specific actions made by MCS either to select an EV for charging services or access to a battery swapping station (purple filled cell). Additionally, the requested energy of EVs is shown by red numbers which are asked for the next time interval.

Figure 4 illustrates the amount of energy stored in the MCS at each time step and the sequence of replacements with batteries $B_1$ and $B_2$ available at the charging center throughout the scheduling period. At each time step where the MCS swaps its battery for a fully charged one, the MCS battery charge is reset to its maximum level, while the charge of the replaced battery is updated to match the MCS’s charge level from the previous time step. In six time steps, the battery is permitted to reach full charge using the resources within the charging station. This scheduling is supported through power purchases from the upstream grid and PV at each time step. Figure 5 presents the cost of grid power procurement, the available solar energy (a 100 kW PV system is considered for the roof of the parking structure [42]), and the changes in stored energy for batteries $B_1$ and $B_2$ at each time step. The energy allocated for battery charging is depicted as bars in Figure 5.

Figure 6 shows the power received by the storage system from the upstream grid, the power dispatched from the storage system to the upstream grid, the charge and discharge levels of each battery separately, and the power exchanged between batteries $B_1$ and $B_2$.

In

Table 2. The result comparison.

	Optimal	DQN
The MCS Power Delivery Incom	USD 787.5	USD 619
The Power Exchange Income	USD 144.5	USD 155
The Battery Package Charging Cost	USD −79.5	USD −44.5
Total Revenue	USD 852.59	USD 729.5

, the gap between the proposed DQN solution and the traditional optimal solution using the GLPK solver is demonstrated. It should be noted that the GLPK result represents a benchmark under full-day request information, while the DQN operates with only one-step-ahead requests; therefore, the comparison is intended as a reference rather than a fully equivalent setting. Although the traditional optimal solution generates higher revenue, it requires full knowledge of EV requests for the entire day, which is impractical in real-world scenarios. Throughout the DQN solution model, the MCS successfully serviced 18 EVs, resulting in a total revenue of USD 619. The revenue generated by the charging station is from selling power to the upstream grid, both through the sale of solar energy and by discharging batteries for grid sales, with a total positive revenue of approximately USD 155. The cost of purchasing power from the upstream grid to charge the batteries within the storage amounts to USD 46.5 for the entire scheduling period.

To further assess the reliability of the proposed model, we computed statistical metrics on the simulation outcomes. The percentage error between the DRL-based results and the optimization benchmark was calculated for each scenario and aggregated across different sample sizes. Figure 7 illustrates the variance and interquartile ranges, while the overlaid lines indicate the evolution of the mean error. The results show that the mean error remains relatively stable and the spread decreases as the number of scenarios increases, thereby demonstrating the robustness and consistency of the proposed framework.

To evaluate the impact of MCS service scheduling and the battery charging plans at the MCS battery charging station, three cases are considered, and the analysis is conducted on the IEEE 15-bus distribution system [43], where buses are modeled as load points and the power flow method is described in Section 4:

• Case 1: Load flow analysis of the network without considering the presence of the MCS.
• Case 2: Scheduling of MCS services and charging of associated batteries, without considering the technical constraints of the power grid.
• Case 3: Scheduling of MCS services and battery charging while accounting for the technical constraints of the power grid.

The MCS battery charging station consistently aims to maximize the number of fully charged batteries available for delivery to MCSs. The more access MCSs have to charged batteries, the more EV charging requests they can fulfill, thereby increasing their revenue. However, higher revenues often correlate with increased power demand from the upstream grid, which can impact the technical constraints of the power system, including voltage and current levels. Therefore, when planning battery charging operations at the charging center, it is essential to evaluate the power system parameters at various nodes in the supplying grid. To mitigate negative impacts on the power network, control strategies such as dynamic pricing must be applied. During this planning, MCSs help reduce the grid load by serving EV requests. Since EVs are charged by MCSs instead of fixed charging stations, the load on the node corresponding to the parking area decreases. This load reduction should also be considered in system studies to accurately assess network parameters during the planning process. To examine both load reduction and increase across different network points, Case 2 and Case 3 have been defined. The study uses the standard IEEE 15-bus network, with 40% of the system’s total load considered. As shown in Figure 8, a fixed EV charging station is assumed to be located at bus 8, and the load at this node decreases when EVs are served by MCSs. The MCS battery charging center is located at bus 14, where power consumption increases the load and power generation reduces it.

The 15-bus network load profile is defined using the realistic demand curve of a section of Chattanooga’s distribution system, with each time step representing 30 min [6]. To isolate the effect of battery charging on the network, no PV generation is considered at the charging center in both Case 2 and Case 3. The battery charging schedules for Case 2 and Case 3 are shown in Figure 9 and Figure 10.

In order to evaluate the impact of MCS scheduling and battery charging (as described in Section 4), voltage and current parameters are calculated at each time step. The acceptable voltage range is assumed to be between 0.95 and 1.05 p.u. In addition to standard line current limits, a specific current constraint of 37.5 A is applied to line 1, the main feeder supplying the entire system. If the battery charging plan violates system constraints, the price of electricity delivered to the MCS battery charging center is increased in 10% steps, encouraging the charging station to shift its demand to other time intervals through self-managed scheduling. Figure 11a shows the current of each network line without MCS scheduling or battery charging center activity. Figure 11b shows the line currents with MCSs and the charging center in operation but without applying voltage or current constraints. As seen in Step 14, the power demand from the charging center causes the current in line 1 to exceed the defined limit. Figure 11c applies the current constraint as part of the charging schedule, using ToU pricing to shift demand.

As a result, line 1 current is reduced by modifying the charging schedule. Figure 12 shows the voltage profiles of buses under different cases. Additionally, upstream electricity price variations for the charging center, with and without system constraints, are presented in Figure 13.

6. Conclusions

In summary, the mathematical model developed in this study provides a comprehensive framework for optimizing the deployment of MCSs and planning their battery recharging in a cost-effective manner. By maximizing profits from EV charging requests and incorporating renewable energy resources into the recharging process, the model addresses key technical and economic challenges. It ensures that MCS operations are both cost-effective and environmentally sustainable. The novelty of this study lies in combining an MILP-based optimization framework with a DQN approach for real-time decision-making while explicitly modeling MCS battery scheduling and its grid impact. This approach not only improves the operational efficiency of MCSs but also contributes to the broader goal of creating a sustainable and scalable EV charging infrastructure for the future. The model effectively manages the limited energy capacity of MCSs by prioritizing high-demand requests and strategically scheduling recharging times. It ensures timely responses to EV users while minimizing idle periods. The proposed solution promotes the intelligent allocation of resources, helping reduce urban congestion and enhance user satisfaction in dynamic traffic environments. Furthermore, this study investigates the impact of MCS battery scheduling on power grid performance using the IEEE 15-bus network and realistic demand data. By modeling the load reduction at fixed EV stations and load increase at MCS charging centers, the study evaluates voltage and current levels across the grid. Dynamic pricing and TOU strategies are applied to ensure that system constraints are satisfied, effectively balancing EV service efficiency with grid reliability.

The limitation of this study is the lack of access to detailed geographical information on traffic conditions along urban routes. Incorporating such data would enable a more precise representation of MCS accessibility and service dynamics. Furthermore, EVs are assumed to be charged only through the plug-in service from MCSs, while the MCS battery packs are managed via swapping at the Parking Battery Station. Future work could build on our framework by integrating spatial and traffic-related information to further enhance the accuracy and applicability of the model.

In future research, we aim to evaluate the impact of our proposed model on grid resilience and cybersecurity, as well as explore multi-agent coordination strategies among MCS fleets and integration with Vehicle-to-Grid (V2G) services. These directions, including the analysis of solar integration with MCS operations and battery swapping for improved load management and grid performance, would further enhance the flexibility and scalability of the proposed framework. In addition, conducting sensitivity analysis on key parameters such as EV demand variability, PV generation, and battery degradation will be considered to assess the robustness of the model under different operational conditions.